How to Measure Execution Performance with Academic Benchmarks

Samples > Rule-Based Programming > How to Measure Execution Performance with Academic Benchmarks

How to Measure Execution Performance with Academic Benchmarks

This sample contains the implementation of the following academic benchmarks:

Puzzle
Tourney
Word
Manner
Waltz

The sample also shows how to tune the performance of the rule engine in RetePlus mode using some of the optimization techniques provided with ILOG JRules.

Note

This sample contains purely academic benchmarks, whose rulesets are traditionally used by rule-based systems to compare their relative performance. However, they are not intended to demonstrate the real performance of a rule engine in operational conditions where numerous rules, numerous objects, and numerous simultaneous engine executions may be involved.

Key Features

Key features include:

Dynamic rule compilation
Hashers
Static agenda
Iterated rules

Importing this Sample

To import the academic sample:

Click File > Import.
In the Import Wizard, select ILOG Rule Studio Samples and Tutorials and click Next.
Select the samples > engine > academic sample.
Click Finish.

The required sample project files are imported.

If one or more files already exist in your workspace, a warning message is displayed indicating that the corresponding file(s) were not re-imported.

The imported sample project files are displayed in the Rule Explorer.

Running this Sample

Make sure you are in the Rule perspective before running this sample.

To run the benchmark sample projects:

Click Run > Open Run Dialog.
Select the required benchmark launch configuration from the following list:

PuzzleBenchmark: This runs the Puzzle benchmark, showing results without optimizations and results with the useJIT optimization.

TournBenchmark: This runs the Tourney benchmark, showing results without optimizations and results with hashers and the useJIT optimizations.

WordBenchmark: This runs the Word benchmark, showing results without optimizations and results with hashers and the useJIT optimizations.

MannerBenchmark: This runs the Manners benchmark, showing results without optimizations and results with several optimizations (including useJIT, hashers, static agenda and iterated rules) for 16, 32, 64, 128 and 256 guests.

WaltzBenchmark: This runs the Waltz benchmark, showing results without optimizations and results with several optimizations (including useJIT, hashers, static agenda and iterated rules) for half-sized and full-sized problem.

Click Run.

Output is displayed in the Console view.

Each benchmark may be run several times, to get average and statistics results. The first run is always neglected in the statistics results, because the class loading times and static class initializations should not be taken into account. All the provided launch configurations require just two runs, so average results are computed on a single run.

You can change the first program argument of each launch configuration to get statistics on more runs.

Each run consists of two executions. This is to differentiate between first execution and normal execution, as the second execution may be quicker (since rule selection and JIT optimization are already done). Each run is divided into several phases:

- PREPARATION: Ruleset parsing and engine construction.

- FIRST.DECLARATION: Insertion of objects in the working memory for the first execution.

- FIRST.EXECUTION: First engine execution.

- FIRST.RESET: Resets the engine at the end of first execution and prepares the engine for subsequent execution.

- NORMAL.DECLARATION: Insertion of objects in the working memory for a normal execution.

- NORMAL.EXECUTION: Normal engine execution.

- NORMAL.RESET: Resets the engine at the end of a normal execution.

By default, the only statistic displayed is the average rules fired per second. To get more information, you can uncomment a line at the end of the BenchmarkRunner class.

Highlights

This sample allows you to assess the impact of different optimization techniques on performance. You can compare execution times between the raw version and the optimized version of different sets of rules.

A shared project, called benchmark, contains the benchmark infrastructure:

A Benchmark interface that is implemented by the actual benchmarks. This interface describes the contract of a rule-based benchmark. It contains a name, a description, a method to configure a benchmark object from a map of properties, the ruleset key (or path), the contents of the working memory at the beginning of execution, the map of input ruleset parameters, and a test method to check that the output ruleset parameters and number of rules fired are as expected.
A BenchmarkSequence interface and the EngineSequence implementation. This interface describes the execution sequence of a rule-based benchmark. It contains methods for each benchmark phase, preparation, working memory declaration, checking of input data against ruleset signature, execution, reset.
A BenchmarkRunner that orchestrates a Benchmark and a BenchmarkSequence together. This will typically resemble the following pseudo code:

benchmark.configure(configurationProperties);

sequence.prepare(benchmark.getRulesetKey());

sequence.declare(benchmark.getWorkingMemory());

sequence.checkInput(benchmark.getWorkingMemory(), benchmark.getInput());

results = sequence.execute(benchmark.getInput());

benchmark.checkOutput(results, sequence.getFiredRulesCount());

sequence.reset();

Instrumentation classes to get time and memory consumption measurements.

Each benchmark is provided as a set of related projects, as shown below (where xxx is the name of the benchmark in question):

xxx-benchmark is the implementation for the benchmark, responsible primarily for providing the ruleset path and the input data. This project also provides the launch configuration to start the execution.
xxx-xom contains the classes for the objects manipulated by the rules.
xxx-rules contains the rules without optimization.
xxx-optim-rules contains either the same rules rewritten with optimizations or some optimization on top of the xxx-rules project

Puzzle

The puzzle consists in finding a digit for a letter to meet the following addition:

  GERALD: 197485
+ DONALD: 526485
----------------
= ROBERT: 723970

The solution found by JRules is printed next to the letters.

In a simple approach to this problem, there are 10 digits and 10 different letters, so there are 100 different letter-digit combinations. The rules choose 10 combinations among those 100 such that the formula holds. This program tests how a rule engine handles combinatory problems.

Tourney

This program deals with the scheduling of a bridge tournament. There are 16 people who are playing bridge over 5 nights. Each player must play with another player exactly once over the five nights.

Intuitively, for a specific player, the 15 other people can be divided into 5 groups of 3, each group of 3 forming a table with the player for one night. There are many solutions, but the rules are written so that only one solution can be found. This program uses rule chaining and builds long-chain conditions.

Word

This program generates 4 letter words according to a set of rules. Some sequences of letters are allowed, and others are not. The rules are simple, but the number of objects matching a condition can be high.

Manner

This program handles the problem of seat assignments for guests at a dinner. It places the people having at least 2 common hobbies next to each other, while keeping people of the opposite sex alternated.

This program uses a data file as input. This file may contain 16, 32, 64, 128 or more guests. The data files are in the data directory.

Waltz

This program is presented with a collection of lines such as might be found by an edge detector in a computer vision system. It applies a set of constraints to the lines and creates labels for them. The labels indicate how the lines correspond to edges in a three-dimensional space. This process is known as Waltz labeling.

The rule conditions in this program are more complex than the conditions used in Manner.