Mongoose Scenario Engine Design

SCENARIO ENGINE

Introduction
Limitations
Approach
- Run a Scenario
- Scenario File Syntax
Configuration
Reporting

INTRODUCTION

The performance testing software should be able to be scripted in order to execute complex tests ("weighted CRUD" or "maxtest", etc). The general way to describe the actions which should be performed in the test is the organizing them to the sequences and parallel sets. So does Apache JMeter, COSBench and other tools. It's decided to rework the way to configure Mongoose in order to make Mongoose more human-friendly and flexible.

LIMITATIONS

The scenario scripting feature available since v2.0.0
"Command" job feature available since v2.2.0
"For" job feature available since v2.3.0
Scenario parametrization with environment variables feature available since v2.4.0

APPROACH

Run a Scenario

There are two ways to feed the scenario to Mongoose:

Specify the file containing the scenario:
java [<DEFAULTS_OVERRIDING>] -jar mongoose.jar [<MODE>] -f <PATH/TO/SCENARIO.json>

Provide the scenario content on the standard input of mongoose (note the "-I" argument):

cat <PATH/TO/SCENARIO.json> | java [<DEFAULTS_OVERRIDING>] -jar mongoose.jar [<MODE>] -I

Mongoose will await the scenario input from the user if there's nor scenario file specified neither scenario data on the standard input. Note the highlighted scenario data specified by user in this semi-interactive mode.

2016-04-14T12:24:32,705 I ScenarioRunner       main                                     Using the s↩
    cenario from the standard input
{
        "type" : "load",
        "config" : { "load" : { "limit" : { "time" : "1m" } } }
}
2016-04-14T12:25:04,982 I HttpBucketHelper     main                                     Bucket "mons↩
    goose-2016.04.14.12.25.04.883" created
2016-04-14T12:25:05,028 I LoadExecutorBase     main                                     0-S3-Write-s↩
    1x1: will use "newDataItemSrc<BasicHttpData>" as an item source
2016-04-14T12:25:05,163 I SingleJobContainer   main                                     Start the js↩
    ob "0-S3-Write-1x1"
2016-04-14T12:25:05,194 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(0/0)s↩
    ; duration[us]=(0/0/0); latency[us]=(0/0/0); TP[op/s]=(0.000/0.000); BW[MB/s]=(0.000/0.000)
2016-04-14T12:25:15,201 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(1654s↩
    6/0); duration[us]=(442/370/1307); latency[us]=(136/47/916); TP[op/s]=(1827.394/1590.264); BW[M↩
    B/s]=(1827.394/1590.249)
2016-04-14T12:25:25,201 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(3865s↩
    0/0); duration[us]=(418/357/1283); latency[us]=(132/47/393); TP[op/s]=(2026.467/1981.767); BW[M↩
    B/s]=(2026.467/1981.761)
2016-04-14T12:25:35,202 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(6068↩
    1/0); duration[us]=(410/357/1245); latency[us]=(131/47/546); TP[op/s]=(2085.863/2121.699); BW[M↩
    B/s]=(2085.863/2121.691)
2016-04-14T12:25:45,204 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(8233↩
    8/0); duration[us]=(407/357/1394); latency[us]=(131/47/632); TP[op/s]=(2105.416/2146.232); BW[M↩
    B/s]=(2105.416/2146.229)
2016-04-14T12:25:55,204 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(1040↩
    74/0); duration[us]=(405/357/1245); latency[us]=(130/47/632); TP[op/s]=(2118.564/2159.117); BW[↩
    MB/s]=(2118.585/2159.206)
2016-04-14T12:26:05,205 I LoadExecutorBase     0-S3-Write-1x1-metrics                   count=(1257↩
    23/0); duration[us]=(405/357/1392); latency[us]=(130/47/632); TP[op/s]=(2125.672/2162.451); BW[↩
    MB/s]=(2125.672/2162.426)
2016-04-14T12:26:05,258 I LoadExecutorBase     main                                     "0-S3-Writes↩
    -1x1" summary: count=(127782/0); duration[us]=(404/357/390/395/403/1392); latency[us]=(130/50/1↩
    22/130/137/632); TP[op/s]=(2126.772/2160.124); BW[MB/s]=(2126.772/2160.102)
2016-04-14T12:26:05,266 I JsonScenario         main                                     Scenario end

A custom scenario to run with mongoose should be in the JSON format.

The only meaningful mode which may be used with a scenario is "client" ("standalone" is used by default when mode is not specified at all). For example, in order to run the scenario in the distributed mode it's enough to add the client argument and the property overriding the default load server address list (which is 127.0.0.1 by default).

java -Dload.server.addrs=192.168.0.3,192.168.0.4,192.168.0.5 -jar mongoose.jar client -f <PATH/T↩
	O/SCENARIO.json>

Scenario File Syntax

Job Types

Basically the scenario file describes the jobs hierarchy to execute.

The root scenario node is always a job:

{
    "type" : <ROOT_JOB_TYPE>
    ...
}

Jobs may include other (children) jobs:

{
    "type" : <CONTAINER_JOB_TYPE>
    "jobs" : [
        {
            "type" : <CHILD_JOB_TYPE>
            ...
        }, {
            "type" : <CHILD_JOB_TYPE>
            ...
        }
    ]
}

The full list of the job types is described in the table below:

Job Type	Description	Nested Jobs	Mandatory Attributes	Optional Attributes
load	The single load job node.	Not Allowed	type	config
precondition	Same as the "load", but the load job is executed in the "precondition" mode. In the precondition mode the metrics are not persisted into the output CSV files.	Required	type	config
parallel	The container job for the nested jobs which should be executed in parallel.	Required	type jobs	config
sequential	The container job for the nested jobs which should be executed sequentially.	Required	type jobs	config
for	The container job which executes the nested jobs for each value taken from the specified list and substituted into specified place of the configuration	Required	type value in jobs	config
command	The container executes the specified shell command. Also allows to start a command not blocking the scenario execution.	Not Allowed	type value	blocking

Configuration Node

Any job type except "command" may contain an optional "config" node:

    {
        "type" : <CONFIGURABLE_JOB_TYPE>
        "config" : {
            // here are the configuration hierarchy
        }
    }

Note that the configuration is inherited by the nested load jobs and overrided by the child job's configuration

The layout of the "config" subtree is the same as for default configuration. The configuration values override (see "Configuration Order" section):

Any default configuration values
Any values passed by CLI
A configuration from the parent job container

Let's realize that Mongoose is launched with the following command:

java -Dstorage.addrs=A,B,C,D -jar mongoose.jar -f scenario/custom.json

And the scenario file "custom.json" contains the following text:

{
    "type" : "load"
    "config" : {
        "storage" : {
            "addrs" : [
                "E", "F", "G", "H"
            ]
        }
    }
}

The effective storage address list will be "E,F,G,H" in this case.

Another example demonstrates the configuration inheritance:

{
    "type" : "sequential"
    "config" : {
        // the configuration will be inherited by nested jobs
        "storage" : {
            "addrs" : [
                "E", "F", "G", "H"
            ]
        }
    },
    "jobs" : [
        {
            "type" : "load",
            "config" : {
                // will override the parent job's configuration
                "storage" : {
                    "addrs" : [
                        "I", "J", "K"
                    ]
                }
            }
        }, {
            "type" : "load"
            // will inherit the parent job's configuration (storage addrs: E, F, G, H)
        }
    ]
}

The effective storage address list will be "I,J,K" for the 1st load job in the sequence and "E,F,G,H" in for the 2nd load job (configuration values inherited from the parent job).

Sequential and Parallel Jobs

Any job of the type "sequential" or "parallel" should include "jobs" element which behaves as the list of the children jobs.The configuration specified for a sequential/parallel job is inherited by the children jobs.

Execute a Shell Command

It may be useful to insert a sleep node into the jobs list between the actual load jobs:

{
    "type" : "sequential"
    "config" : {
        // shared configuration values inherited by the children jobs
    },
    "jobs" : [
        {
            "type" : "load",
            "config" : {
                // specific configuration for the 1st load job
            }
        }, {
            "type" : "command",
            "value" : "sleep 5m"
        }, {
            "type" : "load",
            "config" : {
                // specific configuration for the 2nd load job
            }
        }
    ]
}

Using Items Lists

In the real world the output of a load job should be used by another load job. For example, "create" load job should output the file containing the items have been created. And this list should be used for "read" load job as an input.

The example scenario below consist of 3 sequential steps:

(Preconditon) Create some items. Save the created items info into the destination file "/tmp/items2read.csv".
Read the items specified by the list file "/tmp/items2read.csv". Save the read items info into the destination file "/tmp/items2delete.csv".
Delete the items specified by the list file "/tmp/items2delete.csv". The default items destination will be used (log file).

{
    "type" : "sequential"
    "config" : {
        // shared configuration values inherited by the children jobs
    },
    "jobs" : [
        {
            "type" : "precondition",
            "config" : {
                "item" : {
                    "dst" : {
                        "file" : "/tmp/items2read.csv"
                    }
                }
        }, {
            "type" : "load",
            "config" : {
                "item" : {
                    "src" : {
                        "file" : "/tmp/items2read.csv"
                    },
                    "dst" : {
                        "file" : "/tmp/items2delete.csv"
                    }
                },
                "load" : {
                    "type" : "read"
                }
            }
        }, {
            "type" : "load",
            "config" : {
                "item" : {
                    "src" : {
                        "file" : "/tmp/items2delete.csv"
                    }
                },
                "load" : {
                    "type" : "delete"
                }
            }
        }
    ]
}

Loops

The "for" job performs the recurrent execution of the children jobs. The exact behavior depends on the "in" and "value" attributes values.

in	value	Resulting effect
Not set	Not set	Infinite loop
Not set	Integer >0	Counter based loop
String identifier	Value list or the range	"for each" based loop

The last case ("for each") also is capable to perform the value substitution in its configuration or in the configuration of the children jobs:

{
   "type" : "for",
   "value" : "threads",
   "in" : [
      1, 10, 100, 1000, 10000, 100000
   ],
   "config" : {
      "load" : {
         "threads" : "${threads}"
      }
   },
   "jobs" : [
      {
            "type" : "load"
      }
   ]
}

The example above will try to execute the load job 6 times. Each time the load job will be executed for the next thread count from the list specified (1, 10, 100, ...)

The range syntax is also supported:

{
   "type" : "for",
   "value" : "threads",
   "in" : "-2-7.5,0.5",
   "config" : {
      "load" : {
         "threads" : "${threads}"
      }
   },
   "jobs" : [
      {
            "type" : "load"
      }
   ]
}

The example above is equivalent to the loop defined as (i = -2.0; i < 7.5; i += 0.5)

The exact range pattern is X[-Y[,Z]] where:

X is the start (1st) value, may be negative
Y is the end value (exclusive), may be negative or/and less than start value
Z is the step value, should always be a positive number

If start value is greater than end value the loop is defined as:

(i = X; i > Y; i -= Z)

Parameterizing with Environment Variables

It's possible to substitute a json values in the scenario with the environment variable values. This may help to keep the scenario file unchanged parameterizing the behavior from the command line only.

Example scenario:

{
    "type" : "load",
    "config" : {
        "item" : {
            "data" : {
                "size" : "${ITEM_DATA_SIZE}"
            },
            "dst" : {
                "container" : "${ITEM_DST_CONTAINER}"
            },
            "src" : {
                "file" : "${ITEM_SRC_FILE}"
            }
        },
        "load" : {
            "threads" : "${LOAD_THREADS}",
            "type" : "${LOAD_TYPE}"
        }
    }
}

Example commands to run this scenario:

export ITEM_DST_CONTAINER=bucket1
export ITEM_SRC_FILE=
export LOAD_THREADS=100
export ITEM_DATA_SIZE=1KB
export LOAD_TYPE=create
java -jar mongoose.jar -f <PATH_TO_SCENARIO>.json

CONFIGURATION

Each next step means that the same configuration values from the previous step are overriden by the values from the current step:

Configuration loaded from defaults.json
Configuration loaded from JVM arguments and environment properties
Configuration from any parent job container
Configuration from the current job container

REPORTING

Any "for" job logs the message for each value invocation with the following format:

<TIMESTAMP> I ForJob main Use next value for "<ARG>": <VALUE>