The overall design of the automatic performance testing framework can be outlined succinctly by the following diagram:

A code change in the Repose GitHub repository on the master branch will trigger a Jenkins job to run. The Jenkins job will run the Ansible playbooks for all defined test cases. The Ansible playbooks, in turn, will provision the necessary cloud resources and begin executing the Gatling simulations for each test case. As Gatling runs, it will pipe the performance data it captures to InfluxDB. Once Gatling execution completes, the cloud resources will be cleaned up, and the Ansible playbook will finish executing. Grafana will query InfluxDB to retrieve the performance data which is available in the Performance Tests dashboard.

The directory structure of the performance tests is organized by category and test. That is, each file is located in a directory named after the specific test which is itself located in a directory named after the category of test. For example, the repose configuration files directory:

contains the category sub-directories filters/, services/, and use-cases/.

Armed with this information, these are the basic steps to create a new test:

A Jenkins job is triggered any time that code on the master branch of the Repose Git Repository changes. The Jenkins job will run the performance tests and publish the results automatically. Design describes the full automated process in more detail.

If you have the appropriate permissions on the Repose Jenkins instance, you can run the performance-test Jenkins job to manually start a single performance test.

Grafana serves as the user interface to performance test data which can be seen in the Performance Tests dashboard.

After test execution has completed, Gatling will generate an HTML report. The report can be found inside an archive residing in the home directory with a file name like SamlFilterSimulationRepose-1496319945650.tar.gz. The exact file name will be included in the output of the Fetch the Gatling results task which will resemble the following snippet:

After the report archive is located, the contents must be extracted before the report can be viewed. That can be done with a command like the following:
tar xvf /home/perftester/SamlFilterSimulationRepose-1496319945650.tar.gz

Once the report has been extracted, it can be viewed by opening the index.html file with a web browser.

If you need to manually run Gatling, you can do so from the Gatling host.

Files and directories of interest:

The Ansible directory is prepared by running the Gradle task buildAnsibleDirectory. The base ansible files are processed using an ant based templating, this allows any build time values to be templated in (such as the gatling version to use). The simulations are copied in from the source directory. Any dependencies of the simulations will be copied into place for placement in the Gatling lib directory.

To reduce redundancy, the Repose team created the performance-test-framework for Gatling. This library includes the AbstractReposeSimulation, which is meant to serve as the basis for most, if not all, simulations.

To utilize the performance-test-framework, its JAR file must be placed in the Gatling lib directory. This is handled along with any other dependencies by the Gradle task buildAnsibleDirectory. The lib directory is a sub-directory of the Gatling home directory.

In the context of the Framework, Jenkins will build the performance-test-framework and copy the resulting JAR into the Ansible directory structure where it will then be copied to the hosts running Gatling.

Run the simple use-case test for 15 minutes with a 5 minute warmup (20 minutes total):

By default, both the Repose test and the origin service test (i.e., Repose is bypassed) are run. You can use Ansible tags to specify that only one of those tests should be run.

You can specify which cloud server flavor to use in the configuration overrides.

By default, the Repose start task will wait 5 minutes for Repose to start up. If you expect Repose to take longer to start (e.g., due to a large WADL), you can increase this timeout using a command like the following:

To get Repose to use Saxon, add a saxon-license.lic file to the repose-aggregator/tests/performance-tests/roles/repose/files/ directory and pass in the following configuration override:

You can set the JVM Options (JAVA_OPTS) used by Repose by setting the following configuration override:

Use the master playbook to run a performance test and immediately clean up afterwards without having to running the individual playbooks.

You can re-run a test using the same cloud resources by simply running the start_test.yml playbook again. You can even specify different configuration overrides in subsequent runs, although there are some limitations. For example, you can enable Saxon for a subsequent run, but you can’t disable it afterwards. Also, if you don’t want the Repose JVM already warmed up from the previous run, you should have Ansible restart the Repose service. This feature is considered experimental.

You can specify the base name of the directory containing the Gatling results in the configuration overrides. For example, if you wanted the base name custom-base-name (resulting in a directory name resembling custom-base-name-repose-1487812914968), you would run:

Running a performance test with a unique naming prefix enables you to run a particular test multiple times simultaneously (each run requires a unique prefix):

If you’re using the stop_test.yml playbook to clean up your cloud resources, you’ll need to include the unique prefix to ensure the correct resources are deleted. If the prefix is not specified, the wrong cloud resources or no cloud resources could end up being deleted, and in both cases, no error will be returned (due to idempotency).

When running a Scripting filter test, the perf_test variable will look slightly different than it does for other tests. The reason is that the scripting language to be used in the test is nested one level down in the perf_test variable value.