Capacity Planning

Capacity planning includes analysis of system components, environment, and anticipated load in order to build a system that performs as expected. Load testing is a vital tool to ensure that capacity plans have achieved their intended goals.

Planning an IdP Deployment

TBD

Load Testing

You must load test your system to have confidence that the capacity plan has been satisfied. We offer some guiding principles below that provide a context for the tools and techniques used for load testing.

Know what you're measuring.

In order to know what you're measuring, you must first determine what you want to measure. While most test efforts naturally begin with the goal in mind, the challenge is to develop a test plan and a testing environment that actually accomplishes the goal. Most of the effort of testing should go into this step; after all, the data is only as good as the test. An example may help illustrate the point. It makes little sense for a system load test to be conducted by clients over the public Internet (or even a large institutional network) unless the effect of network latency is intended to be included in the scope of the test. In most cases deployers simply want to know the theoretical maximum throughput of a system; a bottleneck created by network latency could result in reduced system throughput. In that case the test has measured network throughput instead of the intended system throughput.

Keep it simple.

Create the simplest test plan and testing environment that is capable of measuring the desired aspects of a system. Simplicity favors measurement and data analysis and is arguably a goal in itself.

One good result means nothing.

All tests include systematic and random variation. In order to obtain meaningful results, it is necessary to perform multiple test runs for each variable value. Statistical analysis is a natural consequence of multiple test runs and is itself a sign of sound test methodology.

Load Testing Tools and Techniques

Apache JMeter is a free and capable testing tool that can be used to load test the IdP. A simple parameterized test plan, shibboleth-unsolicited-sso-test.jmx, is attached to this page. The test flexes the authentication and SAML messaging subsystems to measure system throughput under common load patterns.

JMeter Unsolicited SSO Test Usage

TEST="/path/to/shibboleth-unsolicited-sso-test.jmx"
LOG="/path/to/jmeter.log"
PROPS="-Dcredentials=/path/to/users-and-passwords.csv"
PROPS="$PROPS -Didphost=idp.example.org"
PROPS="$PROPS -Didpport=443"
PROPS="$PROPS -Didpcontext=/idp"
PROPS="$PROPS -Dproviderid=https://sp-entity-id-you-trust.example.org/shibboleth"
export JVM_ARGS="-Xms1024m -Xmx1024m $PROPS"
export JAVA_HOME="/path/to/jdk"
export PATH="$JAVA_HOME/bin:$PATH"
/path/to/jmeter/bin/jmeter -n -j $LOG -t $TEST

The credentials file should be a comma-delimited list of username/password pairs, one per line. The test by default allocates 100 threads to perform authentication at the unsolicited SSO endpoint using each credential pair in turn. The test runs for 5 minutes (by default) and writes results to a file named shibboleth-unsolicited-sso-test-result-YYYYmmdd-HHMMSS.csv in the user's home directory. The summarize-results.py file, also attached to this page, may be used to process the test results and report a summary for each test run.