IdP Load Testing

Load testing any new production level system is an important part of ensuring that the service is ready for prime time use. The Shibboleth team makes available a set of tests that can be used with The Grinder. While you can get the test scripts and run them in your own installation of Grinder it is much easier to check out a pre-built kit and then run the packaged tests.

The testing framework provided here is provided as a courtesy by the IdP maintainers. By using it you are taking responsibility for reading, and understanding, The Grinder and Shibboleth documentation and diagnosing issues that you may have. Extensive support will not be provided on the mailing list. If you do not know things like your service provider's assertion consumer service or session initiation URL this tool is not for you.

Follow this link for some typical throughput numbers for some environments. Every environment will be different and this should be used to estimate, not as a substitute for your own load tests.

Load Testing Framework Basics

Requirements

Unix OS based machine(s)
Subversion client
Java 1.5

Getting and Installing the Framework

SVN check out the testing kit from https://svn.shibboleth.net/extensions/java-loadtest/trunk
Move the grinder-framework directory to wherever you wish to install it. This location will be referred to as GF_HOME for the rest of the document.
Copy the test packages, the saml1sso_idp, saml2sso_idp, and sp_idp directories, to GF_HOME/tests

Installation is now complete.

Running the Framework

The load testing framework is composed of two parts, an agent and a console. The agent performs the actual work while the console, a GUI application, collects the results from agents and displays some metrics based on those results

To run the console execute the GF_HOME/bin/console.sh command. This will start the console which will listen on port 6372 for agent results. Be sure that this port is accessible to the agents you start.

To run the console execute the GF_HOME/bin/agent.sh <test_name> command. The name of the test is dependent on which test you will run and is given in the documentation for those tests. You can install agents on more than one machine and have them report back to a single client. This is useful if one agent machine is not capable of generating enough load.

Test Setup and Configuration

Each test package contains a test.properties which hold configuration information for the given test. The later half of these options are Grinder specific properties. In most cases you'll need to change the following properties.

Property Name	Value
grinder.consoleHost	The IP address of the machine on which the console is running
grinder.processes	Number of grinder processes to run. Should be equal to the number of CPU cores on the agent machine.
grinder.threads	Number of unique clients, per process, that will connect to the IdP
grinder.runs	Number of times each client will connect to the IdP, 0 (zero) means to keep going until the console indicates the test should be stopped.

SAML 1, IdP Only

This test creates a Shibboleth SSO authentication request (the requests used to initiate an SSO flow in Shibboleth 1), authenticates to the IdP using either HTTP BASIC or FORM based authentication, and checks that a SAML response was returned.

Test Name: saml1sso_idp

Configuration Properties

Property Name	Value
shib.idp.scheme	HTTP scheme for the SSO endpoint on the IdP - allowed values: `http`, `https`
shib.idp.host	host name of the IdP
shib.idp.port	port on which the IdP listens
shib.idp.path	Servlet context path. This is normally `/idp`
shib.idp.auth	type of authentication to be performed - allowed values: `BASIC`, `FORM`
shib.idp.realm	name of the HTTP BASIC authentication realm, unused in the case of FORM authentication
shib.user.name	user name with which to authenticate to the IdP
shib.user.pass	password with which to authenticate to the IdP
shib.sp.id	entity ID of the service provider from which the made up request is said to originate
shib.sp.acsURL	assertion consumer service URL of the service provider from which the made up request is said to originate

SAML 2, IdP Only

This test creates a SAML2 SSO authentication request, authenticates to the IdP using either HTTP BASIC or FORM based authentication, and checks that a SAML response was returned.

Test Name: saml2sso_idp

Configuration Properties

Property Name	Value
shib.idp.scheme	HTTP scheme for the SSO endpoint on the IdP - allowed values: `http`, `https`
shib.idp.host	host name of the IdP
shib.idp.port	port on which the IdP listens
shib.idp.path	Servlet context path. This is normally `/idp`
shib.idp.auth	type of authentication to be performed - allowed values: `BASIC`, `FORM`
shib.idp.realm	name of the HTTP BASIC authentication realm, unused in the case of FORM authentication
shib.user.name	user name with which to authenticate to the IdP
shib.user.pass	password with which to authenticate to the IdP
shib.sp.id	entity ID of the service provider from which the made up request is said to originate
shib.sp.authnreq	authentication request template - this property should not be modified

SP to IdP Request

This test accesses an SP, carries the created request to the IdP, authenticates to the IdP using either HTTP BASIC or FORM based authentication, and checks that a SAML response was returned. Note, the SP endpoint used must NOT require a discovery process.

Test Name: sp_idp

Configuration Properties

Property Name	Value
shib.idp.scheme	HTTP scheme for the SSO endpoint on the IdP - allowed values: `http`, `https`
shib.idp.host	host name of the IdP
shib.idp.port	port on which the IdP listens
shib.idp.path	Servlet context path. This is normally `/idp`
shib.idp.auth	type of authentication to be performed - allowed values: `BASIC`, `FORM`
shib.idp.realm	name of the HTTP BASIC authentication realm, unused in the case of FORM authentication
shib.user.name	user name with which to authenticate to the IdP
shib.user.pass	password with which to authenticate to the IdP
shib.sp.url	URL the will trigger the SP-initiated SSO process. This may be either an SP protected resource or an SP session initiator.

Browser not supported