This page aggregates load test results with the hope that sharing facilitates test practices and supports performance improvements in IdP software.

University of Chicago IdPv3.1.2 Test Results

Test Environment

IdP Configuration:

• CentOS 7 VM, t2.medium (2 core, 4GB RAM)
• Docker 1.6.2
• JVM: Java(TM) SE Runtime Environment (build 1.8.0_45)
• Pooled LDAP for authentication handler and attribute resolver: 12 attributes
• Servlet Container: Jetty 9.2.9.v20150224

Test client:

• Single test node running JMeter 2.13 command line client on Windows 7 with 8G RAM and 2 CPUs.
• Test client connected to IdP via network.

Test Plan

The shibboleth-unsolicited-sso-test.jmx test plan has two different test cases. The first test case issues an HTTP GET to the UChicago NotSilver SSO endpoint, which responds with a login form. The test then sends a POST to the login form to get a SAML2 response which is then sent as a POST to the IdP to complete the authentication and generate an encoded SAML assertion in the response. The second test case spawns threads that issue an HTTP GET to the unsolicited SSO endpoint, which responds with a login form. The test then sends a POST request with credentials to complete the authentication and generate an encoded SAML assertion in the response. These test cases were ran using a 15 minute run-time over a various amount of threads until the SP and the IdP were unable to handle any more threads.

Test Results

Unsolicited SAML Response

Observations/Interpretation

• Throughput remained minimal until a user count of about 25.
• After this initial ramp up of the throughput it remained excellent until the IdP was unable to handle any more traffic.
• The load did have an impact on the average latency, yet this value remained relatively low.
• Average latency remained under 5 seconds until 750 threads test, under 10 seconds until 1500 threads test, and under 15 seconds until 2500 threads test .
• The max latency remained relatively low until about 125 users where it jumped up into non-ideal response times.
• These non-ideal max response times remained consistent until the IdP was unable to handle any more traffic.
• Shown in another test the performance ceiling of the SP was significantly lower than the IdP performance ceiling.
• At 1500 threads the SP for UChicago was unable to respond, and required a restart to resume traffic.
• At 4000 threads the IdP became unresponsive, however after traffic was reduced it was able to recover without a restart.
• The HTTP GET SSO endpoint was the leading failure point of the IdP test case.

University of Memphis IdPv3.1.1 Test Results

Test Environment

IdP Configuration:

• CentOS 6 VM with 12G RAM, 1 CPU socket - 4 cores.
• Cisco ACE SLB.
• JVM: Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
• JVM configuration flags: -server -Xms512 -Xmx4000m
• Pooled LDAP for authentication handler and attribute resolver: 12 attributes
• Servlet Container: Jetty 9.2.7

Test Plan

The shibboleth-unsolicited-sso-test.jmx test plan was executed against three different IdP configurations.   Each configuration was hit successively with 1, 25, 50, 75, 100, and 125 threads. In all cases the IdP was configured to encrypt SAML Assertions.  Latency times are in seconds and reflect the "POST Login Credentials" portion of the test.

Test Results

Single IdP Host - Memory-based StorageService

Single IdP Host - JPA-based StorageService (remote postgres)

Three Node IdP Cluster - JPA-based StorageService (remote postgres)

Observations/Interpretation

• While load had an impact on average latency, throughput remained excellent even in high-load situations.
• Usage of the JPA data store had a small latency cost (Roughly 10% ?).
• In light-load situations, the in-memory session cache had a throughput advantage, while in high-load situations the database cache performed slightly better.
• Throughput seems to be limited by CPU (because of the cryptographic operations) and latency is determined primarily by the responsiveness of back-end infrastructure (LDAP/DBs).
• At no point did the postgres DB server appear to be under significant stress.
• It is unlikely that this test plan found the actual performance ceiling of the 3-node cluster, since the network interface of the JMeter host was pegged during the 125-thread test.  

High Session Count

Another area of interest was the extent to which “high” session counts adversely affect the overall performance of the IdP.  Although no metrics were recorded, an additional test was conducted against the "Single IdP Host - JPA-based StorageService" configuration.  The IdP was preloaded with 100,000 active sessions prior to the running of a test plan with 25 threads.  No discernible degredataion in latency or throughput was found.

Virginia Tech IdPv3 Alpha2 Test Results

Test Environment

IdP Configuration:

• Single IdP node running on CentOS 6 VM (Linux KVM) with 4G RAM and 2 CPUs.
• The IdP host is behind a Brocade ServerIron load balancer.
• JVM configuration flags: -server -Xmn256m -Xms768m -Xmx2560m -XX:MaxPermSize=256M -XX:+DisableExplicitGC -XX:+UseParallelOldGC
• 1 pooled LDAP authentication handler
• 2 pooled LDAP attribute resolvers: 1 for user attributes, 1 for entitlements
• MemoryStorageService
• ComputedPersistentIdGenerationStrategy

Test client:

• Single test node running JMeter command line client on CentOS 6 VM (Linux KVM) with 4G RAM and 2 CPUs.
• Test client physically near IdP connected via 1GbE links.

Test Plan

The shibboleth-unsolicited-sso-test.jmx test plan spawns 100 threads that issue an HTTP GET to the unsolicited SSO endpoint, which responds with a login form. The test then sends a POST request with credentials to complete the authentication and generate an encoded SAML assertion in the response. The assertion was encrypted in our case.

Test Results

We conducted another load test run with the YourKit Java Profiler attached to the Java process running the IdP to facilitate performance analysis under load. The IdP was started and allowed to settle down for approximately 1 minute before the test client started the test run, then the process was given approximately 30s after the test completed before a YourKit snapshot was taken. The profiler was configured for CPU sampling to provide more accurate CPU consumption details.

The YourKit snapshot (compressed) is available for detailed review using the YourKit Java Profiler application, but some highlights of the snapshot are worth noting.

CPU hot spots:

Memory consumption summary:

Heap memory

• Used: 575M
• Allocated: 757M
• Limit: 2.4G

Non-heap memory

• Used: 73M
• Allocated: 73M
• Limit: 304M

Top 20 objects by memory consumption:

Garbage collection summary:

• Minor collections (PS Scavenge): 0.18/s ± 0.39/s
• Major collections (PS MarkSweep): 0