The Shibboleth IdP V4 software has reached its End of Life and is no longer supported. This documentation is available for historical purposes only. See the IDP5 wiki space for current documentation on the supported version.
IdP Heap Management
The Shibboleth IdP's periodic reloading of InCommon metadata makes effective use of heap space and garbage collection less than obvious. This page describes some of the issues and offers some suggestions.
- 1 Java Garbage Collection
- 2 Tuning Garbage Collection for the IdP
- 2.1 CMS Tuning
- 2.1.1 Sizing the Heap
- 2.1.2 Timing Collection of the Old Generation
- 2.1.2.1 Sample CMS 'old' heap space usage
- 2.1.2.2 Suggestions
- 2.2 G1 Tuning
- 2.2.1 Sizing the Heap
- 2.2.2 Timing Collections
- 2.2.2.1 Sample G1 heap space usage
- 2.1 CMS Tuning
- 3 Suggestions for GC Monitoring
- 4 A Day in the Life of an IdP
Java Garbage Collection
Heap memory is Java's temporary storage. Management of the heap is the job of a Garbage Collector (GC). Periodically the collector looks through the heap for allocations that are no longer referenced and reclaims them (a collection), making them available for another allocation.These collections are usually Stop-the-World (STW) events---the application is paused during collection.
Garbage Collector's divide the heap into new and old areas (generations). The new generation is the most dynamic. New allocations and most collections occur here. When an object has survived several collections it is moved into the old generation. When the old generation fills a full collection is necessary to collect and reorganize memory. Old generation collections tend to cause the longest application pauses.
Java has several collector options:
Serial GC. This is the basic, single threaded GC. It is useful on single processor machines with small heaps. Collection is STW.
Parallel GC. This is essentially a multi-threaded version of the Serial GC. It also tends to be STW.
Concurrent Mark Sweep (CMS). This multi-threaded collector is able to perform much of its work on the old generation in parallel with the application's normal processing. This allows it to reduce the length of STW pauses and is most useful for real-time systems, e.g. web services. CMS uses the same new generation collector as the Parallel GC. This is the default GC for Java 1.8.
Garbage-First (G1). This multi-threaded collector attempts to meet configured maximum pause time goals. It is a compacting collector, and can often obviate full STW old generation collections. Although G1 has a sense of new and old allocations, it does not divide the heap into new and old regions.
Regardless of which you use, they all have one architectural design in common: a separation, if only logical, of the heap into 'new' and 'old' spaces, and a sense that old allocations are semi-permanent. The problem is that a Shibboleth IdP doesn't allocate memory that way, Most of the 'old' space in the IdP's heap is InCommon metadata (~500MB per parse). This is dynamic, and is completely replaced daily.
Tuning Garbage Collection for the IdP
Some considerations for IdP GC tuning:
Each InCommon metadata in-core instance consumes about 400-500 MB.
InCommon metadata is replaced daily, usually mid-day in the U.S.
Most IdPs are not uniformly loaded over a 24-hour period.
There's little point in starting out with heap space less than the maximum.
CMS Tuning
Choose CMS with the java arg (-XX:+UseConcMarkSweepGC).
Sizing the Heap
The CMS GC is fairly strict with its allocation of new and old generation heap space. For the Old Generation we need a some score of MB for non-metadata persistent allocations, and enough space for two parses of InCommon metadata—so we can refresh it without triggering a full collection. That's about a minimum of 1.5GB for the old space. The new generation is very dynamic, but the IdP doesn't require large blocks at a time. We could get by with 500MB for the new.
2GB total heap
Set the new ratio so new gets 1/4 of the heap.