...

Regardless of which you use, they all have one architectural design in common: a separation, if only logical, of the heap into 'new' and 'old' spaces, and a sense that old allocations are semi-permanent.  The problem is that a Shibboleth IdP doesn't allocate memory that way,  Most of the 'old' space in the IdP's heap is InCommon metadata (~500MB per parse).  This is dynamic, and is completely replaced daily.

Tuning Garbage Collection for the IdP

...

1. Each InCommon metadata in-core instance consumes about 400-500 MB.
2. InCommon metadata is replaced daily, usually mid-day in the U.S.
3. Most IdPs are not uniformly loaded over a 24-hour period.
4. There's little point in starting out with heap space less than the maximum.

CMS Tuning 

Choose CMS with the java arg (-XX:-UseConcMarkSweepGC).

...

• 2GB total heap
  • Set the new ratio so new gets 1/4 of the heap.

  • Args: (-Xms2g -Xmx2g -XX:NewRatio=34)

• 3GB total heap
  • Set the new ratio so new gets 1/2 of the heap.

  • Args: (-Xms3g -Xmx3g -XX:NewRatio=1)

...

Sample CMS 'old' heap space usage

Image Modified

1. InCommon metadata refresh at 11:17, 11:18:30, 11:19:30, and 11:26.
2. Force GC at 11:27 and 11:28:30.
3. The third refresh triggered a full GC, lasting a total of 7 sec. But recovered only .5GB.
4. A forced GC at 11:27 recovered about .1GB.
5. The second forced GC recovered all unused heap space.

...

• Force a Full Collection Nightly
  • Usually 2 GC runs are required to empty the Old Generation space, 10 or 20 seconds apart.
• Log GC Events
  • GC logging is inexpensive and is very useful.
  • Args: (-XX:+PrintGC -XX:+PrintGCDateStamps -Xloggc:<some_abs_path>)
  • See also: -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles, -XX:GCLogFileSize

G1 Tuning 

Choose G1 with the java arg (-XX:+UseG1GC).

...

Sample G1 heap space usage

Old space	New (Eden) space
Image Modified	Image Modified

1. InCommon metadata refresh at 10:47, 10:53, 10:57, 11:07, 11:21, 13:15.
2. Light load (1-2 login/sec) from start.
3. Heavy load (8-9 login/sec) 11:10-11:23, and 13:30 to end.
4. The heavy load at 11:10 increased Eden space to 1.5GB, reducing old space.
5. The MD refresh at 13:15 shrank Eden space to allow old space to grow without GC.
6. Heavy use at 13:30 grew Eden to 2GB and emptied old of all unused space.
7. No full GC was ever necessary.

...

1. Java settings: -XX:+UseG1GC -Xss24m -Xms3g -Xmx3g -XX:MaxGCPauseMillis=400
2. Load:  logins/min range from about 60 mid day to about 5 mid night.

...

Old space	Eden space
Image Modified	Image Modified

1. Some per-request allocations migrate to old space, but are easily discarded during routine old space GC
   1. These didn't show in load tests
   2. Due to very low number of unique ids used by the load test
2. The old collections at 16:00, 01:00 and 09:30 were due to a threshold (50% of total).
3. Rapid increase of old use at 10:00 was due to a reload of InCommon metadata.
4. Anomaly between 10:30 and 12:00 due to disconnect of jconsole.