The Shibboleth V2 IdP and SP software have reached End of Life and are no longer supported. This documentation is available for historical purposes only. See the IDP v4 and SP v3 wiki spaces for current documentation on the supported versions.

Memcached StorageService

Owned by Former user (Deleted)
Last updated: Feb 04, 2015
5 min read

The IdP Memcached StorageService provides an easy way to connect your Shibboleth IdP to a memcached server, in order to create a stateful cluster. It is intended to be a lightweight alternative to using the Terracotta software.

This extension was announced and is discussed at the shibboleth-users mailing list (see IdP Memcached StorageService implementation and Infinispan Based Storage Service).

Download

The extension is hosted (along with other IdP tools) at the DFN-AAI website.

Current version:
You may download the extension (along with documentation and setup instructions) from this direct link:
https://www.aai.dfn.de/fileadmin/tools/unimr-memcached-idp2.4-rev272.tar.gz (341 KB - Revision 272)

Old versions:
https://www.aai.dfn.de/fileadmin/tools/unimr-memcached.tar.gz (329 KB - Revision 151)
https://www.aai.dfn.de/fileadmin/tools/unimr-memcached-idp2.4-rev218.tar.gz (330 KB - Revision 218)

An in-depth presentation of the extension (only German, though) was held on the 9. DFN-AAI forum in Berlin (Germany) on 19-Oct-2011. It can be found at this link:
http://www.dfn.de/fileadmin/3Beratung/Betriebstagungen/bt55/forum-aai-haim.pdf (811 KB)

Compatibility

This extension has been successfully tested with Shibboleth IdP 2.3 up to 2.4.

Some extensions like the x509 login handler and the Kerberos login handler will not work out-of-the-box, as their data objects utilize transient data fields which are not shared during object serialization, thus a workaround still needs to be implemented.

PLEASE NOTE: If using the default UsernamePassword login handler (as supplied with the Shibboleth IdP), you must not preserve the LDAP principals returned by the vt-ldap JAAS login module, as they contain non-serializable data. In other words, just add the following line to your JAAS configuration (login.config):

setLdapPrincipal="false"

(Alternatively, you may write a custom login handler which does not retain the Set loginSubject.getPrincipals() but rather creates a new java.util.HashSet<Principal>() to be stored within the Subject, containing only the authenticated username.)

Some service providers have been reported e.g. to send SAML AuthnRequest messages with very long but random message IDs (which may then end up as keys within the IdP's replay cache). However, the memcached protocol only allows for keys up to 250 bytes length. In order to handle longer keys, the memcached StorageService will take the key name and calculate the SHA512 hash instead (without collision checking, though). You may turn on INFO logging for the unimr.shib2.UniMrMemcachedStorageService to have oversized keys logged.

Setup instructions

1. Make sure your IdP nodes run behind a load balancer which sticks to the cookie "_idp_authn_lc_key" (for at least a few minutes, if you only want the login to work) or "JSESSIONID" (if you want session stickiness for Shibboleth sessions as well; it will not be sufficient to stick only to the "_idp_session" cookie). If you cannot stick to a cookie, try to stick to the user's IP address (for at least a few minutes), this should work as well.

2. Set up a memcached daemon (or two) and remember the hostname and port. Remember that memcached throws old entries away if it runs out of memory, so
give it at least a few hundred megabytes of RAM.

3. Compile the source and create a jar file (unimr-memcached.jar) from it (this may require additional jars from your Shibboleth IdP), or simply use the precompiled jar file (if available).

4. Then copy this jar to the lib subdirectory from your Shibboleth IdP's setup directory. Also copy the file lib/spymemcached-2.7.jar to that directory.

5. In your webapp/WEB-INF/web.xml , add the following lines below all the other filters:

    <!-- Store the modified session object in the memcached storage service -->
    <filter>
        <filter-name>UniMrMemcachedServletFilter</filter-name>
        <filter-class>unimr.shib2.UniMrMemcachedServletFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>UniMrMemcachedServletFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

6. In your conf/internal.xml , replace the bean "shibboleth.StorageService" with the following bean:

    <bean id="shibboleth.StorageService"
        class="unimr.shib2.UniMrMemcachedStorageService"
        depends-on="shibboleth.LogbackLogging">
        <constructor-arg value="idp-memcached.example.org:11211" />
        <constructor-arg value="sha512" />
    </bean>

The hostname:port of your memcached server is passed as a constructor argument and must be edited to fit your memcached setup. You may even use a space-separated list of servers here in order to spread the data over multiple instances of memcached (using a hash function). If one memcached server fails, its data is lost, but any affected session may still be alive within the local store of the IdP node which the user used to authenticate.
(This is why it is a good idea to have your load balancer stick to the "JSESSIONID" cookie.)
(A failover implementation with one active and several passive servers is not planned anymore, but you may have a look for "repcached" which is a memcached port which includes replication.)

As a second constructor argument, you may select a hashing algorithm which will be used for keys exceeding the memcached limit of 250 bytes:
a) "sha1" will calculate the 40 hex chars long SHA1 hash,
b) "sha512" (default) will calculate the 128 hex chars long SHA512 hash,
c) "sha1mixed" or "sha512mixed" will calculate the respective hash, but instead of replacing the whole key, only the end will be overwritten (so you get a semi-readable key for debugging purposes).
Implementation note: Regardless of the hashing algorithm you choose, there is no check for hash collisions. Each key will still be prepended by the StorageService's partition name (like "session", "transientId" or "replay"), and the regular keys generated by the IdP for session or transientId objects are much shorter than the memcached limit. Thus there is only a very small chance for the replay cache objects (their keys are based on the SAML message ID sent by the service provider which may in some rare cases be of unreasonable length) to get overwritten by a hash collision.

Please note: The "shibboleth.StorageServiceSweeper" in your file conf/internal.xml just stays in place, as it is needed to remove expired entries from UniMrMemcachedStorageService's local object cache.

7. Reinstall your Shibboleth IdP by calling the appropriate install.sh file.

8. Restart tomcat and everything should be fine.

Logging

To enable INFO, DEBUG or TRACE logging only for the new StorageService and servlet filter, add the following entries to your logging.xml configuration file:

<logger name="unimr.shib2" level="DEBUG" />

Finally, restart tomcat.

History

21-Jul-2011: Revision 151 - First release.

03-Nov-2011: Revision 179 - Second release (never published officially).
- Updated readme.
- Removed quick-fix for Shibboleth security advisory
  http://shibboleth.internet2.edu/secadv/secadv_20110718.txt
  (the quick-fix code was never reached AND did not fix the problem).
- Log events are now logged correctly under the respective class names
  (unimr.shib2.UniMrMemcachedServletFilter and
  unimr.shib2.UniMrMemcachedStorageService).
- Reworked the code (removed copy-paste areas and put the code into
  methods, improved log output).
  Thanks to Petra Berg from Humboldt University of Berlin, Germany.

16-May-2013: Revision 218 - Third release.
- Fixed an issue with IdP 2.4's new SLO functionality
  (the SLO removes the session from the StorageService which lead to
  a null-pointer exception in the UniMrMemcachedServletFilter when
  retrieving the session, displaying only the default Shibboleth
  error page).

04-Feb-2015: Current release.
- Added SHA1 / SHA512 hashing for oversized keys (exceeding the 250
  bytes memcached limit).

License

The IdP Memcached StorageService is released under the Apache License, Version 2.0.

See http://www.apache.org/licenses/LICENSE-2.0

Contact

Manuel Haim, haim@hrz.uni-marburg.de