HttpClientConfiguration

Overview

The IdP software uses the Apache HttpClient library anywhere more or less anywhere that this functionality is required, which for most deployers is confined to obtaining metadata from remote sources. The necessary settings to control the behavior of the metadata client code can be handled directly in the metadata-providers.xml file in most cases, so this topic is primarily a reference for people who have very advanced needs or are using other components and features that make use of the client.

Some of the components that require or at least support the injection of a custom client bean include:

• loading service configuration resources from an HTTP server (HTTPResource)
• advanced/custom configuration of remote metadata sources (HTTPMetadataProviders, DynamicHTTPMetadataProvider)
• reporting of metrics via an HTTP collector (MetricsConfiguration)
• a forthcoming HTTPDataConnector for web service access in the attribute resolver

Even then, many basic needs can be met with a set of built-in Spring beans and properties (described below) that provide basic client functionality, at the cost of "global" behavior (meaning all components would be relying on common settings).

You might need to dive into this topic further if you want to finely tune settings for different situations or if you have advanced security requirements that go beyond default behavior. A common example would be if you want to control TLS server authentication at a finely-grained level to avoid dependence on the default trust behavior of Java's TLS implementation. This is particularly true if you ever find yourself modifying the "global" Java trust store. That should never be done, since it makes Java upgrades much more troublesome.

In comparison to some of the IdP's features, the veneer here is very "thin". That is, we don't have a lot of layers of abstraction and simplification in place to hide the gory details (this may come as news, but much of the rest of the configuration is very abstracted and simplified from what it could look like). So two points: this topic will eventually be heavy on examples, and going beyond the examples is more a case of reading javadocs and finding the right settings than reading some documentation to magically impart the right answer.

General Configuration

A set of Spring factory beans are provided that understand how to build an HttpClient instance with a variety of features and settings. For technical reasons the basic caching behavior of the client is determined by selecting from among three different factory bean types:

• net.shibboleth.idp.profile.spring.relyingparty.metadata.HttpClientFactoryBean
• net.shibboleth.idp.profile.spring.relyingparty.metadata.FileCachingHttpClientFactoryBean
• net.shibboleth.idp.profile.spring.relyingparty.metadata.InMemoryCachingHttpClientFactoryBean

As you would expect, the first provides no explicit caching of results, the second caches results on disk (but not across restarts of the software), and the third caches results in memory. This is HTTP caching; that is, it relies on signaling between the client and web server to detect when to reuse results and supports conditional GET requests with cache control headers to redirect requests into the cache. Essentially they act much like a browser would.

Obviously if you expect the content to be fully dynamic, as with a web service, the non-caching choice is appropriate, while components that request large, infrequently-modified files will gain benefits from caching. In no case does this caching supply real high-availability resilience. The IdP tends to implement HA features itself on top of the client library to provide more control and reliability.

Built-In Clients

An instance of each of the above clients is defined by default, either for use directly or as a parent template for your own beans:

• shibboleth.NonCachingHttpClient
• shibboleth.FileCachingHttpClient
• shibboleth.MemoryCachingHttpClient

Many common settings are exposed as properties (mostly commented out for easy definition in services.properties).These properties are a mix of global settings that configure each client type and specific settings controlling the caching behavior of the individual types. This is a suitable approach if you have one use case, or are fine with all use cases sharing client behavior.

Custom Clients

If you have more advanced needs, just define your own bean that inherits from one of these, and override any settings as needed. For example, instead of relying on the global idp.httpclient.socketTimeout property, perhaps you want to define a special client instance with a shorter timeout:

<bean id="ShortTimeoutHttpClient parent="shibboleth.NonCachingHttpClient" p:socketTimeout="PT5S" />

Now you have a bean name you can inject into other components that support an httpClientRef property that will behave differently than the defaults.

Security Configuration

A common use for advanced configuration is to control the server certificate evaluation process. While the IdP typically relies on SAML metadata for this when communicating with servers, this doesn't work for a lot of non-SAML use cases. However, much of the same machinery can be repurposed to support a variety of trust models, and we have integrated this into the HttpClient library.

By default, the HttpClient will rely on Java's built-in TLS behavior and code built into the HttpClient library, and perform basic hostname/certificate checks and will rely on the Java global trust store to determine whether to trust the certificate (or its issuer). This is where most non-Shibboleth software stops. In testing scenarios, you can turn off this checking via the connectionDisregardTLSCertificate property, but this should never be used in production.

For more advanced control over this process, you need to perform additional steps, and add more complex Spring wiring involving the org.opensaml.security.httpclient.HttpClientSecurityParameters class, which provides a mechanism for applying advanced security options to components that use an HttpClient.

You also need to be very careful because leaving out a step can introduce security holes. In particular, if you want an HttpClient bean to use the special TLSSocketFactory we provide that supports these features, you MUST provide an HttpClientSecurityParameters instance to the component using the HttpClient bean to configure the security behavior you want.

Repeating: you can tell the HttpClient bean that you want to support "advanced mode", but you configure that mode not on the HttpClient bean but rather on the component using the client bean. This is because the injection of the rules you want to apply have to be added at runtime when the client gets used and not into the client's own data structures. It's a consequence of the library's design. The risk is that if you tell the client you want "advanced mode" but then don't configure the component that's using it properly, the default Java behavior is also skipped, and you get no security at all.

The examples below, which are necessarily specific to particular components' use of the HttpClient, demonstrate how this works, but in practice what it means is that you should always start by configuring a component using the HttpClient with the HttpClientSecurityParameters wiring needed to implement your needs, let it fail and log something like the following:

19:04:53.364 - 127.0.0.1 - WARN [org.opensaml.security.httpclient.HttpClientSecuritySupport:98] - Configured TLS trust engine was not used to verify server TLS credential, the appropriate socket factory was likely not configured
19:04:53.366 - 127.0.0.1 - ERROR [net.shibboleth.idp.profile.impl.ResolveAttributes:299] - Profile Action ResolveAttributes: Error resolving attributes
net.shibboleth.idp.attribute.resolver.ResolutionException: Data Connector 'webservice': HTTP request failed
 at net.shibboleth.idp.attribute.resolver.dc.http.impl.HTTPDataConnector.retrieveAttributes(HTTPDataConnector.java:90)
Caused by: javax.net.ssl.SSLPeerUnverifiedException: Evaluation of server TLS credential with configured TrustEngine was not performed
 at org.opensaml.security.httpclient.HttpClientSecuritySupport.checkTLSCredentialEvaluated(HttpClientSecuritySupport.java:100)

Then make the final change to the HttpClient bean to fix the error by overriding the tLSSocketFactory property (note the weird mixed case, that's due to the usual "first character is lower case" property convention in Java beans). The IdP includes a pair of socket factories declared for this purpose, shibboleth.SecurityEnhancedTLSSocketFactory and shibboleth.SecurityEnhancedTLSSocketFactoryWithClientTLS

So the final step to adding advanced TLS support is, for example:

<bean id="SecurityEnhancedHttpClient parent="shibboleth.NonCachingHttpClient"
	p:tLSSocketFactory-ref="shibboleth.SecurityEnhancedTLSSocketFactory" />

You won't generally need to configure your own instance of the socket factories, because all the interesting settings are part of the other component's configuration, via HttpClientSecurityParameters.

TLS Server Authentication

TBD

TLS Client Authentication

TBD

HTTP Authentication

TBD

Reference

Beans

Properties