HttpClientConfiguration

1 Overview
2 General Configuration
3 Security Configuration
4 Reference

Overview

The IdP software uses the Apache HttpClient library more or less anywhere that this functionality is required, which for most deployers is confined to obtaining metadata from remote sources. The necessary settings to control the behavior of the metadata client code can be handled directly in the metadata-providers.xml file in most cases, so this topic is primarily a reference for people who have very advanced needs or are using other components and features that make use of HTTP clients.

Some of the components that require or at least support the injection of a custom client bean include:

the HTTPConnector for web service access in the attribute resolver
CAS proxy validation
loading service configuration resources from an HTTP server (HTTPResource)
advanced/custom configuration of remote metadata sources (FileBackedHTTPMetadataProvider, DynamicHTTPMetadataProvider)
reporting of metrics via an HTTP collector (MetricsConfiguration)

You might need to dive into this topic further if you want to finely tune settings for different situations or if you have advanced security requirements that go beyond default behavior. A common example would be if you want to control TLS server authentication at a finely-grained level to avoid dependence on the default trust behavior of Java's TLS implementation. This is particularly true if you ever find yourself modifying the "global" Java trust store. That should never be done, since it makes Java upgrades much more troublesome.

In comparison to some of the IdP's features, the veneer here is very "thin". That is, we don't have a lot of layers of abstraction and simplification in place to hide the gory details (this may come as news, but much of the rest of the configuration is very abstracted and simplified from what it could look like).

As a result, this topic is heavy on examples, and going beyond the examples is more a case of reading javadocs and finding the right settings.

General Configuration

A set of Spring factory beans are provided that understand how to build an HttpClient instance with a variety of features and settings. For technical reasons the basic caching behavior of the client is determined by selecting from among three different factory bean types:

Non-Caching
File-based Caching
In-memory Caching

As you would expect, the first provides no explicit caching of results, the second caches results on disk (but not across restarts of the software), and the third caches results in memory. This is HTTP caching; that is, it relies on signaling between the client and web server to detect when to reuse results and supports conditional GET requests with cache control headers to redirect requests into the cache. Essentially they act much like a browser would.

In the vast majority of cases, we strongly recommend sticking with the non-caching one (the first one). The other two don't really provide benefits in most use cases and they tend to behave unexpectedly and they interfere with commonly-used components that already provide HTTP caching features or file backups for resiliency.

Built-In Clients

This parent bean is available to assist in defining custom HttpClient beans for use with various components:

shibboleth.HttpClientFactory

These two additional parent beans are replacements for the caching clients:

Again, we don’t advise using the caching variants in most cases, as they do not behave in the ways most people have tended to expect, and the dominant use cases for these clients is not with larger documents that benefit from this caching. Metadata resolvers, for example, do not rely on these beans at all because the caching is tailored to those use cases and implemented directly.

These beans merely simplify the declaration of the underlying class, and do NOT predefine any properties or non-default behavior at all. You must define any settings you wish to use yourself. You might want to copy some of the now-removed bean declarations above from the old V4 system resource for your own use, but you should use the new parent beans and define separate clients for any use case for which you want to customize the behavior. (Sharing a single client bean across different components generally can work, but when this happens across different reloadable services, an unfixed Spring bug becomes a problem.)

Some common settings are available as properties (mostly commented out for easy definition in services.properties) and influence the behavior of the internally defined client(s) used by various components to avoid the need for custom configuration. Reusing these properties is a suitable approach if you are fine with all use cases sharing client behavior, and your own beans may chose to reference those properties if you prefer that.

Security Configuration

A common use for advanced configuration is to control the server certificate evaluation process. While the IdP typically relies on SAML metadata for this when communicating with servers, this doesn't work for a lot of non-SAML use cases. However, much of the same machinery can be repurposed to support a variety of trust models, and we have integrated this into the HttpClient library. Another use case is to support HTTP authentication such as a name and password to access a resource.

By default, the HttpClient will rely on Java's built-in TLS behavior and code built into the HttpClient library, and perform basic hostname/certificate checks and will rely on the Java global trust store to determine whether to trust the certificate (or its issuer). This is where most non-Shibboleth software stops. In testing scenarios, you can turn off this checking via the connectionDisregardTLSCertificate property, but this should never be used in production.

For more advanced control over this process, you need to perform additional steps, and add more complex Spring wiring involving the org.opensaml.security.httpclient.HttpClientSecurityParameters class, which provides for using a TrustEngine to evaluate the server's certificate. There are a variety of TrustEngine implementations that can perform simple or advanced checks, but the critical difference is that they're contextual. That is, they can be applied to one component such that the rules it uses can be specific to that component alone and not the whole system.

A particularly useful approach is to abandon the fundamentally flawed use of commercial certificates with non-browser-facing services and use self-signed certificates evaluated on the basis of public key comparisons, much like Shibboleth does with SAML, or even certificates issued on an automated short-term basis by dedicated CAs.

If you want an HttpClient bean to use the special TLSSocketFactory we provide that supports a TrustEngine, you MUST provide an HttpClientSecurityParameters instance to the component using the HttpClient bean to configure the security behavior you want.

Repeating: you can tell the HttpClient bean that you want to support a more "advanced mode" of processing, but you configure the rules not on the HttpClient bean but rather on the component using the client bean. This is because the injection of the rules you want to apply have to be added at runtime when the client gets used and not into the client's own data structures. It's a consequence of the library's design.

TrustEngine Examples

To use a TrustEngine, you need to define an HttpClientSecurityParameters bean with a tLSTrustEngine property. While you can define any compatible object, the two most common cases are supported via a pair of factory beans:

shibboleth.StaticExplicitTrustEngine
- Explicitly compares a set of trusted public keys against the key in the server's certificate, with no regard for the rest of the certificate.
shibboleth.StaticPKIXTrustEngine
- Provides a full suite of PKIX processing to the server's certificate, but with dedicated trust authorities and policy rules.

The examples below are semi-complete, in that they include a portion of the "real" component that the security rules are being supplied for. Most of the components that can support this will carry both httpClient and httpClientSecurityParameters properties. Components defined in custom (non-Spring) XML syntax will usually support httpClientRef and httpClientSecurityParametersRef XML attributes to reference Spring beans of the proper type.

Unsurprisingly, the "explicit" engine is a bit simpler to use. You can provide any number of public keys and certificates via resources (file, classpath, even HTTP, though that gets pretty circular here), to drive the engine.

The common case of a rule to compare the key against a certificate loaded from a file looks like this:

Explicit key comparison via a known certificate

<bean id="CustomHttpSecurity" class="org.opensaml.security.httpclient.HttpClientSecurityParameters">
	<property name="tLSTrustEngine">
		<bean parent="shibboleth.StaticExplicitTrustEngine"
			p:certificates="%{idp.home}/credentials/server.pem" />
	</property>
</bean>

<!-- Sample feature we're actually trying to use, which we inject custom rules into. -->
<bean id="PushReporter" parent="shibboleth.metrics.HTTPReporter" c:name="MyCollector"
	p:httpClient-ref="CustomHttpClient"
	p:httpClientSecurityParameters-ref="CustomHttpSecurity"
	p:collectorURL="https://log.example.org/cgi-bin/collector.cgi" />

An example that might be used to support a server rolling over to a new key:

Explicit key comparison against two keys

<bean id="CustomHttpSecurity" class="org.opensaml.security.httpclient.HttpClientSecurityParameters">
	<property name="tLSTrustEngine">
		<bean parent="shibboleth.StaticExplicitTrustEngine">
			<property name="publicKeys">
				<list>
					<value>%{idp.home}/credentials/pubkey1.pem</value>
					<value>%{idp.home}/credentials/pubkey2.pem</value>
				</list>
			</property>
		</bean>
	</property>
</bean>

<!-- Sample feature we're actually trying to use, which we inject custom rules into. -->
<bean id="PushReporter" parent="shibboleth.metrics.HTTPReporter" c:name="MyCollector"
	p:httpClient-ref="CustomHttpClient"
	p:httpClientSecurityParameters-ref="CustomHttpSecurity"
	p:collectorURL="https://log.example.org/cgi-bin/collector.cgi" />

The full range of PKIX options is quite complex, but for basic use cases a factory bean makes it simple. To validate the server's certificate against a fixed CA (name checking is turned off because the HttpClient is already doing this step for us):

PKIX verification with root CA

<bean id="CustomHttpSecurity" class="org.opensaml.security.httpclient.HttpClientSecurityParameters">
	<property name="tLSTrustEngine">
		<bean parent="shibboleth.StaticPKIXTrustEngine"
			p:certificates="%{idp.home}/credentials/rootca.pem"
			p:checkNames="false" />
	</property>
</bean>

<!-- Sample feature we're actually trying to use, which we inject custom rules into. -->
<bean id="PushReporter" parent="shibboleth.metrics.HTTPReporter" c:name="MyCollector"
	p:httpClient-ref="CustomHttpClient"
	p:httpClientSecurityParameters-ref="CustomHttpSecurity"
	p:collectorURL="https://log.example.org/cgi-bin/collector.cgi" />

Other PKIX options include the verifyDepth property to control the chain length, and the cRLs property to supply certificate revocation lists.

Applying a TrustEngine to HttpClient

You should start by configuring a component using the HttpClient with the HttpClientSecurityParameters wiring needed to implement your needs, as in the above examples.

Once that's in place, temporarily configure the HttpClient with the connectionDisregardTLSCertificate flag on, and finally just test it and let it fail and log something like the following:

The category and details in the ERROR will vary by component, but the WARN and SSLPeerUnverifiedException messages will consistently indicate that the component was configured to apply a TrustEngine to the connection but couldn't do so. This may happen after the connection is established, possibly even after data is sent and consumed, but it should happen at the end. If not, something is wrong and has to be corrected to be confident that the eventual configuration will work.

Once the warning/error occurs, only then make the final change to the HttpClient bean to fix the error by overriding the tLSSocketFactory property (note the weird mixed case, that's due to the usual "first character is lower case" property convention in Java beans). The IdP includes a pair of socket factory beans for this purpose, shibboleth.SecurityEnhancedTLSSocketFactory and shibboleth.SecurityEnhancedTLSSocketFactoryWithClientTLS

So the final step to adding advanced TLS support is, for example:

You won't generally need to configure your own instance of the socket factories, because all the interesting settings are part of the other component's configuration, via HttpClientSecurityParameters.

The code as it stands does not generally support TLS Renegotiation, which is most commonly encountered when using a virtual host that applies client TLS to only a subset of paths and not the host as a whole.

Configuring a component using the HttpClient with a private key and certificate for authenticating itself to a server is a simple two step process:

Make sure the HttpClient bean's tLSSocketFactory-ref property points to the shibboleth.SecurityEnhancedTLSSocketFactoryWithClientTLS bean.
Configure the component's injected HttpClientSecurityParameters instance's clientTLSCredential property with an X.509 credential.

The syntax for supplying a keypair can be essentially copied from the credentials.xml file that contains the more "usual" keys and certificates used by the IdP. Note that the beans defined in that file are not visible outside the RelyingPartyConfiguration so if you try to reuse them elsewhere by reference, you'll get errors. The example builds on the previous one and includes server authentication.

Client TLS example

There is now official support for Basic Authentication using our APIs. Other HTTP authentication mechanisms may work but likely require interacting with lower-level HttpClient library objects.

The supported API for this involves injecting a map keyed by HttpHost objects whose value is the username and password to use. This API supports pre-emptive authentication; that is, the code will offer the credentials automatically when contacting the specified host and will not wait for a challenge from the server to do so. It should therefore be used only with TLS-protected hosts with appropriate verification, as described earlier.

We have defined a pair of parent beans to abstract some of the classes needed to define the map:

shibboleth.HttpHost – a parent bean that wraps the Apache client’s HttpHost class
shibbleth.BasicAuthCredentials – a parent bean that wraps the Apache client’s UsernamePasswordCredentials class

The map must be injected into the HttpClientSecurityParameters bean via the preemptiveBasicAuthMap property.

Example of Basic Authentication along with TLS Verification

Since the map discriminates the credentials by host, it’s safe to define all your various credentials in one map and use it wherever needed.

The HTTPClientBuilder object that is the base for the HTTPClient beans has properties for HTTP proxy settings:

Reference

These properties are used in a set of DEPRECATED parent beans that are no longer supported due to a Spring bug that can impact the reloading of service configurations. They remain present for compatibility and for convenience should you choose to use them in your own bean definitions

Name	Type	Default	Description

Name	Type	Default	Description
idp.httpclient.useSecurityEnhancedTLSSocketFactory	Boolean	false	If true, causes the default clients to be injected with a special socket factory that supports advanced TLS features (requires substantial additional configuration)
idp.httpclient.connectionDisregardTLSCertificate	Boolean	false	If the previous property is false, this allows the default TLS behavior of the client to ignore the TLS server certificate entirely (use with obvious caution, typically only while testing)
idp.httpclient.connectionRequestTimeout	Duration	PT1M	TIme to wait for a connection to be returned from the pool (can be 0 for no imposed value)
idp.httpclient.connectionTimeout	Duration	PT1M	TIme to wait for a connection to be established (can be 0 for no imposed value)
idp.httpclient.socketTimeout	Duration	PT1M	Time to allow between packets on a connection (can be 0 for no imposed value) NOTE: This is NOT an aggregate timeout on the whole request but merely between packets. There is no aggregate request timeout at present as the Apache library doesn’t support it.
idp.httpclient.maxConnectionsTotal	Integer	100	Caps the number of simultaneous connections created by the pooling connection manager
idp.httpclient.maxConnectionsPerRoute	Integer	100	Caps the number of simultaneous connections per route created by the pooling connection manager
idp.httpclient.memorycaching.maxCacheEntries	Integer	50	Size of the in-memory result cache
idp.httpclient.memorycaching.maxCacheEntrySize	Long	1048576 (1MB)	Largest size to allow for an in-memory cache entry
idp.httpclient.filecaching.maxCacheEntries	Integer	100	Size of the on-disk result cache
idp.httpclient.filecaching.maxCacheEntrySize	Long	10485760 (10MB)	Largest sze to allow for an on-disk cache entry
idp.httpclient.filecaching.cacheDirectory	Local directory		Location of on-disk cache