FileBackedHTTPMetadataProvider

The HTTPMetadataProvider type is deprecated

As of V3.4, the HTTPMetadataProvider type is deprecated and will be removed from IdP V4.0. Use a FileBackedHTTPMetadataProvider instead.

The FileBackedHTTPMetadataProvider loads a metadata file from an HTTP server. The provider periodically reloads the metadata file if necessary.

Use this provider with remote metadata

The FileBackedHTTPMetadataProvider is used with remote metadata. See the MetadataManagementBestPractices topic for more information.

The FileBackedHTTPMetadataProvider spools the metadata contents to a local backing file, which is used at startup. As of v3.3.0, the backing file is only used at startup. A refresh operation never consults the backing file since the latter can't possibly represent newer metadata than what is already cached in memory.

Contents

Schema Names and location

The <MetadataProvider> element and the type FileBackedHTTPMetadataProvider are defined by the urn:mace:shibboleth:2.0:metadata schema, which can be located at http://shibboleth.net/schema/idp/shibboleth-metadata.xsd.

Attributes

Any of the Common Attributes, the Reloading Attributes, or the HTTP Attributes may be configured. In addition, a FileBackedHTTPMetadataProvider has a few attributes of its own, some of which are required:

Name	Type	Default	Description
`metadataURL`	URL	required	The URL that the metadata is served from.
`backingFile`	File specificaton	required	Specifies where the backing file is located. If the remote server is unavailable at startup, the backing file is loaded instead.
`initializeFromBackupFile` ^3.3	Boolean	true	Flag indicating whether initialization should first attempt to load metadata from the backup file. If true, foreground initialization will be performed by loading the backing file, and then a refresh from the remote HTTP server will be scheduled to execute in a background thread, after a configured delay. This can improve IdP startup times when the remote HTTP file is large in size.
`backupFileInitNextRefreshDelay` ^3.3	Delay (ISO8601 format)	`PT5S` (5 seconds)	Delay duration after which to schedule next HTTP refresh when initialized from the backing file.

Common Attributes

The following attributes are required on all metadata provider types:

Name	Type	Default	Description
`id`	String	required	Identifier for logging, identification for command line reload, etc.
`xsi:type`	String	required	Specifies the exact type of provider to use (from those listed above, or a custom extension type).

The following attributes are common to all metadata provider types except the ChainingMetadataProvider type:

Name	Type	Default	Description
`requireValidMetadata`	Boolean	true	Whether candidate metadata found by the resolver must be valid in order to be returned (where validity is implementation specific, but in SAML cases generally depends on a `validUntil` attribute.) If this flag is true, then invalid candidate metadata will not be returned.
`failFastInitialization`	Boolean	true	Whether to fail initialization of the underlying MetadataResolverService (and possibly the IdP as a whole) if the initialization of a metadata provider fails. When false, the IdP may start, and will continue to attempt to reload valid metadata if configured to do so, but operations that require valid metadata will fail until it does.
`sortKey`	Integer		Defines the order in which metadata providers are searched (see below), can only be specified on top level `<MetadataProvider>` elements.
The following are advanced settings supporting a new low-level feature allowing metadata lookup by keys other than the unique entityID and are rarely of use to a deployer.
`criterionPredicateRegistryRef` ^3.3	Bean ID		Identifies the a custom `CriterionPredicateRegistry` bean used in resolving predicates from non-predicate input criteria.
`useDefaultPredicateRegistry` ^3.3	Boolean	true	Flag which determines whether the default `CriterionPredicateRegistry` will be used if a custom one is not supplied explicitly.
`satisfyAnyPredicates` ^3.3	Boolean	false	Flag which determines whether predicates used in filtering are connected by a logical 'OR' (true) or by logical 'AND' (false).

Reloading Attributes

The following attributes are common to all reloading "batch-oriented" metadata providers (i.e., FileBackedHTTPMetadataProvider, FilesystemMetadataProvider, and ResourceBackedMetadataProvider):

Name	Type	Default	Description
`parserPoolRef`	Bean ID	shibboleth.ParserPool	Identifies a Spring bean for the XML parser used to parse metadata. Generally should not be changed.
`taskTimerRef`	Bean ID		Identifies a Spring bean containing a Java `TaskTimer` used to schedule reloads. When not set, an internal timer is created. Generally should not be changed.
`minRefreshDelay`	Duration	PT30S	Lower bound on the next refresh from the time calculated based on the metadata's expiration.
`maxRefreshDelay`	Duration	PT4H	Upper bound on the next refresh from the time calculated based on the metadata's expiration.
`refreshDelayFactor`	Real Number (strictly between 0.0 and 1.0)	0.75	A factor applied to the initially determined refresh time in order to determine the next refresh time (typically to ensure refresh takes place prior to the metadata's expiration). Attempts to refresh metadata will generally begin around the product of this number and the maximum refresh delay.
`indexesRef` ^3.3	Bean ID		Identifies an optional `Set<MetadataIndex>` used to support resolution of metadata based on criteria other than an entityID.
`resolveViaPredicatesOnly` ^3.3	Boolean	false	Flag indicating whether resolution may be performed solely by applying predicates to the entire metadata collection, when an entityID input criterion is not supplied.
`expirationWarningThreshold` ^3.4	Duration	PT0S (disabled)	For each attempted metadata refresh (whether or not fresh metadata is obtained), if `requireValidMetadata` is true, and there is a `validUntil` XML attribute on the document root element, and the difference between `validUntil` and the current time is less than `expirationWarningThreshold`, the system logs a warning about the impending expiration.

HTTP Attributes

The following attributes are common to all HTTP metadata providers (i.e., DynamicHTTPMetadataProvider and FileBackedHTTPMetadataProvider).

An HTTP metadata provider includes a default implementation of the org.apache.http.client.HttpClient interface. The attributes in the following subsections control the behavior of the default HTTP client. To override the default client implementation, configure the following attribute:

Name	Type	Default	Description
`httpClientRef`	Bean ID		A reference to an externally defined Spring bean that specifies an `org.apache.http.client.HttpClient` object. This attribute conflicts with and overrides all of the HTTP attributes. See the HttpClientConfiguration topic for more information.

Use of the httpClientRef attribute precludes the use of any and all of the HTTP attributes in the following subsections.

HTTP Connection Attributes

The following attributes apply to the HTTP connections obtained and managed by an HTTP metadata provider:

Name	Type	Default	Description
`connectionRequestTimeout` ^3.3	Duration	Depends on the provider type	The maximum amount of time to wait for a connection to be returned from the HTTP client's connection pool manager. Set to `PT0S` to disable. This attribute is incompatible with `httpClientRef`.
`connectionTimeout` ^3.3	Duration	Depends on the provider type	The maximum amount of time to wait to establish a connection with the remote server. Set to `PT0S` to disable. This attribute is incompatible with `httpClientRef`.
`requestTimeout`	Duration	Depends on the provider type	DEPRECATED: Use `connectionTimeout` instead.
`socketTimeout` ^3.3	Duration	Depends on the provider type	The maximum amount of time to wait between two consecutive packets while reading from the socket connected to the remote server. Set to `PT0S` to disable. This attribute is incompatible with `httpClientRef`.

HTTP Security Attributes

The following security-related attributes apply to any HTTP metadata provider:

Name	Type	Default	Description
`disregardTLSCertificate`	Boolean	false	If true, no TLS certificate checking will take place over an HTTPS connection. This attribute is incompatible with `httpClientRef`. (Be careful with this setting, it is typically only used during testing. See the HttpClientConfiguration topic for more information.)
`disregardSslCertificate`	Boolean	false	DEPRECATED: Use `disregardTLSCertificate` instead.
`basicAuthUser`	String		DEPRECATED: Use `httpClientSecurityParametersRef` instead.
`basicAuthPassword`	String		DEPRECATED: Use `httpClientSecurityParametersRef` instead.
`tlsTrustEngineRef`^3.1	Bean ID		DEPRECATED: Use `httpClientSecurityParametersRef` instead.
`httpClientSecurityParametersRef` ^3.3	Bean ID		A reference to an externally defined Spring bean that specifies an `org.opensaml.security.httpclient.HttpClientSecurityParameters` instance, which consolidates all HTTP security parameters including advanced TLS usage. This attribute conflicts with and overrides any explicit `TrustEngine` implementation configured as an inline <TLSTrustEngine> element. See the HttpClientConfiguration topic for more information.

HTTP Proxy Attributes

The following attributes configure an HTTP proxy for use with an HTTP metadata provider:

Name	Type	Description
`proxyHost`	String	The hostname of the HTTP proxy through which connections will be made. This attribute is incompatible with `httpClientRef`.
`proxyPort`	String	The port of the HTTP proxy through which connections will be made. This attribute is incompatible with `httpClientRef`.
`proxyUser`	String	The username used with the HTTP proxy through which connections will be made. This attribute is incompatible with `httpClientRef`.
`proxyPassword`	String	The password used with the HTTP proxy through which connections will be made. This attribute is incompatible with `httpClientRef`.

HTTP Caching Attributes

The following attributes configure an HTTP cache on an HTTP metadata provider:

Name	Type	Default	Description
`httpCaching`	`"none"`, "`file`", or "`memory`"	Depends on the provider type	The type of low-level HTTP caching to perform. There are three choices: `“none”` indicates the HTTP response is not cached by the client library `"file”` indicates the HTTP response is written to disk (but will not survive a restart) `"memory"` indicates the HTTP response is stored in memory This attribute is incompatible with `httpClientRef` and its value may not be specified as a bean property. Some metadata providers, most notably the reloading "batch-oriented" providers, implement HTTP caching at a higher layer and tend to work best with `httpCaching="none"`.
`httpCacheDirectory`	String		If `httpCaching="file"`, this attribute specifies where retrieved files are to be cached. This attribute is incompatible with `httpClientRef`.
`httpMaxCacheEntries`	Integer	`"memory"`: 50 `"file"`: 100	The maximum number of responses written to cache. This attribute is incompatible with `httpClientRef`.
`httpMaxCacheEntrySize`	Integer	`"memory"`: 1048576 (1MB) `"file"`: 10485760 (10MB)	The maximum response body size that may be cached, in bytes. This attribute is incompatible with `httpClientRef`.

Configuring the HTTP Connection Attributes

For a FileBackedHTTPMetadataProvider, the HTTP Connection Attributes have the following default timeout values:

Name	Default
`connectionRequestTimeout`	`PT60S` (60 seconds)
`connectionTimeout`	`PT60S` (60 seconds)
`socketTimeout`	`PT60S` (60 seconds)

Since the batch metadata refresh process runs in a background thread, a browser user will not notice such a delay.

Configuring the HTTP Caching Attributes

By default, a FileBackedHTTPMetadataProvider does not cache metadata (httpCaching="none"). Unless you've read the code or otherwise know what you're doing, do not override this setting.

Regardless of the value of the httpCaching attribute, a FileBackedHTTPMetadataProvider always issues HTTP conditional GET requests. For this reason, a custom HTTP client is seldom needed.

HTTP conditional GET is independent of caching strategy

Out of the box, a FileBackedHTTPMetadataProvider caches the response headers it needs to subsequently perform HTTP conditional GET but this caching-like behavior is not true HTTP caching. In particular, a FileBackedHTTPMetadataProvider does not cache the full HTTP response by default.

Child Elements

Any of the following child elements may be specified (in order).

Name	Cardinality	Description
`<MetadataFilter>`	0 or more	A metadata filter applied to candidate metadata as it flows through the metadata pipeline
`<TLSTrustEngine>`^3.1	0 or 1	A custom `TrustEngine` used to evaluate TLS server certificates. This element conflicts with and is overridden by the deprecated `tlsTrustEngineRef` attribute.

The former is common to all metadata providers while the latter is common to all HTTP metadata providers. The FileBackedHTTPMetadataProvider type has no child elements of its own.

Examples

A typical use of FileBackedHTTPMetadataProvider is to load (and periodically reload) a metadata aggregate from a remote source via HTTP:

Load a metadata aggregate from a remote server

<!--
    Load (and reload) a signed metadata aggregate from a remote HTTP server.

    This sample configuration assumes: (1) the top-level element of the XML 
    document is signed; (2) the top-level element of the XML document is 
    decorated with a validUntil attribute; (3) the validity interval is two 
    weeks (P14D) in duration; and (4) the server supports HTTP conditional GET.

    The metadata refresh process is influenced by the configured values of 
    the minRefreshDelay attribute (default: PT30S) and the maxRefreshDelay 
    attribute (default: PT4H) and also by any cacheDuration and validUntil 
    attributes in the metadata itself. If the server does not support HTTP 
    conditional GET, the attributes should be adjusted accordingly.
-->
<MetadataProvider id="RemoteMetadataAggregate" xsi:type="FileBackedHTTPMetadataProvider"
                  backingFile="%{idp.home}/metadata/federation-metadata-copy.xml"
                  metadataURL="http://example.org/metadata/federation-metadata.xml">

    <!--
        Verify the signature on the root element of the metadata aggregate 
        using a trusted metadata signing certificate.
    -->
    <MetadataFilter xsi:type="SignatureValidation" requireSignedRoot="true"
		certificateFile="%{idp.home}/conf/metadata/md-cert.pem"/>

    <!--
        Require a validUntil XML attribute on the root element and
        make sure its value is no more than 14 days into the future.
    -->
    <MetadataFilter xsi:type="RequiredValidUntil" maxValidityInterval="P14D"/>

    <!-- Consume all SP metadata in the aggregate -->
    <MetadataFilter xsi:type="EntityRoleWhiteList">
        <RetainedRole>md:SPSSODescriptor</RetainedRole>
    </MetadataFilter>       

</MetadataProvider>

Note that the metadata is loaded (and reloaded) out-of-band and therefore will not interfere with any SAML protocol exchange.

Frequently Asked Questions

What triggers the metadata refresh process?

A FileBackedHTTPMetadataProvider loads (and reloads) metadata in the background, independent of normal IdP operation. The frequency of metadata refresh is influenced by the Reloading Attributes. In particular, the minRefreshDelay and maxRefreshDelay attributes strongly influence the frequency of metadata refresh. Any cacheDuration and validUntil attributes in the metadata itself also influence the process.

What if the metadata resource is large?

Large metadata files consume a significant amount of memory, especially during the reload process (when the IdP must have both the old and the new metadata at hand). Precise memory requirements depend on overall IdP load and other deployment-specific factors but large metadata files (such as those distributed by some federations) may require a system configured with 2GB of memory or more. Ask your federation operator for specific recommendations.

Although metadata refresh can put significant load on the system, all refresh operations are performed in the background, independent of normal IdP operation. Assuming there are no resource limiting factors (such as inadequate memory), end users will not notice the effects of metadata refresh.

What can I do to minimize the impact of metadata refresh?

There are at least two things you can do to help minimize the impact of the metadata refresh process: 1) install and configure adequate memory on your system, and 2) customize the frequency of metadata refresh for optimal performance. In any case, the FileBackedHTTPMetadataProvider implementation has two features that positively affect metadata refresh: background processing and HTTP conditional GET. Neither of these features is configurable, however, so there is nothing further you need to do beyond the two items mentioned above.

What is HTTP conditional GET?

A conforming server will respond to an HTTP conditional GET request (RFC 7232) with a 304 (Not Modified) status code if the target resource has not changed since the last time it was requested. Note that a 304 response does not include a response body, which is more efficient than the corresponding 200 response (especially for large metadata files). More importantly, due to the manner in which the FileBackedHTTPMetadataProvider is implemented, the IdP can safely ignore a 304 response, which precludes the need to redundantly process the metadata. For large signed metadata files, the savings can be quite significant.

How do I customize the frequency of metadata refresh?

To influence the frequency of metadata refresh, configure the following attributes on a FileBackedHTTPMetadataProvider instance: minRefreshDelay, maxRefreshDelay, and refreshDelayFactor. Optimal values for these attributes depend on 1) whether or not the server supports HTTP conditional GET, 2) the life cycle of metadata published on the server, and possibly other factors. Ask the metadata publisher for recommended best practices with respect to published metadata.

What is the backing file used for?

The backing file is only used at startup. If the remote server is unavailable at startup, the backing file is loaded instead and all the configured filters are run on the backing file. If a single filter fails, the backing file is not loaded. For example, if the provider contains a SignatureValidationFilter but the signature on the backing file can not be verified, the entire load operation fails.

What happens if the provider is unable to load the backing file?

If a FileBackedHTTPMetadataProvider is unable to load the backing file at startup, and the failFastInitialization attribute on the provider is set to true (default: true), the metadata initialization process will halt. Additionally, if the service property idp.service.metadata.failFast is set to true (default: false), the IdP as a whole will fail to initialize.

Does the provider support HTTP caching?

Yes, it does so internally, though not across restarts of the system (it will not have a cache tag at startup but will maintain one in memory after the initial request for metadata and do conditional GETs until the next restart).

Identity Provider 3