MetricsConfiguration

Current File(s): conf/admin/metrics.xml, conf/admin/general-admin.xml (V4.0), conf/admin/admin.properties (V4.1+), conf/logback.xml
Format: Native Spring, Logback, Properties (V4.1+)

1 Overview
2 General Configuration
- 2.1 Logger-Driven Filtering
3 Reporting
- 3.1 REST API
- 3.2 HTTP Reporters
4 Per-Profile Instrumentation
5 Reference

Overview

The IdP includes a framework for instrumentation, diagnostics, and performance management that complements the logging support and integrates with it to allow tuning of instrumentation overhead. The framework is based on the Metrics library and includes a number of related features:

some built-in metrics providing low-level system information, software versions, etc.
a small number of built-in metrics exposed by the IdP itself, including a customizeable set of configuration properties
a facillity for installing timers and counters to measure request handling
an API for accessing metrics (all or a subset) in JSON format
the ability to schedule reports of metrics in JSON format out to an http(s) collection point

Most of these features are configured whole or in part via conf/admin/metrics.xml and are discussed below.

General Configuration

The core bean in the metric system is a MetricRegistry that stores all of the active metrics in the system for use by various other reporting objects. Certain kinds of metrics are subject to your control and can be selectively installed or skipped based on what gets registered, at startup. By default various sets of metrics are installed for you via a bean in metrics.xml that inherits from shibboleth.metrics.RegisterMetricSets, which is a way of using Spring to call the registration method. The pre-installed metrics largely overlap with the information available from the now-deprecated status page.

Also provided by default, but commented out, are references to a number of lower level metric sets related to the JVM.

The way to think about these particular metrics is that they're "external" to the IdP for the most part, in the sense that they're objects that access public interfaces of IdP, OpenSAML, and Java components, and use them to report information. Because they're external, they can be selectively installed (or not) just by registering them (or not).

A lot of the other interesting metrics in the system are "internal", in the sense that they're implemented either wholly, or in conjunction with, internal classes themselves, and typically are not something you specifically need to register yourself. Instead, you have the ability to enable or disable all metrics using a MetricFilter.

Logger-Driven Filtering

The registry of metrics is extended by Shibboleth with the ability to "filter" metrics by reusing the logging configuration's ability to adjust the logging level of particular categories hierarchically. By naming metrics in a similar way, it's possible to effectively enable and disable metrics at runtime, which can have very small (or in rare cases, less small) impacts on performance. You may want to leave a metric configured all the time, but toggle it on only at particular times. Using the logging layer is easy, flexible, reloadable, and ignorable if you don't care about the overhead.

Every metric has a name in dotted-separator notation (i.e., the same as a logging category) and you can control the on/off status of a particular metric or set of metrics by manipulating the level of a corresponding logging category with the prefix "metrics." That is, a metric named "net.shibboleth.idp.version" is controlled by a logging category named "metrics.net.shibboleth.idp.version".

By default, all metrics are enabled at the "INFO" level.

Reporting

There are two built-in mechanisms provided for reporting out metrics, along with a variety of options available within the Metrics library itself that deployers may choose to wire in with Spring. For example, it's possible to expose metrics via JMX, through logging, via the console, to databases, etc.

The two generic approaches that come "ready to deploy" are both centered around use of JSON as a reporting format, and both pull (via a REST API) and push (via an HTTP reporter) are provided.

REST API

As a (greatly enhanced) replacement for the status page, an Administrative flow is provided for access to specific metrics, named groups, or all of the metrics in the system, in JSON or JSONP format.

High level configuration of this flow is handled by conf/admin/general-admin.xml, but in most cases the defaults suffice and the real configuration, if any, takes place in conf/admin/metrics.xml.

There are two main features you can configure:

Naming custom groupings of metrics for ease of access, and access control.
Applying specific access control policies to metrics or groups of metrics.

By default, the REST API provides access to either all metrics (via /idp/profile/admin/metrics), or to a specific metric by attaching the name to the end of the path (e.g., /idp/profile/admin/metrics/net.shibboleth.idp.version).

In addition, a number of predefined "groups" are configured for you by default, in a bean named shibboleth.metrics.MetricGroups. The groups associate an object that implements the MetricFilter interface with a shorthand name, and allow you to access all the metrics that satisfy the filter by using the group name on the end of the path in place of a specific metric name. For example, the default configuration allows a number of metrics related to the metadata layer to be accessed via /idp/profile/admin/metrics/metadata.

The other purpose of these groups is for access control. By default, access to the entire REST API is governed by a default policy identified by shibboleth.metrics.DefaultAccessPolicy. If you need more fine-grained control, you can fill in the map defined by shibboleth.metrics.AccessPolicyMap with entries connecting a metric name or metric group name with a named policy to apply.

Note that metrics may be accessed by multiple paths (e.g., directly by themselves or via a group), so typically group or specific metric policies would be used to "open up" broader access to a set of metrics beyond the default policy, which would remain more locked down.

A couple of additional obscure features: it's possible to control the "Access-Control-Allow-Origin" header and turn the feed into a JSONP callback using String beans (see the Reference below) for more exotic uses of the API.

HTTP Reporters

As an alternative to a pull model, the IdP can push metric information out to an HTTP server as a collection point. This is not enabled by default, but some basic configuration support is provided inside a comment in conf/admin/metrics.xml.

You can define any number of reporters using the parent bean shibboleth.metrics.HTTPReporter. The reporter bean is configured with the URL to access, and can be injected with an HttpClient bean for control over many aspects of the HTTP connection, such as timeouts, how to evaluate a server's TLS certificate, etc. It's also possible to add a MetricFilter constructor argument to limit which metrics are actually included.

In addition to defining the bean itself, it must be "scheduled" by calling its "start" method with a timer interval, as illustrated in the default file.

The format of the data is identical to the format returned by the REST API.

Per-Profile Instrumentation

In addition to the built-in metrics, support exists across a number of components (primarily the Action beans and a few of the major services like the Attribute Resolver and Attribute Filter) to add counters and timers to specific requests. This is done by providing a script (or alternative Java code, but a script is simpler) to add the counters and timers to install.

The names of the metrics are arbitrary; the metrics are attached to specific objects based on an identifier for the object that usually will consist of either the Spring bean ID or the simple name of a Java class. This usually requires some specific knowledge of the internals of the system, but advice is available on the support list about how to measure particular parts of the system.

The two kinds of metrics supported are simple counters that measure how often a component runs, and timers that measure the time elapsed between a starting and stopping point. With respect to profile actions, the system will check to see if the execution of the action should be counted and if a timer should be started (at the beginning of execution) or stopped (at the end). The latter means that it's possible to start a timer in one action and stop it in another, bracketing larger parts of a request than just a single action.

(The above is best explained with an example, so keep reading.)

Control over the installation of these metrics is managed by a bean of type Function<ProfileRequestContext,Boolean> named shibboleth.metrics.MetricStrategy. The function's job is to manipulate a MetricContext child context to configure the metrics to activate for the request. The javadoc for that child context class describes the methods available to add counters and timers.

The example provided by default illustrates a simple use case: measuring the time it takes to resolve attributes during the request. In the example, a timer named "idp.attribute.resolution" is installed that starts when the "ResolveAttributes" action starts and stops when the "FilterAttributes" action finishes. This demonstrates that timers can extend beyond a single step in a flow. You could just as easily time only the "ResolveAttributes" action alone by using the same ID for both parameters.

	<bean id="shibboleth.metrics.MetricStrategy" parent="shibboleth.ContextFunctions.Scripted"
			factory-method="inlineScript">
		<constructor-arg>
			<value>
			<![CDATA[
				var metricCtx = input.getSubcontext("org.opensaml.profile.context.MetricContext");
				metricCtx.addTimer("idp.attribute.resolution",
					"ResolveAttributes",
					"FilterAttributes"
					);
				true; // Signals success.
			]]>
			</value>
		</constructor-arg>
	</bean>

Another example demonstrates the use of a counter tracking the number of times the SAML 1.1 Attribute Query flow executes based on counting each time the "DecodeMessage" action runs. It shows how, because the function is run dynamically, it's possible to conditionally enable metrics based on the specific profile flow being run.

	<bean id="shibboleth.metrics.MetricStrategy" parent="shibboleth.ContextFunctions.Scripted"
			factory-method="inlineScript">
		<constructor-arg>
			<value>
			<![CDATA[
				var profileType = Java.type("net.shibboleth.idp.saml.saml1.profile.config.AttributeQueryProfileConfiguration");
				if (profileType.PROFILE_ID.equals(input.getProfileId())) {
					metricCtx = input.getSubcontext("org.opensaml.profile.context.MetricContext");
					metricCtx.addCounter("idp.profile.saml1.attributeQueries", "DecodeMessage");
				}
				true; // Signals success.
			]]>
			</value>
		</constructor-arg>
	</bean>

Reference

V4.0 and upgraded systems include a bean defined in conf/admin/general-admin.xml to control aspects of the flow's behavior.

V4.1 includes properties to control various aspects of the flow's behavior using an internally-defined bean that may be overridden if required.

Name	Type	Description

Name	Type	Description
shibboleth.metrics.MetricRegistry	FilteredMetricRegistry	Registry of all metrics known to the system
shibboleth.metrics.RegisterMetricSets	MethodInvokingBean	Spring parent bean for invoking the `registerMultiple` method on the registry
shibboleth.metrics.RegisterMetricSet	MethodInvokingBean	Spring parent bean for invoking the `register` method on the registry
shibboleth.metrics.CoreGaugeSet	MetricSet / MetricFilter	Low-level gauges for OS, Java, and memory information
shibboleth.metrics.IdPGaugeSet	MetricSet / MetricFilter	Basic IdP system information (version, uptime)
shibboleth.metrics.LoggingGaugeSet	MetricSet / MetricFilter	Information about the logging service
shibboleth.metrics.AccessControlGaugeSet	MetricSet / MetricFilter	Information about the access control service
shibboleth.metrics.MetadataGaugeSet	MetricSet / MetricFilter	Information about the metadata resolver service
shibboleth.metrics.RelyingPartyGaugeSet	MetricSet / MetricFilter	Information about the RelyingParty configuration service
shibboleth.metrics.NameIdentifierGaugeSet	MetricSet / MetricFilter	Information about the Name Identifier generation service
shibboleth.metrics.AttributeResolverGaugeSet	MetricSet / MetricFilter	Information about the attribute resolver service
shibboleth.metrics.AttributeFilterGaugeSet	MetricSet / MetricFilter	Information about the attribute filter service
shibboleth.metrics.AttributeRegistryGaugeSet ^4.1	MetricSet / MetricFilter	Information about the attribute registry service
shibboleth.metrics.HTTPReporter	HTTPReporter	A schedulable background reporter that sends a JSON feed of metrics to a URL
shibboleth.metrics.MetricGroups	Map<String,MetricFilter>	Associates metrics matching a supplied filter with a string label that "names" that set of metrics
shibboleth.metrics.MetricLevelMap	Map<String,Level>	Optional mapping of metric names to logging levels to associate with the metric
shibboleth.metrics.DefaultAccessPolicy	String	Name of the access control policy to apply to the metrics API in the absence of a more specific policy
shibboleth.metrics.AccessPolicyMap	Map<String,String>	Maps a named metric group/filter from the "shibboleth.metrics.MetricGroups" bean to a named access control policy to apply when accessing that group via the API
shibboleth.metrics.AccessPolicyStrategy	Function<ProfileRequestContext,String>	A mechanism to determine the access control policy to apply to a request to the metrics API, normally relies on the two previous beans but can be replaced if desired
shibboleth.metrics.AllowedOrigin	String	Optional "Access-Control-Allow-Origin" header value to return within REST API response
shibboleth.metrics.JSONPCallback	String	Optional name of JSONP callback function to pass the REST API response
shibboleth.metrics.MetricStrategy	Function<ProfileRequestContext,Boolean>	A hook to provide a function to execute at the beginning of every request that can programmatically enable timers and counters for objects during the execution of that request
shibboleth.metrics.ExposedProperties ^4.1	Set<String>	Collection of property names to expose as IdP metric gauges. Take care not to expose confidental information through this hook.

The general properties configuring this flow via admin/admin.properties are:

Name	Default	Description

Name	Default	Description
idp.metrics.logging	Metrics	Audit log identifier for flow
idp.metrics.authenticated	false	Whether authentication should be performed prior to access control evaluation
idp.metrics.nonBrowserSupported	false	Whether the flow should allow for non-browser clients during authentication
idp.metrics.resolveAttributes	false	Whether attributes should be resolved prior to access control evaluation

To replace the internally defined flow descriptor bean, the following XML is required:

<util:list id="shibboleth.AvailableAdminFlows">
 
    <bean parent="shibboleth.AdminFlow"
        c:id="http://shibboleth.net/ns/profiles/metrics"
        p:loggingId="%{idp.metrics.logging:Metrics}"
        p:policyNameLookupStrategy-ref="shibboleth.metrics.AccessPolicyStrategy"
        p:nonBrowserSupported="%{idp.metrics.nonBrowserSupported:false}"
        p:authenticated="%{idp.metrics.authenticated:false}"
        p:resolveAttributes="%{idp.metrics.resolveAttributes:false}" />
 
</util:list>

In older versions and upgraded systems, this list is defined in conf/admin/general-admin.xml. In V4.1+, no default version of the list is provided and it may simply be placed in conf/global.xml if needed.