Field extraction is the process of populating a labeled field in an AuditContext object that is built up over the life of a transaction. The context is just a map of field names to collections of values and the field names are just conventions built-in to the software to provide a standard set of useful fields to audit.

Fields are extracted at various points through the life of a transaction so that adequate detail can be exposed about the request, the processing of the request, and the response. These extraction points are associated with collections of field extraction beans that do the actual work to pull data out of the state of the transaction and store it for output.

The built-in extraction points and beans are as follows (empty beans are not provided to reduce clutter, but you can add them, per the example below):

1. shibboleth.FlowStartAuditExtractors

   • Extractors that run immediately after the flow starts operating and the ability to add fields is possible

2. shibboleth.PostDecodeAuditExtractors

   • Extractors focused on the input message, runs after decoding the input message

3. shibboleth.PostLookupAuditExtractors

   • Runs after establishing the sender's identity, supplements the input side with information about the relying party

4. shibboleth.PostAssertionAuditExtractors

   • Extractors focused on authentication, session, and attribute information, runs after those processes and after the initial creation of the outgoing assertion (if relevant to the request)

5. shibboleth.PostResponseAuditExtractors

   • Extractors focused on the outgoing message, runs after construction of the message

6. shibboleth.LogoutRequestAuditExtractors

   • Extractors focused on information specific to a SAML LogoutRequest message, runs after decoding of the message

7. shibboleth.LogoutAuditExtractors

   • Extractors focused on information in a non-SAML logout

8. shibboleth.ErrorViewAuditExtractors

   • Extracts subject, session, and attribute information, runs in the event that a local error is generated

9. shibboleth.consent.PreConsentAuditExtractors

   • Extracts basic information about the consent step, runs before a decision is made

   • Empty map beans are declared in conf/intercept/consent-intercept-config.xml

10. shibboleth.consent.ConsentAuditExtractors

    • Extracts information about the decision, runs after a decision is made

    • Empty map beans are declared in conf/intercept/consent-intercept-config.xml

11. shibboleth.PostRequestAuditExtractors

    • Extractors for an AuthnRequest built for proxied SAML SSO

12. shibboleth.PostInboundResponseAuditExtractors

    • Extractors for a Response from proxied SAML SSO

13. shibboleth.PostInboundAssertionAuditExtractors

    • Extractors for Assertions from proxied SAML SSO

Each entry in these maps is a key containing the string label to associate (the name of the field), and a value containing a Function<ProfileRequestContext,Object> instance. Functions can return either single objects that get converted to a string for logging, or a Collection of objects that are converted to strings and separated by commas within the logged field.

In this manner, you can add your own fields by implementing a function (or writing a script) to return any data you can find in the context tree and associating it with a custom field label by adding it to one of the maps named above.

In V4.3+, a similar extractor exists for each login flow in the IdP, allowing different auditing behavior per-flow. All these beans have names based on the underlying login flow (e.g. shibboleth.authn.Password.AuditExtractors, etc.)

A skeletal example follows to demonstrate how to create a map to contain a custom field extraction rule:

Adding custom fields to the audit log via audit.xml

<!-- Adds a function to extract a custom field called MYFIELD after the final response has been generated. -->
<bean id="shibboleth.PostResponseAuditExtractors" parent="shibboleth.DefaultPostResponseAuditExtractors">
	<property name="sourceMap">
		<map merge="true">
            <entry key="MYFIELD" value-ref="MyExtractionFunctionBean" /> 
		</map>
	</property>
</bean>

A real world example: if you're logging fields containing a URL, you may need to escape characters in a URL because of the delimiter you choose to use in the log format. You can do this using a script:

Encoding 'pipe' character appearing in CAS service URLs

     <bean id="shibboleth.CASValidationAuditExtractors" parent="shibboleth.DefaultCASValidationAuditExtractors" lazy-init="true">
         <property name="sourceMap">
             <map merge="true">
                 <entry>
                     <key>
                         <util:constant static-field="net.shibboleth.idp.cas.protocol.CASAuditFields.SERVICE_URL"/>
                     </key>
                     <bean parent="shibboleth.ContextFunctions.Scripted" factory-method="inlineScript">
                         <constructor-arg>
                             <value>
                             <![CDATA[
                                 var serviceLookupFunctionClass = Java.type("net.shibboleth.idp.cas.audit.impl.ServiceLookupFunction");
                                 var serviceLookupFunction = new serviceLookupFunctionClass();
                                 var service = serviceLookupFunction.apply(input);
                                 if (service != null) {
                                     var serviceNew = service.replace(/\|/g, '%7C');
                                 }
                                 else {
                                     var serviceNew = null;
                                 }
                                 serviceNew;
                             ]]>
                             </value>
                         </constructor-arg>
                     </bean>
                 </entry>
             </map>
         </property>
     </bean>

A rather more convoluted example is to dump the values or arbitrary attributes into the log.  To do this we define an abstract bean to emit the attribute values (in an arbitrary format) for any attribute (most of this is really error and null handling):

Output Attribute Values

	<bean id="AttributeValueExtraction" parent="shibboleth.ContextFunctions.Scripted" factory-method="inlineScript" abstract="true">
		<constructor-arg>
	        <value>
	        <![CDATA[ 
	        var getinfo = function() {
	        	var rpContext = input.getSubcontext("net.shibboleth.idp.profile.context.RelyingPartyContext");
		        if (null === rpContext) {
		            return null;
		        }
	        	
		        var attrContext = rpContext.getSubcontext("net.shibboleth.idp.attribute.context.AttributeContext");
		        if (null === attrContext) {
		            return null;
		        }
		        var attributes = null
		        attributes = attrContext.getUnfilteredIdPAttributes();
		        if (null === attributes) {
		            return null;
		         }
	            attribute = attributes.get(custom);
	            if (null === attribute || attribute.getValues().isEmpty()) {
	                return null;
	            }
	            var iter = attribute.getValues().iterator();
	            var result = "";
	            while (iter.hasNext()) {
	               result = result + " " + iter.next().toString();
	            }
	            return result;
		     }
		     getinfo();
	         ]]>
	        </value>
	    </constructor-arg>
	</bean>

Then a bean to emit a specific attribute can be easily created:

<bean id="EmailExtraction" parent="AttributeValueExtraction" p:customObject="mail" />

The generic fields that are supported out of the box are as follows (note that not every field is always populated; that depends on the timing of errors and the specific transaction being audited):

The more SAML-oriented fields that are supported out of the box are as follows (note that not every field is always populated; that depends on the timing of errors and the specific transaction being audited):

A few fields are defined for auditing specific information during authentication, which is a V4.3 feature; many are specific to particular login flows.

In the output stage, a bean named shibboleth.AuditFormattingMap is used to write any number of log records containing any fields you configure to the logging API, at which point the Logback configuration takes over and decides how to send that data to particular log files or other log sinks. The bean is a map between logging categories and formatting strings.

The consent audit log relies on its own properties and beans to provide an audit format, so refer to that documentation for those specifics.

Formatting strings contain fields denoted with a '%' character followed by a field label, and any whitespace or punctuation terminates a field label and is included in the record directly.  Using '%%' will output a single percent character.

An additional bean, shibboleth.AuditSuppressedProfiles, contains a list of profile identifiers that should not result in audit log records. The default list is used to exempt the "status" handler from being logged. A partial list of profile IDs can be found in the MetadataDrivenConfiguration topic.

Finally, when errors are handled "locally" (see ErrorHandlingConfiguration), the IdP can be told whether to output an audit log record or not when the error event occurs. This is useful to suppress logging noise when common errors occur that would lead to a lot of useless auditing. The flag controlling this is inside the shibboleth.LocalEventMap bean in conf/errors.xml (the value of each map entry is the flag indicating whether to log).

General beans defined in audit.xml, errors.xml, and related system configuration are:

In addition, there are a set of standard beans containing the map of field extractors for all of the various "interception points" of the processing of the IdP. A pair of beans are reserved for each point, a default set that comes out of the box, and a defined bean ID to override or add to those defaults. Each of these beans is of type Map<String,Function<ProfileRequestContext,Object>>

• shibboleth.FlowStartAuditExtractors / shibboleth.DefaultFlowStartAuditExtractors

• shibboleth.PostDecodeAuditExtractors / shibboleth.DefaultPostDecodeAuditExtractors

• shibboleth.PostLookupAuditExtractors / shibboleth.DefaultPostLookupAuditExtractors

• shibboleth.PostAssertionAuditExtractors / shibboleth.DefaultPostAssertionAuditExtractors

• shibboleth.PostResponseAuditExtractors / shibboleth.DefaultPostResponseAuditExtractors

• shibboleth.PostRequestAuditExtractors / shibboleth.DefaultPostRequestAuditExtractors

• shibboleth.PostInboundResponseAuditExtractors / shibboleth.DefaultPostInboundResponseAuditExtractors

• shibboleth.PostInboundAssertionAuditExtractors / shibboleth.DefaultPostInboundAssertionAuditExtractors

• shibboleth.LogoutRequestAuditExtractors / shibboleth.DefaultLogoutRequestAuditExtractors

• shibboleth.LogoutAuditExtractors / shibboleth.DefaultLogoutAuditExtractors

• shibboleth.ErrorViewAuditExtractors / shibboleth.DefaultErrorViewAuditExtractors

• shibboleth.consent.PreConsentAuditExtractors / shibboleth.consent.DefaultPreConsentAuditExtractors

• shibboleth.consent.ConsentAuditExtractors / shibboleth.consent.DefaultConsentAuditExtractors

With V4.3+, an additional set of beans for each login flow are added:

• shibboleth.authn.Password.AuditExtractors / shibboleth.authn.Password.DefaultAuditExtractors

• etc.

Properties are defined in services.properties to customize various aspects of audit logging:

Additional properties are housed in authn/authn.properties to enable and customize authentication auditing. The <Flow> token in the various property names below is a stand-in for the actual login flow suffix (Password, External, RemoteUser, etc.), as these are per-flow properties.