Shibboleth IdP Probe
This document shows how to download, install, and run probe_shib_idps.sh
, a bash script that probes a sequence of Shibboleth IdPs and determines the version of the Shibboleth software used by each deployment.
The script takes a sequence of entityIDs and a corresponding metadata source. It then iterates over the entityIDs and probes each entity if and only if the entity is a Shibboleth IdP (which is determined by inspecting entity metadata).
Entity metadata is provisioned ahead of time via a local SAML metadata file or obtained just-in-time from a remote SAML metadata query server that supports the SAML Metadata Query Protocol. The examples below illustrate each method.
Contents
How Does It Work?
The script is based on the fact that Shibboleth IdP V2 has a well-known Status endpoint, while later versions of the Shibboleth IdP software do not. For example, click on the following link (or copy-and-paste the following URL into your browser):
https://idp.incommonfederation.org/idp/profile/Status
or, better yet, execute the following command lines:
$ STATUS_URL=https://idp.incommonfederation.org/idp/profile/Status $ /usr/bin/curl --silent --dump-header /dev/tty $STATUS_URL HTTP/1.1 200 OK Date: Fri, 14 Oct 2016 18:15:06 GMT Set-Cookie: JSESSIONID=7BA0BBFB51DC132214AC527F9112DC63; Path=/idp; Secure Expires: 0 Cache-Control: no-cache, no-store, must-revalidate, max-age=0 Pragma: no-cache Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 ok
As you can see, the Status endpoint responds with "ok" if the IdP at that location is up and running. The key observation is that only Shibboleth IdP V2 has such a Status endpoint. In this way, the script is able to distinguish Shibboleth IdP V2 deployments with high probability. (Hat tip to Scott Cantor for suggesting this technique.)
The Algorithm
For each entityID, the script performs the following sequence of steps:
- Obtain entity metadata
- Choose a browser-facing SAML endpoint from metadata
- If this is not a Shibboleth IdP, continue with the next entityID
- Transmit a SAML authentication request to the chosen endpoint
- If the request fails, continue with the next entityID
- Determine the Shibboleth IdP V2 Status endpoint
- Transmit an HTTP request to the Status endpoint
- Determine the Shibboleth IdP software version
The message issued at step 4 is an ordinary SAML authentication request, appropriate for the chosen endpoint. If this initial request fails, the sequence is short-circuited, which helps prevent false positives.
Getting Started
Download and install the following projects from GitHub:
- The bash-library project
- The shib-idp-probe project
The following subsections outline the installation process.
Install Bash Library
First download the bash-library source code. For example, assuming you have git
installed, you can clone the repository as follows:
$ git clone https://github.internet2.edu/InCommon/bash-library.git
Now install the source into /tmp
like this:
$ export BIN_DIR=/tmp/bin $ export LIB_DIR=/tmp/lib $ ./bash-library/install.sh $BIN_DIR $LIB_DIR
or install the source into your home directory:
$ export BIN_DIR=$HOME/bin $ export LIB_DIR=$HOME/lib $ ./bash-library/install.sh $BIN_DIR $LIB_DIR
A given target directory will be created if one doesn't already exist. In any case, the following files will be installed:
$ ls -1 $BIN_DIR cget.sh $ ls -1 $LIB_DIR command_paths.sh compatible_date.sh compatible_mktemp.sh config_tools.sh entity_endpoints_txt.xsl entity_identifiers_txt.xsl entity_idp_names_txt.xsl extract_entity.xsl http_tools.sh md_tools.sh saml_tools.sh
Install Shibboleth IdP Probe
Next download the shib-idp-probe source code:
$ git clone https://github.internet2.edu/InCommon/shib-idp-probe.git
Install the source on top of the previous installation:
$ ./shib-idp-probe/install.sh $BIN_DIR $LIB_DIR
Altogether the following files will be installed:
$ ls -1 $BIN_DIR cget.sh list_local_shib_idps.sh probe_shib_idps.sh summarize_global_shib_idps.sh  $ ls -1 $LIB_DIR command_paths.sh compatible_date.sh compatible_mktemp.sh config_tools.sh entity_endpoints_txt.xsl entity_identifiers_txt.xsl entity_idp_names_txt.xsl entityIDs_IdP_list_txt.xsl extract_entity.xsl extract_IdP_entityIDs.xsl extract_IdP_names.xsl extract_InCommon_IdP_entityIDs.xsl http_tools.sh md_tools.sh saml_tools.sh
Test your installation using the examples below.
Simple Examples
Probe one or more Shibboleth IdPs to determine the version of the Shibboleth software used. Metadata is obtained in one of two ways, by consulting a metadata query server just-in-time or by using a pre-provisioned metadata aggregate.
Example 1a. Using a Metadata Query Server
Run the script on a single entityID:
# The base URL of a Metadata Query Server $ export MDQ_BASE_URL=http://mdq-beta.incommon.org/global # The entityID of a Shibboleth IdP $ id=urn:mace:incommon:nyu.edu # Using metadata from the MDQ server, probe the Shibboleth IdP $ $BIN_DIR/probe_shib_idps.sh $id 0 redirects:0;response:404;dns:0.523;tcp:0.549;ssl:0.789;total:0.818 https://shibboleth.nyu.edu/idp/profile/Status SHIB3
Now run the script on a handful of entityIDs:
# Probe a handful of Shibboleth IdPs (entityIDs specified in a heredoc) $ $BIN_DIR/probe_shib_idps.sh <<ENTITY_IDs urn:mace:incommon:internet2.edu urn:mace:incommon:uchicago.edu https://carmenwiki.osu.edu/shibboleth https://shibboleth-dev-v.musc.edu/shibboleth-idp https://idp.bnl.gov/idp/shibboleth ENTITY_IDs 0 redirects:0;response:404;dns:0.004;tcp:0.006;ssl:0.092;total:0.095 https://origin.internet2.edu/idp/profile/Status SHIB3 0 redirects:0;response:404;dns:0.515;tcp:0.524;ssl:0.601;total:0.610 https://shibboleth2.uchicago.edu/idp/profile/Status SHIB3 WARNING: probe_shib_idps.sh: entity is not an IdP: https://carmenwiki.osu.edu/shibboleth 28 redirects:0;response:000;dns:0.068;tcp:0.000;ssl:0.000;total:1.273 https://shibboleth-dev-v.musc.edu/shibboleth-idp/profile/Status SHIB? 0 redirects:0;response:200;dns:0.518;tcp:0.592;ssl:0.764;total:0.837 https://idp.bnl.gov/idp/profile/Status SHIB2 # To better understand the meaning of the output, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h
Example 1b. Using a Metadata Aggregate
Create a directory for the metadata:
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR
The following code block shows how to fetch the InCommon metadata aggregate but the same commands will work on any aggregate:
# Fetch the main production metadata aggregate at md.incommon.org: $ MD_LOCATION=http://md.incommon.org/InCommon/InCommon-metadata.xml $ MD_PATH=$MD_DIR/InCommon-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Fri, 14 Oct 2016 18:26:09 GMT Server: Apache Last-Modified: Thu, 13 Oct 2016 19:02:50 GMT ETag: "190008-24df972-53ec3c187ca80" Accept-Ranges: bytes Content-Length: 38664562 Connection: close Content-Type: application/samlmetadata+xml
Run the script on a single entityID:
# The entityID of a Shibboleth IdP $ id=https://idp.incommonfederation.org/idp/shibboleth # Using the metadata aggregate, probe the Shibboleth IdP $ $BIN_DIR/probe_shib_idps.sh -f $MD_PATH $id 0 redirects:0;response:200;dns:0.524;tcp:0.527;ssl:0.617;total:0.622 https://idp.incommonfederation.org/idp/profile/Status SHIB2
Run the script on a handful of entityIDs:
# Probe a handful of Shibboleth IdPs (entityIDs specified in a heredoc) $ $BIN_DIR/probe_shib_idps.sh -f $MD_PATH <<ENTITY_IDs > urn:mace:incommon:osu.edu > https://sso.memphis.edu/idp/shibboleth > https://sso.hsc.edu/adfs/services/trust > https://idp.ubalt.edu/idp/shibboleth > https://sso.brown.edu/idp/shibboleth > https://idp.si.edu/idp/shibboleth > ENTITY_IDs 0 redirects:0;response:404;dns:0.530;tcp:0.563;ssl:0.746;total:0.777 https://webauth.service.ohio-state.edu/idp/profile/Status SHIB3 0 redirects:0;response:404;dns:0.131;tcp:0.189;ssl:0.449;total:0.508 https://sso.memphis.edu/idp/profile/Status SHIB3 INFO: probe_shib_idps.sh: entity is not a Shibboleth IdP: https://sso.hsc.edu/adfs/services/trust 0 redirects:3;response:200;dns:0.000;tcp:0.032;ssl:0.072;total:1.034 https://idp.ubalt.edu/idp/profile/Status SHIB? 0 redirects:0;response:200;dns:0.528;tcp:0.563;ssl:0.748;total:0.786 https://sso.brown.edu/idp/profile/Status SHIB2 0 redirects:0;response:200;dns:0.068;tcp:0.093;ssl:0.552;total:0.578 https://idp.si.edu/idp/profile/Status SHIB2 Â # To better understand the meaning of the output, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h
More Examples
Given the location of a metadata aggregate, probe some or all of the Shibboleth IdPs in the aggregate.
Example 2a. Probe all Shibboleth IdPs registered by InCommon
Using an XSLT script, extract the relevant list of IdP entityIDs from the aggregate and then attempt to probe each IdP on that list.
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR # Fetch the main production metadata aggregate at md.incommon.org: $ MD_LOCATION=http://md.incommon.org/InCommon/InCommon-metadata.xml $ MD_PATH=$MD_DIR/InCommon-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Fri, 30 Dec 2016 13:56:53 GMT Server: Apache Last-Modified: Wed, 28 Dec 2016 17:40:50 GMT ETag: "190009-26553e5-544bb78370c80" Accept-Ranges: bytes Content-Length: 40195045 Connection: close Content-Type: application/samlmetadata+xml # How many of the IdPs in InCommon metadata are registered by InCommon? $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_InCommon_IdP_entityIDs.xsl - \ | wc -l 451 # Directory for output $ OUT_DIR=/tmp/out/idps_reg_incommon; mkdir -p $OUT_DIR # Probe every InCommon IdP with a SAML2 HTTP endpoint $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_InCommon_IdP_entityIDs.xsl - \ | $BIN_DIR/probe_shib_idps.sh -f $MD_PATH -d $OUT_DIR -t 8 -m 12 -r 7 # To learn more about this script, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h
For a detailed analysis, expand the code block below:
Convert the output from the previous run to JSON format, suitable for display on a web page:
# Convert the output to JSON format: $ $BIN_DIR/list_local_shib_idps.sh -d $OUT_DIR https://incommon.org # To learn more about this script, read the online help file $ $BIN_DIR/list_local_shib_idps.sh -h
For example, the JSON files produced above were used to construct a Lists of Shibboleth IdPs Registered by InCommon.
Example 2b. Probe all Shibboleth IdPs in eduGAIN metadata
Using an XSLT script, extract a list of all IdP entityIDs from the eduGAIN aggregate and then attempt to probe each IdP on that list.
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR # Fetch the production eduGAIN metadata aggregate: $ MD_LOCATION=http://mds.edugain.org/feed-sha256.xml $ MD_PATH=$MD_DIR/eduGAIN-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Mon, 02 Jan 2017 16:57:33 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16 Last-Modified: Mon, 02 Jan 2017 16:45:01 GMT ETag: "1ac9c20-5451f45daeb38" Accept-Ranges: bytes Content-Length: 28089376 Content-Type: application/samlmetadata+xml # How many IdPs in eduGAIN metadata? $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | wc -l 2262 # custom config parameters $ export SAML2_SP_ENTITY_ID=https://sp24-test.garr.it/shibboleth $ export SAML2_SP_ACS_URL=https://sp24-test.garr.it/Shibboleth.sso/SAML2/POST $ export SAML2_SP_ACS_BINDING=urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST # Directory for output $ OUT_DIR=/tmp/out/all_edugain_idps; mkdir -p $OUT_DIR # Probe every IdP in the metadata file $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | $BIN_DIR/probe_shib_idps.sh -f $MD_PATH -d $OUT_DIR -t 10 -m 15 -r 7 # To learn more about this script, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h
For a detailed analysis, expand the code block below:
Convert the output from the previous run to JSON format, suitable for display on a web page:
# Convert the output to JSON format: $ $BIN_DIR/summarize_global_shib_idps.sh -d $OUT_DIR # To learn more about this script, read the online help file $ $BIN_DIR/summarize_global_shib_idps.sh -h
For example, the JSON files produced above were used to construct a summary of Summary of Shibboleth IdPs in eduGAIN Metadata.