This document shows how to download, install, and run probe_shib_idps.sh
, a bash script that probes a sequence of Shibboleth IdPs and determines the version of the Shibboleth software used by each deployment.
The script takes a sequence of entityIDs and a corresponding metadata source. It then iterates over the entityIDs and probes each entity if and only if the entity is a Shibboleth IdP (which is determined by inspecting entity metadata).
Entity metadata is provisioned ahead of time via a local SAML metadata file or obtained just-in-time from a remote SAML metadata query server that supports the SAML Metadata Query Protocol. The examples below illustrate each method.
Contents
The script is based on the fact that Shibboleth IdP V2 has a well-known Status endpoint, while later versions of the Shibboleth IdP software do not. For example, click on the following link (or copy-and-paste the following URL into your browser):
https://idp.incommonfederation.org/idp/profile/Status
or, better yet, execute the following command lines:
$ STATUS_URL=https://idp.incommonfederation.org/idp/profile/Status $ /usr/bin/curl --silent --dump-header /dev/tty $STATUS_URL HTTP/1.1 200 OK Date: Fri, 14 Oct 2016 18:15:06 GMT Set-Cookie: JSESSIONID=7BA0BBFB51DC132214AC527F9112DC63; Path=/idp; Secure Expires: 0 Cache-Control: no-cache, no-store, must-revalidate, max-age=0 Pragma: no-cache Connection: close Transfer-Encoding: chunked Content-Type: text/plain; charset=UTF-8 ok |
As you can see, the Status endpoint responds with "ok" if the IdP at that location is up and running. The key observation is that only Shibboleth IdP V2 has such a Status endpoint. In this way, the script is able to distinguish Shibboleth IdP V2 deployments with high probability. (Hat tip to Scott Cantor for suggesting this technique.)
For each entityID, the script performs the following sequence of steps:
The message issued at step 4 is an ordinary SAML authentication request, appropriate for the chosen endpoint. If this initial request fails, the sequence is short-circuited, which helps prevent false positives.
Download and install the following projects from GitHub:
The following subsections outline the installation process.
First download the bash-library source code. For example, assuming you have git
installed, you can clone the repository as follows:
$ git clone https://github.internet2.edu/InCommon/bash-library.git |
Now install the source into /tmp
like this:
$ export BIN_DIR=/tmp/bin $ export LIB_DIR=/tmp/lib $ ./bash-library/install.sh $BIN_DIR $LIB_DIR |
or install the source into your home directory:
$ export BIN_DIR=$HOME/bin $ export LIB_DIR=$HOME/lib $ ./bash-library/install.sh $BIN_DIR $LIB_DIR |
A given target directory will be created if one doesn't already exist. In any case, the following files will be installed:
$ ls -1 $BIN_DIR cget.sh $ ls -1 $LIB_DIR command_paths.sh compatible_date.sh compatible_mktemp.sh config_tools.sh entity_endpoints_txt.xsl entity_identifiers_txt.xsl entity_idp_names_txt.xsl extract_entity.xsl http_tools.sh md_tools.sh saml_tools.sh |
Next download the shib-idp-probe source code:
$ git clone https://github.internet2.edu/InCommon/shib-idp-probe.git |
Install the source on top of the previous installation:
$ ./shib-idp-probe/install.sh $BIN_DIR $LIB_DIR |
Altogether the following files will be installed:
$ ls -1 $BIN_DIR cget.sh list_local_shib_idps.sh probe_shib_idps.sh summarize_global_shib_idps.sh $ ls -1 $LIB_DIR command_paths.sh compatible_date.sh compatible_mktemp.sh config_tools.sh entity_endpoints_txt.xsl entity_identifiers_txt.xsl entity_idp_names_txt.xsl entityIDs_IdP_list_txt.xsl extract_entity.xsl extract_IdP_entityIDs.xsl extract_IdP_names.xsl extract_InCommon_IdP_entityIDs.xsl http_tools.sh md_tools.sh saml_tools.sh |
Test your installation using the examples below.
Probe one or more Shibboleth IdPs to determine the version of the Shibboleth software used. Metadata is obtained in one of two ways, by consulting a metadata query server just-in-time or by using a pre-provisioned metadata aggregate.
Run the script on a single entityID:
# The base URL of a Metadata Query Server $ export MDQ_BASE_URL=http://mdq-beta.incommon.org/global # The entityID of a Shibboleth IdP $ id=urn:mace:incommon:nyu.edu # Using metadata from the MDQ server, probe the Shibboleth IdP $ $BIN_DIR/probe_shib_idps.sh $id 0 redirects:0;response:404;dns:0.523;tcp:0.549;ssl:0.789;total:0.818 https://shibboleth.nyu.edu/idp/profile/Status SHIB3 |
Now run the script on a handful of entityIDs:
# Probe a handful of Shibboleth IdPs (entityIDs specified in a heredoc) $ $BIN_DIR/probe_shib_idps.sh <<ENTITY_IDs urn:mace:incommon:internet2.edu urn:mace:incommon:uchicago.edu https://carmenwiki.osu.edu/shibboleth https://shibboleth-dev-v.musc.edu/shibboleth-idp https://idp.bnl.gov/idp/shibboleth ENTITY_IDs 0 redirects:0;response:404;dns:0.004;tcp:0.006;ssl:0.092;total:0.095 https://origin.internet2.edu/idp/profile/Status SHIB3 0 redirects:0;response:404;dns:0.515;tcp:0.524;ssl:0.601;total:0.610 https://shibboleth2.uchicago.edu/idp/profile/Status SHIB3 WARNING: probe_shib_idps.sh: entity is not an IdP: https://carmenwiki.osu.edu/shibboleth 28 redirects:0;response:000;dns:0.068;tcp:0.000;ssl:0.000;total:1.273 https://shibboleth-dev-v.musc.edu/shibboleth-idp/profile/Status SHIB? 0 redirects:0;response:200;dns:0.518;tcp:0.592;ssl:0.764;total:0.837 https://idp.bnl.gov/idp/profile/Status SHIB2 # To better understand the meaning of the output, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h |
Create a directory for the metadata:
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR |
The following code block shows how to fetch the InCommon metadata aggregate but the same commands will work on any aggregate:
# Fetch the main production metadata aggregate at md.incommon.org: $ MD_LOCATION=http://md.incommon.org/InCommon/InCommon-metadata.xml $ MD_PATH=$MD_DIR/InCommon-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Fri, 14 Oct 2016 18:26:09 GMT Server: Apache Last-Modified: Thu, 13 Oct 2016 19:02:50 GMT ETag: "190008-24df972-53ec3c187ca80" Accept-Ranges: bytes Content-Length: 38664562 Connection: close Content-Type: application/samlmetadata+xml |
Run the script on a single entityID:
# The entityID of a Shibboleth IdP $ id=https://idp.incommonfederation.org/idp/shibboleth # Using the metadata aggregate, probe the Shibboleth IdP $ $BIN_DIR/probe_shib_idps.sh -f $MD_PATH $id 0 redirects:0;response:200;dns:0.524;tcp:0.527;ssl:0.617;total:0.622 https://idp.incommonfederation.org/idp/profile/Status SHIB2 |
Run the script on a handful of entityIDs:
# Probe a handful of Shibboleth IdPs (entityIDs specified in a heredoc) $ $BIN_DIR/probe_shib_idps.sh -f $MD_PATH <<ENTITY_IDs > urn:mace:incommon:osu.edu > https://sso.memphis.edu/idp/shibboleth > https://sso.hsc.edu/adfs/services/trust > https://idp.ubalt.edu/idp/shibboleth > https://sso.brown.edu/idp/shibboleth > https://idp.si.edu/idp/shibboleth > ENTITY_IDs 0 redirects:0;response:404;dns:0.530;tcp:0.563;ssl:0.746;total:0.777 https://webauth.service.ohio-state.edu/idp/profile/Status SHIB3 0 redirects:0;response:404;dns:0.131;tcp:0.189;ssl:0.449;total:0.508 https://sso.memphis.edu/idp/profile/Status SHIB3 INFO: probe_shib_idps.sh: entity is not a Shibboleth IdP: https://sso.hsc.edu/adfs/services/trust 0 redirects:3;response:200;dns:0.000;tcp:0.032;ssl:0.072;total:1.034 https://idp.ubalt.edu/idp/profile/Status SHIB? 0 redirects:0;response:200;dns:0.528;tcp:0.563;ssl:0.748;total:0.786 https://sso.brown.edu/idp/profile/Status SHIB2 0 redirects:0;response:200;dns:0.068;tcp:0.093;ssl:0.552;total:0.578 https://idp.si.edu/idp/profile/Status SHIB2 # To better understand the meaning of the output, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h |
Given the location of a metadata aggregate, probe some or all of the Shibboleth IdPs in the aggregate.
Using an XSLT script, extract the relevant list of IdP entityIDs from the aggregate and then attempt to probe each IdP on that list.
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR # Fetch the main production metadata aggregate at md.incommon.org: $ MD_LOCATION=http://md.incommon.org/InCommon/InCommon-metadata.xml $ MD_PATH=$MD_DIR/InCommon-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Fri, 30 Dec 2016 13:56:53 GMT Server: Apache Last-Modified: Wed, 28 Dec 2016 17:40:50 GMT ETag: "190009-26553e5-544bb78370c80" Accept-Ranges: bytes Content-Length: 40195045 Connection: close Content-Type: application/samlmetadata+xml # How many of the IdPs in InCommon metadata are registered by InCommon? $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_InCommon_IdP_entityIDs.xsl - \ | wc -l 451 # Directory for output $ OUT_DIR=/tmp/out/idps_reg_incommon; mkdir -p $OUT_DIR # Probe every InCommon IdP with a SAML2 HTTP endpoint $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_InCommon_IdP_entityIDs.xsl - \ | $BIN_DIR/probe_shib_idps.sh -f $MD_PATH -d $OUT_DIR -t 8 -m 12 -r 7 # To learn more about this script, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h |
For a detailed analysis, expand the code block below:
# How many InCommon IdPs are running the Shibboleth software? $ /bin/cat $OUT_DIR/idps-shib-log.txt | wc -l 403 # How many Shibboleth IdPs are running V3 software? $ /bin/cat $OUT_DIR/idps-shib-v3-list.txt | wc -l 225 # How many Shibboleth IdPs are running V2 software? $ /bin/cat $OUT_DIR/idps-shib-v2-list.txt | wc -l 149 # How many Shibboleth IdPs are running an unknown software version? $ /bin/cat $OUT_DIR/idps-shib-v-unknown-list.txt | wc -l 29 # Tally curl status codes $ /bin/cat $OUT_DIR/idps-shib-log.txt \ | cut -f1 -d" " \ | sed \ -e 's/^0$/00/' \ -e 's/^1$/01/' \ -e 's/^6$/06/' \ -e 's/^7$/07/' \ | sort | uniq -c \ | sed \ -e 's/ 00$/ 0 Successful/' \ -e 's/ 01$/ 1 Unsupported protocol/' \ -e 's/ 06$/ 6 DNS name resolution failed/' \ -e 's/ 07$/ 7 Host unreachable/' \ -e 's/ 28$/ 28 Timeout/' \ -e 's/ 35$/ 35 SSL connection error/' \ -e 's/ 51$/ 51 SSL certificate of peer was unacceptable/' \ -e 's/ 52$/ 52 The server did not reply to the request/' \ -e 's/ 56$/ 56 Receipt of network data failed/' 385 0 Successful 2 6 DNS name resolution failed 2 7 Host unreachable 12 28 Timeout 1 35 SSL connection error 1 56 Receipt of network data failed # Tally HTTP response codes $ /bin/cat $OUT_DIR/idps-shib-log.txt \ | sed -e 's/^.*;response:\([^;]*\).*$/\1/' \ | sort | uniq -c \ | sed \ -e 's/ 000$/ 000 No response/' \ -e 's/ 200$/ 200 OK/' \ -e 's/ 302$/ 302 Found/' \ -e 's/ 400$/ 400 Bad Request/' \ -e 's/ 401$/ 401 Unauthorized/' \ -e 's/ 403$/ 403 Forbidden/' \ -e 's/ 404$/ 404 Not Found/' \ -e 's/ 500$/ 500 Internal Server Error/' \ -e 's/ 503$/ 503 Service Unavailable/' 17 000 No response 365 200 OK 1 302 Found 4 400 Bad Request 4 401 Unauthorized 6 404 Not Found 5 500 Internal Server Error 1 503 Service Unavailable |
Convert the output from the previous run to JSON format, suitable for display on a web page:
# Convert the output to JSON format: $ $BIN_DIR/list_local_shib_idps.sh -d $OUT_DIR https://incommon.org # To learn more about this script, read the online help file $ $BIN_DIR/list_local_shib_idps.sh -h |
For example, the JSON files produced above were used to construct a Lists of Shibboleth IdPs Registered by InCommon.
Using an XSLT script, extract a list of all IdP entityIDs from the eduGAIN aggregate and then attempt to probe each IdP on that list.
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR # Fetch the production eduGAIN metadata aggregate: $ MD_LOCATION=http://mds.edugain.org/feed-sha256.xml $ MD_PATH=$MD_DIR/eduGAIN-metadata.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Mon, 02 Jan 2017 16:57:33 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips PHP/5.4.16 Last-Modified: Mon, 02 Jan 2017 16:45:01 GMT ETag: "1ac9c20-5451f45daeb38" Accept-Ranges: bytes Content-Length: 28089376 Content-Type: application/samlmetadata+xml # How many IdPs in eduGAIN metadata? $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | wc -l 2262 # custom config parameters $ export SAML2_SP_ENTITY_ID=https://sp24-test.garr.it/shibboleth $ export SAML2_SP_ACS_URL=https://sp24-test.garr.it/Shibboleth.sso/SAML2/POST $ export SAML2_SP_ACS_BINDING=urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST # Directory for output $ OUT_DIR=/tmp/out/all_edugain_idps; mkdir -p $OUT_DIR # Probe every IdP in the metadata file $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | $BIN_DIR/probe_shib_idps.sh -f $MD_PATH -d $OUT_DIR -t 10 -m 15 -r 7 # To learn more about this script, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h |
For a detailed analysis, expand the code block below:
# How many eduGAIN IdPs are Shibboleth IdPs? $ /bin/cat $OUT_DIR/idps-shib-log.txt | wc -l 1690 # How many eduGAIN Shibboleth IdPs are running V3 software? $ /bin/cat $OUT_DIR/idps-shib-v3-list.txt | wc -l 818 # How many eduGAIN Shibboleth IdPs are running V2 software? $ /bin/cat $OUT_DIR/idps-shib-v2-list.txt | wc -l 696 # How many eduGAIN Shibboleth IdPs are running an unknown software version? $ /bin/cat $OUT_DIR/idps-shib-v-unknown-list.txt | wc -l 176 # Compute # of Shibboleth IdPs per registrar $ /bin/cat $OUT_DIR/idps-shib-log.txt \ | cut -f5 -d" " | sort | uniq -c 39 http://aai.grnet.gr/ 169 http://cafe.rnp.br 2 http://cofre.reuna.cl 1 http://colfire.co 18 http://eduid.at 5 http://eduid.hu 16 http://federation.belnet.be/ 41 http://rr.aai.switch.ch/ 459 http://ukfederation.org.uk 19 http://www.canarie.ca 12 http://www.csc.fi/haka 62 http://www.eduid.cz/ 29 http://www.heanet.ie 66 http://www.idem.garr.it/ 39 http://www.swamid.se/ 1 https://aaf.edu.au 1 https://aai.asnet.am 9 https://aai.pionier.net.pl 263 https://federation.renater.fr/ 381 https://incommon.org 1 https://peano.uran.ua 1 https://rr.aaiedu.mk 52 https://www.aai.dfn.de 2 https://www.fccn.pt 2 https://www.gakunin.jp # Compute # of Shibboleth V3 IdPs per registrar $ /bin/cat $OUT_DIR/idps-shib-v3-list.txt \ | cut -f7 | sort | uniq -c 6 http://aai.grnet.gr/ 159 http://cafe.rnp.br 6 http://eduid.at 7 http://federation.belnet.be/ 38 http://rr.aai.switch.ch/ 167 http://ukfederation.org.uk 9 http://www.canarie.ca 9 http://www.csc.fi/haka 52 http://www.eduid.cz/ 5 http://www.heanet.ie 31 http://www.idem.garr.it/ 17 http://www.swamid.se/ 7 https://aai.pionier.net.pl 68 https://federation.renater.fr/ 218 https://incommon.org 17 https://www.aai.dfn.de 1 https://www.fccn.pt 1 https://www.gakunin.jp # Compute # of Shibboleth V2 IdPs per registrar $ /bin/cat $OUT_DIR/idps-shib-v2-list.txt \ | cut -f7 | sort | uniq -c 31 http://aai.grnet.gr/ 1 http://cafe.rnp.br 2 http://cofre.reuna.cl 1 http://colfire.co 12 http://eduid.at 5 http://eduid.hu 7 http://federation.belnet.be/ 3 http://rr.aai.switch.ch/ 189 http://ukfederation.org.uk 9 http://www.canarie.ca 3 http://www.csc.fi/haka 9 http://www.eduid.cz/ 22 http://www.heanet.ie 33 http://www.idem.garr.it/ 22 http://www.swamid.se/ 1 https://aaf.edu.au 1 https://aai.asnet.am 1 https://aai.pionier.net.pl 166 https://federation.renater.fr/ 142 https://incommon.org 1 https://rr.aaiedu.mk 33 https://www.aai.dfn.de 1 https://www.fccn.pt 1 https://www.gakunin.jp |
Convert the output from the previous run to JSON format, suitable for display on a web page:
# Convert the output to JSON format: $ $BIN_DIR/summarize_global_shib_idps.sh -d $OUT_DIR # To learn more about this script, read the online help file $ $BIN_DIR/summarize_global_shib_idps.sh -h |
For example, the JSON files produced above were used to construct a summary of Summary of Shibboleth IdPs in eduGAIN Metadata.
# Directory for metadata: $ MD_DIR=/tmp/md; mkdir $MD_DIR # Fetch the main production metadata aggregate at md.incommon.org: $ MD_LOCATION=http://md.swamid.se/md/swamid-idp.xml $ MD_PATH=$MD_DIR/swamid-idp.xml $ /usr/bin/curl --silent --dump-header /dev/tty $MD_LOCATION > $MD_PATH HTTP/1.1 200 OK Date: Mon, 02 Jan 2017 20:38:25 GMT Server: Apache/2.2.14 (Ubuntu) Last-Modified: Mon, 02 Jan 2017 20:35:35 GMT ETag: "51072-d487b-545227e6305cd" Accept-Ranges: bytes Content-Length: 870523 Content-Type: application/xml # How many of the IdPs in SWAMID metadata? $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | wc -l 60 # Directory for output $ OUT_DIR=/tmp/out/idps_reg_swamid; mkdir -p $OUT_DIR # Probe every SWAMID IdP with a SAML2 HTTP endpoint $ /bin/cat $MD_PATH \ | /usr/bin/xsltproc $LIB_DIR/extract_IdP_entityIDs.xsl - \ | $BIN_DIR/probe_shib_idps.sh -f $MD_PATH -d $OUT_DIR -t 8 -m 12 # To learn more about this script, read the online help file $ $BIN_DIR/probe_shib_idps.sh -h # How many SWAMID IdPs are running the Shibboleth software? $ /bin/cat $OUT_DIR/idps-shib-log.txt | wc -l 50 # How many Shibboleth IdPs are running V3 software? $ /bin/cat $OUT_DIR/idps-shib-v3-list.txt | wc -l 23 # How many Shibboleth IdPs are running V2 software? $ /bin/cat $OUT_DIR/idps-shib-v2-list.txt | wc -l 26 # How many Shibboleth IdPs are running an unknown software version? $ /bin/cat $OUT_DIR/idps-shib-v-unknown-list.txt | wc -l 1 # Tally curl status codes $ /bin/cat $OUT_DIR/idps-shib-log.txt \ | cut -f1 -d" " \ | sed \ -e 's/^0$/00/' \ -e 's/^1$/01/' \ -e 's/^6$/06/' \ -e 's/^7$/07/' \ | sort | uniq -c \ | sed \ -e 's/ 00$/ 0 Successful/' \ -e 's/ 01$/ 1 Unsupported protocol/' \ -e 's/ 06$/ 6 DNS name resolution failed/' \ -e 's/ 07$/ 7 Host unreachable/' \ -e 's/ 28$/ 28 Timeout/' \ -e 's/ 35$/ 35 SSL connection error/' \ -e 's/ 51$/ 51 SSL certificate of peer was unacceptable/' \ -e 's/ 52$/ 52 The server did not reply to the request/' \ -e 's/ 56$/ 56 Receipt of network data failed/' 49 0 Successful 1 28 Timeout # Tally HTTP response codes $ /bin/cat $OUT_DIR/idps-shib-log.txt \ | sed -e 's/^.*;response:\([^;]*\).*$/\1/' \ | sort | uniq -c \ | sed \ -e 's/ 000$/ 000 No response/' \ -e 's/ 200$/ 200 OK/' \ -e 's/ 302$/ 302 Found/' \ -e 's/ 400$/ 400 Bad Request/' \ -e 's/ 401$/ 401 Unauthorized/' \ -e 's/ 403$/ 403 Forbidden/' \ -e 's/ 404$/ 404 Not Found/' \ -e 's/ 503$/ 503 Service Unavailable/' 49 200 OK 1 302 Found # Convert the output to JSON format: # (there are no mdrpi:RegistrationInfo elements, so the registrar ID is NULL) $ $BIN_DIR/list_local_shib_idps.sh -d $OUT_DIR NULL # To learn more about this script, read the online help file $ $BIN_DIR/list_local_shib_idps.sh -h |