Introduction

During the life of IdP V4 we want to put the framework in place to allow us move away from a monolithic approach where everything anybody might want is available to a slightly more modular approach. where deployers can select what services they want. 

Examples might be

• Nashorn support (post JDK15)
• Rhino support
• OIDC
• Duo Plugin
• Database backed Storage (consent &c).

This documents explores the area.

Requirements

• Easy to use as a deployer and as a developer
• Allow secure (manual) update from a server
• Flexible update detection (for instance, a module's IdP version dependency must not be statically defined when the module is installed).
• Flexible enough to do OIDC (with maybe some hand config)
• Stackable (multiple modules)
• KISS
• Start development for 4.1; Full functionality ready for 5.0

Observations

Two key things have resulted from discussion so far

1) The plugin name is critical

This name must be

• Unique
• Capable of being part of a file name and part of a URL without escaping.

The suggestion is to use the reverse DNS notation that the author controls hence net.shibboleth.plugins.scripting.rhino  could be used by the Shibboleth project because it owns the *.shibboleth.net  domain tree.

2) Our thoughts on trust and signing have developed.

See below

After-installation end-state

For a product called net.shibboleth.foo 

• A new directory dist/webapp-net.shibboleth.foo 
  • Build-war will incorporate all of dist/webapp-*  (in a random order).  This is this done after  the population from dist/webapp  (as now) and before  edit-webapp.
  • We might want to add a post-edit-webapp  folder to our distribution to allow us to force the jars we want.
  • The plugin can thus use preConfig.xml  and postConfig.xml to setup global beans
  • The plugin can use this to define flows
  • The plugin can do that to make jars available
• A new file credentials/plugins-trust/net.shibboleth.foo/truststore.asc which contains the public PGP keys that the deployer has accepted for use with this plugin. See PluginTrust
• Files may have been added across the installation (but *not* in system or dist)
  • These will NOT overwrite existing files.
• Properties files auto-edited
  • Files added to idp.additionalProperties 

  • List properties can be edited.  For example

    idp.service.attribute.filter.resources = shibboleth.AttributeFilterResources
    
    (or)
    
    #idp.service.attribute.filter.resources = shibboleth.AttributeFilterResources

    becomes 

    idp.service.attribute.filter.resources = BEAN.idp.service.attribute.filter.resources.net.shibboleth.foo
    idp.service.attribute.filter.resources.OLD.net.shibboleth.foo = shibboleth.AttributeFilterResources

    
    And the installation is required (via preConfig/postConfig) to define BEAN.idp.service.attribute.filter.resources.net.shibboleth.foo  which is the list merge of what it needs plus %{idp.service.attribute.filter.resources.OLD.net.shibboleth.foo} 

    List merging is done using net.shibboleth.ext.spring.factory.CombiningListFactoryBean
    
    This list merging is primarily aimed at adding files to existing service definitions

Driving the Installation

The install package is shipped as .tgz (Linux line ending) or .zip (Windows).  The distributiin must be accompanied with an armoured detached PGP signatures.

The format of an unpacked package is 

Example package layout

The Service Interface

The installation is driven from within the (unzip'd) package itself via the service interface.  So the installer just needs to unpack, add edit-webapp/WEB_INF/lib  (in the package) to the classpath and fire up service API to see what it has to do. 

The Interface will have methods to return (at least) the following 

[ { “service.properties,                            // File
    [ “idp.service.attribute.filter.resources” , ]  // List of properties in that file
  } , 
]

Bootstrap region

The plugin ID and the initial keystore are included as text (so no suspect jar need be on the classpath) in a directory with is part of the package (but not installed) called "bootstrap".  This contains two files

• id.property a file with a single property in it pluginid 
• keys.txt  the signing keys.  This is optional.  See PluginTrust

Uninstalling

Uninstallation cannot occur because of any changes to the configuration will render the system unusable.  As part of an upgrade the webapp parts of the previous distribution will be removed but nothing else.

Update

Doing an update consists of collecting the new version, uninstalling the old version, installing the new one.

• Because the config files are unchanged, configuration is rolled forward.
• Depending on how much we care, we can do the usual ACID tricks to be able to roll forward or back from crashes during an update (mostly using file rename as the ATOMIC operation).  See below for an example
• So this requires that an install (and indeed an uninstall) is idempotent.

Versioning

This has many different aspects.  The four most important are:

1) Jar versions. 

Jars can now be sourced from four different types of place

1. The IdP.
2. This plugin
3. Other plugins
4. Edit-webapp

If each supplies a jar which provides the same class, precisely which class will be used will be random.

I do not believe that it is sensible to check for this. so the best we can do is provide mitigation, probably via some orthogonal configuration.  This might (for instance) be a series of regexps which would make up a black list of jars not to copy.  How this is communicated is tbd.

2) IdP to Module versioning 

There is a new IdP version.  Will the modules work with it?

• We can be told (by the deployer) the new IdP version to check for
• We know our current version (see above)
• We can determine (see below) what the IdP version limits are (as of today, not as of when the plugin was made)

3) Module to IdP versioning

This Module has a new version.  Can it be installed against the IdP?

• We know the IdP version (it is written during install and upgrade to dist): 

  idp.installed.properties

  #Version file written at 2020-03-11T09:59:08.023Z
  #Wed Mar 11 09:59:08 GMT 2020
  idp.installed.version=4.0.0
  idp.previous.installed.version=3
  

• We know the new versions which are available (see below)
• We can determine (see below) dynamically what the version limits are.
• Therefore we can propose the latest version known to work with this IdP version

Which of course doesn't help the "How do I update my IdP" story unless the plugin writers allow some spread of supported versions so allow staggered upgrade.

4) Module to Module versioning

Out of scope however we should make the following recommendations (tbd whether this is a SHOULD or a MUST 

• People building plugins should eschew any dependencies which are not "part of the IdP" (where that needs to be defined - for instance what about Spring?)
• Where possible, consider shadowing dependencies.

The Update Property file

The file is is located at the URLs pointed to by getUpdateURLs().  Multiple locations allow for redundancy.  The property names are derived from the plugin id , with the specific version appended where this is relevant.  All three digits of the version information should be present and dot-separated.

Rules for version comparison

• Assume a patch of zero if not defined
• Version minimum is INCLUSIVE, maximum is EXCLUSIVE
  •  so min 4.1, max 5.0 would allow anything starting at 4.1 but not an upgrade to 5.0
• Patch versions do matter (but should not be specified if not needed)
• Iff major versions are the same then compare minor versions
• Iff major & minor versions are the same then compare patch versions

Detecting the need to update

This is all done from within the IdP in an admin flow. 

• All installed plugins will be in the IdP classpath
• Then use the service interface to get all plugins 
• Collect the version.matrix.properties file.
• Open the this as a property file and check the version/subversion/patch for increasing values against what the plugin claims
• If the number have changed we want to propose the upgrade,

In a later release we might chose to have an external program - this would build a classpath from idp.home/dist/webapp-*/WEB_INF/lib 

Updating

# plugin update <pluginid>

• Collect URL+/version/package.tgz  and URL+/package.jar.tgz.sig (or zip equivalents)
• Check the signature against the keystore
• Unpack to a folder for update and do the obvious thing
  • Check versioning
  • Uninstall old,
  • Install new
  • Do as much ACID dance as we want to make this crash proof:

Example of being crash proof

1. Unpackage to idp.home/unpackage-net.shibboleth.foo 
   • If discovered on restart this is deleted (undo)
2. Rename (atomic) to idp.home/unpackage-net.shibboleth.foo
   • If discovered on restart operation restarted here (redo)
3. Uninstall old (from idp.home/edit-webapp-net.shibboleth.foo 
   • This is idempotent.  The existence of the unpackage folder causes a post crash restart from (2).  
4. Install from idp.home/unpackage-net.shibboleth.foo
   • This must be idempotent (so care needed with property manipulation)
5. Commit changed by deleting idp.home/unpackage-net.shibboleth.foo
   • Or (destructively) renaming to dist-net.shibboleth.foo

Initial Install

The initial install is made fraught by the whole "bootstrapping trust" thing.  You have an artefact which you don't trust which is the only place you can do to find out it.   For the initial bootstrapping of trust we do not want to run any code which we do not trust.  The trick is therefore to

• Open the distribution as a zip or tgz file (needs commons-compress)
• Extract the plugin id and the certificate used to sign the distro
• Query the user to add the certificate to the (per plugin) trust store
• We now have "trust" established
• The distribution can be unpacked and the services interface checked.

• Check versioning
• Install in the canonical manner

Blocking IdP upgrades

Given a putative IdP version "it can be seen that" one can test whether the installed plugins are compatible.  This leads us to two options

• An admin flow to "test for update".  This is the easier option
• A way to probing an IdP installation for it's version and using that before the install starts to block the update.  This is a harder option and I'd sooner not do it to start with

Behind all this is a feeling that there is a "turtles all the way down" solution by which the IdP also exports most of the Service Interface and this can therefore be used to drive update detection and testing and even "download and update" the IdP.  

IdP Updates Insert

IdP update needs to be modfiied to preserve the dist\edit-webapp-pluginID  folders.

Windows.

Nothing special.  We should have no interest in deploying special windows installation technology.  Windows user can just do what everyone else does.  Anyone smart enough to want a plugin probably wants to (a) not be running Windows and (b) owning their jetty install.

Testing

My ideas here are not well formed yet, but there seem to be at least three different areas

• Testing the plugins themselves (as stand alone items)
  • I think that we will discover how best to do this as we develop the plugins. 
  • We probably want to test not just the functionality the plugin provides but also its plug-in-ness
• Testing the IdPs interaction with the plugins
  • The trick here is that we can do this testing by adding the plugin jars as test dependencies of the areas we want to excercise the IdP in
• Testing the module installer
  • I this this might be best addressed by building a testing framework for the IdP Installer and extending that.

Of course there are two axes of "one off testing" and regression/unit testing,  I think I'd sooner concentrate on the latter and leave the former as ad hoc testing as part of the development process.

Project Process Issues

Again, this is an area where my ideas are not well formed yet.  Some observations

• We agreed that plugins live in separate git repositories.  Small plugins (like nashorn or rhino) share a repository
• I Created IDP-1595 - Getting issue details... STATUS to capture the work items I envisage as being required.  That case, and specifically the sub tasks,  might (or might not) provide a suitable location for discussions of the details.
• I have created the following "under contruction" documentation pages
  • PluginPOMGuidance
  • PluginTestbedGuidance
  • PluginTrust
• There is the usual issue of where to put the classes.  Development is happening in the idp-installer project.  But if we want to use classes from this in an administrative flow then they will need to be moved to "somewhere else" and it is not obvious where that somewhere else is.

Licensing considerations

These come in two flavors

Plugin licensing

The plugins need to obey relevant licensing restrictions.  The plugin architecture allow the optional install time download of files which can help this

Plugin Installer licensing

Building the plugin will require some new dependencies.  We need to be aware of them

• Bouncy castle (for GPG signature checking).
  • This is already a dependency, so we assume that we are covered
• Commons-compress (needed to unpick tgz files) 
  • This is Apache licensed
  • But its dependencies are not and work is need to fulfill their requirements. (one 2-Clause BSD , one MIT)
    • It looks as thought the stated dependencies are not real,  They do not live in the POM and they are not in our repository.
• The is the perpetual where to put the classes question.
  • This is all being developed in the idp-installer project
  • But if we want to report about plugin status in  an admin page or jsp or some sy

Further documentation

• Plugin Archetype
• PluginPOMGuidance
• PluginTestbedGuidance
• PluginTrust
• PluginBuildSteps