This page is out of date and replaced by Plugin Installation

Introduction

As discussions have continued the shape of the proposed solution has changed. Rather that rewrite the document I have added sections. It doesn't make for great readability, but it does make for easier re-review.

During the life of IdP V4 we want to put the framework in place to allow us move away from a monolithic approach where everything anybody might want is available to a slightly more modular approach. where deployers can select what services they want.

Examples might be

Nashorn support (post JDK15)
Rhino support
OIDC
Duo Plugin
Database backed Storage (consent &c).

This documents explores the area.

Requirements

Easy to use as a deployer and as a developer
Allow secure (manual) update from a server
Flexible update detection (for instance, a module's IdP version dependency must not be statically defined when the module is installed).
Flexible enough to do OIDC (with maybe some hand config)
Stackable (multiple modules)
KISS
Start development for 4.1; Full functionality ready for 5.0

Observations

Two key things have resulted from discussion so far

1) The plugin name is critical

This name must be

Unique
Capable of being part of a file name and part of a URL without escaping.

The suggestion is to use the reverse DNS notation that the author controls hence net.shibboleth.plugins.scripting.rhino could be used by the Shibboleth project because it owns the *.shibboleth.net domain tree.

2) Our thoughts on trust and signing have developed.

See below

After-installation end-state

For a product called net.shibboleth.foo

A new directory dist/webapp-net.shibboleth.foo
- Build-war will incorporate all of dist/webapp-* (in a random order). This is this done after the population from dist/webapp (as now) and before edit-webapp.
- We might want to add a post-edit-webapp folder to our distribution to allow us to force the jars we want.
- The plugin can thus use preConfig.xml and postConfig.xml to setup global beans
- The plugin can use this to define flows
- The plugin can do that to make jars available
A new file credentials/plugins-trust/net.shibboleth.foo/truststore.asc which contains the public PGP keys that the deployer has accepted for use with this plugin. See PluginTrust
Files may have been added across the installation (but *not* in system or dist)
- These will NOT overwrite existing files.
Properties files auto-edited
- Files added to idp.additionalProperties
- List properties can be edited. For example
```
idp.service.attribute.filter.resources = shibboleth.AttributeFilterResources

(or)

#idp.service.attribute.filter.resources = shibboleth.AttributeFilterResources
```
  becomes
```
idp.service.attribute.filter.resources = BEAN.idp.service.attribute.filter.resources.net.shibboleth.foo
idp.service.attribute.filter.resources.OLD.net.shibboleth.foo = shibboleth.AttributeFilterResources
```
  And the installation is required (via preConfig/postConfig) to define BEAN.idp.service.attribute.filter.resources.net.shibboleth.foo which is the list merge of what it needs plus %{idp.service.attribute.filter.resources.OLD.net.shibboleth.foo}
  List merging is done using net.shibboleth.ext.spring.factory.CombiningListFactoryBean
  
  This list merging is primarily aimed at adding files to existing service definitions

Driving the Installation

The install package is shipped as .tgz (Linux line ending) or .zip (Windows). The distributiin must be accompanied with an armoured detached PGP signatures.

The format of an unpacked package is

Example package layout

Directory	Installation notes
`bin`	optional.
`bin/lib`	not allowed (it is refreshed by the installation)
`bootstrap`	NOT COPIED Contains the bootstrap information for the plugin installer
`conf` `conf/*`	optional.
`dist`	not allowed
`credentials`	deprecated
`doc`	recommended (for licenses) Note: changes to this file will not be installed and so the name should change with changes to ensure that the customer site has up to date licensing documentation
`flows/*`	optional
`logs`	not disallowed
`messages`	optional
`metadata`	not disallowed
`system`	not allowed
`view`	optional
`webapp`	Required wildcard copied to `dist/webapp-pluginId`

The Service Interface

The installation is driven from within the (unzip'd) package itself via the service interface. So the installer just needs to unpack, add edit-webapp/WEB_INF/lib (in the package) to the classpath and fire up service API to see what it has to do.

The Interface will have methods to return (at least) the following

Method	Description
`String getPluginId()`	Used as per above to uniquify file and property names
`List<String> getAdditionalPropertyFiles()`	0 or more property file names to add to `idp.additional.properties` Not supported in currently
`List<Path> getFilePathsToCopy()`	0 or more files to add to the distribution Install will fail if files are going where they shouldn’t (see above) The install will NOT do a wildcard copy Webapp is special cased, and wildcard copied, so it must not be mentioned
`List<Pair<URL, Path>> getExternalFilePathsToCopy()`	0 or more files and their location. The <Path> is relative to idp-home . The user will be prompted to download these (leap of faith). The alternative for them is to download this by hand.
`List<URL> getUpdateURLs()`	1 or more locations where a property file can be located. The properties will include those with a name derived from the plugin Id.
`List<Pair<Path, List<String>>> getPropertyMerges()`	A list of property files and paired with a list of properties to merge. So in the example above if would be [ { “service.properties, // File [ “idp.service.attribute.filter.resources” , ] // List of properties in that file } , ] Not currently supported
`int getMajorVersion()`
`int getMinorVersion()`
`int getPatchVersion()`

Bootstrap region

The plugin ID and the initial keystore are included as text (so no suspect jar need be on the classpath) in a directory with is part of the package (but not installed) called "bootstrap". This contains two files

id.property a file with a single property in it pluginid
keys.txt the signing keys. This is optional. See PluginTrust

Uninstalling

Uninstallation cannot occur because of any changes to the configuration will render the system unusable. As part of an upgrade the webapp parts of the previous distribution will be removed but nothing else.

Update

Doing an update consists of collecting the new version, uninstalling the old version, installing the new one.

Because the config files are unchanged, configuration is rolled forward.
Depending on how much we care, we can do the usual ACID tricks to be able to roll forward or back from crashes during an update (mostly using file rename as the ATOMIC operation). See below for an example
So this requires that an install (and indeed an uninstall) is idempotent.

Versioning

This has many different aspects. The four most important are:

1) Jar versions.

Jars can now be sourced from four different types of place

The IdP.
This plugin
Other plugins
Edit-webapp

If each supplies a jar which provides the same class, precisely which class will be used will be random.

I do not believe that it is sensible to check for this. so the best we can do is provide mitigation, probably via some orthogonal configuration. This might (for instance) be a series of regexps which would make up a black list of jars not to copy. How this is communicated is tbd.

2) IdP to Module versioning

There is a new IdP version. Will the modules work with it?

We can be told (by the deployer) the new IdP version to check for
We know our current version (see above)
We can determine (see below) what the IdP version limits are (as of today, not as of when the plugin was made)

3) Module to IdP versioning

This Module has a new version. Can it be installed against the IdP?

We know the IdP version (it is written during install and upgrade to dist):

idp.installed.properties

#Version file written at 2020-03-11T09:59:08.023Z
#Wed Mar 11 09:59:08 GMT 2020
idp.installed.version=4.0.0
idp.previous.installed.version=3

We know the new versions which are available (see below)
We can determine (see below) dynamically what the version limits are.
Therefore we can propose the latest version known to work with this IdP version

Which of course doesn't help the "How do I update my IdP" story unless the plugin writers allow some spread of supported versions so allow staggered upgrade.

4) Module to Module versioning

Out of scope however we should make the following recommendations (tbd whether this is a SHOULD or a MUST

People building plugins should eschew any dependencies which are not "part of the IdP" (where that needs to be defined - for instance what about Spring?)
Where possible, consider shadowing dependencies.

The Update Property file

The file is is located at the URLs pointed to by getUpdateURLs(). Multiple locations allow for redundancy. The property names are derived from the plugin id , with the specific version appended where this is relevant. All three digits of the version information should be present and dot-separated.

Property Name	Property Value Description	Example
`<pluginid>.versions`	A list of versions in arbitrary order. All three digits of the version must be present	`net.shibboleth.plugin.totp=1.0.0 2.0.0 2.1.0 2.1.1 2.1.2`
`<pluginid>.downloadURL.<version>`	Space separated list of directories where this version is to be found. Multiple values allow redundancy.	`net.shibboleth.idp.plugin.rhino.downloadURL.1.1.0=\ https://build.shibboleth.net/nexus/service/local/\ repositories/releases/content/net/shibboleth/idp/plugin/idp-scripting`
`<pluginid>.baseName.<version>`	The file name. the following four suffixes will be added during download `.tar.gz` (Linux line endings) `.zip` (Windows line endings) `.tar.gz.asc (`signature) `.zip.asc`	`net.shibboleth.idp.plugin.rhino.baseName.1.1.0=\ shibboleth-idp-plugin-rhino-1.1.0`
`<pluginid>.idpVersionMax.<version>`	The maximum (exclusive) IdP version supported. Trailing version and ppatch levels of zero are inferred	`net.shibboleth.idp.plugin.rhino.idpVersionMax.1.1.0=5`
`<pluginid>.idpVersionMin.<version>`	The minimum (inclusive) IdP version supported. Trailing version and ppatch levels of zero are inferred.	`net.shibboleth.idp.plugin.rhino.idpVersionMin.1.1.0=4.1.1`
`<pluginid>.supportLevel.<version>`	Values are: Current OutOfDate Unsupported Secadv Withdrawn,	`net.shibboleth.idp.plugin.rhino.supportLevel.1.1.0=Current`

Rules for version comparison

Assume a patch of zero if not defined
Version minimum is INCLUSIVE, maximum is EXCLUSIVE
- so min 4.1, max 5.0 would allow anything starting at 4.1 but not an upgrade to 5.0
Patch versions do matter (but should not be specified if not needed)
Iff major versions are the same then compare minor versions
Iff major & minor versions are the same then compare patch versions

Detecting the need to update

This is all done from within the IdP in an admin flow.

All installed plugins will be in the IdP classpath
Then use the service interface to get all plugins
Collect the version.matrix.properties file.
Open the this as a property file and check the version/subversion/patch for increasing values against what the plugin claims
If the number have changed we want to propose the upgrade,

In a later release we might chose to have an external program - this would build a classpath from idp.home/dist/webapp-*/WEB_INF/lib

Updating

# plugin update <pluginid>

Collect URL+/version/package.tgz and URL+/package.jar.tgz.sig (or zip equivalents)
Check the signature against the keystore
Unpack to a folder for update and do the obvious thing
- Check versioning
- Uninstall old,
- Install new
- Do as much ACID dance as we want to make this crash proof:

Example of being crash proof

Unpackage to idp.home/unpackage-net.shibboleth.foo
- If discovered on restart this is deleted (undo)
Rename (atomic) to idp.home/unpackage-net.shibboleth.foo
- If discovered on restart operation restarted here (redo)
Uninstall old (from idp.home/edit-webapp-net.shibboleth.foo
- This is idempotent. The existence of the unpackage folder causes a post crash restart from (2).
Install from idp.home/unpackage-net.shibboleth.foo
- This must be idempotent (so care needed with property manipulation)
Commit changed by deleting idp.home/unpackage-net.shibboleth.foo
- Or (destructively) renaming to dist-net.shibboleth.foo

Initial Install

The initial install is made fraught by the whole "bootstrapping trust" thing. You have an artefact which you don't trust which is the only place you can do to find out it. For the initial bootstrapping of trust we do not want to run any code which we do not trust. The trick is therefore to

Open the distribution as a zip or tgz file (needs commons-compress)
Extract the plugin id and the certificate used to sign the distro
Query the user to add the certificate to the (per plugin) trust store
We now have "trust" established
The distribution can be unpacked and the services interface checked.

Check versioning
Install in the canonical manner

Blocking IdP upgrades

handwaving ahead.

Given a putative IdP version "it can be seen that" one can test whether the installed plugins are compatible. This leads us to two options

An admin flow to "test for update". This is the easier option
A way to probing an IdP installation for it's version and using that before the install starts to block the update. This is a harder option and I'd sooner not do it to start with

Behind all this is a feeling that there is a "turtles all the way down" solution by which the IdP also exports most of the Service Interface and this can therefore be used to drive update detection and testing and even "download and update" the IdP.

IdP Updates Insert

IdP update needs to be modfiied to preserve the dist\edit-webapp-pluginID folders.

Windows.

Nothing special. We should have no interest in deploying special windows installation technology. Windows user can just do what everyone else does. Anyone smart enough to want a plugin probably wants to (a) not be running Windows and (b) owning their jetty install.

Testing

My ideas here are not well formed yet, but there seem to be at least three different areas

Testing the plugins themselves (as stand alone items)
- I think that we will discover how best to do this as we develop the plugins.
- We probably want to test not just the functionality the plugin provides but also its plug-in-ness
Testing the IdPs interaction with the plugins
- The trick here is that we can do this testing by adding the plugin jars as test dependencies of the areas we want to excercise the IdP in
Testing the module installer
- I this this might be best addressed by building a testing framework for the IdP Installer and extending that.

Of course there are two axes of "one off testing" and regression/unit testing, I think I'd sooner concentrate on the latter and leave the former as ad hoc testing as part of the development process.

Project Process Issues

Again, this is an area where my ideas are not well formed yet. Some observations

We agreed that plugins live in separate git repositories. Small plugins (like nashorn or rhino) share a repository
I Created IDP-1595 - Getting issue details... STATUS to capture the work items I envisage as being required. That case, and specifically the sub tasks, might (or might not) provide a suitable location for discussions of the details.
I have created the following "under contruction" documentation pages
There is the usual issue of where to put the classes. Development is happening in the idp-installer project. But if we want to use classes from this in an administrative flow then they will need to be moved to "somewhere else" and it is not obvious where that somewhere else is.

Licensing considerations

These come in two flavors

Plugin licensing

The plugins need to obey relevant licensing restrictions. The plugin architecture allow the optional install time download of files which can help this

Plugin Installer licensing

Building the plugin will require some new dependencies. We need to be aware of them

Bouncy castle (for GPG signature checking).
- This is already a dependency, so we assume that we are covered
Commons-compress (needed to unpick tgz files)
- This is Apache licensed
- But its dependencies are not and work is need to fulfill their requirements. (one 2-Clause BSD , one MIT)
  - It looks as thought the stated dependencies are not real, They do not live in the POM and they are not in our repository.
The is the perpetual where to put the classes question.
- This is all being developed in the idp-installer project
- But if we want to report about plugin status in an admin page or jsp or some sy

Further documentation