The Shibboleth IdP V4 software has reached its End of Life and is no longer supported. This documentation is available for historical purposes only. See the IDP5 wiki space for current documentation on the supported version.

Upgrading

Owned by Scott Cantor
Last updated: May 04, 2023
6 min read

Before You Begin

Upgrading from pre-V4 releases

If you are upgrading from a pre-V4 release, you must upgrade to the latest V3 release first and remove all deprecation warnings.

In addition, you must ensure that your versions of Java and servlet container meet the system requirements for V4. If they do not, you should generally upgrade those before attempting to upgrade the IdP software.

For example, Java 11 is now required, and although Jetty 8 was sufficient to run IdP V3, IdP V4 supports only Jetty 9.4+ (and Jetty 9 is essentially end of life, so Jetty 10 is strongly recommended).

You must install the new version on top of your previous installation. This is not only safe but essential to properly maintain a working system.

One caveat is that you may have customizations in edit-webapp, and those customizations are almost certainly not compatible with V4 without changes. This may require removing them first and making other configuration changes temporarily until the customizations are ported or re-validated on the new version.

Finally, you must always ensure the servlet container is stopped before making any changes to the warfile, including upgrades.

In the event that you refuse to adhere to this approach, please be aware that you MUST copy back all previously existing files in the conf directory that were present before, not just the ones you may have changed. Failure to do so is the primary cause of issues after upgrading when the proper approach is not followed.

In effect, make sure that you capture all of the files in your configuration management system, not just changed files. Once you install once, you own and effectively have changed all of the files because future upgrades will make that assumption. This will increase the likelihood, but not guarantee, that an improper upgrade "from scratch" will work, but it is still not a supported approach and may lead to outright failure at any time.

It's a good idea to review the Installation material to refamiliarize yourself with the general process. Most importantly, review the ReleaseNotes carefully for any important changes you might need to account for or make. In general, you should not expect upgrades to require changes ahead of time, but security issues or other major bugs might occasionally require special care.

The upgrade process is designed to be very safe, and will never (barring exigent need and clear documentation) overwrite any configuration files, views/templates, properties, etc. that are already in place.

By design, the idp.home/edit-webapp directory can be used to preserve changes across upgrades, but if you modify an existing file, principally web.xml, (as opposed to adding files), you should always compare your changed versions to the upgraded files to understand if any changes are important, though we will make every effort to highlight any such changes in the ReleaseNotes.

Be aware that rolling upgrades (that result in online servers on different versions) are generally guaranteed to work only for patch upgrades (changes in the final digit). Minor upgrades may sometimes include internal changes to storage formats or other implementation details that could prevent certain features from working interoperably between versions. It is best to either plan for a relatively fast rolling upgrade within a maintenance window, or plan for a short period of downtime.

Upgrading to V4.1 and Beyond

Because V4.1 introduces a large number of configuration changes and new methods for installing and enabling new features via PluginInstallation and ModuleConfiguration, some special notes are warranted for those upgrading from earlier versions. There is also a HowTo from @Scott Cantor covering the set of changes made to a real world IdP to migrate it from pre-4.1 state.

  • All of the configuration changes are meant to be backward-compatible with V4.0 and any regressions (save for very unusual cases called out in the Release Notes) will be dealt with promptly.

  • We don't know of any reasons why an upgrade directly from a remediated V3.4 installation (i.e., one free of warnings) won't work, but this has been tested much less than upgrading from V4.0.

  • It's not required to migrate some of the XML required in previous releases to properties and this can easily be done later, but you will notice a number of INFO messages in the log during startup in the form "INFO [net.shibboleth.ext.spring.util.IdentifiedComponentManager:89] - Replacing auto-wired component:" followed by the ID of a flow. This is just noting that the XML definitions in your files are overriding the built-in beans that are part of the system's internals now. That's by design to ensure your XML customizations are honored. In most cases you will have made very few changes to the XML and many of the beans are entirely unused because you may not be using the corresponding feature anyway.

  • If you have unsupported changes to system files or web flows, this is where they break. Essentially all of the internals are now inside jars and are harder to override (it generally works to place files in WEB-INF/classes but we haven't tested that extensively).

  • The new ModuleConfiguration machinery is installed automatically and the module.sh/module.bat command will typically report that all of the modules that are included with the IdP are enabled because this is based on the presence of configuration files that were always installed by older versions. You will find a number of new files created with ".idpnew" extensions because the new versions don't match the older ones. If you aren't using particular features, you can just disable those modules and remove the ".idpsave" and ".idpnew" files to clean the tree of unneeded noise. In most cases, features in modules are not really "active" without other settings. The purpose of enabling or disabling modules is generally more to include or exclude configuration files that can be excluded when features are not in use.

  • There are no plugins installed by default. This extends to the OIDC OP extension available from previous releases and this extension is NOT compatible with V4.1 and will not load. Rather, you will have to start from scratch by "undoing" changes made for that extension and removing it, and then installing the V3.0 OP plugin that is now official and compatible with V4.1. This is a necessary inconvenience to allow us to deliver an officially supported configuration, and the new version should be simpler to configure in most cases.

Once V4.1 is installed, future upgrades will take into consideration the state of modules and plugins going forward and it will be routine to see ".idpnew" (and less commonly ".idpsave") extensions after upgrades as modules upgrade themselves and install new default files without overwriting your copies. This will be more targeted than previous upgrades, so modules that you have disabled will stay disabled.

Note on Plugins

Assuming you are already on V4.1 or later, generally it’s best to update all installed plugins to the latest supported versions for your IdP before you actually upgrade the IdP, as there may be minor adjustments to them to account for changes to the later IdP release. At times you may notice more warnings in the log if you upgrade the IdP first.

Windows Upgrade

The installer available for Microsoft Windows handles upgrades from older releases. This also supports upgrading Jetty, if that option was selected.

Non-Windows Upgrade

  1. Download the latest Identity Provider software package (the zip file has Windows line endings, the tarball Unix line endings).

  2. Unpack the archive you downloaded to a convenient location. It will not be needed after installation and none of its content should be used outside the scope of the installation process.

  3. Change into the newly created distribution directory, shibboleth-identity-provider-VERSION

  4. You should take a backup of the idp.home directory prior to the upgrade in case anything goes wrong. Remember that there are files in this directory tree with highly sensitive information.

  5. Run either ./bin/install.sh (on non-Windows systems) or .\bin\install.bat (on Windows systems).

    • Make sure to specify the same installation directory you used originally (the idp.home directory). This will cause the installer to perform an upgrade. Using a different installation directory will essentially perform a new installation and this is not a supported mechanism for doing upgrades.

  6. After reviewing any necessary further changes, rebuild the warfile with any edits you've applied by running either idp.home/bin/build.sh or idp.home\bin\build.bat

  7. Re-deploy the new IdP warfile, located in idp.home/war/idp.war

Background

The rules for upgrades (which in turn drive your upgrade procedure) are derived from the Java Product Version Policy. This means that (except in exceptional circumstances):

  • It will be possible to move between patch versions of the same major and minor version (e.g.. from 4.3.2 to 4.3.1 or 4.3.3)

  • It will be possible to move to more recent minor versions of the same major version (e.g. from 4.3.2 to 4.4.1, but not from 4.4.1 to 4.3.2)

Exceptions would be:

  • Use of reserved ('impl') class names (as per the Java Product Version Policy) in spring configuration

  • Use of reserved ('impl') classes by products

  • Very rarely if required for security reasons.

In addition, major version releases (V3 to V4, or V5 to V6) will be compatibly upgraded subject to the following constraints:

  • The starting installation is the very latest release of the previous major release (at time of writing this is V3.4.8 for a V4.0.0 upgrade).

  • The installation starts and operates with no deprecation warnings.