Jetty and Windows

We currently only worry about Jetty for our Windows deployers. The motivation is historic but boils down to an assumption that Windows sysadmins are less confident when working with open source software. To that end the installer started being (and still is IMO) “too helpful”. For instance there is a “I want to work with AD” button which would be better replaced with just following the documentation.

Anyway, as far as the installed Jetty goes:

1. There is exactly one user configurable file: idp.ini. This contains the paths to access the keystores, the keystore passwords and so on. The contents of this file are preserved (and updated) during installs. Nothing else is preserved.

2. The assumption is that the user doesn’t care about Jetty beyond “it has to be there” so an optimal solutions do not matter. For instance you always get a backchannel.

3. We have always aimed for the “It kinda-works once the installation is done” approach. Now that there are things like the “Hello” module, I think this is wrong and we should rely on explicit post install stages (after all the user is expected to put in a proper user facing cert).

The Installation

If the Windows Installer does install Jetty it:

1. Installs an unpacked distribution (jetty-home) alongside the IdP.

2. Installs a jetty-base inside the IdP (as a peer to ‘conf’ and ‘edit-webapp’ and so on). As noted above the jetty base has one user adjustable file.

3. Installs & configures Procrun (an Apache projects to allow you to run java programs as daemons).

   • They keep threatening to end of life it and assure people that there are much nicer solutions.
     My particular favorite recommendation is the one that allows you to inject arbitrary shell scripts to be run at system startup because that's really safe and secure. Frankly if Procrun is ever end of lifed it would be quicker to write the 200+ lines of C to do what we need ourselves.

   • The only interesting thing here is that it does require that Jetty not fork - it is unclear whether this is a Windows thing or a Procrun thing.

   • Actually there is another interesting thing. How much of what we are struggling with is because the limits that Procrun puts on us?

The installation gets a bit grubby on Windows because we want to be able to run the daemon from a low privilege account.

The Jetty base that is used in the installer comes from the “Jetty base” repo. It is, however and entirely different branch (“10-windows”) from the “11” branch. I'll note that the Windows branch has been shown to work on jetty11 and jetty 10.

The substantive changes between the “11” branch and the “10-windows” branch are:

• idp-backchannel.xml: The default credentials location is different. I cannot remember why but there is reason

• Some minor differences in idp.xml which I could collect (SystemProperties, not Properties)

• And 6 files

jetty-ssl-context.xml (Windows Only)

This is a changed version of the one distributed with jetty and its maintenance represents the biggest ongoing cost.
https://shibboleth.atlassian.net/wiki/spaces/DEV/pages/2937454597/JettyBase10ForWindows#etc/jetty-ssl-context.xml describes what needs to be done. As I recall it is all because at one stage (probably even now) Jetty got confused about

• Absolute vs relative paths (which really matters on windows)

• How to do defaulting of values. I think that there are 3 ways to do defaults now

idp.ini.windows (Windows Only)

The installer turns this into idp.ini on a fresh install. It is modified to contain any collected passwords (keystores and so on)

idp-system.ini (Windows Only)

Contains the “don't tinker with these” settings from the mainline idp.ini

idp.ini.rewrite.property.names (Windows Only)

This deals with the issue when jetty rename properties (or indeed if one of our xml files changes a property name). The (property) name changes this file contains are made to an existing idp.ini

idp-backchannel.ini, idp.ini (not Windows)

These are not there for Windows, instead they are subsumed into idp.ini.windows and idp-system.ini

Installing Jetty base

Install the jetty base is made complicated since

• The Jetty project have been known to changes property names between release and we may need to rewrite the user’s idp.ini. This is historic in as much as they have done it once and we need to be able to always make the changes.

• We need to merge in things like the passwords the user was prompted for.

• We create a keystore for the https port with a self signed cert in it.

• Its all done with ant (which feels old)

• But basically it ‘just works’ .

Other shit

• I cannot remember why but for historic reasons we install into the 32 bit folder. Of course this is java so we are actually keyed into what JVM is installed.

What are the costs?

Cranking a Windows release is pretty quick (Download the IdP, Jetty, Jetty-Base and Procrun and run two bat files). I have a note that it might be nice to bundle that into a maven plugin so it can be done by anyone with a Windows machine. Critically this would allow us to use the enforcer to do the signature checks on Jetty and procrun

The costs are the ancillary ones:

• Making sure that the files we do ship have not diverged from jetty (https://shibboleth.atlassian.net/wiki/spaces/DEV/pages/2937454597/JettyBase10ForWindows#Check-Changes )

• Smoke testing the install

• My general unease in building a any thing on my dev box.

And we have this costs every time that the IdP ships and every time that Jetty ships. Subjectively, it is the jetty changes that are the biggest pain because they are no made on our timeline.

So what are the options?

I list these in order of programming effort, least to most.

• Walk away from shipping jetty on any platform

  • I am so bored with Jetty and this installer that this appeals to me. I’ll support the real installer for ever but….

• Bludgeon what we have now into working with V5.

  • The path of least resistance but it does mean that we need to carry on shipping windows specific files (jettty-ant.xml) in the idp. I could even fix that at the same time.

  • I’d actually like to walk back from all the helpful stuff that the windows installer does like editing ldap.properties.

• Bludgeon what we have into two packages. One for the IdP and one for jetty

  • More work to get there (because we are effectively splitting one installer into two, and we still have the same costs with every jetty release.

  • And we inadvertently introduce a test matrix (which we solve by saying “We only support the latest IdP with the latest Jetty” thus gaining us nothing)

• Try to bring the jetty base's together.

  • I have no idea what the problems would be and the wounds from last time still hurt, but I have a gut feeling that the real problem is that on Windows we cannot allow jetty to fork (I don’t even know whether that is because of Windows or because of procrun)

If we were starting with a clean slate

• I would be looking to extensions to the plugin and module infrastructure so that the user basically said “plugin -I jetty”

• Is there some sort of inversion whereby the IdP itself starts jetty and we start the IdP in some clever way.

  • Again on Windows I have no issue with walking away from procrun which is overly complicated for our needs and may well get in our way.

I think we need to think about what we are doing with Jetty at a project level, then I can fit windows around it.

…. And we haven’t even talked about jetty12. If this really works it does change the game. Or at least the questions.

Minor details on porting the installer to V5

(For my information only)

• In V5 there is no install -nocopy so we need to unpack to a location in \ProgramData and run the installer from there

• The rest of the user interface could stay the same (we use property driver installs)

  • I’d dearly like to dump all the AD stuff (which involves LDAP jiggery pokery)

  •