Jenkins CI Server

The Shibboleth project runs a private instance of the Jenkins continuous integration server. Previously, this server provided unauthenticated read-only access to anyone with an interest in the project, however, access has been restricted to project committers due to frequent advisories and updates.

The Jenkins server runs three main classes of job:

• Commit rebuild jobs are run as soon as possible after a commit is performed to a Subversion repository.  These jobs verify that the commit has not "broken the build", and their main output is an e-mailed notification to the developer performing the most recent commit.  Example: java-identity-provider.

• Multi-JDK rebuild jobs are more intensive versions of the normal commit rebuild jobs, in which the software is further verified across different build and execution environments.  Because they are more intensive, these jobs are run less frequently.  Again, the main output is an e-mailed notification.  Example: java-identity-provider-multi-jdk.

• Nightly jobs are run to deploy snapshot artifacts to the project's Maven repository.  Example: java-identity-provider-nightly.

In most cases, each project will be associated with one job of each of these three types.

A few other jobs have specific one-off purposes.

Job Configuration Guidelines

When creating a new Jenkins job, you should follow the guidelines below for each type of job.

Commit Rebuild Job Configuration

Commit rebuild jobs are intended to provide immediate feedback to the developers when a commit is made to the source repository, so that immediate action can be taken if problems have been introduced. Commit rebuild jobs no longer perform much in the way of reporting (other than a go/no-go and test result summary); for detailed reporting, look to the nightly jobs.

• Commit rebuild jobs are "Maven" jobs in Jenkins terms.

• The job name should be the name of the corresponding project, e.g., java-identity-provider.

• So that they run as soon as possible after a commit, these jobs are triggered by SCM polling, with the following interval specifier:

• The "H/15" formulation is Jenkins "hash notation" meaning that the poll should be executed every 15 minutes, but otherwise randomised so that for example all the commit rebuild jobs don't fire at exactly the same time.  This helps to even out the load on our shared infrastructure server, which hosts our Jenkins and Subversion servers as well as things like JIRA and Confluence.

• In the "Advanced" options of the "Build" section, select "Disable automatic artifact archiving".  These artifacts don't have value for the commit jobs, and they can take a lot of space.  (Older jobs also have the number of builds retaining archived artifacts set to 2 as a transitional measure.)

• Typical Maven goals: clean -Prelease install

Note that the commit rebuild jobs take advantage of the Jenkins Maven plugin's ability to compute relationships between jobs and trigger "downstream" jobs when an "upstream" job completes.  For example, a commit to the java-parent-v3 project will ultimately cause all of the V3 Jenkins jobs to be run. The dependency graph should be checked from time to time to make sure that jobs aren't being missed out due to dependency issues.

Multi-JDK Job Configuration

Multi-JDK jobs are intended to provide a more in-depth verification of the current state of a source repository by building and testing in different Java environments.  Because these jobs are much more expensive to run, and relatively rarely pick up problems that the corresponding commit rebuild job does not, multi-jdk jobs are run much less frequently.

• Multi-JDK jobs are "multi-jdk" jobs in Jenkins terms.

• The job name should be the name of the corresponding project suffixed with -multi-jdk, e.g., java-identity-provider-multi-jdk.

• These jobs should be triggered by SCM polling so that they only run if changes have been made, but polled only once per day:

• The number of builds retained should be 30, to allow good tracking of trends.

• Typical Maven goals: clean verify -Prelease

Nightly Job Configuration

Nightly jobs are run once per day.  Their principal functions are:

• Build SNAPSHOT artifacts and deploy them to our Nexus Maven repository for use by active developers, both within the Shibboleth team and externally.  Nightly jobs never deploy release artifacts.

• Generate and publish reports such as the project's site report, test reports, Checkstyle reports, etc.

Nightly jobs are configured as follows:

• Nightly jobs are "freeform" jobs in Jenkins terms.

• The job name should be the name of the corresponding project suffixed with -nightly, e.g., java-identity-provider-nightly.

• Nightly jobs are arranged in groups consisting of a sequence of jobs associated with a particular parent POM:

  • A trigger job (trigger-nightly-v7,trigger-nightly-v8) which determines the time the group is rebuilt. This triggers the building of the parent project's nightly.

  • The parent project's nightly job, which once built triggers the first dependent nightly,

  • Subsequent dependent nightlies, each of which triggers the next once it has been built.

  • The group is structured mainly as a chain, but jobs which themselves have no dependencies may be added as branches at any point where all of their own dependencies have already been built. This is useful for jobs which we might want to re-trigger the nightly for without rebuilding everything.

• Within each group, the triggering of the subsequent job is:

  • An unconditional post-built action; this forces all jobs in the chain to be performed even if one should fail.

  • Ordered so that a dependency is always built before all of its dependent jobs.

• The number of builds retained should be 30, to allow good tracking of trends.

• Typical Maven goals: clean -Prelease -U deploy site

  • The site goal enables things like the Checkstyle and Cobertura results to be published.

• Post-build actions:

  • Publish Checkstyle analysis results

    • Leave box blank

    • Select "Detect modules" advanced warning for multi-module projects.

  • Build other projects:

    • next project in the chain

    • Select "Trigger even if the build fails" option.

  • Publish HTML reports:

    • target/site | index.html | Site Report

  • Publish Javadoc:

    • target/apidocs or target/site/apidocs (which?) for single module project

    • foo-parent/target/site/apidocs for multi-module project

  • Publish TestNG Results:

    • **/target/surefire-reports/testng-results.xml

    • Select "Escape Test description string"

    • Select "Escape exception messages"

  • E-mail notification:

    • commits@shibboleth.net

    • Send e-mail for every unstable build

The V7 nightly job group executes in the following order, with indentation indicating branching from the main chain:

• trigger-nightly-v7 (at 20:00 Eastern every day)

• java-parent-project-v7-nightly

  • ant-extensions-nightly

  • jetty7-dta-ssl-nightly

  • jetty9-dta-ssl-nightly

  • tomcat6-dta-ssl-nightly

• java-support-v7-nightly

• spring-extensions-v5-nightly

  • java-metadata-aggregator-v0.9-nightly

• java-opensaml-nightly

• java-identity-provider-nightly

• java-idp-testbed-nightly

The V8 nightly group executes in the following order:

• trigger-nightly-v8 (at 20:00 Eastern every day)

• java-parent-project-v8-nightly

Other Jenkins Jobs

Plugin version matrix tests

By default the SNAPSHOT version of a plugin is tested against one IdP version (usually the current SNAPSHOT). The matrix tests allow testing of any Plugin and IdP version. Specifically this allows testing that

• A change to the plugin has not broken down stream IdP compatibility

• A change to the IdP has not broken down stream plugin compatibility

These tests are run as a pipeline driven from a file in the appropriate git repository.

Currently only two test are run.

• The scripting plugin java-idp-plugin-scripting-v2-idp-versions-java17

• The OIDC storage service java-plugin-storage-jdbc-v2-idp-versions-java17

Maven Tooling Tests

Currently the only test is java-versions-set-nightly This exercises the mvn versions:set plugin which is used during builds but which is not used in any other CI build. This test ensures that this plugin does not bring in aby untrusted dependencies.

Jenkins System Configuration Notes

Jenkins is configured to use only one "executor" in our deployment; the default shipped with Jenkins is to use two.

• This reduces the peak load Jenkins can impose on our shared infrastructure machine, to avoid interfering with other services.

• In general, multiple jobs containing mvn install goals cannot execute safely in parallel.

Upgrading Jenkins

Jenkins and its plugins are updated fairly regularly, but it's probably only worth paying attention if there's something we need, or a known security issue.  Ian intends to do an update sweep every couple of months.  Another reason to do an update sweep is that a new plugin to be installed requires a later version of the core of Jenkins than the one we're running.

Always update the core of Jenkins before updating plugins.  If you go to the plugin administration page first, you will quite often find that the latest updates to modules require the latest version of the Jenkins core, and disclaim behaviour on an older version.

To update the core of Jenkins:

• read the changelog for anything scary

• in the web application, Manage Jenkins | Prepare for shutdown.  This will tell Jenkins to stop executing new jobs, so that it will become safe to shut it down once any running at present have completed.  You will start seeing a red banner saying "Jenkins is going to shut down" on every page.

• log in to the infrastructure machine and run : sudo yum update jenkins

To stop or start Jenkins :

• sudo systemctl stop jenkins

• sudo systemctl start jenkins

Once this has restarted, check that things are basically functional before proceeding to update the plugins.

Now update the plugins:

• in the web application, Manage Jenkins | Prepare for shutdown.  This will tell Jenkins to stop executing new jobs, so that it will become safe to shut it down once any running at present have completed.  You will start seeing a red banner saying "Jenkins is going to shut down" on every page.

• Go to Manage Jenkins | Manage Plugins

• Review the web pages for each plugin to see what has changed

• select the plugins you want to update and select "Download now and install after restart".

Snapshot Signing

Signing of snapshots involves the copying in of a GPG “secret” keyring file and the password to decrypt it. At the moment, this key has been set to expire annually so has to be extended and reimported into Jenkins when it expires.

The commands to do this require importing the current key into the “new” format used by GPG, then exporting it back into the old format for Jenkins to use because the Maven support requires this.

As jenins on the server:

At that point the new file can be used to replace the old one in Jenkins and staged in ~jenkins for future extension.

Once that’s done, the updated key should be exported and the file in ~shibwww/html/downloads/SHIBBOLETH_SNAPSHOT_PGP_KEYS updated.