C++ Developer's Guide

Source Control Organization

The SP is split across three primary projects, cpp-xmltooling, cpp-opensaml, and cpp-sp. At any given time, development is occurring on one or more branches, including master. Each project is a self-contained automake/autoconf/libtool project (for non-Windows) and a Visual Studio C++ solution containing various subprojects (for Windows). Each project builds on the next, so first dependencies are built and installed, then xmltooling, opensaml, and finally the SP.

Releases within a major version and the latest minor version are in a branch called maint-x where x is the major release. For example the 2.x SP code is in the maint-2 branch.

Releases within a previous minor version (patch releases) are in a branch called maint-x.y where x is the major release and y is the minor release. For example, the 1.4.x xmltooling code is in the maint-1.4 branch.

Build Environment and Compiler Notes

Non-Windows

The Git checkout is in an "unbootstrapped" state and building requires the complete set of autotools (automake, autoconf, libtool). With modern autotools, running autoreconf -if is sufficient to bootstrap the process. There should be few if any warnings, so noisy output indicates an environmental problem. To generate the "cleanest" (meaning the most portable) scripts and makefiles, FreeBSD or a Mac are ironically the best place to do the bootstrap and generation of a distribution file. Using Solaris is a lot of work to get setup, and not terribly portable, and using Linux tends to introduce Linux-only shell script features. The BSD results seem to have the best chance of working on all the officially supported platforms.

For all but Solaris, the platform default GNU compiler collection should work fine. macOS is a bit more complex in that the built-in compiler is based on gcc + llvm, while the macports default compiler is clang. The latter is usually more strict, so just building from the shell isn't sufficient to verify the build.

Solaris is a nightmare (and no longer even supported). Rather than reproducing all the details, see NativeSPSolarisSourceBuild.

Windows

See WindowsBuild for the details.

The expected layout of a Windows build is to place all the dependencies in a common directory, and then checkout or unpack the Shibboleth sources together in a common directory. To build the various Apache modules, installations of Apache should be unpacked to the root of the drive. The newer versions can be obtained from http://www.apachelounge.com/

Release Engineering

The release process for the SP involves a number of steps that can be performed in a relatively loose order, but experience has led to the following outline to minimize the chance of overlooking a portability problem and having to repeat steps. In general, a failure in any of the steps means starting over again, or there's a good chance of introducing a regression on one of the platforms.

The basic summary follows, and then each step is elaborated.

1. Generate source distributions for use by the later steps.

2. Do the Windows build and installers.

3. Generate prerelease "final" RPM packages.

4. Upload final source distributions and Windows installers to public site.

5. Generate official RPM packages.

6. Update the macports.

Source Prep

The first step is to prepare candidate source releases. The Mac is recommended for this purpose as discussed earlier, because it seems to produce the best autoconf results for the most platforms.

1. Ensure all the dependencies are available.

2. Pull the candidate sources from Git.

3. In sequence, bootstrap each project with autoreconf, then run the configure, make, make install sequence.

4. After fixing any build issues, prepare each of the distributions with make dist.

Windows Build

The Windows build tends to be in better shape if its the primary development platform, but for other developers this may not be the case. It may be advantageous to move the Windows build earlier to catch regressions there, because any fixes end up risking regressions to the rest of the builds, so the goal once this step is reached is for the chance of regressions to be low.

Preferably long before this point, but as a final check, make sure that all of the Windows resource files in the projects have been updated with final version numbers for the release, including the Windows installer files.

The "final" build is done on an EC2 instance (currently running Server 2016) that has the tools installed. The builds are done out of the Administrator account, with all the dependencies and source inside C:\Users\Administrator\Shibboleth

Linux / RPM Builds

The RPM packages are now built using a Docker-based process that is driven by a standard makefile in the cpp-linbuild project. It produces packages native to the host’s architecture and we have servers created in EC2 for both x86_64 and aarch64. The packages can be copied via EFS to the staging folders used by the project server to mirror them out.

Release Publication

The initial set of release artifacts (and the appropriate release location) consists of:

• shibwww@shibboleth.net/home/shibwww/html/downloads/c++-opensaml/ver/xmltooling-ver.tar.gz

• shibwww@shibboleth.net/home/shibwww/html/downloads/c++-opensaml/ver/opensaml-ver.tar.gz

• shibwww@shibboleth.net/home/shibwww/html/downloads/service-provider/ver/shibboleth-sp-ver.tar.gz

• shibwww@shibboleth.net/home/shibwww/html/downloads/service-provider/ver/win32/shibboleth-sp-ver-win32.msi

• shibwww@shibboleth.net/home/shibwww/html/downloads/service-provider/ver/win64/shibboleth-sp-ver-win64.msi

• shibwww@shibboleth.net/home/shibwww/html/downloads/service-provider/extensions/shibresolver/ver/shibboleth-resolver-ver.tar.gz

ver is replaced by the version number. A symlink called latest is also reset in the two product directories to point to the updated version directory.

Each artifact is accompanied by a parallel zip, tar.bz2, a PGP signature (with ".asc" added to the filename) and a SHA2 checksum (with ".sha256" added).

The builds generally use the bz2 version because it's smaller.

Once these are published, the macport Porfiles can be updated in the official macport tree safely, and the final RPM packages built and published (as described in the previous section).

If any showstopper bugs appear at this stage, they have to be addressed either with patches applied during the packaging process, or by issuing an updated patch release and essentially deprecating the release that's in progress. In other words, don't upload the sources until everything is 99.9% certain to be working. For sure, don't do it until RPM generation in a prerelease project is confirmed.

Finally, the three projects are tagged in git to complete the process.

Once the RPM build process is completed, a complicated set of scripts is used to copy all the RPM and SRPM files into place locally and then push them out to a couple of mirrors.