Codeberg migration

Shibboleth project Git repositories have for some time been hosted on the project’s own infrastructure. We are currently migrating away from this towards hosting them primarily on the public Codeberg service. This is an attractive destination amongst other reasons because there are no indemnity clauses in their terms of use.

As of 2025-10-21, each of our repositories may be:

• Canonically located at git.shibboleth.net and mirrored regularly to a read-only mirror on Codeberg.

• Canonically located at Codeberg and immediately mirrored on change to a read-only mirror on git.shibboleth.net.

• Canonically located at Codeberg but not mirrored in either direction (archived and frozen repositories).

• A special case of some kind.

The different cases are laid out in sections below, and a table is presented with the intended final disposition for each repository, along with its current state.

If you are a project committer, please:

• Set up a Codeberg account at https://codeberg.org by clicking on “Register”.

• I would recommend picking the same username as you have on GitHub, if you have one. I would not recommend any other connection with GitHub for this new account.

• The association between commits in repositories and your Codeberg account is made through the e-mail address you register with Codeberg, so if possible you should use the same e-mail address as you use for Git. Note that you can add more than one e-mail address to Codeberg if necessary.

• You should set up two-factor authentication and optionally a Passkey. This will be required for project committers at some point.

• It’s always worth providing an avatar so that we can see which user is which.

• You should upload both SSH and GPG keys in the usual way.

• Let Ian know that you have done this, and what your user name is on Codeberg, so that you can be added to appropriate teams.

Codeberg makes it possible for your membership in an organization to be either public or not. This is under your control, but unless you tell Ian otherwise you will be added to a team publicly in the first instance.

Shibboleth organization and teams

There is a Shibboleth organization on Codeberg, owned and administered by Ian Young and Scott Cantor (this is the “Owners team”).

The organization also contains a “Committers” team. This is intended to be analogous to the @committers group in the existing Gitolite configuration, with write access to most of the active repositories. Archived and Frozen repositories are excluded.

The “Translators” team is used to manage the group of project contributors who are not full Committers but are associated with the java-idp-translations repository.

A “Collaborators” team is used to enumerate those project contributors who are not full Committers. These contributors are typically given specific access to the particular repository they work with.

Mirroring from Codeberg to git.shibboleth.net

The final state for any “active” repository is to use Codeberg as their canonical location; any pushes to Codeberg are immediately mirrored to git.shibboleth.net.

Note: although pushes to Codeberg containing new commits are always mirrored to git.shibboleth.net immediately, pushes without new content may not be mirrored for up to 8h. This means that, for example, the deletion of a topic branch may not be immediately mirrored to git.shibboleth.net.

Repositories mirrored from Codeberg are distinguished in GitWeb by prefixing their descriptions with (M), for example (M) Shibboleth Metadata Aggregator.

To switch the direction of mirroring so that a repository which was mirrored from git.shibboleth.net is instead mirrored to git.shibboleth.net:

• Remove the repository from the list in parameters.sh in the infra-codeberg project in Ian’s GitHub. Push, run build and then bump to put the new list into effect.

• In the repository’s settings on Codeberg:

  • Repository > Basic settings > Description: remove the text Read-only mirror of from the beginning of the repository’s description. Save settings.

  • Repository > Mirror settings: add a new push mirror:

    • Repository URL of the form ssh://git@git.shibboleth.net/reponame.git

    • Authorisation:

      • Username: (leave blank)

      • Password: (leave blank)

      • Use SSH authentication: checked.

    • Sync when commits are pushed: checked.

    • Leave “Mirror interval” unchanged at 8h0m0s.

    • Selecting “Add push mirror” will add a new push mirroring destination and create an SSH key pair. Use the “Copy public key” option to copy the public part of this. Create a text file called codeberg@reponame.pub for this value.

  • This is not quite what you’d expect from the documentation, or from looking at existing examples. The username part of the ssh:// URL is stripped despite the documentation indicating that it should be included in the Authentication section. If you include it in the Authentication section, it won’t in fact be registered as a SSH destination and no public key will be available. If you make this mistake, just delete the mirror and add it again.

  • This will not prompt an immediate mirroring of the repository. That’s a good thing, because the destination needs to be told about the key, see below.

  • Branches > Branch protection:

    • Add rule for main.

      • Protected branch name pattern: main

      • Push: Enable push (does not allow force push)

      • Enforcement: Enforce this rule for repository admins: checked.

  • Collaborators: Add the Committers team. Add any individual external collaborators (rare).

• In the Gitolite configuration:

  • Add the codeberg@reponame.pub file to the keydir/mirror directory along with the other codeberg@ keys. Because Codeberg generates a separate key for every mirror, they all need to be individually included in the Gitolite configuration. It would have been possible to retain the per-repository semantics, but merging them all as alternative keys for a single codeberg user in Gitolite simplifies configuration.

  • In gitolite.conf, remove the repository from the @foss group (around line 200) and add it to the @mirrored group (around line 770).

  • In gitolite.conf, remove any special handling for the individual repository. This is relatively rare.

  • In gitolite.conf, add a prefix (M) to the desc property for the repository (line 780+).

  • Commit and push the Gitolite configuration to make it active.

• You can now attempt a push mirror sync from Settings > Repository > Mirror settings. Click the “Synchronise now” button, wait a few seconds and reload the page. The “Last update” field should read “now”.

• People with SSH clones of the repository should be able to switch to reference the new one like this.

  • git remote set-url origin ssh://git@codeberg.org/Shibboleth/reponame.git

• A git fetch should then succeed for them but have no effect, assuming they were up-to-date before the switch.

• git push should work if they are part of an appropriate team, or have appropriate explicit permissions. Any such push should be immediately visible in the Gitweb interface on git.shibboleth.net.

Mirroring special cases

• The info-codeberg repository which contains the code behind the mirroring service lives in a private repository on Ian’s GitLab instance: this is primarily to give an associated location for the container image it builds and deploys. It is mirrored by GitLab into an info-codeberg repository on Codeberg. This is again private but visible to the Owners team. This mirroring is performed near-instantly whenever the source repository is updated.

Mirroring from git.shibboleth.net to Codeberg

Active repositories whose canonical location is still on git.shibboleth.net are periodically mirrored to Codeberg. Note that we expect to entirely retire this process once every repository has moved to using Codeberg as its canonical location.

Mirroring is performed roughly every three hours. All repositories being mirrored are fetched into a cache, then those which have “recent changes” are pushed to Codeberg. Repositories which do not have “recent changes” are not mirrored to Codeberg on every pass, to avoid hitting suspected limits there (we have a lot of repositories). “Recent changes” is computed modelled on Gitweb’s “Last change” column: the heuristic is that date of the most recent last commit on any of the repository’s branches must be within the last seven days. This is imperfect (it won’t notice the creation of an isolated tag, and it won’t notice pathological reconstruction operations) but seems adequate for present purposes.

Adding any new repositories to the mirrored list is a fairly easy process, although it is manual (I have partial keyboard automation to help out):

• Start a “New migration” from a Shibboleth owner (currently just Ian).

• Set the origin HTTPS URL and destination namespace

• Set a description of the form “Read-only mirror of…” followed by the description from Gitolite.

• After import is complete

  • Settings > Deploy keys: add a deploy key called “Mirroring” and set it to read/write permissions.

  • Settings > Units > Issues: un-tick “Enable repository issue tracker” and save settings. (Note: at a later date, we can set this up to point to the appropriate Jira URL for each project.)

  • Settings > Units > Pull requests: un-tick “Enable repository pull requests” and save settings. (This facility makes little sense for a read-only mirror.)

  • Settings > Units > Wiki: this should already be disabled by default. (Note: at a later date, we can set this up to point to the appropriate Confluence URL for each project.)

The mirroring task runs as a Docker Swarm service in Ian’s Docker Swarm cluster. The SSH private key used for the push is held as a swarm secret. The SSH public key is in the infra-codeberg repository, currently only in Ian’s GitLab instance.

What is not mirrored to Codeberg:

• Project-private repositories: gitolite-admin (administrative).

• Gitolite personal repositories (if these are needed at Codeberg in the long run, they can be moved to personal namespaces there).

Repository disposition

This table indicates the intended final disposition for each of our repositories, as well as its current state. The underlying assumption is that whatever the final disposition, all repositories are at some point copied to Codeberg, whether by mirroring or some other means. After that:

• Active means mirrored from Codeberg to git.shibboleth.net so that Jenkins jobs can access them. Read-only in Gitolite; available both in GitWeb and over the Git protocol. Associated with the Committers team on Codeberg so that anyone in the team can work on it.

• Archived means no mirroring; in the “Archive” state in Codeberg (online if you know about it, but read-only and only appears in the dashboard for the Owners team). Locked in Gitolite: available to @committers but not via GitWeb or over the Git protocol. Once GitWeb has been removed, these could probably be deleted from the Shibboleth machine.

• Frozen means no mirroring; not associated with any teams in Codeberg (so, effectively read-only except by the Owners team). Locked in Gitolite but visible to both @committers and over GitWeb and the Git protocol. In GitWeb, described as `Once Gitweb has been removed, these could probably be deleted from the Shibboleth machine.