IdPv5 upgrade guide

Last Update: 2024-04-16

Introduction

This page contains our guidance for operators of Shibboleth Identity Providers (IdPs) within the UK federation.

Specifically, it is aimed at operators of an existing IdP running the Shibboleth IdP v4 software, that will need to be upgraded to v5.

This upgrade should be completed before September 1, 2024 when the Shibboleth IdP v4 software will reach end of support with the Shibboleth project.

Whilst the UK federation support desk will provide support for operators upgrading their IdP beyond Autumn 2024, we would advise using the following guide, in addition to documentation on the official Shibboleth IdP v5 Wiki, and upgrading as soon as possible, to ensure that you can receive necessary security updates to the software.

For organisations requiring in-depth and hands-on support we offer a Trust and Identity consultancy service

We have made every effort to provide guides that will work for you; however you are encouraged to read in full the guides before starting work, confirm their validity for your deployment and to make backups of your system before beginning.

Using the guide

• Linux-specific
  • Linux platform support
  • Java upgrade - Linux
  • Tomcat upgrade for Linux deployers
  • Jetty upgrade for Linux deployers
  • Shibboleth v4.3.1 to v4.3.3 upgrade for Linux deployers
  • Shibboleth v4.3.3 to v5.1.2 upgrade for Linux deployers
• Windows-specific
  • Windows platform support
  • Java upgrade - Windows
  • Jetty upgrade - Windows
  • Upgrade from earlier 4.x.x to 4.3.3 steps
  • Upgrade from 4.3.3 to 5.1.2 steps
• General
  • IdP deployers should use MDQ rather than metadata aggregates
  • SAML Proxying Issue

Linux-specific

Linux platform support

You should be running a current, supported distribution and release of Linux. It can be quite hard to determine when support will end for a given platform. Check https://endoflife.software/operating-systems/ for your platform and release.

You will need to know the Linux distribution and release you are running. Most distributions provide a standard command:

 lsb_release -a

This will provide output similar to:

 Distributor ID:  Ubuntu
 Description:     Ubuntu 22.04.4 LTS
 Release:         22.04
 Codename:	  jammy

If that doesn't provide the information then you may find it by running:

 cat /etc/os-release

Does my Linux distribution supply a recent enough Java and web container?

This guide covers both the Tomcat and Jetty web containers. Different Linux distributions and releases have different versions of Java and Tomcat or Jetty available.

To run IdP5 You will need Java 17 (or later) and either Tomcat 10.1 (or later) or Jetty 11 (or later) so you may need to upgrade various components.

The tables below will allow you to check if your distribution makes suitable versions of the web containers available.

If you're running a generally supported distribution release which doesn't provide the relevant supporting packages then you will need to either upgrade/migrate to one that does or manage the software dependencies yourself.

If you've visited our pages before you may note that CentOS is missing. Cloud providers (eg AWS) along with Rocky are now down-stream distributions of RedHat.

Tomcat

If you already have a suitable version of Java installed, you can skip ahead to Tomcat upgrade for Linux deployers.

Jetty

It can be tricky obtaining a suitable version of Jetty, the following table describes the situation for major distributions.

If you already have a suitable version of Java installed, you can skip ahead to Jetty upgrade for Linux deployers.

Java upgrade - Linux

In general, you should select a Linux distribution and release which provides a supported Java runtime.

Support matrices for popular combinations of platform, Java and web container are given above.

Using the distributions package manager (yum, dnf, apt etc) is the most convenient way to manage the available Java.

The following Java distributions are fully supported:

• Amazon Corretto 17 for Linux
• Red Hat's OpenJDK 17 for Linux as supplied under Red Hat Enterprise Linux (or compatible rebuild)

Note that this specifically refers to actual Java 17, and not “latest”. Some older versions include “floating” versions but not a specifically supported package for Java 17.

Other Java distributions that are substantially identical or meant to be fully compatible with OpenJDK 17 are of course likely to work, but are officially regarded as unsupported.

First check which version is currently installed:

 $ java -version
openjdk version "11.0.22" 2024-01-16
OpenJDK Runtime Environment (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1)
OpenJDK 64-Bit Server VM (build 11.0.22+7-post-Ubuntu-0ubuntu222.04.1, mixed mode, sharing)

If you need to upgrade then find the appropriate package name using the package manager's search function:

• CentOS / RedHat: yum search openjdk
• Debian / Ubuntu: apt-cache search openjdk

When you've found the package, install it using the package manager. Below are some examples but check first:

 # CentOS / RedHat 
 yum install java-17-openjdk

 # Debian / Ubuntu
 apt-get install openjdk-17-jdk

Verify the new version is being used by default:

 $ java -version
 openjdk version "17.0.10" 2024-01-16
 OpenJDK Runtime Environment (build 17.0.10+7-Ubuntu-122.04.1)
 OpenJDK 64-Bit Server VM (build 17.0.10+7-Ubuntu-122.04.1, mixed mode, sharing)

If the install process has not enabled by default then you will need to use the update-alternatives system to select it:

 update-alternatives --config java

Remember to check your JAVA_HOME is set correctly after upgrading.

Note: Java 17 does not include a JavaScript Engine like it did in earlier versions, you will need a plugin like Nashorn, there are instructions how to do this in the Shibboleth IdP v4.3.3 -> v5.1.2 upgrade guide on this page.

Tomcat upgrade for Linux deployers

Tomcat upgrade/install

If your distribution has the required version of Tomcat available then your first choice should be that package. Please see your distribution's documentation on how to install or upgrade packages.

However, should you have Tomcat installed from your distribution's package manager and this version is not appropriate for Shibboleth IdP v5, you will need to manually install the latest version (you can of course run the two versions side by side but you will need to put the configuration and installation and logs in their own directories in order to avoid clashes)

Manual installation of Tomcat

Should your distribution not provide an appropriate Tomcat package then you will need to install and maintain this yourself. The procedure below is just one way of doing this and may need altering to suit your environment.

Note

• Before doing this, ensure you have a supported version of Java available.

Overview

We're assuming you are upgrading from v4 and Tomcat9

We will be using Tomcat10 as both webserver and servlet container, removing the need for Apache to act as a proxy.

1. Download the latest version from Tomcat 10 and verify
2. Decompress the archive
3. Create a tomcat user if required
4. Prepare the new directory tree (move bundled directories and symlink to system locations)
5. Create master symlink
6. Create service
7. Configure Tomcat
8. Disable (or uninstall) Tomcat9 (if it's already installed)
9. Start Tomcat service

Details

Download the latest version

Check the latest version of Tomcat 10 available. Download the package and verify the signature to ensure it's not been tampered with:

For example:

 $ cd /usr/local/src
 $ curl -O https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.19/bin/apache-tomcat-10.1.19.tar.gz
 $ curl -O https://downloads.apache.org/tomcat/tomcat-10/v10.1.19/bin/apache-tomcat-10.1.19.tar.gz.sha512

 $ sha512sum -c apache-tomcat-10.1.19.tar.gz.sha512

 apache-tomcat-10.1.19.tar.gz: OK

Decompress the archive

 cd /opt
 tar xfz /usr/local/src/apache-tomcat-10.1.19.tar.gz

If required, create a tomcat user

Check if a tomcat user is already available:

 getent passwd tomcat

If nothing is returned then you will need to create a user to run tomcat as. This can vary between distributions but generally looks like:

 useradd --system --no-create-home --home-dir /opt/tomcat10 tomcat

Prepare the directory tree

To make upgrades easier, move the components that need to persist across Tomcat upgrades out to their traditional system locations and replace them with symlinks. Lastly, ensure the required directories are writeable by the tomcat user.

 mkdir -p /var/cache/tomcat10 /var/lib/tomcat10

 cd /opt/apache-tomcat-10.{rest of directory name}

 mv -i conf /etc/tomcat10
 mv -i logs /var/log/tomcat10/
 mv -i temp /var/cache/tomcat10/
 mv -i webapps /var/lib/tomcat10/
 mv -i work /var/cache/tomcat10/

 ln -s /etc/tomcat10 conf
 ln -s /var/log/tomcat10 logs
 ln -s /var/cache/tomcat10/temp temp
 ln -s /var/lib/tomcat10/webapps webapps
 ln -s /var/cache/tomcat10/work work

 chown -R tomcat /var/log/tomcat10 /var/cache/tomcat10 /var/lib/tomcat10 /etc/tomcat10

 chown -R tomcat /opt/apache-tomcat-10.{rest of directory name}

• note that if you have an existing tomcat user on your system that is not "tomcat" - replace with that user in the above chown command.

Create the master symlink

To make upgrades easier, create a symlink to the current version:

 cd /opt
 ln -s apache-tomcat-10.{rest of directory name} tomcat10

Create the Tomcat service

Most modern Linux distributions use systemd to manage services. Some distributions use "RC" scripts instead.

To see if your system uses systemd, run:

 systemctl status

If this returns information about running services then you can add a new service by creating a new service unit file.

 nano /etc/systemd/system/tomcat10.service

Make the contents of the file:

 [Unit]
 Description=Tomcat10
 After=syslog.target network.target

 [Service]
 Type=forking

 Environment="JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64"
 Environment="JAVA_OPTS=-Djava.security.egd=file:///dev/urandom"
 Environment="CATALINA_BASE=/opt/tomcat10"
 Environment="CATALINA_HOME=/opt/tomcat10"
 Environment="CATALINA_PID=/opt/tomcat10/temp/tomcat.pid"
 Environment="CATALINA_OPTS=-Xms512M -Xmx1024M -server -XX:+UseParallelGC"

 ExecStart=/opt/tomcat10/bin/startup.sh
 ExecStop=/opt/tomcat10/bin/shutdown.sh

 RestartSec=10
 Restart=always

 User=tomcat

 AmbientCapabilities=CAP_NET_BIND_SERVICE


 ReadWritePaths=/opt/shibboleth-idp


 [Install]
 WantedBy=multi-user.target

• NOTE that JAVA_HOME in the above can be found by running: ls -l /etc/alternatives/java and taking the path=].
• NOTE that again, if a preexisting tomcat user is different to "tomcat", replace in the above text.
• NOTE the line above AmbientCapabilities=CAP_NET_BIND_SERVICE which tells systemd to allow Tomcat10 to access privileged network ports.

Save and quit (Ctrl-X and follow the prompts).

Next, stop Apache HTTP Server:

 apachectl stop

Next, tell systemd to enable your Tomcat v10 installation. You may also want to disable any existing Tomcat v9 installation:

 systemctl enable tomcat10
 systemctl disable tomcat9

You may need to uninstall Tomcat9

On Ubuntu (and possibly other distributions) this is important as it removes Tomcat/Apache library(-ies) which Tomcat10 may complain are out of date. Do this if disabling doesn't work in the first instance.

Lastly tell systemd to reload its configuration:

 systemctl daemon-reload

Configure Tomcat

The Tomcat configuration requires a few tweaks to support the IdP.

Note that we're assuming an installation of Tomcat v10 (note the directory names in the following). You will need to alter the directory names if you've chosen a different naming convention for any reason.

Note The following moves away from using Apache as a proxy for web requests on port 443 being passed into Tomcat and onto the IdP. The following uses Tomcat as the Web server and Servlet container. This means that Apache will not be able to run alongside.

idp.xml

 mkdir -p /etc/tomcat10/Catalina/localhost
 nano /etc/tomcat10/Catalina/localhost/idp.xml

Start the Tomcat service

 systemctl start tomcat10

Check the startup logs:

• journalctl -f
• cat /var/log/tomcat10/catalina.out

Tomcat Troubleshooting

If Tomcat cannot deploy the warfile with the set docBase in the idp.xml file in /etc/tomcat10/Catalina/localhost/idp.xml you can try copying the idp.war file into the /opt/tomcat10/webapps directory to see if it will load, this will let you know if there is a permissions issue accessing the warfile directly directly from the docBase path.

Check to see if the idp.war loads and is referenced in /opt/tomcat10/logs/catalina.out like below.

 INFO [main] org.apache.catalina.startup.HostConfig.deployWAR Deploying web application archive 
 [/var/lib/tomcat10/webapps/idp.war]

Note: /var/lib/tomcat10/webapps was symlinked earlier from /opt/tomcat10/webapps

Upgrade of manually installed Tomcat - beyond initial v9 -> v10

Once you're managing Tomcat as described above, the upgrade process is:

1. Download the latest version (check Shibboleth IdP requirements) from Tomcat and verify
2. Decompress the archive
3. Prepare the new directory tree (remove bundled directories and symlink system locations)
4. Replace master symlink
5. Restart Tomcat

This may look like:

 # Download the latest version
 cd /usr/local/src
 curl -O {tar.gz file}
 curl -O {tar.gz.sha512 file}

 sha512sum -c {tar.gz.sha512 file}

 # Decompress the archive
 cd /opt
 tar xfz /usr/local/src/{tar.gz file}

 # Prepare the new directory tree
 cd {directory that you uncompress the tar.gz file into}
 mkdir ~/tomcat10-backup-1970-01-01
 mv -fr conf logs temp webapps work ~/tomcat10-backup--1970-01-01
 ln -s /etc/tomcat10 conf
 ln -s /var/log/tomcat10 logs
 ln -s /var/cache/tomcat10/temp temp
 ln -s /var/lib/tomcat10/webapps webapps
 ln -s /var/cache/tomcat10/work work
 chown -R tomcat .

 # Replace master symlink
 cd /opt
 ls -l tomcat10 # note current version
 lrwxrwxrwx. 1 root root YY MMM DD HH:MM tomcat10 -> apache-tomcat-10.something
 rm -f tomcat10
 ln -s {directory that you uncompressed the tar.gz file into} tomcat10

 # Restart Tomcat
 systemctl restart tomcat10

Jetty upgrade for Linux deployers

As of April 2024 Jetty 11 has already reached "end of community support" although security updates will still be provided until it is deemed end of life, which we expect will be end of this year going on previous lifespans.

Jetty version 12 is not available for the majority of distributions . The Shibboleth project is in the process of releasing an updated Shibboleth IdP Jetty Configuration Baseline for Jetty 12 (jetty-base) and when this is available we will publish the upgrade instructions for Jetty.

Our advice to deployers of Jetty is that you should if not running latest Jetty 11, upgrade to latest Jetty 11, and utilise until such a time as the ability to support Jetty 12 improves.

In the meantime the UK Federation can supply instructions for installing Version 5 Shibboleth with Jetty Version 11 but you will need to contact the UK federation service desk to discuss this requirement to see if this is the right approach.

Linux - Upgrading Shibboleth

Linux - Upgrading Shibboleth IdP from v4.3.1 to v4.3.3

Before you can upgrade Shibboleth IdP to v5.1.2 you must upgrade to v4.3.3, this update was released in March 2024 and must be installed prior to proceeding to the main v5.1.2 update.

After the upgrade to v4.3.3 you will be required to resolve deprecation warnings. Note that some of these warnings may only be visible in your servlet container’s log. This is due to the order of operations; the IdP’s logging service cannot be started until the global Spring context is running, and some of the configuration triggering warnings may be in that root context.

There is one set of warnings that are an exception to this: the undocumented “SAML2NameID” and “SAML1NameID” attribute encoders produce warnings indicating they were removed from the next major version.

 DEPRECATED xsi:type ''''SAML2NameID'''', (file [/opt/shibboleth-idp/conf/attribute-resolver.xml]): 
 This will be removed in the next major version of this software; replacement is (none)

In fact they were not, and their use is still valid and there are no explicit plans to remove them.

This upgrade assumes Ubuntu 22.04 LTS and Tomcat10 are installed.

Back up your current system either by taking a snapshot of your virtual machine, or creating tar archives, or a combination of these.

Check the currently installed version

Do not proceed if it less than v4.3.1:

 cd /opt/shibboleth-idp
 ./bin/version.sh
 4.3.1

Download the files

 cd /usr/local/src
 curl -O https://shibboleth.net/downloads/identity-provider/latest4/shibboleth-identity-provider-4.3.3.tar.gz
 curl -O https://shibboleth.net/downloads/identity-provider/latest4/shibboleth-identity-provider-4.3.3.tar.gz.sha256

Check SHA256 message digest matches:

 sha256sum -c shibboleth-identity-provider-4.3.3.tar.gz.sha256

Decompress the files

Untar the files and change to the directory:

 tar xf shibboleth-identity-provider-4.3.3.tar.gz
 cd shibboleth-identity-provider-4.3.3

Run the installer for v4.3.3

It should detect your current source/install dir and display that on screen as below:

 ./bin/install.sh

 Source (Distribution) Directory (press <enter> to accept default): [/usr/local/src/shibboleth-identity-provider-4.3.3]

 Installation Directory: [/opt/shibboleth-idp]

 Update from version 4.3.1 to version 4.3.3
 Rebuilding /opt/shibboleth-idp/war/idp.war, Version 4.3.3
 Initial populate from /opt/shibboleth-idp/dist/webapp to /opt/shibboleth-idp/webpapp.tmp
 Overlay from /opt/shibboleth-idp/dist/plugin-webapp to /opt/shibboleth-idp/webpapp.tmp
 Overlay from /opt/shibboleth-idp/edit-webapp to /opt/shibboleth-idp/webpapp.tmp
 Creating war file /opt/shibboleth-idp/war/idp.war
 ...done

Check the version has been upgraded

 cd /opt/shibboleth-idp
 ./bin/version.sh
 4.3.3

Although this is a minor update, please check that everything still works before proceeding to the v5.1.2 upgrade.

Reload Attribute Resolver service

This is so there are no issues being flagged on-screen or in the log files

 ./bin/reload-service.sh -id shibboleth.AttributeResolverService

If reloaded correctly you should see:

 Configuration reloaded for 'shibboleth.AttributeResolverService'

Reload Attribute Filter service

Check what attributes are being released and that there are no issues being flagged on-screen or in the log files:

 ./bin/reload-service.sh -id shibboleth.AttributeFilterService

If reloaded correctly you should see:

 Configuration reloaded for 'shibboleth.AttributeFilterService'

See what attributes are being released:

 ./bin/aacli.sh --configDir=conf/ --principal=* --requester=*

You should get something similar to below showing all the attributes you're releasing:

 {
    "name": "schacHomeOrganization",
    "values": [
        "uni.ac.uk"
    ]
  },


  {
    "name": "eduPersonTargetedID",
    "values": [
        "R45453FGEGE5GEGGSFSDF"
    ]
  }

Restart Tomcat

Check the log for any errors:

 systemctl restart tomcat10

Check the IdP process log for any errors and review the warning log for deprecation issues and resolve:

 tail -f ./logs/idp-process.log

Check the Warn log:

 tail -f ./logs/idp-warn.log

Check the Catalina log (Tomcat):

 tail -f /opt/tomcat10/logs/catalina.out

When the IdP has started, you can exit the tail using Control-C.

Linux - Upgrading Shibboleth IdP from v4.3.3 to v5.1.2:

Back up your current system either by taking a snapshot of your virtual machine, or creating tar archives, or a combination of these.

You must ensure that your versions of Java and servlet container (Jetty or Tomcat) meet the system requirements for v5. This is more complex than usual because there are relatively few servlet containers right now that support both Java and Jakarta Servlet specifications, and v4 requires Java while v5 requires Jakarta. At a minimum, it’s a good idea to upgrade a v4 system to Java 17 first, if you have just moved to Java 17 install one of the JavaScript engine plugins (Nashorn or Rhino) be aware that previous versions contained a JavaScript engine, from Java 17 onwards you require a separate JavaScript engine (like Nashorn or Rhino).

You must be on Shibboleth IdP v4.3.3 before proceeding with the upgrade to v5.1.2.

Check the currently installed version

Change to the IdP installed directory and check the currently installed version, do not proceed if it's less that v4.3.3:

 cd /opt/shibboleth-idp
 ./bin/version.sh
 4.3.3

Check attributes

Make a note of the attributes that are currently being released, within v4.3.3, for comparison after upgrading:

 ./bin/aacli.sh --configDir=conf/ --principal=* --requester=*

Note down the output, for comparison later.

Check plugins

Make sure that all plugins are up to date, firstly, see what plugins you have installed:

 ./bin/plugin.sh -l

For further information about IdP Plugins, please refer to this page: https://shibboleth.atlassian.net/wiki/spaces/IDP5/pages/3199500688/PluginInstallation#Overview

You may have Nashorn v1.1.0 currently installed, this is not compatible with v5.1.2 and needs upgraded to Nashorn v2.0.0 to work.

Upgrade any plugins

Upgrade the Nashorn plugin to v2.0.0 if not already up-to-date:

 ./bin/plugin.sh -u net.shibboleth.idp.plugin.nashorn 

 INFO  - Including auto-located properties in ./../conf/authn/authn.properties
 INFO  - Including auto-located properties in ./../conf/saml-nameid.properties
 INFO  - Including auto-located properties in ./../conf/admin/admin.properties
 INFO  - Including auto-located properties in ./../conf/c14n/subject-c14n.properties
 INFO  - Including auto-located properties in ./../conf/ldap.properties
 INFO  - Including auto-located properties in ./../conf/services.properties
 INFO  - Downloading from HTTPResource [https://shibboleth.net/downloads/identity-provider/plugins/scripting/2.0.0/idp-plugin-nashorn-jdk-dist-2.0.0.tar.gz]  
 INFO  - Downloading from HTTPResource [https://shibboleth.net/downloads/identity-provider/plugins/scripting/2.0.0/idp-plugin-nashorn-jdk-dist-2.0.0.tar.gz.asc]
 INFO  - TrustStore does not contain signature 0xxxxxxxxxxx
 Accept this key:
 Signature:	0xxxxxxxxxxxxxxxxxx
 FingerPrint:	xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 [yN] y
 INFO  - Installing Plugin 'net.shibboleth.idp.plugin.nashorn' version 2.0.0
 INFO  - Rebuilding /opt/shibboleth-idp/war/idp.war, Version 5.1.2
 INFO  - Initial populate from /opt/shibboleth-idp/dist/webapp to /opt/shibboleth-idp/webpapp.tmp
 INFO  - Overlay from /opt/shibboleth-idp/dist/plugin-webapp to /opt/shibboleth-idp/webpapp.tmp
 INFO  - Overlay from /opt/shibboleth-idp/edit-webapp to /opt/shibboleth-idp/webpapp.tmp
 INFO  - Creating war file /opt/shibboleth-idp/war/idp.war

Now list the plugins to see if Nashorn v2.0.0 is installed correctly:

 ./bin/plugin.sh -l
 INFO  - Including auto-located properties in ./bin/../conf/authn/authn.properties
 INFO  - Including auto-located properties in ./bin/../conf/saml-nameid.properties
 INFO  - Including auto-located properties in ./bin/../conf/admin/admin.properties
 INFO  - Including auto-located properties in ./bin/../conf/c14n/subject-c14n.properties
 INFO  - Including auto-located properties in ./bin/../conf/ldap.properties
 INFO  - Including auto-located properties in ./bin/../conf/services.properties
 Plugin: net.shibboleth.idp.plugin.nashorn	Current Version: 2.0.0 

Download IdP install files

Change to the src directory and download IdP install files:

Note: Always check to see if there's an update available as the version below may not represent the actual latest version that's available:

https://shibboleth.net/downloads/identity-provider/latest/

 cd /usr/local/src
 curl -O https://shibboleth.net/downloads/identity-provider/latest5/shibboleth-identity-provider-5.1.2.tar.gz
 curl -O https://shibboleth.net/downloads/identity-provider/latest5/shibboleth-identity-provider-5.1.2.tar.gz.sha256

Check SHA256 message digest matches:

 sha256sum -c shibboleth-identity-provider-5.1.2.tar.gz.sha256

Decompress the files

Untar the files and change to the 5.1.2 directory path:

 tar xf shibboleth-identity-provider-5.1.2.tar.gz
 cd shibboleth-identity-provider-5.1.2

Run the installer for v5.1.2

It should detect your current source/install dir and display that on screen as below:

 ./bin/install.sh

 Source (Distribution) Directory (press <enter> to accept default): [/usr/local/src/shibboleth-identity-provider-5.1.2]

 Installation Directory: [/opt/shibboleth-idp]

 Update from version 4.3.3 to version 5.1.2
 Rebuilding /opt/shibboleth-idp/war/idp.war, Version 5.1.2
 Initial populate from /opt/shibboleth-idp/dist/webapp to /opt/shibboleth-idp/webpapp.tmp
 Overlay from /opt/shibboleth-idp/dist/plugin-webapp to /opt/shibboleth-idp/webpapp.tmp
 Overlay from /opt/shibboleth-idp/edit-webapp to /opt/shibboleth-idp/webpapp.tmp
 Creating war file /opt/shibboleth-idp/war/idp.war
 ...done

 BUILD SUCCESSFUL
 Total time: 13 seconds

Check the version has been upgraded:

 cd /opt/shibboleth-idp
 ./bin/version.sh
 5.1.2

Restart the servlet container

Since this is a major update, it's best to restart everything.

  systemctl restart tomcat10 

See what attributes are being released:

 ./bin/aacli.sh --configDir=conf/ --principal=* --requester=*

You should get something similar to below showing all the attributes you're releasing and they should match what the IdP was releasing when at v4.3.3:

 {
    "name": "schacHomeOrganization",
    "values": [
        "uni.ac.uk"
    ]
  },


  {
    "name": "eduPersonTargetedID",
    "values": [
        "R45453FGEGE5GEGGSFSDF"
    ]
  }

Check the IdP process log for any errors and review the warning log for deprecation issues and resolve:

 tail -f ./logs/idp-process.log

Check the Warn log:

 tail -f ./logs/idp-warn.log

Check the Catalina log (Tomcat):

 tail -f /opt/tomcat10/logs/catalina.out

When the IdP has started, you can exit the tail using Control-C.

Note: The v5.1.2 upgrade will have kept your existing configuration files like attribute-resolver.xml, ldp.properties etc. however you will notice new similarly named files in the conf directory with .idpnew.512 on the end of the file, this is the latest v5.1.2 configuration files, it's highly recommended you migrate your existing configuration settings to the same format as these files, ensuring you have a full backup of your existing configuration before doing so.

IdP Troubleshooting:

Some common errors:

• Has the IdP started correctly and releasing attributes?
  Connection refused when restarting attribute-resolver
  Normally caused by a permissions issue with the p12 certificate, check permissions on credentials directory and files.
• Web page showing correct at https://your.idp.url/idp/status with no errors?
  Servlet exception on web status page
  You may need to add the Jakarta Servlet 5 file, see Tomcat Troubleshooting.
• Any errors showing in the catalina.out (Tomcat) or idp-process.log? (Shibboleth)
• Are directory permissions correct?

IdP browser status error

If you check the IdP web page status https://your.idp.url/idp/status you may notice an error similar to below in the /opt/shibboleth-idp/logs/idp-process.log which points to a servlet exception:

 ERROR [jakarta.servlet.ServletException:144] -
 jakarta.servlet.ServletException: Handler dispatch failed: java.lang.NoClassDefFoundError: 
 jakarta/servlet/jsp/jstl/core/Config
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1104)
 Caused by: java.lang.NoClassDefFoundError: jakarta/servlet/jsp/jstl/core/Config
        at org.springframework.web.servlet.support.JstlUtils.exposeLocalizationContext(JstlUtils.java:103)
 Caused by: java.lang.ClassNotFoundException: jakarta.servlet.jsp.jstl.core.Config
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1353)

It will show in the browser similar to the below:

 Web Login Service - Error
 An error occurred: ServletException 

This is because Jakarta Servlet 5 is looking for the Jakarta Standard Tag Library API, to correct this, change to the correct directory and download the following file:

 cd /opt/shibboleth-idp/edit-webapp/WEB-INF/lib
 wget https://repo1.maven.org/maven2/jakarta/servlet/jsp/jstl/jakarta.servlet.jsp.jstl-api/3.0.0/jakarta.servlet.jsp.jstl-api-3.0.0.jar

Rebuild the idp.war file so it contains the new jar file required for the servlet to run correctly:

 cd /opt/shibboleth-idp/bin/
 ./build.sh

The idp.war file will then be rebuilt in /opt/shibboleth-idp/war containing the newly added jar file. If your current idp.war file is located in /opt/tomcat10/webapps you can copy this new idp.war file over your existing idp.war file.

Windows-specific

Windows platform support

OS Support

Shibboleth IdP v5 requires a 64-bit operating system with Java 17 64-bit installed (32-bit will not work). Supported windows operating systems (from a Java standpoint) are Windows Server 2016 (64-bit, any architecture) or newer. You should be running a current supported Windows environment, search for your environment from here to see life cycle of the product

https://support.microsoft.com/en-gb/lifecycle/search?alpha=Windows

Proposed steps for upgrade from Shibboleth IdP v4:

1. Prepare OS:
   1. Upgrade windows to 2016+
2. Fulfil prereqs:
   1. Upgrade java to 17
   2. Upgrade to latest Shib v4
3. Upgrade Shibboleth

Take snapshots (if VM) and/or ensure you have usable backups before carrying out any changes to your system. There will be several periods of service downtime during the upgrade process (beyond the usual ~20 seconds on unresponsiveness for a service restart)

Windows server

Ensure your OS is Windows server 2016 64-bit or newer. OS upgrades are outside the scope of this document, there are many guides as to how to do this on the internet

Java upgrade - Windows

Upgrading Java

Amazon Corretto 17 for Windows is the fully supported Java Distribution for shibboleth 5 Identity Provider. Installation instructions are available from Amazon Corretto.

In contrast, Shib v4 will run on Java 11 or 17 - the main difference in running Shib v4 on these 2 Java versions is the handling of Javascript within configs (for instance in attribute resolver) – Java 17 will require the Nashorn plugin to be installed - admin command window:

 "c:\Program Files (x86)\Shibboleth\IdP)bin\plugin.bat" -I net.shibboleth.idp.plugin.nashorn

On startup, the Shibboleth and Java version will be stated in the idp-process.log file: Shibboleth IdP Version 4.2.1 Java version='17.0.9' vendor='Amazon.com Inc.'

Jetty upgrade for Windows

As of April 2024 Jetty 11 has already reached "end of community support" although security updates will still be provided until it is deemed end of life, which we expect will be end of this year going on previous lifespans.

The Shibboleth project is in the process of releasing an updated Shibboleth IdP Jetty Configuration Baseline for Jetty 12 (jetty-base) and when this is available we will publish the upgrade instructions for Jetty.

Our advice to deployers of Jetty is that you should if not running latest Jetty 11, upgrade to latest Jetty 11, and utilise until such a time as the ability to support Jetty 12 improves.

In the meantime the Shibboleth project UK Federation can supply instructions for installing Version 5 Shibboleth with Jetty Version 11 but you will need to contact the UK federation service desk to discuss this requirement to see if this is the right approach.

Upgrade to latest Shibboleth v4 (v4.3.3)

Part of the process of upgrading to version 5 of the Shibboleth Identity Provider (IdP) should involve updating to the latest version 4 release.

The Shibboleth IdP Windows install/upgrade process is well documented on the

Check the currently installed version

Run an Administrator command prompt window (cmd.exe)

Do not proceed if it is less than v4.3.1:

 cd "C:\Program Files (x86)\Shibboleth\IdP\bin\"
 version.bat"

Download the files

Check if your deployment includes Jetty (does a Jetty directory exist under C:\Program Files (x86)\Shibboleth?)

Download the latest v4 msi

Run the installer for v4.3.3

It should detect your current source/install dir and display that on screen:

 shibboleth-identity-provider-4.3.3-x64.msi

It will install over the top of the existing installation. Make sure you tick the ‘install Jetty’ option when prompted.

After the install completes, restart the Shibboleth 4 IdP service using the standard Windows tools.

Check for deprecation warnings in idp-warn.log and address them. Of relevance to eduGAIN organisations is warnings regarding as shown below this is still valid in v5 configs so can be ignored.

 DEPRECATED xsi:type 'SAML2NameID', (file [C:\Program Files (x86)\Shibboleth\IdP\conf\attribute-resolver.xml]): 
 This will be removed in the next major version of this software; replacement is (none)

Upgrade to the latest Shibboleth 5

Unlike Shibboleth v4, this install is split into 2 parts – the Shibboleth container and the Jetty service. You need to install both of these.

Back up your current system either by taking a snapshot of your physical/virtual machine before attempting any upgrade.

Download IdP install files

Note: Always check to see if there's an update available as the version below may not represent the actual latest version available:

Download the msi installer from https://shibboleth.net/downloads/identity-provider/latest5/ and install over the top of the existing v4.3.3 installation (this will stop the existing shibboleth Windows service)

There are Windows upgrade instructions available on the Shibboleth site for reference: https://shibboleth.atlassian.net/wiki/spaces/IDP5/pages/3199500925/Upgrading

You must be on Shibboleth IdP v4.3.3 before proceeding with the upgrade to v5.1.2

Check the currently installed version

Run an Administrator command prompt window (cmd.exe)

Change to the IdP installed directory and check the currently installed version, do not proceed if it's less that v4.3.3:

Do not proceed if it's less than v4.3.3:

 cd "C:\Program Files (x86)\Shibboleth\IdP\bin\"
  version.bat"

Check attributes

Make a note of the attributes that are currently being released, within v4.3.3, for comparison after upgrading:

 aacli.bat --configDir=conf/ --principal=* --requester=*"

Note down the output, for comparison later.

Install files

Install v5.1.2 using the following file with Administrator privileges:

 shibboleth-idp-5.1.2-x64.msi

Next, install the latest Jetty base msi from https://shibboleth.net/downloads/identity-provider/jetty-windows/

There are instructions on the Shibboleth site for the Jetty Base install: https://shibboleth.atlassian.net/wiki/spaces/IDP5/pages/3255435265/Jetty-Base+Installation

Check plugins

At this point you will need to ensure that all plugins and modules are up to date.

Make sure that all plugins are up to date, firstly, see what plugins you have installed:

 cd "C:\Program Files (x86)\Shibboleth\IdP\bin\"

 plugin.bat -l

For further information about IdP Plugins, please refer to this page: https://shibboleth.atlassian.net/wiki/spaces/IDP5/pages/3199500688/PluginInstallation#Overview

You may have Nashorn v1.1.0 currently installed, this is not compatible with v5.1.2 and needs upgraded to Nashorn v2.0.0 to work.

Upgrade any plugins

Upgrade the Nashorn plugin to v2.0.0 if not already up-to-date:

 plugin.bat -u net.shibboleth.idp.plugin.nashorn

Now list the plugins to see if Nashorn v2.0.0 is installed correctly:

 plugin.bat -l

Now that the install has completed, restart the Shibboleth 4 IdP service using the standard Windows tools and check for deprecation messages in the log files.

Check the IdP process log for any errors and review the warning log for deprecation issues and resolve:

 notepad.exe idp-process.log

Check the Warn log:

 notepad.exe idp-warn.log

General

This section contains those steps of the upgrade process that are not platform-dependent: which is to say, those steps that relate to the IdP itself.

SAML Proxying Issue:

If your Shibboleth IdP leverages the SAML proxying feature to pull attributes from Azure AD then your Azure AD metadata may fail to load. If you have this issue then you will need to add the following line to your conf/idp.properties

# opensaml.config.xml.unmarshall.strictMode = false

IdP deployers should use MDQ rather than metadata aggregates

The UK federation publishes metadata via its Metadata Publication Service describing all participating entities. This metadata file (ukfederation-metadata.xml) provides the information required for entities to know how to communicate with each other, and establishes a trust fabric permitting entities to verify each other’s identities. The IdP relies on the regular downloading of this metadata aggregate which is currently (Septermber 2020) over 58MB and will continue to grow with new entities added to the federation. This download is then processed locally within Java, including signature verification, which overall requires a large amount of RAM and CPU resources.

We have a new publication which use a newer protocol called Metadata Query (MDQ) to retrieve metadata. Rather than downloading the whole aggregate XML file, the metadata is looked up on demand by the IdP by querying a UK federation MDQ publication service. Thus, the required CPU and memory footprint is massively reduced.

Further details on MDQ and specifically on configuring metadata providers in the IdP

Note: that deployers who continue to use the federation metadata aggregate will eventually hit either the limits of either CPU and Memory available on the host or limits within the configuration such as Java memory assignment, resulting in a service outage or needing to size-up their environment to cope, and incurring further costs.