Shibboleth Identity Provider (IdP) 3 Installation Guide

    This guide describes the installation of the Shibboleth Identity Provider (IdP) for deployments in the SWITCHaai federation. It is targeted at system administrators who are familiar with Unix shell commands and XML syntax – the IdP is mostly configured through XML files –, and optionally have some experience with running server-side Java Web applications.

    For the Shibboleth IdP version 3, SWITCH recommends relying on an enterprise-grade Linux distribution with long term support: specifically, either Ubuntu Server LTS or Red Hat Enterprise Linux / CentOS. For both distributions, the versions supported by our guide were released in 2014 and will receive security updates from the vendor through April 2019 or June 2024, respectively. The instructions for installing the IdP on these distributions are fairly similar, and depending on the option selected below, the sections of this guide will automatically be updated with the proper information. Our IdP Guide is not intended for Ubuntu Server 16.04 LTS as it contains OpenJDK 8 (and even 9) and Tomcat 8. Covering these new versions together with the current ones would make the guide more complex and harder to follow. Ubuntu Server 16.04 LTS is a perfectly valid platform for the IdP, but at the moment we provide no offical guide for it.

    For upgrades from version 2 of the Shibboleth IdP: (permalink)

    For Upgrade instructions for specific versions of the Shibboleth IdP, see

    Upgrading from a v2.x IdP

    For upgrades, SWITCH discourages from performing an in-place upgrade of a Shibboleth 2.x IdP deployment. The recommended procedure consists of setting up version 3.x on a completely new system (virtual or physical), and carrying over / adapting the configuration from 2.x where applicable. A switchover to v3 should only take place after having thoroughly tested the new system with the most commonly used service providers.

    Specific upgrade instructions can be found below in the boxes at the end of each relevant section. They should not be considered all-encompassing, but are meant to provide further guidance on things which have to be taken into account when preparing for a migration. The SWITCHaai team welcomes feedback from administrators of v2 IdPs with their experiences of the v3 upgrade, so that the instructions in this guide can hopefully be improved further and made more useful for other members of the SWITCH community.

    Based on the information supplied above, the sections below will configure the IdP with the following parameters:

    • the name of the home organization is example.org
    • the service name of the IdP is aai-login.example.org
    • the IdP has its entityID set to https://aai-login.example.org/idp/shibboleth
    • metadata is retrieved from and http://metadata.aai.switch.ch/entities/interfederation+sp

    System requirements

    For the Shibboleth IdP 3, we recommend a system with at least 4 GByte of memory. For an interfederation-enabled IdP consider to add more memory.
    Make sure that the following tools are also installed:

    sudo
    We recommend installing sudo for commands that require root privileges.
    curl
    To download software and configuration files our instructions rely on using curl.
    OpenSSL
    Package: openssl, the command-line tool will be used to deal with server certificates.
    unzip
    Used to unpack ZIP archives (and also useful for listing contents of .war/.jar files).

    Finally, please note that time synchronization via NTP is essential when operating an IdP. SAML messages include timestamps which are checked by the service providers, and clock offsets of more than 5 minutes are guaranteed to cause troubles (messages being rejected, and preventing a user from logging in successfully with his SWITCHaai account). Depending on your environment, you might have to install the ntpd package, in particular when running the IdP on a non-virtualized system.

    Recommended server software

    The Shibboleth IdP is a Java-based Web application and therefore requires a Servlet container. Our recommended setup consists of the following components:

    • Apache HTTP Server 2.4 for the Web frontend
    • Apache Tomcat 7 for the Java Servlet container
    • PostgreSQL 9 for the database which stores persistent IDs and user consent

    We first cover the steps for installing and configuring the above components, and then proceed to installing and configuring the Shibboleth IdP.

    Apache HTTP Server installation

    Before installing the Apache HTTP Server, get an SSL certificate for the service name of the IdP (aai-login.example.org) from a publicly trusted CA. Organizations from the SWITCH community will find further information on how to request such a server certificate under How to request a SWITCHpki QuoVadis server certificate. Store the private key under /etc/sslpki/tls/private/aai-login.example.org.key and the certificate under /etc/sslpki/tls/certs/aai-login.example.org.crt, and make sure that the certificate file includes both the server certificate and any intermediate CA certificate(s). If SELinux is enabled on your system, verify that the two files have the correct context: use restorecon -nv /etc/pki/tls/private/aai-login.example.org.key /etc/pki/tls/certs/aai-login.example.org.crt to check, and if there's output from this command, rerun without -n to fix the context(s).

    Then, install the Apache HTTP Server 2.4 with the following command:

    sudo apt-get install apache2
    sudo yum install httpd mod_ssl

    We recommend to enable the headers module to make the X-Frame-Options of the VirtualHost configuration work. This will disallow embedding your IdP's login page within an iframe. Also enable HSTS so that browsers will always use HTTPS when connecting to the IdP.

    Next, add a VirtualHost configuration for the IdP by creating a file named /etc/apache2/sites-available/etc/httpd/conf.d/aai-login.example.org.conf and populating it with the following contents:

    <VirtualHost *:443>
      ServerName aai-login.example.org
      ServerAdmin idp-admin@example.org
      CustomLog /var/log/apache2/aai-login.example.org.access.log combined
      ErrorLog /var/log/apache2/aai-login.example.org.error.log
    
      SSLEngine On
      SSLCipherSuite HIGH:MEDIUM:!aNULL:!kRSA:!MD5:!RC4
      SSLProtocol all -SSLv2 -SSLv3
      SSLCertificateKeyFile /etc/ssl/private/aai-login.example.org.key
      SSLCertificateFile /etc/ssl/certs/aai-login.example.org.crt
      SSLCertificateChainFile /etc/ssl/certs/aai-login.example.org.crt
    
      <IfModule headers_module>
      Header set X-Frame-Options DENY
      Header set Strict-Transport-Security "max-age=31536000 ; includeSubDomains"
      </IfModule>
    
      ProxyPass /idp ajp://localhost:8009/idp retry=5
      <Proxy ajp://localhost:8009>
        Require all granted
      </Proxy>
    </VirtualHost>
    <VirtualHost *:443>
      ServerName aai-login.example.org
      ServerAdmin idp-admin@example.org
      CustomLog /var/log/httpd/aai-login.example.org.access.log combined
      ErrorLog /var/log/httpd/aai-login.example.org.error.log
    
      SSLEngine On
      SSLCipherSuite HIGH:MEDIUM:!aNULL:!kRSA:!MD5:!RC4
      SSLProtocol all -SSLv2 -SSLv3
      SSLCertificateKeyFile /etc/pki/tls/private/aai-login.example.org.key
      SSLCertificateFile /etc/pki/tls/certs/aai-login.example.org.crt
      SSLCertificateChainFile /etc/pki/tls/certs/aai-login.example.org.crt
    
      <IfModule headers_module>
      Header set X-Frame-Options DENY
      Header set Strict-Transport-Security "max-age=31536000 ; includeSubDomains"
      </IfModule>
    
      ProxyPass /idp ajp://localhost:8009/idp retry=5
      <Proxy ajp://localhost:8009>
        Require all granted
      </Proxy>
    </VirtualHost>

    In addition to adding the above VirtualHost definition, we suggest setting ServerTokens Prod and ServerSignature off in /etc/apache2/conf-enabled/security.conf/etc/httpd/conf/httpd.conf to make the server not disclose/announce version number details etc. by default.

    Finally, enable the Apache HTTP server and check its configuration with the following commands:

    sudo a2enmod ssl proxy_ajp headers
    sudo a2ensite aai-login.example.org
    sudo apachectl configtest
    sudo systemctl enable httpd.service
    sudo apachectl configtest

    If the output of the last command states Syntax OK, then proceed with restarting the Apache HTTP server:

    sudo apachectl restart
    sudo systemctl restart httpd.service

    Upgrading from a v2.x IdP

    If you keep the service name of your v2 IdP – which is highly preferred unless you have really good reasons for changing the name –, you can use the publicly-trusted certificate from your existing IdP. For v2 deployments which were set up according to the previous SWITCH deployment guide, the certificate and private key can be found in the /etc/ssl/certs and /etc/ssl/private directories.

    Java Servlet container installation

    Please install in the order indicated below (otherwise Tomcat will pull Java 8 as dependency)

    Java runtime environment: OpenJDK 7

    Both Linux distributions supported in this guide provide OpenJDK 7 as their default Java runtime environment. To install, execute the following command:

    sudo apt-get install openjdk-7-jdk
    sudo yum install java-1.7.0-openjdk
    If you need to install Java 8 for some reason, you need to check your scripted attribute definitions and probably adapt them.

    Servlet container: Apache Tomcat 7

    The default supported version of the Tomcat Servlet container shipped with the recommended operating systems is version 7. While newer versions of Tomcat – as well as alternative Servlet containers such as Jetty – are available, they have the disadvantage of requiring manual security updates. For this reason, SWITCH recommends relying on the Servlet container which is provided in the form of a vendor-supported package.

    To install the Apache Tomcat Servlet container, execute:

    sudo apt-get install tomcat7
    sudo yum install tomcat

    As a next step, adapt the following configuration defaults in /etc/tomcat/server.xml:

    • Disable the HTTP/1.1 connector on port 8080 by commenting out the <Connector port="8080" ...>element
    • Add an AJP connector on port 8009, by inserting the following lineadapting the <Connector port="8009" ...> line:
      <Connector port="8009" address="127.0.0.1" protocol="AJP/1.3" />
    • Disable automatic application deployment by changing autoDeploy="true" to autoDeploy="false" on the <Host name="localhost" ...> element

    We recommend to allocate a large part of the memory for Tomcat and spare one 1024 MB for the OS and Apache. Configure JAVA_OPTS in /etc/default/tomcat7/etc/sysconfig/tomcat as follows:

    JAVA_OPTS="-server -Xmx1024m -Djava.security.egd=file:/dev/./urandom"

    -Xmx1024m - this is the maximum amount of heap memory that Tomcat may use.

    If your system is not short of “true” entropy (cf. cat /proc/sys/kernel/random/entropy_avail), you can leave out the -Djava.security.egd setting (for further background, see the HowTo FasterStartUp page in the Tomcat Wiki).

    If you need to install Tomcat 8 for some reason, please refer to the Shibboleth Wiki.

    SQL database: PostgreSQL 9

    A Shibboleth IdP in the SWITCHaai federation relies on database-backed storage for two categories of information: persistent identifiers and user consent data. For the IdP version 3, SWITCH recommends deploying PostgreSQL as the database backend; other relational databases may be used for this purpose as well, as long as they can be made available to the IdP through Hibernate ORM. If you prefer using a different RDBMS, then we assume that you – or your DBA – is able to adapt the steps in this section to your environment as needed.

    Installation

    For simplicity's sake, we recommend and show instructions to install the version of PostgreSQL provided by your Linux distribution, which may not be the most recent. If you prefer running the latest stable version, PostgreSQL developers provide packages repositories for all major distributions. Additionally, Red Hat Enterprise Linux subscribers can use Red Hat Software Collections to obtain recent PostgreSQL packages from Red Hat.

    Be aware: Recent postgresql-jdbc RPM from the disto requires Java 8 instead of Java 7

    Since January 2018, the postgresql-jdbc RPM from the distro requires Java 8 instead of Java 7. Therefore, any future IdP rebuild will fail. You will need to download and install the PostgreSQL JDBC driver for Java 7 manually.

    1. Go to https://jdbc.postgresql.org/download.html and pick the latest version with the .jre7 extension, currently postgresql-42.2.2.jre7.jar.
    2. Remove the existing JDBC driver from the IdP:
      rm /opt/shibboleth-idp/edit-webapp/WEB-INF/lib/postgresql*
      rm /opt/shibboleth-idp/webapp/WEB-INF/lib/postgresql* 
    3. Copy the new JAR into edit-webapp:
      cp postgresql-42.2.2.jre7.jar /opt/shibboleth-idp/edit-webapp/WEB-INF/lib
    4. Finally rebuild the idp.war file:
      sudo JAVACMD=/usr/bin/java /opt/shibboleth-idp/bin/build.sh -Didp.target.dir=/opt/shibboleth-idp

    To prepare PostgreSQL for use with the IdP, install the package as follows:

    sudo apt-get install postgresql libpostgresql-jdbc-java
    sudo yum install postgresql-server postgresql-jdbc
    sudo postgresql-setup initdb
    sudo systemctl enable postgresql.service
    sudo systemctl start postgresql.service

    Database creation and settings

    Then, create a shibboleth database account, the shibboleth database and the shibpid and storagerecords tables with the following command sequence:

    sudo -iu postgres psql <<EOF
    CREATE ROLE shibboleth WITH LOGIN;
    CREATE DATABASE shibboleth WITH OWNER shibboleth ENCODING 'UTF8' TEMPLATE template0;
    \c shibboleth
    SET ROLE shibboleth;
    CREATE TABLE shibpid (
        localEntity VARCHAR(1024) NOT NULL,
        peerEntity VARCHAR(1024) NOT NULL,
        principalName VARCHAR(255) NOT NULL,
        localId VARCHAR(255) NOT NULL,
        persistentId VARCHAR(36) NOT NULL,
        peerProvidedId VARCHAR(255) NULL,
        creationDate TIMESTAMP NOT NULL DEFAULT LOCALTIMESTAMP,
        deactivationDate TIMESTAMP NULL DEFAULT NULL,
        PRIMARY KEY (localEntity, peerEntity, persistentId)
    );
    CREATE INDEX shibpid_getbysourcevalue_index ON shibpid(localEntity, peerEntity, localId, deactivationDate);
    CREATE TABLE storagerecords (
        context VARCHAR(255) NOT NULL,
        id VARCHAR(255) NOT NULL,
        expires BIGINT DEFAULT NULL,
        value TEXT NOT NULL,
        version BIGINT NOT NULL,
        PRIMARY KEY (context, id)
    );
    CREATE INDEX storagerecords_expires_index ON storagerecords(expires);
    EOF

    On Red Hat Enterprise Linux / CentOS, you need to modify the default authentication settings: in /var/lib/pgsql/data/pg_hba.conf, change the METHOD for local connections from ident to md5, so that the lines read as follows:

    # IPv4 local connections:
    host    all             all             127.0.0.1/32            md5
    # IPv6 local connections:
    host    all             all             ::1/128                 md5

    Execute

    sudo systemctl reload postgresql.service

    afterwards to make PostgreSQL re-read the changes to pg_hba.conf.

    Enable autovacuum and autoanalyze to make sure that data gets cleaned out and indexes are updated correctly. The default thresholds are set rather high (at least 20%/10% of a table needs to change before triggering autovacuum/autoanalyze) so we recommend lowering them to 0.2% and 0.1%, respectively. Edit PostgreSQL's configuration in /etc/postgresql/9.3/main/postgresql.conf /var/lib/pgsql/data/postgresql.conf so that the two lines are uncommented and the numbers changed like below:

    autovacuum_vacuum_scale_factor = 0.002  # fraction of table size before vacuum
    autovacuum_analyze_scale_factor = 0.001 # fraction of table size before analyze

    Reload the configuration to apply changes:

    sudo service postgresql reload
    sudo systemctl reload postgresql.service

    Don't use large objects!

    Out-of-the box, the IdP version 3 will use “large objects” with the storagerecords.value column without enabling PostgreSQL's large object management (see issue OSJ-135 for further details). This results in orphaned large objects and guaranteed data loss if the vacuumlo command is ever run. In order to prevent such an accident, we urge administrators to apply the workaround described later before starting their IdP for the first time.

    Database backup

    Finally, it is highly recommended to create backups (dumps) of the database at least on a daily basis, so that the database can be restored from a known good state in case of an emergency. A lightweight solution can be implemented by an /etc/cron.d/postgres-backup file with a pg_dumpall command which dumps all databases:

    40 4 * * * postgres pg_dumpall | gzip > /var/backups/postgresql/dumpall-`date +\%a`.sql.gz
    20 * * * * postgres pg_dumpall | gzip > /var/backups/postgresql/dumpall-latest.sql.gz
    40 4 * * * postgres pg_dumpall | gzip > /var/lib/pgsql/backups/dumpall-`date +\%a`.sql.gz
    20 * * * * postgres pg_dumpall | gzip > /var/lib/pgsql/backups/dumpall-latest.sql.gz

    The first cron entry creates a daily dump of all PostgreSQL databases at 4:40 in the morning and writes them in compressed format to the backup directory (with file names such as dumpall-Mon.sql.gz, dumpall-Tue.sql.gz etc). The second cron entry creates additionally an hourly backup. On Ubuntu, you also need to create the directory with the appropriate ownership and permissions first:

    sudo mkdir /var/backups/postgresql
    sudo chown postgres:postgres /var/backups/postgresql
    sudo chmod 700 /var/backups/postgresql

    Upgrading from a v2.x IdP

    An important step when upgrading from an existing IdP is to make sure that the persistent IDs are retained. To populate the newly created PostgreSQL database with the existing shibpid records, perform these steps (under the assumption that your v2 IdP has been set up with a MySQL database backend, as suggested in previous versions of this guide):

    1. On the current IdP, dump the persistent IDs stored from the MySQL database to a file:
      sudo mysqldump --compatible postgres --compact --no-create-info --result-file shibpid.sql shibboleth shibpid
    2. Copy the shibpid.sql file to your new IdP
    3. Import the persistent IDs into the PostgreSQL database:
      sudo -iu postgres psql shibboleth --file /path/to/shibpid.sql

    Be sure to only execute the import with psql once, otherwise the shibpid table will end up with duplicate records. To start the import from scratch, the table can be emptied first with sudo -u postgres psql shibboleth -c "truncate shibpid".

    Shibboleth IdP

    Deployment of the Shibboleth IdP is a two-part procedure: first, the distribution is unpacked and its installer is run, and second, the default configuration needs to be customized for an IdP in the SWITCHaai (or AAI Test) federation. The next subsection is about the first part, while the remaining ones cover various aspects of the configuration in more detail. Generally speaking, this section is focusing on the specifics for an IdP in the SWITCHaai federation, it should not be considered a systematic description of the vast number of configuration options provided by the Shibboleth IdP version 3.

    Readers interested in a comprehensive documentation of the IdP configuration settings are referred to the Configuration page in the Shibboleth Wiki, and might find the ConfigurationFileSummary page useful to get an overview of the fairly complex configuration file structure. Making oneself familiar with some SpringConfiguration basics is also helpful when trying to get a better understanding of the syntax used in the IdP configuration files.

    Upgrading from a v2.x IdP

    Depending on the customization of your v2.x IdP, migrating its configuration to a v3-based setup can be a fairly time-consuming task. In the upgrade instructions in this guide, we put the focus on the steps required for a typical v2 IdP which was set up according to the previous version of this guide. If you have further customized your IdP configuration (e.g. by supporting additional login methods or similar), then we strongly recommend diffing your configuration against the default versions of the files as shipped in the respective distribution from https://shibboleth.net/downloads/identity-provider/, to get a better understanding of what configuration changes might also have to be taken into account.

    IdP installation

    Download the distribution archive in the tar.gz format from the Shibboleth site. We recommend creating a /usr/local/dist directory for storing the downloads:

    [ -d /usr/local/dist ] || sudo mkdir /usr/local/dist
    cd /usr/local/dist
    sudo curl -O https://shibboleth.net/downloads/identity-provider/3.x.y/shibboleth-identity-provider-3.x.y.tar.gz
    sudo tar -zxf shibboleth-identity-provider-3.x.y.tar.gz

    Running the install.sh script included in the IdP distribution without any parameters would create a default configuration which is not very suitable for an IdP in the SWITCHaai federation. We therefore recommend using a simple idp-install.sh wrapper script which ensures that the basic configuration parameters are set to SWITCHaai compatible defaults. To run the custom installer, execute

    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/idp-install.sh
    sudo bash idp-install.sh shibboleth-identity-provider-3.x.y

    The custom installer script will first prompt for two parameters and ask for confirmation, and then install the IdP to /opt/shibboleth-idp:

    Please enter the name of your home organization [example.org]: example.org
    Please specify the service name for your IdP [aai-login.example.org]: aai-login.example.org
    
    Proceed with the installation using the following parameters?
    
    Service name:             aai-login.example.org
    Home organization name:   example.org
    Installation directory:   /opt/shibboleth-idp
    
    Enter 'yes' to continue: yes
    
    Warning: /opt/shibboleth-idp/bin does not exist.
    Warning: /opt/shibboleth-idp/dist does not exist.
    Warning: /opt/shibboleth-idp/doc does not exist.
    Warning: /opt/shibboleth-idp/system does not exist.
    Warning: /opt/shibboleth-idp/webapp does not exist.
    Creating cookie encryption key files...
    ...done
    Rebuilding /opt/shibboleth-idp/war/idp.war ...
    ...done
    
    BUILD SUCCESSFUL
    Total time: 4 seconds
    
    Creating self-signed certificate...
    ...done

    The idp-install.sh script will also adjust the ownership of some files and directories, which you should verify with the following command:

    ls -ld /opt/shibboleth-idp/{conf/credentials.properties,credentials/{*.key,sealer.*},logs,metadata}

    The output should look similar to:

    -rw------- 1 tomcat root 1675 Mar 25 13:05 /opt/shibboleth-idp/conf/credentials.properties
    -rw------- 1 tomcat root 1675 Mar 25 13:05 /opt/shibboleth-idp/credentials/idp.key
    -rw-r--r-- 1 tomcat root  500 Mar 25 13:05 /opt/shibboleth-idp/credentials/sealer.jks
    -rw-r--r-- 1 tomcat root   47 Mar 25 13:05 /opt/shibboleth-idp/credentials/sealer.kver
    drwxr-xr-x 2 tomcat root 4096 Mar 25 13:05 /opt/shibboleth-idp/logs
    drwxr-xr-x 2 tomcat root 4096 Mar 25 13:05 /opt/shibboleth-idp/metadata

    Upgrading from a v2.x IdP

    SWITCH recommends leaving the entity ID of the v2 IdP unchanged, in particular when taking into account that modifying this ID would break continuity of existing persistent IDs. In this case, it's also easiest to retain the certificate and private key from the v2 IdP, as this makes a key rollover unnecessary. Copy /opt/shibboleth-idp/credentials/idp.crt and /opt/shibboleth-idp/credentials/idp.key from the existing IdP and make sure that they have the proper ownership and permissions (see above for the idp.key).

    General IdP settings: services.xml and global.xml

    In /opt/shibboleth-idp/conf/services.xml there are two settings that need adjustment:

    a), the shibboleth.MetadataResolverResources list needs to be substituted by the following definition (to enable metadata selection with the idp.metadata property in /opt/shibboleth-idp/conf/idp.properties):

        <util:list id="shibboleth.MetadataResolverResources">
            <value>#{ '%{idp.metadata}' matches '.*\bswitchaai\b.*' ?
                      '%{idp.home}/conf/metadata-provider-switchaai.xml' : '' }</value>
            <value>#{ '%{idp.metadata}' matches '.*\baaitest\b.*' ?
                      '%{idp.home}/conf/metadata-provider-aaitest.xml' : '' }</value>
            <value>#{ '%{idp.metadata}' matches '.*\binterfederation\b.*' ?
                      '%{idp.home}/conf/metadata-provider-interfederation.xml' : '' }</value>
            <value>%{idp.home}/system/conf/metadata-providers-system.xml</value>
        </util:list>

    and b), the shibboleth.AttributeFilterResources list in the same file needs to be replaced with the following lines:

        <util:list id="shibboleth.AttributeFilterResources">
            <ref bean="FileBackedSWITCHaaiAttributeFilter"/>
        </util:list>
    
        <bean id="FileBackedSWITCHaaiAttributeFilter"
              class="net.shibboleth.ext.spring.resource.FileBackedHTTPResource"
              c:client-ref="#{ '%{idp.http.proxyhost:null}' != 'null' ?
                               'shibboleth.ProxyHttpClient' : 'shibboleth.HttpClient' }"
              c:url="https://rr.aai.switch.ch/switchaai/example.org/attribute-filter_v4.xml"
              c:backingFile="%{idp.home}/conf/attribute-filter.switchaai.xml">
        </bean>

    The attribute-filter.switchaai.xml file will be dynamically updated by the IdP, so it needs to be writable by the tomcat user:

    sudo touch /opt/shibboleth-idp/conf/attribute-filter.switchaai.xml
    sudo chown tomcat /opt/shibboleth-idp/conf/attribute-filter.switchaai.xml

    Finally, there are a couple of configuration elements related to the database and HTTP client configuration which need to be added to global.xml. Instead of listing them verbatim here, the complete file can be downloaded as follows:

    cd /opt/shibboleth-idp/conf
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/global.xml

    Federation metadata configuration

    Download the configuration file(s) with the respective <MetadataProvider> element:

    cd /opt/shibboleth-idp/conf
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/metadata-provider-switchaai.xml

    For interfederation also download the interfederation <MetadataProvider> element:

    cd /opt/shibboleth-idp/conf
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/metadata-provider-interfederation.xml

    Then, download the trust anchor, the “root certificate”, which is needed for verifying the signature of SWITCHaai metadata (see the SWITCHaai Root CA Repository for background information):

    cd /opt/shibboleth-idp/credentials
    sudo curl -O http://ca.aai.switch.ch/SWITCHaaiRootCA.crt.pem

    And verify that the fingerprint shown from the following OpenSSL command output matches exactly:

    openssl x509 -in /opt/shibboleth-idp/credentials/SWITCHaaiRootCA.crt.pem -noout -fingerprint -sha256
    SHA256 Fingerprint=37:DC:E4:D7:1C:24:42:32:6A:0F:85:B6:12:00:22:C7:54:AA:FF:B2:8C:BF:CF:69:EB:F3:F7:31:90:3C:09:5A

    Then, insert in /opt/shibboleth-idp/conf/idp.properties the following line with the idp.metadata property to the pick the metadata files to load:

    idp.metadata = switchaai, interfederation

    Proxy settings for outgoing HTTP connections: for normal operations, the IdP needs to retrieve settings from both the metadata download site, http://metadata.aai.switch.ch, and the AAI Resource Registry, https://rr.aai.switch.ch. If your network policy does not allow outgoing connections on ports 80 and 443 by default, then it is recommended to rely on an HTTP proxy for this purpose (the IP addresses for these DNS names are subject to change without notice, so it is strongly discouraged to add IP-based rules to the firewall configuration). To enable the use of an HTTP proxy server, add the following two parameters to the /opt/shibboleth-idp/conf/idp.properties file:

    idp.http.proxyhost = proxy.example.org
    idp.http.proxyport = 8080

    These settings are taken into account for both metadata downloads (HTTP) and attribute filter downloads (HTTPS). Omitting these two parameters from idp.properties – the default setting – will make the IdP open direct connections for HTTP(S) requests.

    Relying party configuration

    Download an adapted relying-party.xml configuration file which includes overrides for the special-purpose https://www.switch.ch/aai/SAML1/Attribute-Push relying party group (see SAML 1 Attribute Push Relying Party for further information):

    cd /opt/shibboleth-idp/conf
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/relying-party.xml
    If necessary, configure in the relying-party.xml the “trusted” SPs for which the user consent dialog should be suppressed. In general, SWITCH recommends using the new IdP v3 default settings, i.e. enabling user consent for all SPs.

    Upgrading from a v2.x IdP

    To determine if your v2 IdP includes custom modifications which might have to be recreated in the new configuration, we recommend comparing your current v2 configuration file against the default version SWITCH provided in the v2.4 deployment guide. To do so, execute on your existing v2 IdP:

    curl -o rp-v2-default.xml https://www.switch.ch/aai/docs/shibboleth/SWITCH/2.4/idp/deployment/relying-party.xml
    diff -bu rp-v2-default.xml /opt/shibboleth-idp/conf/relying-party.xml

    Only changes to the Relying Party Configurations part are relevant (metadata providers are configured separately in v3, see above), and only if they relate to something other than the provider value. If you see additional differences in the relying-party.xml file of your existing IdP compared to rp-v2-default.xml, then have a look at Upgrade Tasks for relying-party.xml in the Shibboleth Wiki for further information.

    The relying-party.xml file is also the place where user consent for attribute release is configured (formerly known as uApprove, now included in the standard release of the Shibboleth IdP). The attribute release page is enabled for all SPs by default, through the postAuthenticationFlows setting for the default relying party. To disable user consent for specific SPs, follow the instructions in the relying-party.xml file itself, by enabling and configuring the shibboleth.NoUserConsentRelyingParty bean as needed. Generic user consent settings are documented on the ConsentConfiguration page in the Shibboleth Wiki and are specified through the idp.consent.* parameters in the idp.properties file, or via configuration in /opt/shibboleth-idp/conf/intercept/consent-intercept-config.xml. For most of the uApprove settings previously specified in /opt/shibboleth-idp/conf/uApprove.properties, equivalent options for the IdP v3 exist.

    User authentication

    The user authentication component in the Shibboleth IdP has undergone major rework for version 3. While JAAS (Java Authentication and Authorization Service) continues to be available for a flexible configuration of authentication methods, the IdP now ships with a native LDAP-based authentication component, which is configured in /opt/shibboleth-idp/conf/ldap.properties.

    We assume that a typical IdP deployment in the SWITCHaai federation relies on authentication against an LDAP directory which is accessed through an LDAPS connection, with the LDAP server requiring a service account for binding to the directory and being configured with a server certificate chaining to a root that is configured with the property 'idp.authn.LDAP.trustCertificates'.

    For this explicit certificate trust configuration, get the Root CA certificate of the issuing CA of your LDAP server's SSL certificate and store it into a file.
    If you have obtained the LDAP server's certificate via SWITCHpki, you can download the QuoVadis Root CA 2 certificate from https://www.switch.ch/pki/manage/download/qvrca2.crt.pem.

    For this case, the settings in /opt/shibboleth-idp/conf/ldap.properties need to be adjusted as follows:

    idp.authn.LDAP.authenticator                    = bindSearchAuthenticator
    [...]
    idp.authn.LDAP.ldapURL                          = ldaps://ldap.example.org
    idp.authn.LDAP.useStartTLS                      = false
    idp.authn.LDAP.useSSL                           = true
    idp.authn.LDAP.sslConfig                        = certificateTrust
    idp.authn.LDAP.trustCertificates                = /opt/shibboleth-idp/credentials/qvrca2.crt.pem
    idp.authn.LDAP.baseDN                           = ou=people,dc=example,dc=org
    idp.authn.LDAP.bindDN                           = cn=idp,dc=example,dc=org
    [...]
    idp.attribute.resolver.LDAP.trustCertificates   = %{idp.authn.LDAP.trustCertificates:undefined}
    

    The LDAP bind password should be added to /opt/shibboleth-idp/conf/credentials.properties, by inserting the following line (and the respective line being removed from ldap.properties at the same time, for the sake of consistency):

    idp.authn.LDAP.bindDNCredential = secret

    For additional LDAP authentication options supported by version 3 of the Shibboleth IdP, see the LDAPAuthnConfiguration page in the Shibboleth Wiki. The full list of available login flows is covered on the AuthenticationConfiguration page in the Shibboleth Wiki.

    The default settings shipping with IdP version 3 do not log the success or failure of an LDAP authentication. If you're interested in seeing this information recorded in the idp-process.log file, (Authentication (succeeded|failed) for dn: ... lines), add the following element to /opt/shibboleth-idp/conf/logback.xml (after the line which sets org.ldaptive to WARN, for reasons of consistency):

        <logger name="org.ldaptive.auth.Authenticator" level="INFO" />

    Upgrading from a v2.x IdP

    The relevant settings for the ldap.properties file can be found in the /opt/shibboleth-idp/conf/login.config file of an existing v2 IdP. For setups which go beyond authenticating against a single LDAP directory, see PasswordAuthnConfiguration in the Shibboleth Wiki. In particular, note that JAAS continues to be supported with v3, so it should also be possible to carry over a configuration from a v2 IdP. To switch to JAAS, adjust the setting at the beginning of the /opt/shibboleth-idp/conf/authn/password-authn-config.xml file; by default, the JAAS configuration is now loaded from /opt/shibboleth-idp/conf/authn/jaas.config.

    Attribute resolution configuration

    The configuration of the attribute resolution component is the most complex, but also the most important part of the IdP configuration – the primary task of an IdP is to make statements about its users and their attributes. There isn't a one-size-fits-all approach to configuring attribute resolution on an IdP in the SWITCHaai federation, as it very much depends on what attributes are available within the organization, and what sources they are retrieved from.

    Make sure that the attributes configured in this section are aligned with the supported attributes you configured/will configure in the Resource Registry.

    While the configuration files provided in this section can be used as templates for setting up attribute resolution, they will need further customization in almost all cases. The Shibboleth IdP is very flexible in terms of configuring retrieval of attributes from other sources; the reference documentation can be found in the Shibboleth Wiki under AttributeResolverConfiguration and the respective subpages, in particular the description of the various AttributeDefinitionConfiguration options.

    To get started, configure the shibboleth.AttributeResolverResources list in /opt/shibboleth-idp/conf/services.xml as follows:

        <util:list id="shibboleth.AttributeResolverResources">
            <value>%{idp.home}/conf/attribute-resolver-switchaai-core.xml</value>
            <value>%{idp.home}/conf/attribute-resolver-interfederation-core.xml</value>
            <value>%{idp.home}/conf/attribute-resolver-connectors.xml</value>
            <value>%{idp.home}/conf/attribute-resolver-other.xml</value>
        </util:list>

    and download the files with:

    cd /opt/shibboleth-idp/conf
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/attribute-resolver-switchaai-core.xml
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/attribute-resolver-interfederation-core.xml
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/attribute-resolver-connectors.xml
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/attribute-resolver-other.xml

    Enabling attribute-resolver-switchaai-core.xml and attribute-resolver-connectors.xml will add support for the SWITCHaai Core attributes as listed on the AAI attributes page: Affiliation, E-mail, Given name, Home organization, Home organization type, Scoped affiliation, Surname, Targeted ID/Persistent ID, and Unique ID. The two files assume that the user-specific attributes are retrieved from the LDAP directory (as defined via ldap.properties), and two static values are configured: at least the proper value for swissEduPersonHomeOrganizationType must be added in attribute-resolver-connectors.xml, see the swissEduPersonHomeOrganizationType page for the list of possible values. To extend your LDAP server configuration fetch the LDIF schema file

    The attribute-resolver-interfederation.xml file includes definitions for the five Interfederation core attributes: Common name, Display name, Principal name, SCHAC home organisation, and SCHAC home organisation type. The default definitions in this file are based on existing SWITCHaai core attributes (Common Name and Display name e.g. are assembled from Given name and surname), but you should check each AttributeDefinition block in any case – in particular if “genuine” values are available from your LDAP directory, you should switch to retrieving the values from the directory.

    If your organization supports the common-lib-terms entitlement, you find its definition inside the attribute-resolver-other.xml. The attribute-resolver-other.xml file includes definitions for (most of) the attributes listed under Other attributes on the AAI attributes page. All attributes (except common-lib-terms entitlement) are commented out by default and should only be enabled after having carefully checked each AttributeDefinition (it assumes that most of these values are available from the LDAP directory, and all definitions which do not apply to your organization should be removed).

    Upgrading from a v2.x IdP

    In theory, it should be possible to reuse an attribute-resolver.xml configuration file from a v2 IdP as is, but SWITCH strongly encourages administrators to carefully check / clean up existing v2 configuration files on the occasion of upgrading to the IdP version 3. As a first step, it is helpful to know what modifications your current v2 configuration includes compared to the default template we provided in the previous version of this guide. On your existing v2 IdP, execute

    curl -o ar-v2-default.xml https://www.switch.ch/aai/docs/shibboleth/SWITCH/2.4/idp/deployment/attribute-resolver.xml
    diff -bu ar-v2-default.xml /opt/shibboleth-idp/conf/attribute-resolver.xml

    to examine them. Then, migrate your customizations to the new versions of attribute-resolver-switchaai-core.xml, attribute-resolver-connectors.xml etc. For script-based attribute definitions, check in particular the ScriptedAttributeDefinition page and its V2 Compatibility section in the Shibboleth Wiki and where feasible, consider replacing a scripted attribute with one of the “simpler” types, e.g. Mapped or Template).

    For local or bilaterally agreed attributes, we suggest creating separate configuration files (attribute-resolver-local.xml or similar), which can be loaded by adding them to the shibboleth.AttributeResolverResources list shown above.

    Consent intercept configuration

    Download an adapted consent-intercept-config.xml configuration file that contains a defined ordering for the attribute release consent dialog as well as an extended blacklist that covers also the usually cryptic unique identifiers (see 'Shibboleth IdPv3 User Consent Configuration Guide' for further information):

    cd /opt/shibboleth-idp/conf/intercept
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/consent-intercept-config.xml

    Enabling support for generating persistent IDs

    To enable support for generating persistent IDs through the new method recommended for version 3 of the Shibboleth IdP (see also the V2 Compatibility section on the respective page in the Shibboleth Wiki), the following configuration changes are required:

    1. in /opt/shibboleth-idp/conf/saml-nameid.properties, enable/configure the following parameters:
      idp.persistentId.generator = shibboleth.StoredPersistentIdGenerator
      idp.persistentId.dataSource = shibboleth.PostgreSQLDataSource
      idp.persistentId.sourceAttribute = swissEduPersonUniqueID
    2. in /opt/shibboleth-idp/conf/saml-nameid.xml, remove the comments for the shibboleth.SAML2PersistentGenerator bean reference, so that it looks as follows:
          <util:list id="shibboleth.SAML2NameIDGenerators">
      
              <ref bean="shibboleth.SAML2TransientGenerator" />
      
              <ref bean="shibboleth.SAML2PersistentGenerator" />
      
              [...]
    3. in /opt/shibboleth-idp/conf/c14n/subject-c14n.xml, remove the comments for the c14n/SAML2Persistent bean reference, so that it reads:
              <!-- Handle a SAML 2 persistent ID, provided a stored strategy is in use. -->
              <ref bean="c14n/SAML2Persistent" />
    4. in /opt/shibboleth-idp/conf/credentials.properties, add a line which specifies the salt to use for generating the persistent ID. A suitable value can be generated with openssl rand -base64 36.
      idp.persistentId.salt = ...random value...

    Upgrading from a v2.x IdP

    For the idp.persistentId.salt setting, use the value defined for the myStoredId data connector in the /opt/shibboleth-idp/conf/attribute-resolver.xml file of your IdP v2 configuration. The transfer of the persistent IDs from the database of your current IdP was already addressed in the section on the PostgreSQL database. Before putting the new v3 deployment into production, do not forget to repeat the import with the latest shibpid records from the v2 IdP (use sudo -u postgres psql shibboleth -c "truncate shibpid" to empty the table first).

    Configuring periodic encryption key rotation

    When using the default setting for the idp.session.StorageService in the idp.properties file, the Shibboleth IdP stores session information on the client side, in a cookie with encrypted data. The (symmetric AES) key to encrypt the client-side information should be rotated regularly, so it's best to set up a cron for this purpose. We recommend creating a shell script with the following contents and store it under /opt/shibboleth-idp/credentials/rotate-sealer.sh:

    #!/bin/sh
    
    IDP_HOME=/opt/shibboleth-idp
    export JAVA_HOME=/usr/lib/jvm/jre
    
    $IDP_HOME/bin/seckeygen.sh \
        --storefile $IDP_HOME/credentials/sealer.jks \
        --storepass "$(grep '^idp.sealer.password' $IDP_HOME/conf/credentials.properties | cut -d = -f 2 | awk '{print $1}')" \
        --versionfile $IDP_HOME/credentials/sealer.kver \
        --alias secret

    Set the execute permissions on the file with chmod +x and configure a corresponding entry under /etc/cron.d/idp-rotate-sealer:

    50 4 * * 0 tomcat /opt/shibboleth-idp/credentials/rotate-sealer.sh

    This will rotate the key each Monday morning at 4:50 - depending on your local needs, you might want to adapt the time and/or the interval for this cron entry.

    With the default logging settings, the IdP will log a message of type Default key version has not changed to the idp-process.log file every 15 minutes, printing the alias of the currently used key (such as secret65, with the number being increased whenever the key is rotated). If you consider this sort of logging too noisy, then these messages can be suppressed by adding

    <logger name="net.shibboleth.utilities.java.support.security.BasicKeystoreKeyStrategy" level="WARN" />

    to the /opt/shibboleth-idp/conf/logback.xml file.

    Limit Cookies to Secure Connections

    The default setting for idp.cookie.secure is false.
    Add the following line to the /opt/shibboleth-idp/conf/idp.properties file to prevent browsers from sending IdP cookies via http instead of https:

    idp.cookie.secure = true

    Enable HTML Local Session Storage

    The default setting for idp.storage.htmlLocalStorage is false. That SSO also works properly with new browser versions that default to SameSite=Lax you need to activate HTML Local Storage for IdP Session information. Read all the details about the Same Site issue in the Shibboleth Wiki.
    Add the following line to the /opt/shibboleth-idp/conf/idp.properties file:

    idp.storage.htmlLocalStorage = true

    In addition, consider to tailor the idp.footer messages property (or set it to the empty string) for all languages your IdP supports. This property gets used by the two views distributed in the /opt/shibboleth-idp/dist/views/client-storage directory. In case you want to modify the views, copy them first into /opt/shibboleth-idp/views/client-storage and modify them there.

    Messages Translation (supporting multiple languages)

    Translations of the messages properties files in different languages are collected in the Shibboleth Wiki.

    Download the messages_XX.properties files for the languages relevant for your user base and install the downloaded files into the directory /opt/shibboleth-idp/messages.

    By default, the IdP evaluates the Accept-Language header as provided by the browser to identify the appropriate locale (language) to use.

    Optional configuration: If you want to store local changes to messages.properties and messages_[de_CH|fr|it].properties files into separate files, e.g. local.properties and local_[de_CH|fr|it].properties in the directory /opt/shibboleth-idp/messages, you can add such local files as separate bundle with the following configuration in /opt/shibboleth-idp/conf/services.xml:

    <util:list id="shibboleth.MessageSourceResources">
        <value>%{idp.home}/messages/local</value>
        <value>%{idp.home}/messages/messages</value>
    </util:list>

    Login form customization

    In version 3 of the Shibboleth IdP, the default mechanism for rendering the login and error pages has been changed from JSP to Velocity templates, with the benefit that modifications to the login page do no longer require a restart of the Servlet container (Tomcat), but become active immediately after the editing the respective .vm file in the /opt/shibboleth-idp/views directory.

    The default templates included in the IdP are provided as samples, but it is assumed that they are customized for local use. You are welcome to use your organization's own corporate design for the login form; alternatively, you may use the templates which reflect the recommendations from the AAI Design Guidelines page. To install these custom SWITCHaai Velocity templates and CSS files, proceed as follows:

    cd /usr/local/dist
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/idpv3-switchaai-login-templates.zip
    cd /opt/shibboleth-idp
    sudo unzip -o /usr/local/dist/idpv3-switchaai-login-templates.zip

    The ZIP archive includes the following files which have been customized for SWITCHaai use (and may be further adapted):

    • three Velocity template files (replacing the default files in the /opt/shibboleth-idp/views directory): login.vm, login-error.vm and error.vm. Changes to these files do not require a Servlet container restart (with the exception of strings which are configured via entries in .properties files and rendered through #springMessageText).
    • two CSS files in the /opt/shibboleth-idp/edit-webapp/css subdirectory: main.css for use by most of the Velocity templates in the views subdirectory, and consent.css for the attribute release consent page
    • a small SWITCHaai logo (/opt/shibboleth-idp/edit-webapp/images/switchaai-logo.png) for use on the login and error pages
    • a replacement for the index.jsp file which is used when someone is trying to access the IdP's “root page” (https://aai-login.example.org/idp/, does not happen under normal circumstances, and the page will only display No services are available at this location in reply to such requests).

    After unpacking the ZIP archive, make sure that at least the following strings are set to correct values:

    In /opt/shibboleth-idp/system/messages/messages.properties and all the /opt/shibboleth-idp/messages/messages_XX.properties files for other languages you installed
    • configure three logo related parameters:
      Insert the new parameter idp.logo.target.url since it is not included in the default configuration.
      idp.logo = /images/example.org.png
      idp.logo.alt-text = Friendly name of your organization
      idp.logo.target.url = http://www.example.org
      You also need to place a suitable PNG image with your organization's logo under the location referenced by idp.logo, i.e. /opt/shibboleth-idp/edit-webapp/images/example.org.png, with a maximum suggested size of 200 by 100 pixel.
    • configure three target link parameters:
      Add three lines which define the target for the Forgot your password?, Change your password and Need Help? links:
      idp.login.forgotPassword.url = https://support.example.org/password-assistance
      idp.login.needHelp.url = https://support.example.org/help
      idp.login.changePassword.url = https://support.example.org/password-assistance
      

    Note: If the above parameters are the same for all languages, you can keep them in messages.properties and remove them from all other files such that they have to be defined only in the default messages file.

    Provided you installed support for multiple languages as described in the chapter Messages Translations above, you might want to integrate an explicit choice of language into the login form. Consult the Shibboleth Wiki for detailed instructions: Switching locale on the login page.

    These steps should be sufficient as a starting point, though further modifications to the above mentioned files might be necessary at a later stage (in login.vm e.g., there's an element which can optionally be enabled for better support of logins from “public” systems, allowing the user to disable SSO and only have the IdP create a one-time session for a single SP). More information on what properties are available through VTL – Velocity Template Language – can be found on the VelocityVariables page in the Shibboleth Wiki.

    Rebuilding the idp.war file is essential after any changes to files in the edit-webapp subdirectory, but we will execute this step after the installation of the configuration of the status page in the next section.

    Upgrading from a v2.x IdP

    With v2 of the IdP, the login and error pages always had to be implemented in the form of JSP (Java Server Pages), and customization was done by modifying the relevant files in the src/main/webapp subdirectory of the distribution, followed by re-running install.sh and restarting Tomcat after every modification. The Shibboleth IdP v3 now relies on Spring Web Flow for handling the login, and the default rendering method has been changed to Velocity (with the corresponding templates living under /opt/shibboleth-idp/views). If you have customized the login.jsp, login.css, error.jspfiles etc. in the past and did not use the SWITCH-provided templates, then we recommend porting your changes to the new-style .vm files, as they have the benefit of not requiring a Servlet container restart to take effect. Alternatively, you may continue to use existing JSP versions of these pages, in which case you need to remove the equivalent Velocity files in the views subdirectory, so that the IdP falls back to a JSP-based rendering (see also the User Interface section on the PasswordAuthnConfiguration Wiki page). Be aware that all customized JSP files must be placed under /opt/shibboleth-idp/edit-webapp subdirectory, not /opt/shibboleth-idp/webapp – the latter directory gets removed when doing an upgrade from 3.x to 3.y.

    IdP status URL configuration

    The IdP status page depends on the JSP Standard Tag Library (JSTL), which is not part of the Shibboleth IdP distribution. The status page provides useful diagnostic information, and it's strongly recommended to enable this feature. To do so, download the jstl-1.2.jar file to the /opt/shibboleth-idp/edit-webapp/WEB-INF/lib directory, and rebuild the idp.war file:

    cd /opt/shibboleth-idp/edit-webapp/WEB-INF/lib
    sudo curl -O https://repo1.maven.org/maven2/jstl/jstl/1.2/jstl-1.2.jar
    sudo JAVACMD=/usr/bin/java /opt/shibboleth-idp/bin/build.sh -Didp.target.dir=/opt/shibboleth-idp

    which should result in the following output:

    Warning: JAVA_HOME environment variable is not set.
      If build fails because sun.* classes could not be found
      you will need to set the JAVA_HOME environment variable
      to the installation directory of java.
    Rebuilding /opt/shibboleth-idp/war/idp.war ...
    ...done
    
    BUILD SUCCESSFUL
    Total time: 3 seconds

    Second, add in /opt/shibboleth-idp/conf/idp.properties this new property to enable the access method:

    idp.status.accessPolicy = status.AccessByIPAddress

    Third, add in /opt/shibboleth-idp/conf/access-control.xml a new entry with the key status.AccessByIPAddress, after the first entry with key AccessByIPAddress. This enables access to the status page for the system running the AAI Resource Registry. The first IPv4/IPv6 address pair allows access from localhost. The remaining addresses are used by the two instances of the AAI Resource Registry:

            <entry key="AccessByIPAddress">
                [...]
            </entry>
    
            <entry key="status.AccessByIPAddress">
                <!--
                  Accepted hosts: localhost & two Resource Registry hosts
                -->
                <bean id="status.AccessByIPAddress" parent="shibboleth.IPRangeAccessControl"
                    p:allowedRanges="#{ {'127.0.0.1/32', '::1/128',
                    '130.59.118.224/32', '2001:620:0:1006::255/128',
                    '130.59.198.12/32', '2001:620:0:3006::256/128'} }" />
            </entry>

    You might also want to add IP addresses from your local network, in case you want to check the status page from additional systems. According to your needs, you can also configure in analogy further entries with key values resolvertest.AccessByIPAddress or reload.AccessByIPAddress to specificly enable access to these IdP services.
    The latter one is for the new IdP v3 reload feature permitting fine-grained control over the reloads of specific configuration changes. Alternatively to IP addresses, you can configure a separate accessPolicy, as documented in the Shibboleth Wiki under AccessControlConfiguration.

    Hibernate ORM workaround for JPAStorageService

    There is an unfortunate interaction between the value column of the storagerecords database table being declared as “large object” in the IdP code, how Hibernate ORM interprets it and PostgreSQL's JDBC driver way of handling BLOBs. Combination of these three elements has the effect of storing large object identifiers in the value column without enabling PostgreSQL's large object management (see issue OSJ-135 for further details). This results in orphaned large objects and guaranteed data loss if the vacuumlo command is ever run. In order to prevent such an accident, we urge administrators to implement the following workaround before starting their IdP for the first time.

    To ensure Hibernate ORM does not use large objects with PostgreSQL, download the file orm.xml and place it so that it will be packaged in idp.war:

    sudo mkdir -p /opt/shibboleth-idp/edit-webapp/WEB-INF/classes/META-INF
    cd /opt/shibboleth-idp/edit-webapp/WEB-INF/classes/META-INF/
    sudo curl -O https://www.switch.ch/aai/guides/idp/installation/orm.xml

    The orm.xml override will make sure that the storagerecords.value column is used to store the actual object content as text instead of an identifier, with the additional benefit that the value is easily readable.

    Finally, rebuild the idp.war file again to include those changes:

    sudo JAVACMD=/usr/bin/java /opt/shibboleth-idp/bin/build.sh -Didp.target.dir=/opt/shibboleth-idp

    Web application enablement

    Enable the IdP Web application in the Tomcat configuration by creating a file /etc/tomcat/Catalina/localhost/idp.xml and populating it with the following contents:

    <Context docBase="/opt/shibboleth-idp/war/idp.war"
             unpackWAR="false"
             swallowOutput="true">
      <Manager pathname="" />
    </Context>

    Finally, (re)start Tomcat with:

    sudo service tomcat7 restart
    sudo systemctl enable tomcat.service
    sudo systemctl start tomcat.service

    Testing the status page

    At this point, you're able to check if the IdP Web application is at least starting properly. On the IdP, execute

    curl -sk https://localhost/idp/status

    which should generate an output like

    ### Operating Environment Information
    operating_system: Linux
    operating_system_version: 3.10.0-123.20.1.el7.x86_64
    operating_system_architecture: amd64
    jdk_version: 1.7.0_75
    available_cores: 1
    used_memory: 494 MB
    maximum_memory: 494 MB
    
    ### Identity Provider Information
    idp_version: 3.3.3
    start_time: 2017-12-21T07:10:32+01:00
    current_time: 2017-12-22T09:18:12+01:00
    uptime: 94059315 ms
    
    [...]

    If the status page is not shown as expected, then something went wrong when Tomcat was trying to load the IdP Web application. In such a case, you should examine the following log files, in this recommended order:

    • /opt/shibboleth-idp/logs/idp-warn.log: messages from the IdP of level WARN or ERROR, i.e. a (small) subset of the idp-process.log file which can be useful in getting a quick overview of potential issues.
    • /opt/shibboleth-idp/logs/idp-process.log: the log file written by the IdP itself. Depending on what startup problem you're experiencing, the files in /opt/shibboleth-idp/logs might not have been created yet, or there could be a problem with the directory or file permissions (make sure that it is writable by the tomcat user).
    • /var/log/tomcat/localhost.YYYY-MM-DD.log: messages from Tomcat's Servlet implementation (see Logging in Tomcat for further information).
    • /var/log/tomcat/catalina.out: console output from Tomcat.

    In case the problem does not become apparent from examining the log messages, the first step we recommend is having a look at the Troubleshooting Tips page in the Shibboleth Wiki, or search for the error message in the Identity Provider 3 Wiki space. Alternatively, feeding the error message to your favourite search engine and limiting the results with +site:shibboleth.net/pipermail/users (or similar) can be helpful in determining if the problem has already been brought up on the Shibboleth users mailing list.

    Last but not least, it is worth mentioning that syntax errors in XML files also happen to be the source of (sometimes fairly cryptic) error messages. To make sure that your configuration files are well-formed XML documents, we recommend using the xmllint --noout somefile.xml command, which will print out syntax errors (if there are any) in a relatively human-friendly form. Use apt-get install libxml2-utilsyum install libxml2 to install this utility in case it is not yet available on your system. Alternatively you can also use the command xmlwf somefile.xml to check the syntax of your XML files.

    Registering the IdP in the federation

    The last step in making your IdP fully functional is making it known to the service providers ("resources") in the federation, so that these trust the authentication and attribute statements issued by the IdP. This step only applies to Swiss-based deployments and organizations, as SWITCH is operating the AAI Resource Registry for IdPs of its community members only.

    Proceed to the Home Organization Description bootstrap page on the AAI Resource Registry, and fill in the details as follows:

    • Under Home Organization Name, enter
      example.org
    • Select the appropriate federation: SWITCHaai Federation for production IdPs, AAI Test Federation otherwise.
    • For the entityID, specify
      https://aai-login.example.org/idp/shibboleth
    • For the HTTP POST binding URL, specify
      https://aai-login.example.org/idp/profile/SAML2/POST/SSO
    • For the HTTP Redirect binding URL, specify
      https://aai-login.example.org/idp/profile/SAML2/Redirect/SSO
    • Paste the contents of the /opt/shibboleth-idp/credentials/idp.crt file into the Identity Provider Certificate field
    • For the Attribute Authority URL, specify
      https://aai-login.example.org/idp/profile/SAML2/SOAP/AttributeQuery
      Note: do not specify port 8443 for IdP version 3 deployments, it is no longer necessary to configure the endpoint for the attribute authority on a separate TCP port – the standard SSL/TLS port (443) is used for this purpose.
    • Paste the contents of the /opt/shibboleth-idp/credentials/idp.crt file into the Attribute Authority certificate field
    • Provide your contact information in the last section and submit the form.

    After completion of the bootstrap step, the SWITCHaai team will check the information you provided and approve your request (or contact you in case the data need to be modified). As soon as your request has been approved, continue with the configuration of the IdP in the AAI Resource Registry. You will find the entry for your IdP in the Resource Registry under Home Organizations, from where you can proceed to the menu which allows configuring the details of your home organization (see also the AAI Resource Registry Guide for additional background).

    At the very least, you should adapt/complete the information under 1. General Information, 2. Descriptive Information and 5. Contacts, as these sections are only populated with preliminary information after the bootstrap step (the display name for your IdP e.g. is set to Bootstrapped Home Organisation example.org, so it's essential that you to go through these sections and modify the settings accordingly).

    When you have successfully completed the registration of the IdP, it is recommended to try a login on the AAI Attribute Viewer to see if authentication and attribute release is working properly.

    Upgrading from a v2.x IdP

    To test the v3 setup, SWITCH recommends setting up the new IdP with the same entity ID and certificate as the existing v2 deployment (as mentioned in the IdP installation section above), and use an entry in the local hosts file for basic testing. While this procedure does not allow complete testing of the back-channel functionality, it is a light-weight method of trying out the new IdP without requiring extensive modifications in the AAI Resource Registry. To do so, add an entry like

    192.0.2.3 aai-login.example.org

    (substitute the proper IP address) to the /etc/hosts file of the system which is running your browser (under Windows, the path is %SystemRoot%\system32\drivers\etc\hosts). Then, test the login as usual with the SPs of your choice (the AAI Attribute Viewer can serve as a useful starting point), and verify with the /opt/shibboleth-idp/logs/idp-process.log file on the new IdP that the login was indeed processed by the new system – if it's not the case, then you might have to restart the browser so that it picks up the IP address change.

    The next important point relates to the “backchannel” functionality (SOAP endpoints for attribute queries and artifact resolution, more precisely). The suggested v3 setup requires a potentially disruptive change to the Technical Information section of your home organization description in the AAI Resource Registry: instead of using a separate TCP port (8443) as with previous versions of the Shibboleth IdP, backchannel requests are now supposed to be submitted on the standard HTTPS port (443), which means that references to port 8443 for either the Attribute Service or the Artifact Resolution Service endpoints are removed (replace https://aai-login.example.org:8443/idp/profile/SAML2/SOAP/AttributeQuery with https://aai-login.example.org/idp/profile/SAML2/SOAP/AttributeQuery etc.). Before changing these URLs in the AAI Resource Registry, you should check the Apache HTTP Server logs of your v2 IdP for the frequency of attribute queries / artifact resolution requests, though. A command like

    grep -hE 'AttributeQuery|ArtifactResolution' /path/to/apache_httpd.log

    should help in determining if changing the backchannel endpoints can be done without causing relevant disrupting the operations of your current v2 IdP, or whether additional precautions need to be taken. If you're finding a non-negligible amount of attribute queries / artifact resolution requests in the logs of your existing IdP, then you're encouraged to contact the SWITCHaai Team to discuss the best strategy for migrating to the new SOAP endpoints.

    Keeping up to date & Upgrades

    Operating an IdP in the SWITCHaai federation also means keeping up with further developments in this area, as well as making sure that security updates are applied in time. All administrators of IdPs in the SWITCHaai federation are expected to subscribe to the AAI-Operations mailing list, where SWITCH is making announcements about important changes in the federation. Furthermore, it can be useful to subscribe to the Shibboleth Announce mailing list, which is used by the members of the Shibboleth Project team to announce new releases or send out security advisories.

    Additionally, subscribing to the security announcements mailing list of the OS vendor (ubuntu-security-announce, enterprise-watch-list) is highly recommended for keeping up with information about security updates for the OS-provided packages.

    Upgrade instructions for specific versions

    Upgrading from version 3.1.x to 3.2.x

    Version 3.2.x code contains a few bug fixes and changes that require some configurations changes that were introduced with version 3.1. To upgrade from version 3.1.x the IdP will have a short downtime. The recommended procedure is to first backup your /opt/shibboleth-idp/conf directory. Then change your configuration according to the following steps. Steps 1 until 7 (configurations settings) can be prepared without stopping the IdP.

    1. Unless you applied any changes in /opt/shibboleth-idp/edit-webapp/WEB-INF/web.xml, remove this file, so that the default version from the IdP distribution will be installed. Add the following line to your /opt/shibboleth-idp/conf/idp.properties to disable serving auto-generated metadata under the URL of the IdP's entityID:

      idp.entityID.metadataFile = 
    2. In /opt/shibboleth-idp/conf/services.xml within the bean FileBackedSWITCHaaiAttributeFilter, the bold printed text

        <bean id="FileBackedSWITCHaaiAttributeFilter"
                class="net.shibboleth.ext.spring.resource.FileBackedHTTPResource"
                c:client-ref="#{ '%{idp.http.proxyhost:null}' != 'null' ?
                                 'shibboleth.ProxyHttpClient' : 'shibboleth.HttpClient' }"
                c:url="https://rr.aai.switch.ch/switchaai/example.org/attribute-filter.xml"
                c:resource="file://%{idp.home}/conf/attribute-filter.switchaai.xml">
        </bean>

      must be changed to

        <bean id="FileBackedSWITCHaaiAttributeFilter"
                class="net.shibboleth.ext.spring.resource.FileBackedHTTPResource"
                c:client-ref="#{ '%{idp.http.proxyhost:null}' != 'null' ?
                                 'shibboleth.ProxyHttpClient' : 'shibboleth.HttpClient' }"
                c:url="https://rr.aai.switch.ch/switchaai/example.org/attribute-filter.xml"
                c:backingFile="%{idp.home}/conf/attribute-filter.switchaai.xml">
        </bean>
    3. We added the file attribute-resolver-other.xml to the standard configuration for version 3.2.x. All attributes but common-lib-terms entitlement are commented out by default and need to be enabled only when needed. If you already have the attribute-resolver-other.xml downloaded and enabled for the IdP version 3.1, there is nothing to do. If you do not have that file in the /opt/shibboleth-idp/conf directory, download the attribute-resolver-other.xml now.

      cd /opt/shibboleth-idp/conf
      sudo curl -O https://www.switch.ch/aai/guides/idp/installation/attribute-resolver-other.xml

      and enable it in services.xml:

          <util:list id="shibboleth.AttributeResolverResources">
              <value>%{idp.home}/conf/attribute-resolver-switchaai-core.xml</value>
              <value>%{idp.home}/conf/attribute-resolver-interfederation-core.xml</value>
              <value>%{idp.home}/conf/attribute-resolver-connectors.xml</value>
              <value>%{idp.home}/conf/attribute-resolver-other.xml</value>
          </util:list>
    4. To generate the persistentID we no longer need to detour the additional attribute definition for swissEduPersonUniqueID.withoutAttributeEncoder. In the file /opt/shibboleth-idp/conf/attribute-resolver-switchaai-core.xml file remove the following <resolver:AttributeDefinition> element:

      <resolver:AttributeDefinition id="%{idp.persistentId.sourceAttribute}" xsi:type="ad:Simple" sourceAttributeID="swissEduPersonUniqueID">
          <resolver:Dependency ref="swissEduPersonUniqueID"/>
      </resolver:AttributeDefinition>

      Replace in /opt/shibboleth-idp/conf/saml-nameid.properties the following property:

      idp.persistentId.sourceAttribute = swissEduPersonUniqueID.withoutAttributeEncoder

      with:

      idp.persistentId.sourceAttribute = swissEduPersonUniqueID
    5. Also for persistentID storage, there is a new property that should be used in order to avoid a warning appearing in the logs. The old configuration still works, but triggers the warning: net.shibboleth.idp.saml.nameid.impl.StoredPersistentIdGenerationStrategy: Transferring settings from deprecated JDBCPersistentStore, please update configuration. In order to get rid of it, replace in /opt/shibboleth-idp/conf/saml-nameid.properties the property:

      idp.persistentId.store = PersistentIdStore

      with:

      idp.persistentId.dataSource = shibboleth.PostgreSQLDataSource

      The identifier shibboleth.PostgreSQLDataSource references a bean in /opt/shibboleth-idp/conf/global.xml. Then in /opt/shibboleth-idp/conf/global.xml, delete the bean definition for PersistentIdStore which is no longer needed:

          <!-- Bean for accessing the "shibpid" table -->
          <bean id="PersistentIdStore"
                class="net.shibboleth.idp.saml.nameid.impl.JDBCPersistentIdStore"
                p:dataSource-ref="shibboleth.PostgreSQLDataSource" />

      The “old” property idp.persistentId.store now allows changing some settings related to database queries like timeout, SQL and column names. Please refer to the Shibboleth wiki page PersistentNameIDGenerationConfiguration for details if you need to customize these.

    6. Undo the temporary workaround from 2015-07-17: Change in /opt/shibboleth-idp/conf/attribute-resolver-connectors.xml

          <resolver:DataConnector id="myStoredId"
              [...]
              salt="...random value..."
              queryTimeout="0">
      

      back to

          <resolver:DataConnector id="myStoredId"
              [...]
              salt="%{idp.persistentId.salt}"
              queryTimeout="0">
      
    7. There is a new template (updated for 3.2.1) for /opt/shibboleth-idp/conf/intercept/consent-intercept-config.xml available. We recommend downloading the new consent-intercept-config.xml and merge with your possible changes. If you stick to your existing consent-intercept-config.xml, replace the following entry:

      <value>swissEduPersonUniqueID.withoutAttributeEncoder</value>

      with

      <value>swissEduPersonUniqueID</value>
    8. At the moment, the user consent and terms-of-use are stored inside so called large objects. As this leads to data loss when running vacuumlo, we decided to change the structure of the Postgres Database. The following steps are needed for this migration:
      • Go inside your IdP's installation directory /opt/shibboleth-idp/ and create the following directories:

        mkdir -p edit-webapp/WEB-INF/classes/META-INF
      • Download the file orm.xml inside your /opt/shibboleth-idp/edit-webapp/WEB-INF/classes/META-INF/

        cd /opt/shibboleth-idp/edit-webapp/WEB-INF/classes/META-INF/
        sudo curl -O https://www.switch.ch/aai/guides/idp/installation/orm.xml
        

        This will instruct the IdP to use storagerecords.value as text instead of as a large object reference.

      • Backup your PostgreSQL server, e.g using the following command:

        sudo -iu postgres pg_dumpall > dumpall-`date +\%Y-\%m-\%d-\%H\%M\%S`.sql
      • Stop Tomcat now to avoid inconsistencies during migration of the database.

        sudo service tomcat7 stop
        sudo systemctl stop tomcat.service

        Create a new storagerecords table with text instead of oid by running the following queries:

        sudo -iu postgres psql shibboleth
        
        DROP RULE valuedeletecleanup on storagerecords;
        DROP RULE valueupdatecleanup on storagerecords;
        
        DROP TABLE IF EXISTS storagerecords_new;
        DROP TABLE IF EXISTS storagerecords_old;
        
        CREATE TABLE storagerecords_new (LIKE storagerecords INCLUDING ALL);
        INSERT INTO storagerecords_new
            SELECT a.context as context,
                a.id as id,
                a.expires as expires,
                convert_from(b.data, 'UTF8') as value,
                a.version as version
            FROM storagerecords a, pg_largeobject b
                WHERE a.value::oid=b.loid;
        
      • Then swap that table with the old storagerecords:

        ALTER TABLE storagerecords RENAME TO storagerecords_old;
        ALTER INDEX storagerecords_pkey RENAME TO storagerecords_old_pkey;
        ALTER TABLE storagerecords_new RENAME TO storagerecords;
        ALTER INDEX storagerecords_new_pkey RENAME TO storagerecords_pkey;
        ALTER TABLE storagerecords OWNER TO shibboleth;
        
      • Then create a new index on the storagerecords.expires column. This will help the cleanup task that runs every 10 minutes find expired records faster when the storagerecords table has thousands of rows.

        CREATE INDEX storagerecords_expires_index ON storagerecords(expires);
        
      • Another new requirement with version 3.2.x is the shibpid table must have a primary key defined over columns localEntity, peerEntity, persistentId. Add it with the following query:

        ALTER TABLE shibpid ADD PRIMARY KEY (localEntity, peerEntity, persistentId);
        

        Now that there is a primary key index on this table, most other indexes are no longer used during queries, according to our testing. It is better to remove these obsolete indexes as their maintenance incurs a cost on every change on the table. The last remaining index is renamed for clarity.

        DROP INDEX persistentId;
        DROP INDEX persistentId_2;
        DROP INDEX localentity;
        ALTER INDEX localentity_2 RENAME TO shibpid_getbysourcevalue_index;
        

        Exit PostgreSQL:

        \q
        

    Then proceed with the steps in the section below Common upgrade instructions for the Shibboleth IdP 3 (patch or minor versions).

    Upgrading from version 3.2.x to 3.3.x

    Version 3.3.x code contains a few bug fixes and changes that may require some configurations changes. Most of them are not relevant for a typical Identity Provider deployment in the SWITCHaai or AAI Test federations.
    Still, four changes are recommended to apply to increase the security and keep the configuration forward compatible.

    XML Namespace Cleanup in Attribute Resolution Configuration

    Cleaning up the XML namespaces in the attribute resolution configuration is highly recommended to keep the configuration forward compatible. Although the current configuration using namespace prefixes is still supported by version 3.3.x, some new features, like the trustFile attribute of the LDAPDirectory Data Connector won't be available without this cleanup.

    The cleanup process is as simple as updating the header of the attribute resolution XML files and removing any occurences of the resolver:, dc:, ad: and enc: namespace prefixes from any XML element and type identifier in the attribute resolution configuration files.

    The following instructions describe in detail how to cleanup the XML namespaces.

    For each of the following files, process the steps below:

    • /opt/shibboleth-idp/conf/attribute-resolver-connectors.xml
    • /opt/shibboleth-idp/conf/attribute-resolver-switchaai-core.xml
    • /opt/shibboleth-idp/conf/attribute-resolver-interfederation-core.xml
    • /opt/shibboleth-idp/conf/attribute-resolver-other.xml
    • (possibly further files you added yourself)

    Steps:

    1. Locate the following <resolver:AttributeResolver ...> element at the beginning of the file (starting on the second line)
      <resolver:AttributeResolver
              xmlns:resolver="urn:mace:shibboleth:2.0:resolver"
              xmlns:pc="urn:mace:shibboleth:2.0:resolver:pc"
              xmlns:ad="urn:mace:shibboleth:2.0:resolver:ad"
              xmlns:dc="urn:mace:shibboleth:2.0:resolver:dc"
              xmlns:enc="urn:mace:shibboleth:2.0:attribute:encoder"
              xmlns:sec="urn:mace:shibboleth:2.0:security"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="urn:mace:shibboleth:2.0:resolver http://shibboleth.net/schema/idp/shibboleth-attribute-resolver.xsd
                                  urn:mace:shibboleth:2.0:resolver:pc http://shibboleth.net/schema/idp/shibboleth-attribute-resolver-pc.xsd
                                  urn:mace:shibboleth:2.0:resolver:ad http://shibboleth.net/schema/idp/shibboleth-attribute-resolver-ad.xsd
                                  urn:mace:shibboleth:2.0:resolver:dc http://shibboleth.net/schema/idp/shibboleth-attribute-resolver-dc.xsd
                                  urn:mace:shibboleth:2.0:attribute:encoder http://shibboleth.net/schema/idp/shibboleth-attribute-encoder.xsd
                                  urn:mace:shibboleth:2.0:security http://shibboleth.net/schema/idp/shibboleth-security.xsd">
      and replace it with the following (much simpler) contents:
      <AttributeResolver
              xmlns="urn:mace:shibboleth:2.0:resolver"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
              xsi:schemaLocation="urn:mace:shibboleth:2.0:resolver http://shibboleth.net/schema/idp/shibboleth-attribute-resolver.xsd">
      
    2. At the end of the file, replace </resolver:AttributeResolver> by </AttributeResolver>.
    3. Remove all namespace prefixes resolver:, dc:, ad: and enc: from XML elements and XML type identifiers. You may do this be searching and replacing in your editor.
      • Remove the namespace prefix resolver: from XML elements.
        Examples:
        • Change <resolver:DataConnector ...> to <DataConnector ...> and
          change </resolver:DataConnector> to </DataConnector>.
        • Change <resolver:AttributeDefinition ...> to <AttributeDefinition ...> and
          change </resolver:AttributeDefinition> to </AttributeDefinition>.
        • etc.
      • Remove the namespace prefix dc: from XML elements and XML type identifiers.
        Examples:
        • Change <dc:FilterTemplate ...> to <FilterTemplate ...> and
          change </dc:FilterTemaplte> to </FilterTemplate>.
        • Change xsi:type="dc:LDAPDirectory" to xsi:type="LDAPDirectory".
        • etc.
      • Remove the namespace prefix ad: from XML elements and XML type identifiers.
        Examples:
        • Change <ad:Script> to <Script> and
          change </ad:Script> to </Script>.
        • Change <ad:ValueMap ...> to <ValueMap ...> and
          change </ad:ValueMap> to </ValueMap>.
        • Change xsi:type="ad:Simple" to xsi:type="Simple".
        • Change xsi:type="ad:Script" to xsi:type="ScriptedAttribute"
          Note: In the type definition you need to change the value 'Script' to 'ScriptedAttribute'!
        • etc.
      • Remove the namespace prefix enc: from XML elements and XML type identifiers.
        Examples:
        • Change xsi:type="enc:SAML2String" to xsi:type="SAML2String".
        • etc.
    4. The pc: prefix refers to the Principal Connector deprecated in IdPv3.
      Replace any configuration with a pc: prefix with the new mechanism.
    Explicit Certificate Trust Configuration

    It's highly recommended to configure explicit certificate trust configuration for the verification of the LDAP server certificates (instead of relying on Java's built-in mechanism 'jvmTrust').

    Follow these instructions to configure explicit certificate trust configuration:

    1. Get the Root CA certificate of the issuing CA of your LDAP server's SSL certificate and store it into a file.
      If you have obtained the LDAP server's certificate via SWITCHpki, you can download the QuoVadis Root CA 2 certificate from https://www.switch.ch/pki/manage/download/qvrca2.crt.pem.
    2. Copy the file to the directory /opt/shibboleth-idp/credentials (e.g. /opt/shibboleth-idp/credentials/qvrca2.crt.pem).
    3. In the file /opt/shibboleth-idp/conf/ldap.properties, update the following properties, i.e. change the property 'idp.authn.LDAP.sslConfig' from 'jvmTrust' to 'certificateTrust' and set the property 'idp.authn.LDAP.trustCertificates' to the path of your Root CA's certificate file (adapt the filename 'qvrca2.crt.pem' as appropriate):
      idp.authn.LDAP.sslConfig          = certificateTrust
      idp.authn.LDAP.trustCertificates  = /opt/shibboleth-idp/credentials/qvrca2.crt.pem
      
      This configuration makes sure that the LDAP connections used for authentication use the explicit trust configuration.
      Further down in the same file, make sure that the following property exists (add it just after the property 'idp.attribute.resolver.LDAP.useStartTLS' if it doesn't yet exist):
      idp.attribute.resolver.LDAP.trustCertificates   = %{idp.authn.LDAP.trustCertificates:undefined}
      
      This configuraiton, together with the following step, makes sure that the LDAP connections used for attribute resolution use the explicit trust configuration.
    4. In the file /opt/shibboleth-idp/conf/attribute-resolver-connectors.xml, add the attribute trustFile to the LDAP data connector refering to the property idp.attribute.resolver.LDAP.trustCertificates (i.e. refering to the file of the Root CA certificate):
          <DataConnector id="myLDAP" xsi:type="LDAPDirectory"
              [...]
              useStartTLS="%{idp.attribute.resolver.LDAP.useStartTLS:true}"
              trustFile="%{idp.attribute.resolver.LDAP.trustCertificates}">
              [...]
          </DataConnector>
      
      Note: The attribute 'trustFile' is only available if you've cleaned up the XML namespaces, as described in the section XML Namespace Cleanup in Attribute Resolution Configuration just above.
      In case your LDAP data connector configuration still contains a <dc:StartTLSTrustCredential> element, remove it (the attribute trustFile provides this functionality).
    Configure an Attribute Resolver Connection Pool

    It is recommended to insert a <ConnectionPool> element after the <FilterTemplate> element in the /opt/shibboleth-idp/conf/attribute-resolver-connectors.xml file.

        <DataConnector id="myLDAP" xsi:type="LDAPDirectory"
          [...]
            <FilterTemplate>
                <![CDATA[
                    %{idp.attribute.resolver.LDAP.searchFilter}
                ]]>
            </FilterTemplate>
            <ConnectionPool
                minPoolSize="%{idp.pool.LDAP.minSize:3}"
                maxPoolSize="%{idp.pool.LDAP.maxSize:10}"
                blockWaitTime="%{idp.pool.LDAP.blockWaitTime:PT3S}"
                validatePeriodically="%{idp.pool.LDAP.validatePeriodically:true}"
                validateTimerPeriod="%{idp.pool.LDAP.validatePeriod:PT5M}"
                expirationTime="%{idp.pool.LDAP.idleTime:PT10M}"
                failFastInitialize="%{idp.pool.LDAP.failFastInitialize:false}" />
        </DataConnector>
    
    Limit Cookies to Secure Connections

    The default setting for idp.cookie.secure is false. To prevent browsers from sending IdP cookies via http instead of https, add the following line to the /opt/shibboleth-idp/conf/idp.properties file:

    idp.cookie.secure = true
    Update the messages.properties configuration
    1. Go to the Shibboleth Message Translation page
    2. Then download the relevant messages_[de_CH|fr|it].properties files
    3. Move the downloaded messages files into the /opt/shibboleth-idp/messages/ directory:
      mv messages_*.properties /opt/shibboleth-idp/messages/
    4. Adapt the messages files. E.g. you need to adapt the generic terms of use if your organisation asks users to accept them before first use. A future software upgrade will not touch them but you may have to manually fetch new versions as they become available. and then reapply your modifications.
    5. Replace in /opt/shibboleth-idp/conf/services.xml the following lines
      <util:list id="shibboleth.MessageSourceResources">
          <value>%{idp.home}/messages/authn-messages</value>
          <value>%{idp.home}/messages/consent-messages</value>
          <value>%{idp.home}/messages/error-messages</value>
      </util:list>
      with these lines to load the new messages.properties files:
      <util:list id="shibboleth.MessageSourceResources">
          <value>%{idp.home}/messages/messages</value>
      </util:list>
    6. Optional step: If you want to store local changes to messages.properties and messages_[de_CH|fr|it].properties files into separate files, e.g. local.properties and local_[de_CH|fr|it].properties in the directory /opt/shibboleth-idp/messages, you can add a separate bundle of files by the following configuration in /opt/shibboleth-idp/conf/services.xml:
      <util:list id="shibboleth.MessageSourceResources">
          <value>%{idp.home}/messages/local</value>
          <value>%{idp.home}/messages/messages</value>
      </util:list>
    Software Upgrade

    Now proceed with the steps from the Common upgrade instructions for the Shibboleth IdP 3 (patch or minor versions) below and return back here for the remaining post-upgrade steps.

    Note: This update will overwrite any changes you applied to the directory system. This is relevant in case you for example changed some text in the file system/messages/messages.properties.

    Post-upgrade instructions for the messages.properties configuration
    1. You may want to remove the previous messages files from /opt/shibboleth-idp/messages if you were using older translations. These could include authn-messages_[de_CH|fr|it].properties, consent-messages_[de_CH|fr|it].properties, error-messages_[de_CH|fr|it].properties.
    2. Only in case you use IdP logout: You should manually fetch the latest logout.vm from the distribution /usr/local/dist/shibboleth-identity-provider-3.x.y/views/logout.vm. This view contains a patch. Then reapply your local changes, if any. The upgrade did not overwrite what was already in the /opt/shibboleth-idp/views directory.

    Upgrading from version 3.3.x to 3.4.x

    Note: These upgrade instructions for 3.4 are rather new, they might be incomplete. Please share your experience with us.

    The Shibboleth IdP Release Notes (3.4.0-3.x.y) summarize the many changes.
    Many tags or elements in the IdP configuration are now marked as deprecated, they will disappear in IdPv4. They are all documented on the DeprecatedIdPV4 page in the Shibboleth Wiki. A helpful reference with instructions on how to update it is the deprecated Dependency element page.
    So best update your configuration already now to be forward compatible. Ideally, no WARN entries show up in the idp-process.log file after restarting the IdP.

    Update properties in idp.properties

    In the file /opt/shibboleth-idp/conf/idp.properties replace these two properity names:
    Old: idp.consent.userStorageKey, New: idp.consent.attribute-release.userStorageKey
    Old: idp.consent.userStorageKeyAttribute, New: idp.consent.attribute-release.userStorageKeyAttribute

    Update the rotate-sealer.sh script

    The script to rotate the key used to encrypt the cookie information needs an update. Replace the script in /opt/shibboleth-idp/credentials/rotate-sealer.sh with the one in chapter Configuring periodic encryption key rotation above.

    Disable the LegacyPrincipalConnector

    In the file /opt/shibboleth-idp/conf/c14n/subject-c14n.xml disable the deprecated LegacyPrincipalConnector.

            <!-- <ref bean="c14n/LegacyPrincipalConnector" /> -->
    
    Update the IdP status URL configuration

    Check out the updated configuration for the IdP status URL in the conf/access-control.xml file and consider to apply it as well.

    Modify the attribute-filter download URL

    IdPv3.4 supports a new, more compact, attribute filter schema. The former schema is deprecated, IdPv4 will not be able to parse it. Therefore, change the attribute-filter download URL now, to be forward compatible.
    In /opt/shibboleth-idp/conf/services.xml in the bean FileBackedSWITCHaaiAttributeFilter insert _v4 into the string in c:url, so that it ends like this: attribute-filter_v4.xml. The filename in c:backingFile you do not need to modify.

    Activate HTML Local Storage to store the IdP's session information

    Check out the updated configuration for the HTML Local Session Storage in the conf/idp.properties file. Activate it to ensure that SSO also works properly with new browser versions that default to SameSite=Lax.

    Replace requestContext.principalName method with resolutionContext.principal

    Replace all occurances of the deprecated requestContext.principalName method with resolutionContext.principal. You will likely find it in the idp.attribute.resolver.LDAP.searchFilter in the file ldap.properties and perhaps also in the attribute-resolver-connectors.xml file.

    Update the messages.properties configuration
    1. Go to the Shibboleth Message Translation page
    2. Download the relevant messages_[de_CH|fr|it].properties files
    3. In case you loclly modified some strings, you need to reapply these changes bevore your Copy the downloaded messages files into the /opt/shibboleth-idp/messages/ directory.

    Note: These upgrade instructions for 3.4 are rather new, they might be incomplete. Please share your experience with us.

    Common upgrade instructions for the Shibboleth IdP 3 (patch or minor versions)

    Updating from 3.x.y to a newer patch or minor version (3.x.y) of the Shibboleth IdP 3 is very simple, in contrast to the fairly convoluted procedure which was necessary with IdP v2. For a summary of the changes, see the ReleaseNotes page in the Shibboleth Wiki. In general, no modification of an existing configuration is required when updating to the latest 3.x.y release (unless new features are introduced, in which case it might be necessary to explicitly enable them through configuration changes).

    Warning for RHEL/CentOS: Checkout the changed Java dependency of the distribution-supplied postgresql-jdbc driver in chapter PostgreSQL Installation. You need to manually install a new driver compatible again with Java 7.

    Please note: The upgrade from 3.1 to 3.2 and from 3.2 to 3.3 includes some configuration changes (e.g. for PostgreSQL). If you are upgrading from one of these version, please apply the configuration as described in the section above and then follow the generic upgrade instructions below.

    cd /usr/local/dist
    sudo curl -O https://shibboleth.net/downloads/identity-provider/3.x.y/shibboleth-identity-provider-3.x.y.tar.gz
    sudo tar -zxf shibboleth-identity-provider-3.x.y.tar.gz
    sudo JAVACMD=/usr/bin/java /usr/local/dist/shibboleth-identity-provider-3.x.y/bin/install.sh -Didp.conf.filemode=644

    Make sure to supply the -Didp.conf.filemode=644 command line option, and confirm the defaults, by pressing enter, for the source and installation directory when prompted. This should result in the following output:

    Warning: JAVA_HOME environment variable is not set.
      If build fails because sun.* classes could not be found
      you will need to set the JAVA_HOME environment variable
      to the installation directory of java.
    Source (Distribution) Directory: [/usr/local/dist/shibboleth-identity-provider-3.x.y]
    
    Installation Directory: [/opt/shibboleth-idp]
    
    Rebuilding /opt/shibboleth-idp/war/idp.war ...
    ...done
    
    BUILD SUCCESSFUL
    Total time: 5 seconds

    Note that specifying the proper idp.conf.filemode parameter is essential – otherwise the permissions of the configuration files will be adjusted in a way which is unsuitable for the setup recommended by this guide (see IdP installation above). In case you inadvertently forgot to specify this option, rerun install.sh with this parameter appended. Finally, adjust the permissions of credentials.properties and restart the Servlet container with:

    sudo chmod 600 /opt/shibboleth-idp/conf/credentials.properties
    sudo service tomcat7 restart
    sudo chmod 600 /opt/shibboleth-idp/conf/credentials.properties
    sudo systemctl restart tomcat.service

    Final Tests

    After having completed the installation and upgrade, it is highly recommended to perform the following tests to see if the Identity Provider is configured properly:

    1. Check your IdP's status page to see if the Identity Provider is loaded and a reasonable reponse is returned.
    2. Check if a standard SAML login works by accessing the AAI Attribute Viewer. Ensure that there is no error message or warning displayed. Also especially check whether the values of the persistentIDs are still the same like before the upgrade.
    3. Check whether your IdP properly responds to SAML Attribute Queries. This feature is not widely used yet but is required for account checking and the support for SWITCH edu-ID.
    4. Check the interfederation readiness of your IdP by going to the Interfederation Test Page. This page will show you whether all the attributes recommended for interfederation are available.
    5. If your IdP's metadata is already published in eduGAIN, you should test the access to a test SP kindly provided by GARR and registered in the Italian IDEM federation.

    Further information

    The following links can serve as starting points for further exploring the Shibboleth IdP and its impressive set of customizable features:

    Version Information

    Copyright:  SWITCH
    Author:     aai@switch.ch
    Date:       2021-08-03
                Changelog