MDQ (Metadata Query) in the UK federation

The size of the UK federation metadata aggregate has grown sufficiently large that scaling issues are appearing for some metadata consumers. We are piloting another approach where entities can request metadata for individual entities as required from our MDQ (Metadata Query) service, instead of periodically downloading a single aggregate file that contains metadata for all the entities in the world.

Problem statement

An entity (IdP or SP) can be described by SAML metadata, which contains all the technical details required for other entities to interoperate with it, for example certificates, endpoints, security algorithms, and logos. An IdP and SP wishing to interoperate will, in the simplest case, exchange metadata bilaterally. In the UK's research and education sector, the UK federation has traditionally collated metadata from each of the entities in the federation and published that metadata in a single metadata aggregate.

However, as the UK federation membership has grown, and our reach has become global through the eduGAIN metadata exchange, our aggregate file now contains a few thousand entities and is approximately 30 MB in size. Several problems are becoming apparent:

  • The memory requirement for entities consuming our metadata aggregate is increasing, and out of memory errors are difficult to diagnose.
  • A new aggregate must be generated if any entity updates its metadata. Entities download the new aggregate, even if they do not interoperate with any of the entities that have changed
  • The federation operator has significant network load (30MB file x 2000 entities downloading it = 56 GB network traffic per updated aggregate)

Mechanisms have already been put in place that can partially alleviate some of these problems, for example we publish updated metadata once per day, and we support downloading of compressed metadata. However these are at-best temporary measures.

What is MDQ?

MDQ is a mechanism that allows entities to request only the metadata they need, as they need it.

Traditionally, your entity has been configured to regularly download and verify the whole metadata aggregate. In the MDQ model, you configure your entity to request metadata from the UK federation MDQ server using the MDQ protocol.

Benefits and risks

The major benefit to deployers is that you require a much lower memory footprint for your entity. We do not have definitive figures, although we have anecdotal evidence that a Shibboleth IdP can run with a Java heap size of 500MB, compared to the current recommendation to use a minimum heap size of 1.5 GB when consuming large metadata aggregates like the UK federation metadata.

The major risk is that you move from a situation where your entity downloads metadata in the background, which allows retries if the metadata server is unavailable, to a situation where you query metadata just-in-time. To mitigate this risk, the UK federation has deployed a MDQ server with several mechanisms for resilience. However, we cannot guarantee delivery.

Another benefit of the MDQ protocol is that it allows the metadata for an entity to be available at a specific URL. You can pre-fetch metadata for high-value, commonly-used entities, and we provide a configuration examples below.

The InCommon per-entity Metadata Working Group final report details much more about the operational aspects of an MDQ server.

New signing key

Metadata from the MDQ server is signed using a different key to our metadata aggregates. The certificate is available from http://mdq.ukfederation.org.uk/ as is other information. Please remember to check the fingerprint of the certificate when you download it, either by checking our Technical Recommendations for Participants, or by phoning the UK federation helpdesk.

Configuration example for Shibboleth IdP

The Shibboleth IdP has supported the MDQ protocol since version 3.0.0. Here is an example of how to configure your IdP to use the UK federation MDQ server:

    <!-- UK federation MDQ service -->
    <MetadataProvider id="ukfMDQ" xsi:type="DynamicHTTPMetadataProvider">
        <!-- Verify the signature on the root element (i.e., the EntityDescriptor element) -->
        <MetadataFilter xsi:type="SignatureValidation" requireSignedRoot="true"
                certificateFile="%{idp.home}/credentials/ukfederation-mdq.pem" />

        <!-- Require a validUntil XML attribute no more than 30 days into the future -->
        <MetadataFilter xsi:type="RequiredValidUntil" maxValidityInterval="P30D" />

        <!-- The MetadataQueryProtocol element specifies the base URL for the query protocol -->
        <MetadataQueryProtocol>http://mdq.ukfederation.org.uk/</MetadataQueryProtocol>
    </MetadataProvider>

Configuration example for Shibboleth IdP that pre-fetches a high-value entity

Assuming you wish to pre-fetch metadata for the UK federation Test SP, you could use the following configuration example. This can obviously be generalised to multiple high-value SPs. Reference documentation for the Shibboleth IdP MetadataProvider configuration is available from the Shibboleth wiki.

    <!-- UK federation MDQ service -->
    <MetadataProvider id="ukfMDQ" xsi:type="DynamicHTTPMetadataProvider">

        <!-- Verify the signature on the root element (i.e., the EntityDescriptor element) -->
        <MetadataFilter xsi:type="SignatureValidation" requireSignedRoot="true"
                certificateFile="%{idp.home}/credentials/ukfederation-mdq.pem" />

        <!-- Require a validUntil XML attribute no more than 30 days into the future -->
        <MetadataFilter xsi:type="RequiredValidUntil" maxValidityInterval="P30D" />

        <!-- The MetadataQueryProtocol element specifies the base URL for the query protocol -->
        <MetadataQueryProtocol>http://mdq.ukfederation.org.uk/</MetadataQueryProtocol>

    </MetadataProvider>

    <!-- for UK federation Test SP, entityID https://test.ukfederation.org.uk/entity -->
    <MetadataProvider id="ukf-uk001896" xsi:type="FileBackedHTTPMetadataProvider"
            metadataURL="http://mdq.ukfederation.org.uk/entities/https:%2F%2Ftest.ukfederation.org.uk%2Fentity"
            backingFile="%{idp.home}/metadata/ukf-uk001896.xml"
            maxRefreshDelay="PT6H">

        <!-- Verify the signature on the root element (i.e., the EntityDescriptor element) -->
        <MetadataFilter xsi:type="SignatureValidation"
                        requireSignedRoot="true"
                        certificateFile="%{idp.home}/credentials/ukfederation-mdq.pem" />

        <!-- Require a validUntil XML attribute no more than 30 days into the future -->
        <MetadataFilter xsi:type="RequiredValidUntil" maxValidityInterval="P30D"/>

    </MetadataProvider>


Please note that the requireSignedRoot attribute on the SignatureValidation filter was added in v3.2.0. You cannot use this configuration with an older version of the IdP because the MetadataResolverService will fail to load. If this affects you, we recommend that you upgrade your IdP to a supported version. Do not remove the SignatureValidation filter or you open your IdP to MITM (man in the middle) attacks. If you are unable to upgrade your IdP to a supported version of the IdP, you can use the requireSignedMetadata although this is a deprecated attribute that will be removed at some point.

Configuration example for Shibboleth SP

This example configures a Shibboleth SP to use the UK federation MDQ service for all entities. The SP will query the MDQ server when it needs metadata for a specific IdP, and will cache the result.

        <!-- UK federation MDQ service -->
        <MetadataProvider type="Dynamic" ignoreTransport="true">
                <Subst>http://mdq.ukfederation.org.uk/entities/$entityID</Subst>
                <MetadataFilter type="RequireValidUntil" maxValidityInterval="2592000"/>
                <MetadataFilter type="Signature" certificate="ukfederation-mdq.pem"/>
        </MetadataProvider>

Configuration example for Shibboleth SP that pre-fetches a high-value entity

One criticism of the MDQ model is that the SP must always have a connection to the MDQ server to obtain metadata just-in-time. As we saw earlier, the SP as an MDQ client will cache metadata it downloads, but the SP must initially obtain that metadata. And whilst we deploy several mechanisms to provide resilience in the MDQ service itself, we cannot guarantee 100% uptime. There is also a risk that you will lose end-to-end connectivity between your SP and the MDQ service. To mitigate against these classes of failure you can pre-fetch metadata for high value entities that your SP interoperates with.

The following example would configure your SP to pre-fetch metadata for the UK federation Test IdP and use the just-in-time MDQ service to obtain metadata for any other entity. The pre-fetch is configured using a traditional MetadataProvider with type="XML" that will download the IdP's metadata from the MDQ service at start-up and periodically afterwards. It includes the usual functions of retrying the MDQ service if it is unavailable. The example here configures an SP to refresh metadata for the UK federation Test IdP after approximately 2 hours, which is far less than the maximum validity interval of our metadata (typically 14 days). Reference documentation for the MetadataProvider is available form the Shibboleth wiki.

        <!-- UKf MDQ service -->
        <MetadataProvider type="Dynamic" ignoreTransport="true">
                <Subst>http://mdq.ukfederation.org.uk/entities/$entityID</Subst>
                <MetadataFilter type="RequireValidUntil" maxValidityInterval="2592000"/>
                <MetadataFilter type="Signature" certificate="ukfederation-mdq.pem"/>
        </MetadataProvider>

        <!-- Pre-fetches metadata for the UKf Test IdP, entityID https://test-idp.ukfederation.org.uk/idp/shibboleth, from the UK federation MDQ service -->
        <MetadataProvider type="XML" validate="true"
              uri="http://mdq.ukfederation.org.uk/entities/https:%2F%2Ftest-idp.ukfederation.org.uk%2Fidp%2Fshibboleth"
              backingFilePath="/etc/shibboleth/ukf-uk002386.xml" reloadInterval="7200">
            <MetadataFilter type="RequireValidUntil" maxValidityInterval="2419200"/>
            <MetadataFilter type="Signature" certificate="ukfederation-mdq.pem"/>
        </MetadataProvider>

Please note that if your SP interacts only with a few specified IdPs, you can just configure it to pre-fetch metadata and not have the MetadataProvider with type="Dynamic" at all.

Configuration for non-Shibboleth software

Please contact your software vendor.