UNICORE and S3 configuration
UNICORE installation
For serving S3 requests through UNICORE, only a small installation of UNICORE is necessary, comprising the UNICORE Gateway and the UNICORE/X components. The setup described here, for integrating the WOS S3 storage with the Human Brain Project's UNICORE infrastructure, is depicted in the picture below. Authentication is done via Unity or OIDC. The UNICORE installation will only accept users authenticated via the HBP Unified Portal. The services should also be available via the UNICORE REST API.
The two UNICORE components can be installed from the core server bundle at [1] or the rpm/deb packages found at [2]. On Debian, for example, download the latest packages and install them:
wget http://unicore-dev.zam.kfa-juelich.de/release-candidates/core/7.2.0-packages/deb/unicore-gateway_7.2.0-2_all.deb wget http://unicore-dev.zam.kfa-juelich.de/release-candidates/core/7.2.0-packages/deb/unicore-unicorex_7.2.0-2_all.deb dpkg -i unicore-gateway_7.2.0-2_all.deb dpkg -i unicore-unicorex_7.2.0-2_all.deb
The components need grid server certificates. Our host unicore.data.kit.edu has a server certificate signed by the gridka CA [3].
UNICORE Gateway setup
The following settings are needed:
- enabling certificate-less user access, in file /etc/unicore/gateway/gateway.properties:
gateway.httpServer.requireClientAuthn=false
- setting the gateway credential and truststore locations, in file /etc/unicore/gateway/security.properties:
#Set the trust store here gateway.truststore.type=directory gateway.truststore.directoryLocations.1=/etc/unicore/gateway/truststore/*.pem gateway.truststore.crlLocations.1=/etc/unicore/gateway/truststore/*.crl # and the gateway credential gateway.credential.path=/etc/unicore/certs/unicore.data.kit.edu.pem
- configuring the sites in file /etc/unicore/gateway/connections.properties:
DEFAULT-SITE = https://unicore.data.kit.edu:7777
Starting and stopping the service can be done by running the scripts:
unicore-gateway-start.sh unicore-gateway-stop.sh
The gateway needs to be accessible from the internet.
UNICORE/X setup
The following two files need to be edited for setting up the UNICORE/X server:
/etc/unicore/unicorex/uas.config - for general settings, storages /etc/unicore/unicorex/wsrflite.xml - for services to be deployed, SSL settings
First, set the gateway URL and identity, as well as the UNICORE/X hostname, in file wsrflite.xml:
<property name="container.baseurl" value="https://unicore.data.kit.edu:8080/DEFAULT-SITE/services"/> <property name="container.security.gateway.certificate" value="/etc/unicore/certs/unicore.data.kit.edu.pem"/> <property name="container.host" value="unicore.data.kit.edu"/> <property name="container.port" value="7777"/>
Then, set the server identity and truststore, in wsrflite.xml:
<property name="container.security.credential.path" value="/etc/unicore/certs/unicore.data.kit.edu.pem"/> <property name="container.security.truststore.directoryLocations.1" value="/etc/unicore/unicorex/truststore/*.pem"/> <property name="container.security.truststore.crlLocations.1" value="/etc/unicore/unicorex/truststore/*.crl"/>
Let's move now to the authentication via Unity. Configure UNICORE to trust the Unity server by pointing to it's public key (unity.pem) in PEM format (also in wsrflite.xml):
<property name="container.security.trustedAssertionIssuers.type" value="directory" /> <property name="container.security.trustedAssertionIssuers.directoryLocations.1" value="/etc/unicore/unicorex/unity/unity.pem" />
To enable the UNICORE REST API, edit wsrflite.xml:
<service name="core" type="rest" enabled="true"> <interface class=""/> <implementation class="de.fzj.unicore.uas.rest.CoreServices"/> </service>
To configure access control for the REST services, make sure that the file "02restServices.xml" in the "unicorex/conf/xacml2Policies" directory is present (or copy it from the core server bundle). Next, we will configure the authentication for the REST services by editing the uas.config file:
# Configured authenticators and their order container.security.rest.authentication.order=UNITY-OIDC # Unity with OIDC container.security.rest.authentication.UNITY-OIDC.class=eu.unicore.services.rest.security.UnityOAuthAuthenticator container.security.rest.authentication.UNITY-OIDC.address=https://hbp-unic.fz-juelich.de:7112/UNITY/unicore-soapidp-oidc/saml2unicoreidp-soap/AuthenticationService container.security.rest.authentication.UNITY-OIDC.validate=true
We will disable the HOME storage, which is enabled by default, and comment out all settings related to it:
coreServices.targetsystem.homeDisable=true # Home Storage (these are minimal config options, see manual for info) #coreServices.targetsystem.storage.0.name=Home #coreServices.targetsystem.storage.0.type=HOME #coreServices.targetsystem.storage.0.protocols=BFT
Adding S3 as storage backend
An existing S3 storage can be added as backend for the StorageFactory service in UNICORE. In our set-up, this is the only type of storage supported by the StorageFactoryService. You need to provide the endpoint, a description and optionally allow users to overwrite the endpoint when they create a new storage through the StorageFactory service.
coreServices.targetsystem.uniqueStorageIds=true # Available storage types coreServices.sms.factory.storagetypes=S3 # S3 storage configuration coreServices.sms.factory.S3.description=S3 at KIT coreServices.sms.factory.S3.class=de.fzj.unicore.uas.jclouds.s3.S3StorageImpl coreServices.sms.factory.S3.type=CUSTOM coreServices.sms.factory.S3.infoProviderClass=de.fzj.unicore.uas.jclouds.s3.S3InfoProvider coreServices.sms.factory.S3.path=/ coreServices.sms.factory.S3.cleanup=false coreServices.sms.factory.S3.protocols=BFT # the next four settings depend on your S3 backend # provider is "s3" or "aws-s3" coreServices.sms.factory.S3.settings.provider=s3 # endpoint of the S3 coreServices.sms.factory.S3.settings.endpoint=https://s3.data.kit.edu # may user overwrite endpoint and provider? OPTIONAL, false by default coreServices.sms.factory.S3.settings.allowUserdefinedEndpoint=true
UNICORE offers several ways to configure the S3 authentication keys. We chose to do it via UNICORE's map file attribute sources because it offers the best single-sign-on experience for the user by managing everything server-side.
First make sure that the following lines are present in your uas.config file:
container.security.attributes.FILE.class=eu.unicore.uas.security.file.FileAttributeSource container.security.attributes.FILE.file=/etc/unicore/unicorex/simpleuudb container.security.attributes.FILE.matching=strict
Then edit the simpleuudb file and add an entry for each user that looks like this:
<fileAttributeSource>
<entry key="...user's DN...">
<attribute name="role">
<value>user</value>
</attribute>
<attribute name="xlogin">
<value>...local uid...</value>
</attribute>
<attribute name="s3.accessKey">
<value>...access key...</value>
</attribute>
<attribute name="s3.secretKey">
<value>...secret key...</value>
</attribute>
</entry>
</fileAttributeSource>
The DNs for the UNITY/OIDC users currently look like this:
CN=Name Surname 123456, O=HBP
