ARCS Data Fabric Pauline Mak pauline makarcs org

ARCS Data Fabric Pauline Mak pauline. mak@arcs. org. au ARCS Data Services Pauline Mak

Outline • Introduction to the ARCS Data Fabric • Migration from SRB to i. RODS • Davis • Hermes Pauline Mak

ARCS Data Fabric Pauline Mak

Why SRB? • In June 2008, i. RODS lacked - Full GSI Authentication - Federation • Now migrating to i. RODS 2. 0 - Supports both GSI and federation - Open-sourced - Microservices and Rules Engine Pauline Mak

Migration • Modified migration script handles: - Users - SRB Objects and Collections - Metadata - Groups • Does not migrate - Permissions - User DNs Pauline Mak

Migration • Moving other processes from SRB to i. RODS - Automatic account creation - Usage statistics - User sync Pauline Mak

Automatic Account (SRB) MCAT Checks MCAT for user with the same DN srb. tpac. org. au (patched SRB server) If no account matches, calls external scripts to generate account User attempts to login with a valid certificate Pauline Mak

Automatic Accounts (i. RODS) 5. The server retrieves new username from script output (or query ICAT by DN) and logs in i. RODS Server 1. Client connects using a certificate 2. Fires the rule if the DN is unknown ac. Get. User. By. DN 3. Executes external script (Create. User) Pauline Mak ICAT 4. Create. User generates a new account

Shibboleth & SLCS 2. Forwards user to institution's Shibboleth login page Short Lived Certificate Server (SLCS) TPAC Identify Provider (Id. P) 4. Forwards result back to SLCS server 1. Connects to SLCS server to select Id. P 3. User enters Id. P username and password Pauline Mak 5. (If valid) Creates a certificate and sends it back to the user

Usage Scripts (SRB) srb. tpac. org. au srb. sapac. edu. au Each zone runs a script daily that will query MCAT for usage info Uploads usage XML file to a central location in SRB Central collection zone srb. hpsc. uq. edu. au My. SQL DB Pauline Mak Use Scommands to collect users and resource info, then ingest XML data into DB status. arcs. or g. au

Usage Scripts (i. RODS) srb. tpac. org. au srb. sapac. edu. au Each zone runs a script daily that will query ICAT for usage info Uploads usage XML file to a central location in SRB Central collection zone srb. hpsc. uq. edu. au Ingests XML data into DB My. SQL DB Pauline Mak status. arcs. or g. au

status. arcs. org. au Pauline Mak

Migration – Zone sync • We only sync users • SRB - Szonesync: executed hourly - Federation can lag for up to an hour • i. RODS - Executing rules immediately when a user is added Pauline Mak

Davis • Web. DAV • Supports SRB and i. RODS • Browser and client mode • Supports Shibboleth • Multi-threaded and resumable downloads • Simplifies access Pauline Mak

Davis Architecture SRB/i. RODS Jargon API Browser Mode Handler Web Browser Authentication Processor Web. DAV Handler Web. DAV Client Pauline Mak SLCS Server Id. P

Davis – Modes Pauline Mak

Davis - Permissions Pauline Mak

Davis - Metadata Pauline Mak

Hermes/commons-vfs-grid • Originally from JCU (ARCHER project) – Mathew Wyatt • Commons-vfs - local, SFTP, Web. DAV • Commons-vfs-grid: - grid-related protocols - Grid. FTP contributed by David Meredith Pauline Mak

Hermes Architecture Local proxy SLCS + Shib My. Proxy S/FTP SRBi. RODS Grid. FTP JSCH Jargon Globus Protocol specific libraries GSIProxy. Manager commons-vfs(-grid) GSI Based Credentials Account File. Object Connectors Pauline Mak

Hermes – Browser Pauline Mak

Hermes - Permission Pauline Mak

Hermes - Metadata Pauline Mak

Hermes – metadata search Pauline Mak

Questions? Pauline Mak

Links • http: //www. arcs. org. au (ARCS) • http: //www. tpac. org. au (TPAC) • http: //projects. arcs. org. au/trac/davis/ (Davis) • http: //projects. arcs. org. au/trac/commons-vfs-grid/ (Hermes) • http: //projects. arcs. org. au/trac/systems/wiki/Data. S ervices/SRB • http: //projects. arcs. org. au/trac/systems/wiki/Data. S ervices/i. RODS Pauline Mak