XRoot D Release 4 5 And Beyond XRoot

XRoot. D Release 4. 5 And Beyond XRoot. D Workshop Tokyo Stanford University/SLAC November 10, 2016 Andrew Hanushevsky, SLAC http: //xrootd. org

November Release 4. 5 Highlights Request signing Dual stack networking option Client reporting of release at login time Separate negative cache timeout in cmsd Allow file names with spaces Allow host names starting with digit Automatic URL prefixing Zip archive support November 10, 2016 XRoot. D Workshop Tokyo 2

Request Signing Protects XRoot. D servers from bad actors Request Delete Response File x Client Server Client can cryptographically sign requests Sever verifies request came from the same authenticated client Bad actor problem avoided November 10, 2016 XRoot. D Workshop Tokyo 3

Enabling Request Signing Request signing gives you peace of mind n When allowing R/W access to the server n Especially on the WAN Server configuration option n sec. level {all|local|remote] [relaxed] level n all n applies level to local and remote client local and remote provide split options n relaxed n November 10, 2016 provides a migration path It requires signing only for 4. 5 and up clients XRoot. D Workshop Tokyo 4

Request Signing Levels none n The default compatible n Only destructive operations n Is compatible with R/O access for old clients standard | intense | pedantic n Each requires more operations to be signed November 10, 2016 XRoot. D Workshop Tokyo 5

Request Verification by Level Operation admin auth bind chmod close decrypt dirlist endsess getfile locate login mkdir mv open read open Write ping prepare protocol putfile query special readv rm rmdir set special sigver statx sync truncate verifyw write November 10, 2016 Compatible verified --------verified --verified ----verified ----- Standard verified --------verified verified --------verified ----verified ----- XRoot. D Workshop Tokyo Intense verified --verified ----verified verified ------verified verified ----verified Pedantic verified --verified verified --verified verified verified --verified verified 6

Dual Stack Networking Option Pre-4. 3 clients may report as IPv 6 only n This is a big headache for IPv 4 -only servers New server-side option xrd. network assumev 4 n Server will assume client has IPv 4 n Only applied to 4. 4 or older clients n n Server November 10, 2016 can’t detect a client’s release level until… XRoot. D Workshop Tokyo 7

Client Release Reporting Client will report it’s release level n Happens at login time This allows future server-side bypasses When a particular release has a bug n Upgrading server can bypass old client bugs n Make client migration much easier n November 10, 2016 XRoot. D Workshop Tokyo 8

Separate Negative Cache Timeout The cmsd caches file location n Implicitly caches missing files as well Default is 8 hours n Incorrectly missing files will be missing 8 hours n Unless data server updates the cahe New server configuration option n cms. fxhold noloc ntime[h|m|s] n ntime November 10, 2016 expiration for cached missing files only XRoot. D Workshop Tokyo 9

Allow File Names With Spaces XRoot. D allows spaces in file names n Every operation except rename Protocol extension covers rename now n All 4. 5 plus clients use the protocol extension November 10, 2016 XRoot. D Workshop Tokyo 10

Host Names Starting With Digits XRoot. D originally adhered to RFC 952 n Hostnames may contain letters, digits, dashes n But may not start with a digit Now it adheres to RFC 1123 n Supplants RFC 952 allows 1 st char as a digit Required for auto-generated hostnames n Typically a problem for VM’s and containers November 10, 2016 XRoot. D Workshop Tokyo 11

Automatic URL Prefixing Required by fully firewalled sites Outside World Forwarding Proxy Server Inside Client Much like HTTP proxy Open(“root: //x//file”) Automatically converted to Open(“root: //proxy//root: //x//file”) Done as a configurable client plug-in Usable by 4. 0 and above clients Will address multi-tenant sites later November 10, 2016 XRoot. D Workshop Tokyo 12

Zip Archive Support Fully implemented as a client feature n Allows extraction of file from archive n No n need to transmit the whole archive Covered by Elvin’s talk November 10, 2016 XRoot. D Workshop Tokyo 13

January Release 4. 6 Async I/O Proxy Handling Disk Caching Proxy Possible other addition n Extreme (multi-source) copy Perhaps others to be determined November 10, 2016 XRoot. D Workshop Tokyo 14

Async I/O Proxy Handling Proxy now handles async I/O requests n Previously converted async to sync I/O Motivation Improve streaming performance of xrdcp n Improve Disk Caching Proxy performance n n Certain background operations (e. g. pre-fetch) May require tuning to get best performance See xrootd. async directive November 10, 2016 XRoot. D Workshop Tokyo 15

Disk Caching Proxy File or block level caching November 10, 2016 XRoot. D Workshop Tokyo 16

Typical Disk Caching Proxy Uses Speed up Remote XRoot. D Remote Access Clusters Self-Managed SE’s Caching Proxy Local Clients Caching Proxy November 10, 2016 XRoot. D Server SSD XRoot. D Workshop Tokyo XRoot. D Cluster HDFS Speed up Random Access FS Speed up HD Access 17

More On Disk Caching Proxy High potential to solve vexing problems Reduced remote latency for user analysis n Just in time data n n Avoids n pre-placement delays Optimized disk space utilization Cached data access via xroot and http Currently being tested at scale n MWT 2, SLAC, University of Notre Dame November 10, 2016 XRoot. D Workshop Tokyo 18

Disk Caching Proxy Caveats Needs installation of tcmalloc or jemalloc n Avoids memory growth problems in glibc It’s very easy to overload the proxy n Lager sites should consider caching clusters n Two or more proxies clustered together n Fully supported upon release November 10, 2016 XRoot. D Workshop Tokyo 19

Disk Caching Proxy Deployment Target sites without ATLAS managed disk Opportunistic sites n OSG diskless sites n Pacific Research Platform sites (NSF funded program) n n NASA, n NREN, U Washington, UC School System Includes University California LHC sites (3 ATLAS T 3’s) n Interconnected n November 10, 2016 via Cal. REN, ESNET, & Pacific Wave Sites and networking may expand XRoot. D Workshop Tokyo 20

In The Pipeline for 4. 7 or 4. 8 Space quotas Scalable Service Interface November 10, 2016 XRoot. D Workshop Tokyo 21

Space Quotas Experiments want write access to XRoot. D Current development Non-hierarchical logical path based quotas n Quota is soft n n Roughly within a specified resolution n Periodic recalibration n Will be a plug-in so can be replaced This is much harder than imagined! November 10, 2016 XRoot. D Workshop Tokyo 22

Scalable Service Interface Framework for distributed services n Builds on robust & recoverable XRoot. D n Uses a remote object execution model Current deployed for LSST qserv n Distributed unshared my. SQL servers n Successfully being used with 100’s of nodes API is still being refined n Will be released when finalized November 10, 2016 XRoot. D Workshop Tokyo 23

Future Enhancements (not yet set) Multi-source copy Multi-Source load balancing client Eliminating cmsd write lookup delays Tracking file ownership Eliminating 64 -node limit (exploratory) HTTP 2 plug-in November 10, 2016 XRoot. D Workshop Tokyo 24

Multi-Source Copy Implement the –sources xrdcp option XRoot. D Source 1 XRoot. D Source 2 XRoot. D Source 3 XRoot. D Source 4 xrdcp –sources 4 source target The source can be a redirector or metalink I/O automatically balanced across sources n Advanced algorithm to avoid ending tail n November 10, 2016 XRoot. D Workshop Tokyo 25

Multi-Source Load Balancing Client Similar to xrdcp but with a big twist XRoot. D Source 1 XRoot. D Source 2 XRoot. D Source 3 XRoot. D Source 4 TFile. open(source); The source can be a redirector or metalink n A new source is added only if current one slow n Can n November 10, 2016 bounce around multiple sources Determines by real-time performance metrics XRoot. D Workshop Tokyo 26

The Missing File Problem Application xroot Server xroot Client Linux Client Machine open(“/foo”); 2 Nope! Linux Redirector 1 Linux Server Machine A Who has /foo? Server Machine R 5 File deemed not to exist if there is no response after 5 seconds! OK for read access not so much for file creation. November 10, 2016 Data Files xroot Server xrdcp root: //R//foo /tmp /foo Data Files xroot Server Linux Server Machine B XRoot. D Workshop Tokyo 27

Eliminating cmsd Write Lookup Delay The cmsd uses a no response model n No response -> file does not exist Extremely scalable for analysis use case n Usually always looking for existing files Not so good for creating files n A small change in protocol can fix this Required for efficient handling of experiments desire for writable clusters November 10, 2016 XRoot. D Workshop Tokyo 28

Tracking File Ownership UID/GID tracking of ownership n Available for certain authentication methods n GSI with a gridmap file n Kerberos (in domain only) n Simple Shared Secret n Unix n Must start XRoot. D as root n Security n considerations abound May allow of uid/gid based quotas November 10, 2016 XRoot. D Workshop Tokyo 29

New Third Party Transfer New 3 rd Party Transfer n Plan to use forwardable credentials n X. 509 n Allows almost universal 3 rd party access n Only n n (i. e. Grid Certificates) one of three parties needs to support it The File Residency Manager already does this Will coexist with current mechanism November 10, 2016 XRoot. D Workshop Tokyo 30

Eliminating 64 -Node Limit I xrootd cmsd Manager (Root Node) cmsd xrootd 641 = 64 cmsd 642 = 4096 xrootd xrootd cmsd cmsd November 10, 2016 xrootd 643 = 262144 Supervisors (Interior Nodes) xrootd cmsd 644 = 16777216 XRoot. D Workshop Tokyo xrootd cmsd Data Server (Leaf Nodes) 31

Eliminating 64 -Node Limit II A B 64 tree architecture is generally ideal n Fast scaling and highly parallel file search But it’s cumbersome for large clusters n Need to deploy sufficient supervisor nodes Exploring different type of trees n B 128 B 256 B 512 etc Parallelism is the biggest stumbling block n However, it would simplify configuration November 10, 2016 XRoot. D Workshop Tokyo 32

That’s All! What’s Your Wish List? November 10, 2016 XRoot. D Workshop Tokyo 33