SDNSF LANL Tasks LANL Research Tasks Explore parallel

SDN-SF LANL Tasks

LANL Research Tasks • Explore parallel file system networking (e. g. LNet peer credits) in order to give preferential treatment to isolated routes within the storage area network. This may be done by adding additional virtual routes to existing LNet routers. Further, LANL will expose this tuning as a service that can be invoked by a custom software-based network controller. • Explore dynamically adjusting Lustre network request schedulers to allow preferential storage operation en-queuing. Replicate SDN-SF rack at the local site and collaborate with ORNL in developing and customizing the emulation test bed. • Evaluate the role of Data Center TCP in ensuring high-performance flows can be constructed within the data center infrastructure. • Extend the concepts within Data Center TCP to high-speed networking technology. In particular, LANL plans to develop a practical reservationaware congestion control for Data Center TCP, and then extend Data Center TCP techniques to the Lustre networking (LNet) protocol to alleviate bottlenecks (e. g. the parking lot problem).

LANL Task Timetable • Year 1: I/O and File System Orchestrator module; Lustre performance optimization for intra-datacenter transfers • Year 2: I/O and FS testing; SDN-SF site installation • Year 3: I/O testing with remote computation

LANL Year 1 Overview • Frequent data movement within the data center – We’ve examined ~30 scientist allocations for LANL’s data center – 3 basic types of science: Simulation, Uncertainty quantification, and High-throughput computing – Each generates massive long-lived data sets that flow throughout the data center – (Published in workflow report w/APEX) • Goal: Develop I/O and File System Orchestrator module to improve/reserve storage performance during inter/intra-datacenter transfers

LANL Year 1 Progress To Date: Measuring existing transfers • LANL uses a cluster of file transfer agents to move data between platforms, file systems, archive, DTN staging area, etc. – Transfers use a scheduled pftool session, an MPIbased data mover • Production FTAs pftool instrumented with Darshan, an I/O tracing framework – Capture and profile all data movement between storage systems within LANL’s center

LANL’s Turquoise Enclave Platform Storage Wolf Cluster Mustang Cluster Pinto Cluster L 2 L 1 L 3 FTA Cluster I/O Backbone Network Campaign Storage Tape Archive WAN Staging

LANL’s Turquoise Enclave Platform Storage Wolf Cluster Mustang Cluster Pinto Cluster L 2 L 1 L 3 Darshan Instrumentation FTA Cluster I/O Backbone Network Campaign Storage Tape Archive WAN Staging

Data Retention Time Simulation Science Workflow Forever Sim Input Deck Initial Input Deck Campaign Analysis Data Set Checkpoint Dump 5 - 15 x per pipeline Timestep Data Set Checkpoint Dump Initial State 4 – 8 x per week 5 – 10 x per week Timestep Data Set Checkpoint Dump Temporary Sampled Data Set Data Center Transfers Γ*JMTTI Setup/Parame terize/Create Geometry Phase S 1 Job Begin Simulate Physics Phase S 2 Job End Down. Sample Post. Process Phase S 3 Phase S 4 Simulation Science Pipeline Viz Phase S 5

Data Retention Time Data-Intensive Science Workflow Shared Input Private Input Checkpoint Dump Forever Analysis Data Sets 5 - 15 x per pipeline Analysis Data Sets Checkpoint Dump Campaign 4 – 8 x per week Filebased Comm. Checkpoint Dump Temporary Data Center Transfers Generate and/or Gather Input Data HTC Analysis or UQ Simulation … Analysis … or Phase H 1 Phase U 1 Phase H 2 Phase U 2 or UQ Science Pipeline HTC Science Pipeline Phase H 3 Phase U 3

LANL Year 1: Develop orchestration • FTA cluster provides a natural mechanism to orchestrate science flows within data center – Collect data to describe quantities of flows for provisioning – Techniques for guaranteeing flow Qo. S using FTAs/scheduler/pftool • Opportunity to re-play pftool traces to measure approaches for limiting performance variability – Measure multiple pilot approaches

LANL Year 1: Develop orchestration • Candidate pilot approaches possible due to FTA control of data transfers – File mix (small/large) co-scheduling • Small files limited by MDS throughput • May still generate significant interference – Manage total number of transfer-I/O threads – Network and storage watermarking • Measurements via darshan critical – PFS modifications can be years-long efforts then longer to make it to production usage – Currently, Lnet route changes are expensive – would making it cheaper be worth the effort

Deliverables • Year 1 - 3 – Share PFTool data across complex • Anonymize data from some LANL enclaves • Year 1 – Comparison of performance isolation/maximization techniques • Year 2 – Integrate SDN rack into Darwin – Orchestrate techniques identified as valuable • Goal is to feed this information into the PFS community, so that feasible algorithms are implemented in NRS, etc. • Year 3 – Isolation techniques for remote burst buffer access – BB Software is immature, influence possible • All are based on PLFS • Chance to get small Qo. S hooks added – just need to know what we request

Closing • Questions