The Engagement and Performance Operations Center Overview and

The Engagement and Performance Operations Center: Overview and Opportunities Dr. Jennifer M. Schopf PI, EPOC Indiana University International Networks National Science Foundation Award Jason Zurawski Co-PI, EPOC ESnet / Lawrence Berkeley National Laboratory

Today’s Session • Jennifer Schopf IU • EPOC Basics, User Engagement, and Consulting Services • James Deaton, GPN • Measurement and Monitoring the GPN using Net. Sage • Jennifer Oxenford, KINBER • KINBER and EPOC • Jason Zurawski, ESnet • Managed Services and the KINBER Pilot • Everyone • Discussion and open questions 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 2

Engagement and Performance Operations Center • Joint project between Indiana University and ESnet • co-PI Jent (IU Global. NOC) and Zurawski (ESnet) • Part of CC* program for domestic science support • Program Officer: Kevin Thompson • Award #1826994, $3. 5 M over 3 years • Partnerships with regional, infrastructure, and science communities that span the NSF and DOE continuum of funding • http: //epoc. global 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 3

Why an Engagement Operations Center? • Today’s science is collaborative science • Collaborative science • Multiple partners • Multiple data sets • Many points of connection • Cross agency cooperation • With better access to data we ask harder questions • Interactive data sources change the science we do 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 4

Understanding End-to-End Performance is Hard • Lots of pieces - Host system through networks to host system • No one controls all the pieces • Unknown expectations for what performance should be • Soft failures are hard to find • Many, many points of coordination 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 5

Partners are Needed to Scale Engagement • Regional Network Partners • Large scale collaborations each which supports multiple institutions in a given geographical region • Infrastructure Partners • Collaborative teams that coordinate on special areas • Science Community Partners • Significant groups of application domain specialists • Support that spans federal funding bodies • Sharing knowledge and lessons between communities 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 8

Initial Regional Network Partners • The Indiana State Network (I-Light) • The Ohio State R&E Network (OARnet) • The Keystone Initiative for Network Based Education and Research (KINBER) • The Great Plains Network (GPN) • The Texas State R&E Network (LEARN) • The Front Range Gigapop (FRGP) 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 9

Initial Infrastructure Partners • XSEDE • Uses Campus Champions to supports a single virtual system • Campus Research Computing Consortium (Ca. RC) • Consortium of 30+ campuses to facilitate access to CI • NSF Cybersecurity Center of Excellence (CCOE) • Supports cybersecurity for NSF funded projects • Science Gateway Community Institute (SGCI) • Supports for scientists building/using data portals • Internet 2 • Supports 200+ educational, research and community members • The Quilt • Provides a central organization to share the best practices 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 10

Initial Science Community Partners • Earth Science Information Partners (ESIP) • 180+ member consortium of Earth science data and technologists • World Climate Research - International Climate Network • 1, 000 s of Earth System scientists using climate repositories • IU Grand Challenge Precision Health Initiative • Broad set of precision health applications • University of Hawai’i System Astronomy Community • 15+ astronomy facilities • Midwest Big Data Hub (MBDH) • 12 state collaboration to supports the use of data • The Open Storage Network (OSN) • Will support dozens of applications across a broad set of domains 11 © 2019, Engagement and Performance Operations Center (EPOC) 9/25/2021

Expansion • EPOC is a starting point, not a destination • Fully anticipate bringing on additional partners that provide networking, infrastructure, or with a focus on other scientific areas • No one will be turned away • Mechanisms in place to on-board new participants • Department of Energy Laboratories and Facilities • ESnet Science Engagement has a long history of interaction on addressing reported problems or assisting in evaluating existing and new science efforts • Emerging program (via EPOC and regular operation) in ‘embedding’ resources into a project/facility 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 12

EPOC Five Main Focus Areas 1. Roadside Assistance for Performance Problems and Consulting 2. Application Deep Dives 3. Network Analysis (Net. Sage) 4. Services “in a box” (DMZ, testpoint in a box, etc) 5. Training 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 13

EPOC Five Main Focus Areas 1. Roadside Assistance for Performance Problems and Consulting 2. Application Deep Dives 3. Network Analysis (Net. Sage) 4. Services “in a box” (DMZ, testpoint in a box, etc) 5. Training 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 14

Something’s Broken • “My file transfers were working fine last week… but this morning nothing runs? ” • My side? • Their side? • Something in between? 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 15

EPOC Roadside Assistance – epoc@iu. edu • “This file transfer worked last week, but it doesn’t anymore? ” • Think of this like a flat tire, crash repair • EPOC is a collaboration of 3 teams already supporting this • ESnet Science Engagement (engage@es. net) • IN@IU • IRNC NOC Performance Engagement Team (PET) 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 16

Scaling Engagement – Operations Center Approach • Central point of contact: epoc@iu. edu • Operations Center Approach • Ticketing, Service center framework • Dedicated team- NOT just volunteers with found cycles • Coordination • Ties to the community • Take advantage of the “coalition of the willing” • Strategic partnerships are critical 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 17

Roadside Assistance Process • Anyone can submit • Don’t have to be NSF funded, specific university • Contact epoc@iu. edu • Within 24 hours, gets triaged • Some initial investigation to verify the issues • A Case Manager and Lead Engineer are assigned • Shareable infrastructure set up 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 18

Problem: Many orgs involved results in many tickets, which no one has all the info about • Solution; Local experts on both sides pulled in early • Partner with campus champions, Ca. RC, XSEDE, Regionals etc • Solution: Folder for all engineering docs shared across orgs • No single ticketing system is open to everyone • Folder had docs, lists of tickets (and contacts), maps/diagrams, etc • Anyone working on the case gets access 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 19

Problem: Submitter often doesn’t know status • Solution: online shareable “Customer Case Document” • Written for non-engineers • Updated frequently, generally twice a week • General overview and goal of the case • Contact points (Case manager) • Current status and next steps listed 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 20

Troubleshooting • Understanding the end-to-end path of the data transfer • Use public test and validation services to identify potential issues • perf. SONAR nodes (http: //perfsonar. net) • Router proxies) to identify potential issues • Coordination with a wide set of engineering help staff along the data transfer path 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 21

Outcomes • Ticket stays open until reporter is satisfied with result • Write ups follow • Engineering guidance added to http: //fasterdata. es. net • Sharing how to solve problems is as important as solving them for us! 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 22

Consulting • Lighter weight than a full roadside assistance • Submission process same – contact EPOC@iu. edu Suggestions for DTNs, DMZs, firewalls and DMZs, Data projections for science fields Expected (real) performance between two sites Advice on how to conduct a performance assessment of a network and applications • Or others! • • • Similar operations approach • Results/suggestions will be added to fasterdata. es. net over time 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 23

EPOC Five Main Focus Areas 1. Roadside Assistance for Performance Problems and Consulting 2. Application Deep Dives 3. Network Analysis (Net. Sage) 4. Services “in a box” (DMZ, testpoint in a box, etc) 5. Training 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 24

EPOC Deep Dives • Think of this as regular maintenance, oil change, or planning to buy a new car • Based on seminal work by ESnet to develop Scientific Case Studies • Walk through science workflow with the actual scientists • Way to understand needs and planning • Often identifies issues that have nothing to do with networks, and everything to do with sociology 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 25

Anatomy of a Deep Dive • Two primary components • Narrative • Data Estimation • Built on the ESnet “Requirements Review” template • Contains helper text to guide what is wanted • Items that make sense for DOE/ESnet, may not make sense for another institution – we’re modifying this as we go • https: //fasterdata. es. net/science-dmz/science-and-networkrequirements-review/ 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 26

We Walk Through Scientific Components… 1. Background information • Brief overview of the facility, nature of the science being performed 2. Collaborators • Identify people and institutions that a science group interacts with 3. Instrumentation • Local and remote scientific instruments and facilities. 4. Process of Science • Explain ‘a day in the life’ of the science group • Should tie together the instruments, the people, and the resources 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 27

And Also More Technical Aspects… • 5. Software Infrastructure • 6. Network and Data Architecture • 7. Cloud Services • 8. Outstanding Issues and Pain Points Local and regional IT staff are critical to these parts, and help form valuable partnerships that may not exist, or could use strengthening 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 28

When This Is Done • Better understanding of the science, data movement, who’s using what pieces, dependencies, and time frames • Identification of bottlenecks or pain points becomes more obvious • Relationships build between layers (engineering, science, administration) • Clear path toward improvement and success 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 29

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY What EPOC’s Deployment of Net. Sage did for GPN James Deaton Executive Director jed@greatplains. net

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY ● The Great Plains Network (GPN) is a non-profit consortium aggregating networks through Giga. Po. P connections while advocating research on behalf of universities and community innovators across the Midwest and Great Plains who seek collaboration, cyberinfrastructure and support for big data and big ideas, at the speed of the modern Internet. ● Over two dozen universities ○ Across 9 states ■ More than 20 years of collaborating in research and education networking

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY Connected via 6 state networks and a RON ● ● ● ARE-ON Kan. REN MOREnet Network Nebraska One. Net SD-REED ● BOREAS

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY Goals with Net. Sage and More Broadly via work with EPOC ● ● ● Deeper understanding of R&E data movement across the region Engaging campus IT and state network in both passive and active approaches of improving data movement Increasing awareness of the rich set of science drivers within the region

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY Net. Sage via EPOC Net. Sage is an open privacy-aware network measurement, analysis, and visualization service designed to address the needs of today's international networks. Modern science is increasingly data-driven and collaborative in nature, producing petabytes of data that can be shared by tens to thousands of scientists all over the world. (from http: //www. netsage. global/)

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY

CONNECTING NETWORKS • ADVOCATING RESEARCH • CULTIVATING COMMUNITY Summary • Gained insight into patterns and projects utilizing the connections • Raised interest and commitment in sharing and analyzing flow data • Inspiring future contributions to the Science Registry • Encouraged conversations that have led to much greater amounts of network metric sharing across multiple regionals

The Engagement and Performance Operations Center: Overview and Opportunities Jennifer Oxenford, Director of Community Engagement, KINBER Internet 2 2019 Global Summit March 8, 2019

KINBER Overview www. kinber. org

KINBER CC* perf. SONAR Activities • KINBER NSF CC*IIE Region Accelerating the Adoption of Campus Cyberinfrastructure Technologies in Pennsylvania Grant (Award 1440699) • As part of this grant, KINBER offered a perf. SONAR pilot program from 2015 – 2018 to several schools in PA. • Pilot provided pre-configured perf. SONAR platforms to six campuses in PA in order to understand if pre-configured platforms lead to faster integration into the campus infrastructure. • Participating campuses included: Franklin & Marshall College, Swarthmore College, CMU, Villanova, Harrisburg Area Community College, and The Hill School • Representative campuses included K 12, R 1, and small and underresourced schools. • KINBER perf. SONAR Pilot provided initial key findings on perf. SONAR deployment in region.

KINBER CC* perf. SONAR Activities • Additional KINBER perf. SONAR outreach and engagement activities have included: – – perf. SONAR Workshops perf. SONAR Webinars perf. SONAR Training at KINBER annual conference perf. SONAR Demo and Awareness area at annual conference

KINBER CC* perf. SONAR Activities

KINBER perf. SONAR Activities KINBER Penn. REN perf. SONAR Topology

KINBER and Eastern Regional Network (ERN) Understand support end-to-end performance and data movement in support of research in local, regional and national environments. Project - NSF Funded DIBBS VDC Pennsylvania - Penn. REN ERN

KINBER EPOC Activities • Pilot Partner working with EPOC. • Initial project to offer “perf. SONAR in a Box” to campuses in PA. • Starting first with Franklin and Marshall College, a CC*DNI recipient, and initial KINBER perf. SONAR pilot site, to launch, test, and refine the model. • Goal to scale the “perf. SONAR in a Box” model to additional colleges and universities in PA and then grow to additional offerings in partnership with EPOC.

Services(!) In a Box (!!) Dr. Jennifer M. Schopf PI, EPOC Indiana University International Networks National Science Foundation Award #1826994 Jason Zurawski Co-PI, EPOC ESnet / Lawrence Berkeley National Laboratory

How to increase adoption/deliver value to partners of smaller size? • Observations by IU/ESnet after several years of community events (e. g. OIN Workshops): • Lots of “interest” in new technologies • May not know of (immediate) use cases • Resources (time/$) to design/specify/build is hard to come by • Easier to pay for a service that is build for you, or maintain something someone else builds • “Why would I need that? ” • Unfunded mandates have a way of being ignored 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 58

What is a “Service-in-a-Box”? • Basic idea: • Only large facilities with dedicated funding can afford the time/effort to design/install/operate/maintain a dedicated science infrastructure • Ameliorate the costs of design/install at a higher level (e. g. regional network). • Create infrastructure that can be delivered as a service • Operation can be local or regional (offer flexibility based on the environment and resources available) • Develop a business model that facilitates cost recovery and upgrade schedules 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 59

What is a “Service-in-a-Box”? • Goals • Offer a way for traditionally smaller/less resourced facilities to use emerging technology to support scientific use cases • Create new paths for regionals to interact with/learn about/support scientific use cases • Reduce cost for service deployment and operation • Increase adoption/improve outcomes on a larger scale • E. g. Turn “I can’t do that” into “How did I live without this? ” 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 60

Services “In a Box” • Typical initial setup 9/25/2021 • Larger site that can invest © 2019, Engagement and Performance Operations Center (EPOC) • Proposed regional network operation 61

perf. SONAR in a Box @ KINBER/F&M • General case: work with small/medium campus and regional • Assistance on design, deployment, and support • We do the research on how to do it – regional implements • Help with trouble shooting the problems that are found • We’re trying to understand the *real* cost of offering this • EPOC will do this at no charge, after EPOC ends there choices to be made regarding business model • Measure the usage (qualitative and quantitative) • Try to gauge how to scale based on pilot activities 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 62

SIAB Audience • Deployer • Regional network, national laboratory, resource provider, or other centrally located entity • Has a staff that is available to handle purchasing, design, implementation, operation of advanced technology • “User” • Smaller (not to be pejorative) facility that is already a customer of the deployer (e. g. network service, resource consumer, etc. ) • Doesn’t have the above resources to make an advanced service happen • Has a use case (Single? Multiple? ) that could benefit • Not afraid to treat this as an experiment (since the alternative is nada) 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 63

SIAB Examples • Assuming the following are true • Limited IT Staff & Budget • Interest in services, but not critical 24/7/365 need for them • Measurement and Monitoring Service • Want the ability to understand performance in/out of campus to remote locations && Assistance in fixing local network design/use areas of friction • Data Transmission Service • User with irregular (e. g. not daily) bulk data movement to/from well known scientific facility 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 64

SIAB Locality • Assumptions (? ) • Regional/infrastructure providers and their customers have to ‘touch’ each other somewhere • Someplace on the campus, someplace centrally located where others may be present (e. g. put things where that touch point is) • Hardware specification is time consuming • And most come to the same (good) answers on needs/capability (e. g. there aren’t many ways to skin the DTN/p. S/DMZ cat) • There is power in bulk purchase • “You buy enough meat, they’ll give you anything”, Cosmo Kramer 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 65

SIAB Locality – perf. SONAR (EX) 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 66

SIAB Locality – DTN (EX) 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 67

Anticipated Offerings • perf. SONAR • Science DMZ • Deployment of regional hardware to support campus high-performance needs • Data Transfer Hardware/Software • Rental or co-location of capable hardware and storage • Network Capacity Testing • Use of 10 G/40 G/100 G/(400 G? ) hardware to prove out new circuits, or debug old ones • Security • Regionally deployed IDS infrastructure that doesn’t impact high performance networking. 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 68

What is the End Game? • Wider adoption of scientific networking ideas / improved scientific outcomes (of course …) • Understated Goals: • Want to understand the ‘real’ costs • Want to understand the ‘real’ barriers • Want to see basic things turned into commodity services • In the general case we see this all the time, and adoption varies • Power users will move on, regular users will move in 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 69

Understanding Costs • “I can’t afford this” • Expertise (e. g. people) are expensive • Design • Implementation/Install • ‘watching’ • Hardware has a fixed cost that gets smaller over time. Ages out eventually. • User Support & economies of scale 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 70

Understanding Barriers • For a deployer: • What is the base amount of expected use to do a service? • Is there a similar max value – e. g. too many users and service suffers? • For a user: • Rent, Rent to own, buy • Sharing operational resources is well demonstrated 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 71

EPOC Questions – For now, or Later • Is the effort/cost by one party to simplify this for others ‘worth’ it? • What is the inflection point • Will it be used? • When is ‘too little’ too little. When is ‘too much’ too much? • What is missing? • Technology is ‘easy’ • Policy is not • Sociology is hard 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) 72

Services(!) In a Box (!!) Dr. Jennifer M. Schopf PI, EPOC Indiana University International Networks National Science Foundation Award #1826994 Jason Zurawski Co-PI, EPOC ESnet / Lawrence Berkeley National Laboratory

Take Aways • EPOC is an NSF-funded operations center to help scale science engagement and problem resolution • Single point of contact to help with end-to-end performance issues • epoc@iu. edu • More about EPOC: • http: //epoc. global • Jennifer Schopf, jmschopf@iu. edu • Jason Zurawski, zurawski@es. net 9/25/2021 © 2019, Engagement and Performance Operations Center (EPOC) National Science Foundation Award 74