emulab net Current and Future An Emulation Testbed

emulab. net Current and Future: An Emulation Testbed for Networks and Distributed Systems Jay Lepreau University of Utah December 12, 2001

The Main Players l Undergrads – Chris Alfeld, Chad Barb l Grads – Dave Andersen, Shashi Guruprasad, Abhijeet Joglekar, Indrajeet Kumar, Mac Newbold l Staff – Mike Hibler, Rob Ricci, Leigh Stoller, Kirk Webb l Alumni – Various

What? l A configurable Internet emulator in a room – Today: 328 nodes, 1646 cables, 4 x BFS (switch) – virtualizable topology, links, software l l Bare hardware with lots of tools An instrument for experimental CS research Universally available to any remote experimenter Simple to use

What’s a Node? l Physical hardware: PCs, Strong. ARMs l Virtual node: – Router (network emulation) – Host, middlebox (distributed system) l Future physical hardware: IXP 1200 +

Why? l l l “We evaluated our system on five nodes. ” -job talk from university with 300 -node cluster “We evaluated our Web proxy design with 10 clients on 100 Mbit ethernet. ” “Simulation results indicate. . . ” “Memory and CPU demands on the individual nodes were not measured, but we believe will be modest. ” “The authors ignore interrupt handling overhead in their evaluation, which likely dominates all other costs. ” “Resource control remains an open problem. ”

Why 2 l “You have to know the right people to get access to the cluster. ” l “The cluster is hard to use. ” l “<Experimental network X> runs Free. BSD 2. 2. x. ” l l “October’s schedule for <experimental network Y> is…” “<Experimental network Z> is tunneled through the Internet. ”

Complementary to Other Experimental Environments l Simulation – Fast prototyping, easy to use, but less realistic l Small static testbeds – Real hardware and software, but hard to configure and maintain, and lack scale l Live networks – Realistic, but hard to control, measure, or reproduce results emulab complements and also helps validate these environments

Internet Web/DB/SNMP Switch Mgmt Users Power. Cntl Control Switch/Router Serial Sharks PC 168 Sharks PC 160 “Programmable Patch Panel”

Experiment Creation Process

Zoom In: One Node

Fundamental Leverage: l Extremely Configurable l Easy to Use

Key Design Aspects l Allow experimenter complete control – Configurable link bandwidth, latency, and loss rates, via transparently interposed “traffic shaping” nodes that provide WAN emulation l … but provide fast tools for common cases – OS’s, state mgmt tools, IP, batch, . . . – Disk loading – 6 GB disk image Free. BSD+Linux • Unicast tool: 88 seconds to load • Multicast tool: 40 nodes simultaneously in < 5 minutes l Virtualization – of all experimenter-visible resources – node names, network interface names, network addrs – Allows swapin/swapout, easily scriptable

Key Design Aspects (cont’d) l Flexible, extensible, powerful allocation algorithm – Matches desired “virtual” topology to currently available physical resources l Persistent state maintenance: – none on nodes, all in database – work from known state at boot time l l Familiar, powerful, extensible configuration language: ns Separate, isolated control network

Obligatory Pictures

Then

Now

A Few Research Issues and Challenges l Network management of unknown and untrusted entities l Security (root!) l Scheduling of experiments l Calibration, validation, and scaling l Artifact detection and control l NP-hard virtual --> physical mapping problem l Providing a reasonable user interface l ….

How To Use It. . . l Submit ns script via web form l Relax while emulab … – – – – – l Generates config from script & stores in DB Maps specified virtual topology to physical nodes Allocate resources Provides user accounts for node access Assigns IP addresses and host names Configures VLANs Loads disks, reboots nodes, configures OSs Yet more odds and ends. . . Runs experiment Reports results Takes ~3 min to set up 25 nodes

An “Experiment” l emulab’s central operational entity l Directly generated by an ns script, l l … then represented entirely by database state Steps: Web, compile ns script, map, allocate, provide access, assign IP addrs, host names, configure VLANs, load disks, reboot, configure OS’s, run, report

Mapping Example

Automatic mapping of desired topologies and characteristics to physical resources l l NP-hard problem: graph to graph mapping Algorithm goals: – Minimize likelihood of experimental artifacts (bottlenecks) – “Optimal” packing of many simultaneous experiments – Extensible for heterogeneous hardware, software, new features l l l Randomized heuristic algorithm: simulated annealing Typically completes in < 1 second May move to genetic algorithm

Mapping Results l < 1 second for first solution, 40 nodes l “Good” solution within 5 seconds l Apparently insensitive to number of node “features”

Disk Loading l 13 GB generic IDE 7200 rpm drives l Was 20 minutes for 6 GB image l Now 88 seconds l Unicast – domain-specific compression l Multicast – “Frisbee”

Testbed Users l 26 Active Projects – 20 External – 7 “active” active network projects • • • SANDS (TASC)** Activecast (Kentucky)** AMP Node. OS (NAI Labs)** Active proxies (UMass) XML-based content routing (MIT) Janos, Agile protocols (Utah)** – 3 “not-so-active” DARPA AN projects – 4 other active security projects

Users… l Two OSDI’ 00 and three SOSP’ 01 papers – 20% SOSP general acceptance rate – 60% SOSP acceptance rate for emulab users! l More emulab’s under construction: – Kentucky, Duke, CMU, Cornell – Others intended: MIT, WUSTL, Princeton, HPLabs, Intel/UCB, Mt. Holyoke, …

Ongoing and Future Work l Federation of many diverse “testbeds” – Challenge: heterogeneous sites – Challenge: resource allocation l l Wireless nodes, mobile nodes IXP 1200 nodes, tools, code fragments – Routers, high-capacity shapers l l l Simulation/emulation transparency Event system Scheduling system Topology generation tools and GUI Data capture, logging, visualization tools Microsoft OSs, high speed links, more nodes!

A Global-scale Testbed l l Federation key Bottom-up “organic” growth – Local autonomy and priority – Existing hardware resources – Provides diverse hardware • • • PCs Wireless, mobile Real routers, switches (Wisconsin, …) Network processors (IXP’s) Research switches (WUSTL)

NSF ITR Proposal (Nov 01) l Global-scale testbed l Utah primary l Subcontractors: – – l Brown co-PI (resource allocation) MIT (RON overlay, wireless) Duke (Model. Net, early adopter) Mt. Holyoke (diverse users, education) $5 M, 5 years, almost no hardware

Types of Sites l High-end facilities l Generic clusters l Generic labs l l “Virtual machines” (leverage ANETS R&D) Internet 2 links between some sites

Result… l Loosely coupled distributed system l Controlled isolation l “Internet Petri Dish”

New Stuff: Extending to Wireless and Mobile Problems with existing approaches q q q Same problems as wired domain But worse (simulation scaling, . . . ) And more (no models for new technologies, . . . )

Our Approach: Exploit a Dense Mesh of Devices § Density enables broad range of emulation § Wireless q q q Deploy devices throughout campus Measure Nx. N path characteristics (e. g. power, interference, bit error rate) Employ diversity: 900 MHz, Bluetooth, IEEE 802. 11 § Mobile q Leverage passive “couriers” • Assign PDAs to students walking to class • Equip public transit system with higher-end devices q Provides a realistic, predictable mobile testbed

Possible User Interfaces § Specify desired device and path properties q emulab selects closest approximation § Specify desired spatial layout q emulab selects closest mapping § Manually select from deployed devices

Wireless Virtual to Physical Mapping

Available for universities, labs, and companies, for research and teaching, at: www. emulab. net