The Case for Open Infrastructure Services in Java

The Case for Open Infrastructure Services in Java David Culler Computer Science Division U. C. Berkeley www. cs. berkeley. edu/~culler Java Grande Dinner Keynote, June 2000 6/4/2000 Java Grande

Appetizer • ‘Grande’-scale computing dominated by internet services • Delivered to millions per day on well-engineered clusters over service interfaces Clients Clients Servers 6/4/2000 Java Grande 2

Opportunity: infrastructure services • Prehistoric: DNS, IP route tables, … • Historic: crawl, index, search, • Emerging: compose and manipulate data and services Clients Infrastructure Services Servers Clients Servers And client diversity has just begun! 6/4/2000 Servers Java Grande 3

Danger: loss of distributed innovation • PC generation of individual authoring & distr. • vs ATT, IBM, AOL scale service engineering • … Open Clients Infrastructure Services Servers Clients Servers 6/4/2000 Java Grande 4

UCB Ninja Vision • Open platform architecture for world-scale internet services • receptive execution environment – push services into the platform • scalability and availability “built-in” • service composition as a first-class programming concept => make it easy to author and publish high quality services into a well-engineered infrastructure. . for example 6/4/2000 Java Grande 5

Example: Ninja Jukebox 98 3. au/. mp 3 player WWW Browser HTTPd service Web page with song playlists Music Directory service Ninja i. Space Collaborative Community: anyone can add content => mp 3. com, real jukebox, napster 4 Music stream Pushes an index of Authentication and authorization was built-in (. au or. mp 3) locally available songs to Jukebox 99: Music similarity query engine the master directory. => mongomusic. com, . . . CD “ripper” service 6/4/2000 CDDB service Java Grande Ninja i. Space 2 Fetches track/title & artist information from an online DB. 1 6

Santio: universal instant messaging S. Gribble AOL AOL protocol worker AOL client english toto english to spanish profile DDS ICQ ICQ protocol worker 6/4/2000 ICQ client Java Grande sanctio service (cluster) 7

Composable, Secure Proxy Architecture for Post-PC devices S. Ross, J. Hill Internet Diverse Clients Services Personal Appl Embeded Untrusted Client Trusted Client 6/4/2000 S A F T Transient Identity Store Service Filter and Control Modifier Format Transcoders F T S A DATEK (Trust Contract) Security Adpaters https Java Grande 8

Reduce value of the information DATEK 6/4/2000 Java Grande 9

Example: e. Science Services ‘Sugar’ MEMS simulation Service 6/4/2000 Nodal LAPACK Modeling Services Java Grande Netsolver 10

Outline • Call for distributed innovation of scalable, composable services • Wandering Down the Java Garden Path • Returning to robust building blocks and design patterns • Postprandial thoughts 6/4/2000 Java Grande 11

A ‘Structured Architecture’ Approach • Bases (1 M’s) – – – scalable, highly available persistent state databases, agents “home” base per user service programming environment Wide-Area Path • Active Proxies (100 M’s) – not packet routers – bootstrap thin devices into infrastructure – soft-state and well-connected • Units (1 B’s) – – sensors / actuators PDAs / smartphones / PCs heterogeneous Minimal functionality: “Smart Clients” 6/4/2000 Java Grande 12

Guided by the CAP lemma • Consider – Consistency – Availability – Operation in the presence of network Partitions You may have any two of the three, but not all three • Example: replicate for availability – lose consistency upon update during partition – or can defer the updates till healed – or can engineer the system so no partition between replicas 6/4/2000 Java Grande 13

The Java “Apple” • strong typing • automatic memory management • Concurrency built-in: Threads and Synchronized Methods – finally! • Elegant remote access built-in: RMI – service lookup yields service object stub – transparent access • Code mobility – traditionally for pulling down applets on demand 6/4/2000 Java Grande 14

i. Space Execution Environment Untrusted Services Security Mgr Ninja i. Space + RMI Loader Sandbox that contains untrusted, uploaded services. Trusted Services Service is an interface, plus objects that implement that interface. Name service, RMI stub registry, and service control API: • Load. Service (URL) • interf. [ ]=List. Services • stub=Get. Service(name ) • Kill. Service(name) JVM + persistent store APIs RMI + authentication, i. Space encryption, multicast, JVM provides service upload capability, plus strong typing of user-level SAN speed. service interfaces. Distributed hash table API provide scalable, available hard state 6/4/2000 Java Grande 15

Multispace Cluster Platform m-RMI stub Multi. Space Loader DDS client RMI “Redirector Stubs run -time compiled RMI superstub • stub selection policy • fail-over, • broadcast, multicast, fork, etc. 6/4/2000 i. Space Java Grande SAN 16

After the garden: Post-Prototype Reality • Powerful, attractive, tantalizing possibilities… – see examples. . . • Didn’t scale – service concurrency – client population – service diversity • Wasn’t robust • Lessons – – 6/4/2000 Thread-per-task considered harmful Woes of blocking interfaces The Transparency trap Versions really matter Java Grande 17

Java RMI Thread-per-task services Client RMI Service Blocking RMI • Server Thread per client thread – familiar per-task programming model, including RMI and I/O • Socket per client JVM (or per thread, per stub!) Java Grande 6/4/2000 18

The transparency trap • Server commits thread regardless of client load • Client places demand regardless of server concurrency • || resource || to blocking composition depth • ease leads to fine grain use of remote objects • RMI “call backs” make client a server • lifetime and scope of remote object unlimited • inexpressive error model (wait or Remote. Exception) 6/4/2000 19 • serialization is costly Java Grande

Blocking + Thread = Non-blocking ? ? ? • JAVA i/o and comm APIs all blocking! • need JNI for select! Keep going to the “thread well” 6/4/2000 Java Grande 20

Study a Service “test problem” task arrivals rate: A tasks / sec closed loop implies S=A Threaded server • A: popularity • L: I/O, network, or service composition depth dispatch( ) or create( ) latency: L sec task completions # concurrent tasks in server: T = A x L rate: S tasks / sec 6/4/2000 Java Grande 21

Response time vs S (= T/L) 6/4/2000 Java Grande 22

Threads are a limited Resource • Fix L = 10 ms, for each T measure max A = S • Cluster parallelism just raises the threshold * CPU bound tasks saturate early 6/4/2000 23 Java Grandeultra 170 and E 450, Solaris 7. 2, jdk 1. 2. 2 * focus on threads, footprint follows

Alternative: queues, events, typed msgs • server provides bounded resources at request interface – chooses when to assign resources to request event – imposes load-conditioning or admission control • client retains control of its thread Explicit request queue 6/4/2000 – chooses when to block – permits negotiation protocol – key to service composition • queues absorb load and decouple operations • provide non-blocking interface • RMI as syntax sugar Java Grande 24

Java Event-based Server task arrivals rate: A tasks / sec timer queue with latency: L seconds closed loop implies S=A task completions rate: S tasks / sec 6/4/2000 • Fixed # threads , independent of # concurrent tasks in server (A x L) Java Grande 25

Event-per-task saturates gracefully • Better and more robust performance – Use cluster parallelism to match demand • Decompose task into multiple events – circulate or pipeline 6/4/2000 • but. . . Java Grande 26

Down side of event approach • Lose the familiar sequential programming (plus synchronization) – need a handler per stage of the task • Does not naturally exploit SMP parallelism – must pipeline multiple event handler blocks • Blocking interfaces (or faults) cause throughput to follow 1/L in an event block! 6/4/2000 Java Grande 27

Hybrid, Robust building block Explicit event queue absorbs bursts of tasks allows introspection Load conditioning point # concurrent tasks decoupled from # concurrent threads in server: Bounded thread pool of T < T’ threads • Compose service as graph of task handlers – Decouple stages of task within a node – Replicate across cluster nodes for scale and availability • Thread parallelism and latency tolerance within task handler block (i. e. , A x L < T per node) 6/4/2000 Java Grande 28

Hybrid Performance Ultra 1 • Competitive with pure event block – small overhead due to extra threads • Upon blocking op, throughput tracks T/L 6/4/2000 Java Grande 29

Four key task handler design patterns • • Wrap Pipeline Replicate Combine 6/4/2000 Java Grande 30

Wrap => • Take arbitrary piece of code: • place queue in front • encapsulate with bounded thread pool T < T’ => get ‘robust’ service with non-blocking interface 6/4/2000 Java Grande 31

Wrap (thread-per-task server) => • Get robust hybrid task handler with T/L tolerance • Preserve conventional task sequencing • Building block for composed services 6/4/2000 Java Grande 32

Pipeline => • Decouple stages within task handler across multiple task handlers • Wrapped Blocking call is natural boundary 6/4/2000 Java Grande 33

Why Pipeline? • Functional parallelism across stages – when thread blocks in one. . . • Functional parallelism across processors • Functional parallelism across nodes • Increase locality (cache, VM, TLB, …) within node – tend to perform operation (stage) on “convoy” of tasks • Limit number of threads devoted to “low concurrency” operation – ex: file system can only handle 40 -50 concurrent write requests, so this limits useful T – additional threads can be applied to remainder of stage 6/4/2000 Java Grande 34

Replicate => • Scale throughput across nodes • Provide fault isolation boundary • Mediate thread-pool bottleneck within node 6/4/2000 Java Grande 35

Combine => • Two task handlers share pool and queue • Common use is before/after wrapped call • Avoid wasting threads 6/4/2000 Java Grande 36

A Prescription Well-conditioned node • Wrap to introduce load conditioning • Pipeline to avoid wasting threads at bottlenecks • Pipeline to enhance locality Available Service • Replicate for Fault Tolerance Scaling • Replicate to meet concurrency demand Tuning • Combine to limit threads per node • Pipeline for functional specialization 6/4/2000 Java Grande 37

Ninja v. SPACE design • Each blocking interface is wrapped • Service described by collection of task handler modules • Each module implements a set of task types – includes completion events – module clones are replicated on demand • Most task handlers are state free • Persistent state provided by DDS • Explicit queues are the fundamental means of introspection 6/4/2000 Java Grande 38

Example: Hash Table Distr. Data Struct. Clustered Service DDS lib Distr Hash table API Redundant low latency high xput network System Area Network Storage Storage “brick” “brick” 6/4/2000 Java Grande Single-node durable hash table 39

DDS Hash Table Brick Design I/O core disk distributed hashtable “RPC” skeletons file system / raw disk single-node HT I/O core network stack Ideal I/O Core buffer cache I/O core disk I/O core network operating system file system / raw disk network stack Pragmetic I/O Core DDS Brick 6/4/2000 I/O core network Java Grande 40

Scalable Throughput 6/4/2000 Java Grande 41

Robust under load 6/4/2000 Java Grande 42

6/4/2000 Java Grande collection Garbage Recovered node cold Recover done Recover start One dies Three nodes Fault and Recovery 43

Dessert thoughts • Performance and efficiency on Java is critical first step, but cannot stay in MPP mode • Huge Opportunity – distributed innovation of widely used services (with I/O) – service composition as new level of programming • Need to deal with resource containment, load, errors, versions and coupling from the beginning – events, queues, types msgs => managed RMI • Event driven execution (encapsulating threads) is exciting & opens a rich set of questions – expressiveness, synthesis – introspection, scheduling, concurrency control – debugging 6/4/2000 Java Grande 44

Where to go for more • http: //ninja. cs. berkeley. edu • A Design Framework for Highly Concurrent Systems, Matt Welsh, Steven Gribble, Eric Brewer, and David Culler. • Scalable, Distributed Data Structures for Internet Service Construction, Steven Gribble, Eric Brewer, Joseph Hellerstein, and David Culler. • A security Architecture for the Post-PC World, S. Ross, J. Hill, M. Chen, D. Culler, A. Joseph, E. Brewer • The Multi. Space: an Evolutionary Platform for Infrastructural Services, Steven Gribble, Matt Welsh, Eric Brewer, and David Culler. 6/4/2000 Java Grande 45

Backup: Mobility not enough • RMI names classes / interfaces in the registry – which class do you get? • Class path management nightmare • Must maintain source web server • distinct services may need distinct instances • service name != class name • versioning is essential • use renaming to allow multiple versions within VM • service publication expresses entire dependence set 6/4/2000 Java Grande 46