JSMeter Characterizing the Behavior of Java Script Web

JSMeter: Characterizing the Behavior of Java. Script Web Applications Paruj Ratanaworabhan Kasetsart University, Thailand Ben Livshits and Ben Zorn Microsoft Research, Redmond in collaboration with David Simmons, Corneliu Barsan, and Allen Wirfs-Brock 1

One Word: Standard for scripting web applications Fast JITs widely available Java. Script Lots of code present in all major web sites Support in every browser 2

ZDNet 29 May 2008 Browser Wars! ghacks. net Dec. 2008 4

Artificial Benchmarks versus Real World Sites 7 V 8 8 Sun. Spider programs: • richards • 3 -draytrace • deltablue • access-nbody • crypto • bitops-nsieve • raytrace • controlflow • earleyboyer • crypto-aes JSMeter Goals of this Talk 11 real sites: Maps • date-xparb • • regexp Show behavior of Java. Script benchmarks versus real sites • splay • math-cordic string-tagcloud • Consider • how benchmarks can mislead design decisions 5

Instrumentation Framework Source-level instrumentation iejscript*. cpp custom jscript. dll website visits custom trace files Offline analyzers custom trace files 6

Visiting the Real Sites • Getting past page load performance • Attempted to use each site in “normal” way: amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Search a book, add to shopping cart, sign in, and sign out Type in a search query and also look for images and news Search for a direction from one city to another Read front page news Search for a notebook, bid, sing in, and sign out Read front page news, view comments Log in, visit a friend pages, browse through photos and comments Sign in, check inbox, delete a mail, and sign out Type in a search query and also look for images and news Search for a direction from one city to another Sign in, check inbox, delete a mail, and sign out 7

Understanding Java. Script Behavior Code Java. Script Events Objects 8

Code Behavior • Function size • Instructions/call Code • Code locality • Instruction mix Java. Script Events Objects 9

Real Sites V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud richards deltablue crypto raytrace earley regexp splay amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Source size (kilobytes) Code|Objects|Events Total Bytes of Java. Script Source 2500 2000 1500 1000 500 0 Sun. Spider 10

Real Sites V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud richards deltablue crypto raytrace earley regexp splay amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Unique Functions Executed Static Unique Functions Executed Code|Objects|Events 4 000 3 500 3 000 2 500 2 000 1 500 1 000 500 0 Sun. Spider 12

Bytecodes / Call 300, 00 Code|Objects|Events function(a, b) { var i=0, elem, pos=a. length; if(D. browser. msie) { while(elem=b[i++]) if(elem. node. Type!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a } 250, 00 Bytecodes / Call 200, 00 150, 00 100, 00 50, 00 V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud Real Sites richards deltablue crypto raytrace earley regexp splay amazon bingmap cnn ebay economist facebook gmail googlemap hotmail 0, 00 Sun. Spider 15

Real Sites V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud 60, 00% richards deltablue crypto raytrace earley regexp splay 70, 00% amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Fraction of Code Executed Code|Objects|Events Fraction of Code Executed 100, 00% 90, 00% 80, 00% Most code not executed 50, 00% 40, 00% 30, 00% 20, 00% 10, 00% Sun. Spider 16

Code|Objects|Events Hot Function Distribution 80% of time in 100+ functions 80% Real Sites 80% of time in < 10 functions 80% V 8 Benchmarks 17

Code|Objects|Events Opcode Distribution Outliers Real Apps Green = Sun. Spider | Blue= Real Web Apps | Red = V 8 18

Object Allocation Behavior • Allocation by types Code • Lifetime distribution • Live heap composition Java. Script Events Objects 19

Real Sites V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud richards deltablue crypto raytrace earley regexp splay 45000 amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Total heap data (kilobytes) Code|Objects|Events Total Bytes Allocated 40000 35000 30000 25000 20000 15000 10000 5000 0 Sun. Spider 20

Many functions Rest are strings Real Sites V 8 3 d-raytrace access-nbody bitops-nsieve controlflow crypto-aes date-xparb math-cordic regexp-dna string-tagcloud aggregate richards deltablue crypto raytrace earley regexp splay V 8 aggragate amazon bingmap cnn ebay economist facebook gmail googlemap hotmail Heap Data by Type Code|Objects|Events Few benchmarks allocate much data 1 0, 9 0, 8 0, 7 Objects 0, 6 Enumarator 0, 5 Reg. Exp 0, 4 Math Error 0, 3 Date 0, 2 Native Func 0, 1 String Arrays 0 Script Func Sun. Spider 21

Code|Objects|Events Live Heap Over Time (gmail) Functions grow steadily GC reduces size of heap Objects grow steadily too 23

Code|Objects|Events Live Heap over Time (ebay) Heap contains mostly functions Heaps repeatedly created, discarded Heap drops to 0 on page load 24

Code|Objects|Events 2 Search Websites, 2 Architectures You stay on the same page during your entire visit Code loaded once Heap is bigger Every transition loads a new page Code loaded repeatedly Heap is smaller Bing (Web 2. 0) Google (Web 1. 0) 25

Event Handlers in Java. Script • Number of events • Sizes of handlers Code Java. Script Events Objects 29

Code|Objects|Events Event-driven Programming Model • Single-threaded, non-preemptive event handlers • Example handlers: onabort, onclick, etc. • Very different from batch processing of benchmarks • Handler responsiveness critical to user experience 30

Real Sites h de ard lta s bl u cr e yp ra to yt ra c ea e rl re ey ge xp sp la y ric ng m ap cn n ec eb on ay o fa mis ce t bo ok gm ai g l go oo og gle le m ho ap tm ai l ng bi 7 000 bi on az am Total Events Handled Code|Objects|Events Total Events Handled 6 000 5 000 4 000 3 000 2 000 Almost no events 1 000 0 V 8 31

l tm ai 506 ho m ap le e og l go l gm ai 500 go og t bo ok ce fa ay om is ec on eb n cn ap gm bi n ng bi on az am Median Bytecodes / Event Handled Code|Objects|Events Median Bytecodes / Event Handled 2137 450 400 350 300 250 200 150 100 50 0 33

Cold-code Experiment • Observation – Real web apps have lots of code (much of it cold) – Benchmarks do not • Question: What happens if the benchmarks have more code? – We added extra, unused to code to 6 Sun. Spider benchmarks – We measured the impact on the benchmark performance 37

Performance Impact of Cold Code Without cold code, 900 Chrome 12 x faster than IE 8 800 400 K 800 K 300 1 M 200 2 M 100 200 K 400 K 800 K 300 1 M 200 2 M 100 gc lo di gta co r rin h- st at m IE 8 ud c s ae opt cr y tro lfl ow ve co n -n sie dy ps to bi ce s s-n bo ce ra yt Cold code has non-uniform impact on execution time ac -ra gc lo di gta co r rin h- Cold code makes Sun. Spider on Chrome up to 4. 5 x slower st at m Chrome ud c s ae opt cr y tro lfl ow ve co n -n sie dy ps bi to ce ss -n bo ra yt ac -ra 3 d 0 K 500 0 ce 0 600 Time (msec) 200 K 400 700 3 d Time (msec) With 700 2 M cold code, Chrome 4. 7 x faster than IE 8 600 Which comparison is more meaningful? 0 K 500 38

Impact of Benchmarks • What gets emphasis – Making tight loops fast – Optimizing small amounts of code • Important issues ignored – – Garbage collection (especially of strings) Managing large amounts of code Optimizing event handling Considering Java. Script context between page loads 39

Conclusions • JSmeter is an instrumentation framework – Used to measure and compare Java. Script applications – High-level views of behavior promote understanding • Benchmarks differ significantly from real sites – Misleads designers, skews implementations • Next steps – Develop and promote better benchmarks – Design and evaluate better Java. Script runtimes – Promote better performance tools for Java. Script developers 40

Conclusions • JSmeter is an instrumentation framework – Used to measure and compare Java. Script applications – High-level views of behavior promote understanding • Behavior in benchmarks significantly differs from real sites – Misleads designers, skews implementations • Next steps – Develop and promote better benchmarks – Design and evaluate better Java. Script runtimes – Promote better performance tools for Java. Script developers 41

Additional Resources • Project: http: //research. microsoft. com/en-us/projects/jsmeter/ • Video: Project JSMeter: Java. Script Performance Analysis in the Real World" - MSDN Channel 9 interview with Erik Meier, Ben Livshits, and Ben Zorn • Papers: – “JSMeter: Comparing the Behavior of Java. Script Benchmarks with Real Web Applications”, Paruj Ratanaworabhan, Benjamin Livshits and Benjamin G. Zorn, USENIX 2010 Conference on Web Application Development (Web. Apps’ 10), June 2010. – "JSMeter: Characterizing Real-World Behavior of Java. Script Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSRTR-2009 -173, December 2009 (49 pages), November 2009. 42

Additional Slides 43

Related Work • Java. Script – “An Analysis of the Dynamic Behavior of Java. Script Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek, PLDI 2010, June 2010/ • C# and Java – Doufour • C++ – Calder et al. 44