Online Displays for the World Publishing online status

Online Displays for the World Publishing online status information to the web Functional requirements m. Implementation requirements m. Chosen technologies m. The publishing mechanism m. Gallery m. Problems m JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb

Functional Requirements m m Publish data from PVSS to web pages Allow users to peek on the experiment status from anywhere q Monitor subdetectors from home institutes without the need to login q Experiment status (DAQ) q Operations status q Subdetector status q … Applications are all “nice to have”, but not critical Necessary: Extract data from PVSS or any DIM service JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 2

Implementation Requirements m m Dynamic web pages Online updating displays q No full page reloads (ugly ) q Was something new (hype factor ) Architecture for the lazy developer q Minimize developments (minimize maintenance) Limit developments to q Data extractor from PVSS or other DIM services q Web pages Use comet (-like) technology q http: //cometdaily. com/maturity. html (a bit old: 2009) JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 3

Comet m m m Technology to feed data to dynamic web pages Data fan-out: one producer, many listeners q Message queuing technology q Topic oriented subscription mechanism q Publishing triggered by first subscription Data provided to web browsers by Javascript / HTML socket The technology selection is determined by the provided q Client APIs q Provided protocols Required API (main limits over technology choice) C/C++ API at the data producer side (DIM/PVSS) Xml. Http. Request objects at the client side (XHR/long poll) Web. Socket not (yet) supported on all browsers JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 4

Data Format m m m All data are strings STOMP or Openwire will internally encode the data q base 64 encoding: blobs etc. become strings Internal format (custom implementation) q timestamp # tag name # PVSS data-type id # value q timestamp currently unused q tag name used to dispatch at client side q PVSS data type used for display optimization (sprintf) q FSM special data type id q Float numbers limited to 4 significant digits Quickly changing numbers become more stable JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 5

Comet m m m Server side architecture Client architecture Maximize the content of the black box JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 6

Server Architecture D I M Producer DIM server(s) Stomp JCOP meeting 6 -Dec-2011 S t o m p / M. Frank H CERN/LHCb Morbit. Q “Out of the box” 7

More Performing Architecture D I M DIM server(s) Producer Open. Wir e Stomp JCOP meeting 6 -Dec-2011 S t o m p Out of the box. Needs config / M. Frank 8 H CERN/LHCb

Web Publishing Mechanism m m Any PVSS Data can be published: PVSS 00 dim A small CTL scripts adds PVSS data type before publish FSM Nodes: q Special multiple item data type to support for more fancy rendering Datapoints: q All Run Control Data q LHC data, etc. Panel + Server: Clara JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 9

WEB Client Architecture m m No “HTML” pages All pages are dynamic q HTML structure is programmed by directly manipulating the DOM tree in Javascript q My choice: all pages have the identical source q Instead of <TABLE> … </TABLE>: var table = document. create. Element(‘table’); …. setup table …. parent. append. Child(table); q Not difficult, but different paradigm q Very flexible, rather complete and easy to use widget library JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 10

WEB Client: First Value Problem m Comet only sends updates No way to enforce sending first value at subscription time of a new client q Or me too stupid ? Chosen solution q Client sends to special topic subscription request q Producer forces update of the newly subscribed topic q Disadvantage: All clients see the update Unnecessary load JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 11

Published Pages m Web Pages Available q m http: //lbcomet. cern. ch/Online Let’s walk through the galery q Page 1 q Run Control q Experiment Control q LHC/LHCb status q Fill Status q L 0 Trigger status q Velo Motion display q Event Display q Browser based desktop JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 12

Page 1 JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 13

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 14

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 15

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 16

ORACLE PVSS WWW JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 17

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 18

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 19

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 20

Browser based desktop JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 21

Monitoring Issues m m Everything can fail Monitoring the main components helps to investigate problems AMQ q Comes with a rather comprehensive monitoring utility q Connection monitoring q Topic summary Orbited q Rather rudimentary JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 22

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 23

JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 24

Trouble and Problems m m Normally things work pretty reliable Limits seen with Orbited+Morbit. Q at LHC startup q Too many clients q Better performance after move to Active. MQ …but we have not seen these peaks anymore Low number of hundreds of clients q Typically ~50 clients q Now during shutdown ~10 No further optimization necessary so far JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 25

Trouble and Problems Web browsers going “ballistic” m About once/week during running q Less frequent during shutdown (request – related ? ) q Single browser sends connection requests with tens of Hz m AMQ closes unused file descriptors after some timeout m Number of open file descriptors explode (>1000) m We investigated a lot (connection limits/client node etc. ) but no real solution found Open FD monitored by watch dog Automated kill and restart if this happens m JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 26

Trouble and Problems m m m Never believe java/Java. Script has no memory leaks q This was a lie for Java and with web browsers and Java. Script it’s no better q References kept “somewhere” (i. e. DOM document) without remembered by user manifest the same way Firefox exploded after some time q But happens on all pages being reloaded Other browsers q IE: Leaks fixed with page reload every few hours q IE: No support for multiple connections from single through html iframes (dynamic popup windows) q Chrome: Very strict security model. But others catch up… JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 27

Summary m m m Overall experience very good Subdetector experts use the web pages quite a lot Software maintenance overhead manageable Binding to PVSS via DIM very simple Performance sufficient for 100 -200 clients Mad browsers are an annoyance JCOP meeting 6 -Dec-2011 M. Frank CERN/LHCb 28