CERN European Organization for Nuclear Research Administrative Information

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 1 Ais. Monitor Giovanni Chierico & Camilla Stenersen March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 2 Background Multiple remarks and demands from both IT-AIS and ITDES after various system failures which often had cascade effects on other systems: • Difficult to monitor (everybody has his own monitoring system) • Difficult to know potential cascade effects (were there two separate problems or one causing the other) • IT-AIS GL and DGL triggered the project using available TECH March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 3 Goals Create a common framework to monitor AIS applications • Simple to integrate in the existing AIS & IT environments • Reliable • Easy to extend • Generic • Keep history March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 4 Process • Gather requirements • Interviews with all the AIS Section leaders • Meetings with IT-DES (Artur Wiecek) • Meetings with IT-FIO (Sebastian Lopienski) • Meeting with IT-AIS and IT-DES GLs • Implement full app stack to monitor some “sample” apps. • DB • Messaging • Applications “hooks” • Interface • Refine the application and implement “hooks” in AIS apps March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 5 Suggestions - • Keep it simple • Don’t reinvent the wheel • Push & Pull monitoring + • Communicate via DB APIs, HTTPS, POP, SMTP, JMS, … • Aware of scheduled downtimes • “Digest” alarms • Monitor batch jobs • Map dependencies • Monitor business processes March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 6 Architecture • Everything is a service • Services are defined by two parameters • The service level (L) • The expected service level (E) • The quality of service is Q=L-E • Service hierarchy defines an oriented weighted graph, with no closed loops. • Services communicate with the world via messages to modify their actual (L) and expected (E) service level. • Level changes are propagated to the children • An alarm defines a Q threshold for a given service, and is triggered accordingly March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 7 Architecture Message contains • Payload alarm DB API Service HTTPS • Expected service level • Comment • Authorization HTTP WS Service • Service level Service Message Service Service • Service name • Auth password Service POP March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 8 Internals Alerts: SMTP Export: XML Pull: Http(s), POP Java Bus&Logic Push: HTTP, WS, SMTP Push: DB APIs Display Interface March 9 th 2006 Repository Meta. Data Interface IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 9 Example level Expected level (E) Actual Level (L) Trigger alarm if threshold equals zero time March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 10 Architecture Key Points • Low “entry cost” • No big initial effort to map dependencies, or modify applications. Monitor and map what you need. • Positive feedback • The more you invest (map deps, create app hooks), the more you get in return • The more value we give back, the more likely the mapped structure will stay up-to-date • Give control to the developers • The developer knows best! No need to depend on others to monitor your app. • Easily manage the alerts and the downtimes March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 11 For the AIS Developer • PULL (Ais. Monitor checks the service): web apps • Expose the test (e. g. on HTTP) • Create Ais. Monitor service • Configure Ais. Monitor PULL (where, when, …) • PUSH (App sends events to Ais. Monitor): batch jobs • Create Ais. Monitor service • Write the app specific test, or hook into existing process • Sending the message can be usually done in 2 -3 lines of code March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 12 Push examples Java Mule. Client client = new Mule. Client(); client. dispatch(“axis: http: //monitor/? ", Payload, null); or Http. Client client = new Http. Client(); Get. Method method = new Get. Method(“http: //monitor/”); //set params client. execute. Method(method); PL/SQL Monitor. Pkg. push@link(name, pwd, level, expected, comment); or SELECT utl_http. request('http: //monitor/? params') FROM dual; March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor 13 Roadmap • What we have today • Push: Run servers and receive messages over HTTP, HTTPS, and file system. • Pull: messages from HTTP and HTTPS, even behind Ais. Login • Propagate messages on Service hierarchy • Persistence (Services, Dependencies, Messages, Statuses saved on DB) • Export service status to xml, accessible over HTTP (needed by IT monitor) • In a week • Basic web interface (Ap. Ex) to manage the services and the pulling params • Pull messages via POP from ais. monitor@cern. ch • Send alerts (SMTP) • In two weeks • Monitor implemented in production apps (CTA & GAD) to test it with real data March 9 th 2006 IT-AIS-HR

CERN – European Organization for Nuclear Research Administrative Information Services Ais. Monitor March 9 th 2006 14 IT-AIS-HR