Panoramix A visualization environment for LHCb CHEP04 Interlaken

Panoramix A visualization environment for LHCb CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 1

Goals • • Handle event display and statistical visualization in a consistent way. LHCb coarse graining organization in term of “projects” : Panoramix: vis Gauss : sim Brunel : rec Da. Vinci : phys LHCb : detector desc & event model Geant 4 • • Gaudi/SEAL/POOL Open. Scientist Panoramix is on top “of everything” (that explains the below “e. Xtreme debugger”). Panoramix subjected to a software agreement between LAL and LHCb. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 2

It works !!! From the collection of Thomas Ruf (done on Windows) CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 3

Engineering choices • Plugged directly to the data framework (Gaudi). No client / server model. • Since the data framework is in C++ (and most of native vis and GUI tools are in C / C++) use C++ for all. • Target the three laptops of the moment : Linuxes, Windows, Mac. OSX by using their native environments. • Used one powerful “scene manager”, Inventor, for all 3 D or 2 D data representation. • Try to put in “reusable elsewhere packages” things not fully dependant of the experiment. Here the “reusable elsewhere packages” being the ones put in the Open. Scientist distribution. • See Open. Scientist slides for more on “Engineering choices”. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 4

Vis and GUI products • Rendering layer : more than ever : Open. GL. • Scene graph manager : Open. Inventor. By using the Coin implementation of System In Motion (Norway). • Do all graphic with the uppers, including histogram and function plotting. • GUI : see Open. Scientist slides for the problematic of a choice. Describe the GUI in XML and use the On. X package to create the GUI by using native toolkit of desktop providers. This is simple and give the best performances. • Scripted callback : mainly dld/c++ (string front end to dlopen/dlsym) : <widget class=“Push. Button”> <activate exec=“C++”>On. X ui_echo hello</activate> </widget> and Python (with LHGDict wrapping) : <activate exec=“Python”>from import On. X *; ui. echo(‘hello’)</activate> CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 5

On. X Xt, Win, Qt, gtk+ XML On. X Dld, Python, CINT GL, Inventor CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 6

GUI paradigm “A la Power. Point” : a compact GUI panel organized around a document area. This is 100% in opposition to a terminal prompt spawning viewers around the screen CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 7

Panoramix on a Mac. OSX by using native Cocoa (Next. Step) / Open. GL (not released yet since all below layers not yet released on OSX). CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 8

Connection to the data framework • “data framework” should be understood as the software permitting to manipulate the “event model” and permitting to connect the “event model” (the “data”) to “facilities” like storage, graphic, GUI, scripting. The data framework is here Gaudi does not do any kind of graphic (no Gaudi: : Data. Object: : draw method around and this is correct). Gaudi manipulates services that permits someone to operate the “transient store” containing, in memory, instances of the “event model”. • On. X is here the “interactivity framework”. It does the connection between GUI (from an XML description), viewers, scene manager (Inventor), renderer (Open. GL) and scripting. • The connection “data framework” “interactivity framework” is done through the Gaudi On. X service, put in a standalone package : Vis/On. XSvc. • Various elements of the event model have a “representation” (a datarep). A datarep code is in general a Gaudi “converter” for the Inventor technology (an So. Converter). An So. Converter, for example So. MCParticle, build from a data instance (here an MCParticle) an Inventor scene graph. When built, the scene graph is send to On. X to be displayed, in general, in the “current viewer”. • A “visualization request” starts from a scripted GUI callback ; the On. XSvc is searched, then a “data convertion for Inventor” request is activated for various pieces of data. • Note that, due to a massive usage of abstract interfaces, things are nicely decoupled (for example an So. Converter is not a Storage. Converter. A direct consequence is that someone can get rid of a technology and migrate to a new one, without touching to everything. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 9

The Vis/So packages • Physicists enter into action by programming some “So. Converter” put in some “Vis/So” package. • So. Det (G. Barrand, S. Ponce) : representation of the detector. This is very generic and is now sparely touched. Coupled to the LHCb XML det description, it offers a very flexible way to enter and view a geometry. Note that there are various interactive goodies that permits to “expand” or “contract” volumes (from J. Boudreau So. Detector. Tree. Kit). The GUI can view the detector and event tree in a tree widget. • So. Event : historical. Used to gather first datareps like So. MCParticle, So. MCHit (P. Mato) and representations for the trackers (J. Van. Tilburg NIKHEF). • So. Calo (I. Belyaev ITEP) : dedicated package to gather reps for calorimeters. • So. Rich (C. Jones Cambridge) : reps for the RICH. • So. Stat (G. Barrand LAL) : to represent histograms of the Gaudi transient store. It uses the HEPVis So. Plotter nodekit. • So. Hep. MC. : to visualize Hep. MCs. Used in conjunction of the Geant 4 simulation (Gauss). • People did not seem to suffer of Inventor programming. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 10

The applications • Two packages remain Vis/Panoramix and Vis/La. Joconde. • Panoramix : it contains the “main” Panoramix. exe program which is a standard “Gaudi main”. This package contains also the Panoramix. onx and other *. onx XML files describing the GUI. It contains also “C++ callbacks” and various Python scripts sufficiently generals to be put “in the common pot”. It is in scripts that people can setup their views (number of regions with their size and position, background colors, etc…) and decide of what they put in their “scenes”. This could be done in Python or directly in C++ (in case some Python wrapping is lacking, too tricky C++ is needed or intensive speed is needed) by using the On. X “dld/C++ callback system”. • La. Joconde : historically introduced to connect to “The Da. Vinci code” (where the secret of LHCb physic is hidden). It had been introduced for “coarse graining” organization but will probably be “transferred” back to Panoramix in future so that physicists manipulate only “one front end program”. This will depend of the capabilities to be able to build coarse graining “Job options” files (by saying easily “I want or I do not want to use Da. Vinci”) and be able to access corresponding packages and libs at run time. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 11

Speed Due to basic choices (stick to desktop providers and be plugged directly to the dataframework) it is fast and the author wait anybody that claims to have something faster…. • BUT, we must always “have an eye” on Inventor. It is a powerful tool and then can be dangerous. A scene graph may become “too fat” and slow down the creation of a data representation. But there are numerous ways to optimize things (share color, line styles nodes, create dedicated nodes to handle huge collections of data, etc…). Experience shows that on local machines scene graph with 10 000 up to 100 000 nodes are ok. For more orders of magnitude, you have to think at optimizing. BUT SOMEONE MUST BE AWARE THAT A CROWDY SCENE GRAPH OFTENLY MEANS A CROWDY PICTURE ON THE SCREEN…. • Then the secret of “being fluent” is not in being able to handle big scene graph, it is in being fluent in “selection of data” (selection of what you really want to see). • Up to now, at least for what is done in LHCb, speed is sufficient. (I am more worried by what is (not) done in more bigger experiments…) • CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 12

Problems (1) • All this works and had been put in place very early. • Problem remains around visualization of boolean operations over volumes. It is the “case of coplanar edges”. Dixit E. Chernyaev, the author of the algorithm (the Boolean. Processor of Geant 4) there are cases where things cannot be decided (easily) and the best way is to avoid to have “edge to edge situations”. Fine, ok. But it is hard to explain to the detector people that they have to add or remove some matter here or there in order to “pass” a visualization algorithm !!!!! • But a non-algorithmic solution perhaps remains ; use the stencil capabilities of Open. GL to do that. Code had been found in Australia but not yet put in an Inventor node and integrated / tested in the system. A side effect ; it probably kill the vector PS production and then only “pixmap” PS production would remain for these views. • If someone has ideas ? Other algorithms to try, etc… CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 13

Problems (2) • Not yet fluent in fully stand alone installation on a (remote) local machine. We remember that a display is on top of everything and that the three common desktop are targeted. Then be fluent in installing the display passes by being fluent in installing “all the below”… • Today, binaries (Windows, Linux) are regularly reconstructed on the /afs/cern. ch areas (lot of thanks to Florence Ranjard) and users run them on their local machine. (Doing remote X 11 on lxplus is definitely not recommended). Then we : – Use CERN as a central integration / binary construction / distribution place : THIS IS GOOD. (I strongly believe that this is the number one role of CERN). – DO NOT USE CERN at run time but use local machines seeing CERN with afs. But CERN anyway around at run time as a STRONG afs server provider. • For more, we strongly rely on efforts done around CMT (and PACMAN ? ) to provide at some time straight forward source or binary local installation “of everything”. (See Christian Arnault poster about ATLAS software distribution). CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 14

Problems (3) • LCG : where are we going here ? • Can we build “bosonic software” with “fermionic people” ? cint root/io lcgdict pool pi root/th gaudi/seal python • Panoramix on top of much more things not really deeply though to work together. • Mac. OSX port had been a pure nightmare… (And the story is not finished here. The only presence of ROOT induces to run in “bind_at_launch” mode that slows down the startup of everything. There are mysteries around dynamic loading and execution of static object constructors done with Gaudi and SEAL / Plugin. Manager). CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 15

A problem ? • In the software agreement, was mentioned that a physicist should be involved (at partial time) to do the sync <engineering> <physic> during the whole story. This never happened. • For example, despite that Panoramix received code contributions from numerous people and that it is used by various people, no physicist “strongly visualization driven” spontaneously emerged to survey that all element of the event model received a representation. The engineer is NOT paid to have all the LHCb detector in head along with all the event model and physic algorithms and to decide how things should be represented. • It would be welcome to avoid a “last minute rush” on that last point. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 16

Problems (4) • http: //www. lal. in 2 p 3. fr/Panoramix. • html pages and paper doc produced with Doxygen from dedicated. h files (and not the ones of the code). Some of the files loadable from the program to form a help pull down menu and contextual helps. • But the content is clearly insufficient. • Nice user page of Thomas Ruf • Basement for a wiki ? • But for programmers (writing So. Converter), doc of Gaudi and Inventor is online. • But once more, is it to the engineer to document everything and in particular all what can be represented with various options, etc…? CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 17

Problems (5) • How to fit 48 hours in one day ? (Transplantation of a superconducting positronic brain ? (*)) • Need manpower ? Sure, the needed army that LCG will not put in place for doing graphic… * Due to the international context, the US Robots refuses to do that on a French for the moment…. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 18

Next… • • Next main issue is going to be the integration of “stat tools” to do “analysis”. For LHCb it will consist to operate “The Da. Vinci code” from the interactive tools. Here the way to deal with events is a little bit different : – Display : pick one event, view things in it, pass to another event, etc… – Analysis : pick and configure one or a bunch of algorithms (that will book histos and tuples), run over thousands of events, look histograms. Repeat again by modifying algorithms. That’s right that what is done BY THE PHYSICIST is different but things to put in place BY THE ENGINEER ARE EXACTLY THE SAME IN BOTH SITUATIONS : a GUI, operate on the event model, produce “graphical representations” from data. The only significant differences are around the data that must be visualized. In one case it is things in the event model, in the other case it is common things like histograms and tuples. In the second case, then a key issue is to INTEGRATE A GOOD PLOTTER, that is to say a good piece of software able to create graphical representation of histograms. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 19

HEPVis / So. Plotter, AIDA • Due to the choice of Inventor as scene manager, the author targets a plotter done with Inventor : here the HEPVis / So. Plotter one developed mainly be the author. So. Plotter covers now a lot of common cases (see Open. PAW poster for pictures). • The Vis / So. Stat package does the connection Gaudi-histos / So. Plotter. This is in fact available for long but not yet heavily used due to the fact that connection to Da. Vinci is a little bit young. (and probably because people had taken some habits with other tools…). • Then, with So. Plotter, the same Panoramix environment can then be used also by working on Da. Vinci. (It opens also the way to do nice things like putting histo within events). • For scripting, someone can already get from Python the On. XSvc and do a vis request over an histo managed by Gaudi. Someone can also customize a lot of the So. Plotter. • What have to be provided now is to “standardize” the way to do that. The plan is to used the AIDA API and interfaces, mainly because it had been though and discussed by various parties and that, being pure abstract interfaces, it does not compel us to tied to another system having orthogonal backend GUI and graphic. It is crucial to keep in sync the LHCb software with AIDA CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 20

Integration of other tools (1) • • • Had been demonstrated than from a python shell, someone can spawn a Da. Vinci task with : – Panoramix (to display an event or plot an histo) – Hippodraw – ROOT Then physicists will have tools. But, shall software engineers be happy with that ? NO. All these tools have incompatible backend technologies (rendering, GUI) and there is no hope to provide one tool within a common GUI that integrate all that in a consistent interactive way. A physicist will be let with a desktop crowed with standalone GUI windows, each having its own logic. An already announced pain. Then, some job (low priority) is done to try to integrate some of these tools within Panoramix… CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 21

Integration of other stat tools (2) • Hippodraw : the design of it is so that someone can easily get rid of the default GUI and rendering (all in Qt), and comes with new ones. Had been done an Open. GL driver to hippodraw. From that it is quite easy to integrate a hippo window within Panoramix. An embryo of reimplementation of the famous “hippo inspector” done with On. X (then mainly in XML) exists now (Open. Scientist / Hippo package). • ROOT : it will go through an implementation over Inventor of the painful TVirtual. Pad (some kind of strange mixture of GUI, rendering and scene manager virtual methods). A prototype exists (Open. Scientist / Mangrove package). Someone can visualize a TGeo with On. X and Inventor and also the bins of a TH. But more work have to be done to have the full THist. Painter over Inventor. But this is feasible. (Any volunteer ? No one, as usual). • Java / JAIDA / jas : hard to see how we can have jas drawing in an Inventor viewer or an Open. GL window not managed by the awt / swing. Moreover it appeared difficult to have Inventor drawing in an awt / swing window…. Strong expertise required to go on. • If no help from the developers of these tools is coming, the author will continue to put the effort on the So. Plotter (and Open. PAW) and to its AIDA front end ; the AIDA: : IPlotter. CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 22

Must be mentioned : Agora • Agora is a prototype of visualization environment for ATLAS using the same technologies than Panoramix (Open. GL, Inventor, On. X, AIDA). • It makes sense since the data-framework is the same than LHCb (Gaudi). • Proof of feasibility done. In fact here, it is a shame that LHCb and ATLAS had not been able to converge on XML for the det desc…. • Anyone interested ? CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 23

Conclusions • Panoramix is regularly released and used and had already given services (for the XML det desc debugging, reconstruction debugging). • Even if not yet an overflow, views appear in meetings and workshops ; and some are very nice. • Engineering choices are done (let us say up to the startup). But the overall architecture will permit to migrate parts if needed. • More work around the release building and distribution are needed if the three common user laptops are targeted (Linux, Windows, Mac). • At the engineering level, efforts will be put now on the final (!) integration of statistical tools : AIDA / Open. PAW for sure, hippodraw probably, ROOT with a gun in the back. Rendez vous at next CHEP for that. • Are physicists happy ? (is a physicist ever happy ? ). It is not to the author to give an answer but there are not too much screams, shouts, laptop thrown on the wall for the moment… CHEP'04 Interlaken G. Barrand, e. Xtreme debugger at LAL 24