JAS Distributed Data Analysis n Grid Enabled Analysis

JAS – Distributed Data Analysis n Grid Enabled Analysis Workshop Caltech - June 23 -25, 2003

Contents n JAS 2 l History l client-server mode l JAS 2 and the Grid n JAS 3 l What’s new l JAS 3 and AIDA l Plans for Gridification

JAS History n First version of JAS 2 released in 2000. l Incremental improvements released over time.

JAS 2 History – Use Cases n With WIRED event display n Online Monitoring

JAS 2 History – Use Cases n Web Servlets n Custom Applications

JAS Client-Server Mode Data Analysis Engine GUI Experiment Java Compiler Extensions + (Event Display) Debugger User’s Java Code Padded Cell DATA

Distributed Analysis System: Goals n Prototype for GRID enabled JAS analysis l Run analysis on a farm of machines t. Use multiple CPU’s in parallel for CPU-intensive analysis t. Access multiple I/O channels for data-intensive analysis l Use standard JAS (Client) as if we are running a local Job l Get interactive feedback t. Create analysis modules (code) t. Control job execution t. View results (Plots/Histograms) l Access distributed datasets as if they were local datasets

Distributed Analysis System: Architecture JAS Data Server … JAS Data Server Network Control Server Network JAS Client … Users JAS Client Catalog Server

JAS 2 – GRID interface (Tech-X)

JAS 3 Overview n A completely new version of JAS l Design based on Application Shell, into which many (optional) modules can be plugged t. Highly customizable for different application domains – HEP/Astrophysics/Other – DST analysis/Online Monitoring/GRID analysis – Experiment/User specific modules t. Modules can be updated independently of shell – Possible to release bug fixes fast l Includes support for programming in many languages t. Scripting: Python, Pnuts, Dynamic Java, …. – Command prompt t. Java (compiled) l Analysis (histograms, tuples, fitting) based on AIDA standard n Not technically backwards compatible with JAS 2 l But migration is straightforward.

AIDA Overview n AIDA = Abstract Interfaces for Data Analysis l Covers key areas for data analysis t. Histograms, Tuples, Fitting, Data Points, Plotting, Management l Developed collaboratively at series of workshops by groups at CERN, LAL, SLAC. t. Next workshop June 30 -July 4 -- CERN l Interfaces developed for C++ and Java ( and maybe Python? ) l Several implementations/tools available t. Anaphe/Lizard/LCG PI – CERN t. Open Scientist – LAL t. JAIDA/JAS/AIDAJNI – SLAC

JAS 3 and AIDA n JAS 3 has adopted AIDA for analysis l AIDA allows us to leverage experience and skill of other developers l AIDA is functionally more complete than JAS 2 analysis package l AIDA allows JAS to exchange data with other AIDA tools l AIDA provides bridge to C++ programs (e. g. Geant 4) l AIDA encourages creativity and innovation n JAS 3 HEP Analysis tools based on JAIDA l JAIDA = Java implementation of AIDA t. JAIDA is part of Free. HEP library t. Usable as standalone library for any Java Application l AIDAJNI = Interface between C++ and Java AIDA t. Allows C++ programs to use JAIDA, JAS 3

JAS 3, AIDA and C++ AIDA Implementation AIDA- JNI AIDA C++ program . aida file (XML) JAIDA Java program JAS 3

JAS 3 and AIDA n JAS 3 supports all AIDA functionality, including l Histograms (includes arithmetic, projections, etc. ) l Clouds (unbinned histograms, scatterplots) l Plotter l Tuples l Fitting – AIDA interfaces allow for multiple fitters t. Uncmin -- pure java minimizer t. Minuit -- Fortran called by Java Native Interface (JNI) l IO t. AIDA XML, PAW, Root n JAS 3 supports user interaction with AIDA in three ways l Scripting (Pnuts, Python etc) l Compiled (Java) code l GUI – Plotting, Fitting, Cuts etc.

JAS 3 Scripting n JAS 3 has multi-language OO scripting support l Command line, Console, Editor l Major components (e. g. AIDA) have scripting interfaces n Currently have plugins to support l Pnuts – syntax almost identical to Java, fast, well documented and feature complete l Python (using Jython) l More scripting languages can be added tnot restricted to Java implementations (e. g. could use C-Python, JPE)

JAS 3 Lightning Tour n Tour designed to give you an overview of the capabilities of JAS 3, you can try them out for yourself this afternoon. Welcome Page, gives initial info and links to example scripts and programs Memory monitor

Opening Files Use file menu Drag from explorer

Graphical Interface to AIDA Histograms, Clouds, Tuples all presented in AIDA tree . aida files, . hbook files, . root files all presented as AIDA objects Drag items onto page, or use (popup) menus

Printing Or copy/paste into Word, Power. Point etc. Can send individual plots or full page direct to printer Or save as PS, EPS, PDF, SWF, SVG, PNG, GIF…

Java Editor, Compiler and Loader Tree shows loaded programs Built-in Java compiler Built-in editor for writing analysis code Unlike JAS 2 which only supported “event analyzers” JAS 3 allows any Java program to be loaded. This example “main routine” is taken directly from the AIDA manual

Scripting Can also write and run scripts Console allows direct interaction with scripting language

Pnuts Language n Currently support Pnuts scripting language l Complete and well documented thttp: //javacenter. sun. co. jp/pnuts/doc/guide. html l Fast (although not as fast as compiled Java) l Syntax very similar to Java l Can easily call compiled Java classes from scripts – best of both worlds n Plan to support other languages in future l In particular Python

Record Sources Opening record (or event) based files causes the run control toolbar to appear Works similarly to JAS 2 Job control, but now also supports random access and “tagged” data sets (mainly for event displays)

Tuple Explorer - Plots Histogram Works with any tuple, read from file or dynamically created Scatter. Plot XY Data (More appropriate for smaller data sets) Profile

Tuple Explorer – Define Columns

Tuple Explorer - Cuts

Tuple Explorer - Tabulate

Tuple Explorer – Record Source To be used with record loop

JAS 3 Spreadsheet n Simple spreadsheet plugin l for t Displaying results t Calculations t Simple Plots l Supports reading/writing t. csv files t Excel files l Cut/Paste with Excel etc l Coming Soon… t Scripting interface t GUI for building plots t User defined functions – Java, scripting

Miscellaneous Features Save/Restore configuration User Preferences Plugin Manager

Status n Currently released JAS 3 version 0. 7. 1 l AIDA functionality is quite solid l Compiler, Loader, Record Loop all quite recently added, t. Certainly still some rough edges n Documentation limited but available l Built-in example scripts and programs l Tutorial on web n If you are used to JAS 2 you will find some functionality not yet ported to JAS 3 l Remote (client/server) access to data. l 3 D Lego/Surface plots

JAS 3 and the GRID n We plan to add client-server/distributed capabilities to JAS 3 similar to those in JAS 2 l Will be based on (distributed) AIDA t. Next AIDA workshop (at CERN next week) will discuss this l Want to use Grid standards where they exist t. Work with others (PPDG-CS 11, ? ? ? ) to define standards where they do not exist l Want to be compatible with C++ servers l Tech-X have submitted phase II SBIR and if approved will work closely if approved

JAS 3 Links, More Info n JAS – Java Analysis Studio - http: //jas. freehep. org n JAS 3 – http: //jas. freehep. org/jas 3 n JAIDA – http: //java. freehep. org/jaida/ n AIDA – http: //aida. freehep. org n Free. HEP - http: //www. freehep. org n Free. HEP Java Libraries - http: //java. freehep. org n WIRED – http: //wired. freehep. org