AMDA Automated Mutiple Dataset Analysis application to data

AMDA Automated Mutiple Dataset Analysis: application to data simultaneously obtained onboard THEMIS, CLUSTER and other missions. C. Jacquey, V. Génot, E. Budnik, R. Hitier, M. Bouchemit CDPP/CESR, Toulouse, France jacquey@cesr. fr http: //cdpp. cesr. fr/ THEMIS launch meeting Cape Canaveral Feb. 13 -14, 2007

Abstract AMDA (Automated Multi-Dataset Analysis) is a service developed at CDPP. AMDA is a suite of integrated tools allowing to perform massive scientific processing of the content of multi-datasets in automated or semi-automated ways. This service is built for working with a local database or (via web-services) distant data sources as well. The functionalities offered in AMDA include (i) user edited visualization browse, (ii) semi-automated event search (visual identification), (iii) automated event search (based on user edited mathematical criteria applied to the content of the data), (iv) data extraction, (v) user edited parameter computation and (vi) access to data stored in distant sites. In this paper, we present AMDA, its functionalities and results obtained in studies based on the use of AMDA. The CDPP (Centre de Données de Physique des Plasmas) is the french data center for solar system plasma data. The CDPP has been created both CNRS and CNES in 1998 and is hosted at CESR (Centre d’Etudes Spatiales des Rayonnements), Toulouse, France. Primary objectives of CDPP is to (i) assure long term preservation of data obtained with french contribution, (ii) to provide technical and scientific services and (iii) to participate to the development of interoperability and virtual observatories.

THEMIS at CESR and CDPP CESR Cluster, Double. Star, Ace, …, database CL Mirror base of the THEMIS space data THEMIS database: K-parameter AMDA CDPP base of standardised data CDPP Distant data and services: CDAWeb, CAA, GAIA, SSCweb, Darts, … CL: server for interactive detailed visualisation and analysis, high resolution data, on flight calibration, high level products of data, user defined parameter or visualisation, … (see poster of E. Penou and J. A. Sauvaud, this session) AMDA: server for automated, semi-automated or simply manual data analysis. AMDA works on a k-parameter database and provides facilities for visualisation, data extraction, semiautomated/automated event or signature search/ characterisation. AMDA produces event list and catalogs. AMDA is developed for being interoperant with other data centers or services. For now, AMDA implemented the web-services offered by the CDAWeb. This allows AMDA to acceess to the CDAWeb database in a way transparent for the user.

AMDA (Automated Mutiple Dataset Analysis ): the goals Multiple Dataset: the study of the (multi-scale) dynamics of plasma objects requires to perform the integrated analysis of multi-point and multi-instrument data Automated (or semi-automated) Analysis: help for ØSearch, characterisation, classification of events ØExtensive exploitation of the vast database (statistics) Ø“historical” studies (example: through solar cycles) ØBuilding catalogues ØBuilding virtual constellations AMDA, fact sheet • Now, very first prototype. Operational version v 1. 0 sould be openend in march. • Forum for user feedback • Already in use for scientific studies • Prototyping development. • Built on an existing data system (DD_System, A. Fedorov) • Output: data and derived parameter files, plots, event lists, catalogues. Underway: • Built in IDL, C, FORTRAN, Javascript • Works on standardised local database through universal requests • Implementation of the CDAWeb web-services • models (Tsyganenko, Shock, MP, NS, …) implementation • Interface development for running AMDA on other databases

Distant services CDAWeb SPDF CASSINI MAPSKP se Distant services Distant database W e rv bice s Distant database External interface User interface USER System interface Local databases Virtual objects AMDA Services Schematic description of the envisioned system

Local database. Themis and Double. Star to be added

AMDA/workspace: user defined parameter computation

AMDA/plot: visualisation and semi-automated event/signature search (1)

AMDA/plot: visualisation and semi-automated event/signature search (2)

AMDA/download: data extraction

AMDA/Conditional search

Example: Looking for possible X-line with a pair of spacecraft Output: event list + gap list In XML format. Other formats are also envisioned BZ 1 * BZ 2 < 0 AND VX 1 * VX 2 < 0 <? xml version="1. 0" ? > - <Time. Table Name="essai 1"> <Created>Sun Feb 11 12: 21: 25 2007</Created> - <Chain> <![CDATA[ B_C 1(2)*B_C 3(2)<0&MOM_C 1(1)*MOM_C 3(1)<0 ]]> </Chain> <Source>AMDA Search</Source> - <Interval> <Start>2001 -08 -01 T 23: 50: 00</Start> <Stop>2001 -08 -02 T 00: 00</Stop> </Interval> - <Interval> <Start>2001 -08 -02 T 00: 05: 00</Start> <Stop>2001 -08 -02 T 00: 30: 00</Stop> </Interval> - <Interval> <Start>2001 -08 -02 T 02: 00</Start> <Stop>2001 -08 -02 T 02: 15: 00</Stop> </Interval> - <Interval> <Start>2001 -08 -02 T 02: 20: 00</Start> <Stop>2001 -08 -02 T 02: 40: 00</Stop> </Interval>

AMDA/implementation of the CDAWeb web-services

Conclusion ØAMDA allows to attack standardised data bases either massively or case by case ØComputation of user defined parameters from multi-datasets ØDescriptive model computation (Tsyganenko, …) ØMulti-dataset visualisation ØData and computed parameter extraction ØCDAWeb web-services ØEvent or signature search, characterisation and classification. Event list creation and managment: %in an interactive way, by visual identification %In an automated way, by editing mathetical criteria on the content of the data ØPerspectives for exchanging event lists with other services (SPDF, CDAWeb, GAIA, CAA, DARTS, …)? AMDA for Themis %Integrated K-parameter database: Themis (space data), Cluster, Double. Star, Geotail, … The level (SP or PP parameters) of Cluster-Double. Star data depends on co-I/non co-I %Transparent connection to CDAWeb… and others? %Providing event list for CL… and others?

Some examples: AMDA + CL (or other softwares): 1) Users produces time-tables on AMDA based on their own defined criteria applied on the content of the data they selected 2) Time-tables sent to CL 3) Users browse user-defined plots on the periods of the timetables and then run detailed analysis on CL SSCWeb (or DARTS) + AMDA: 1) Users produces time-tables on SSCWeb based on criteria applied on the orbitography and conjonctions. 2) Time-tables sent to AMDA 3) Users produces secondary time-tables on AMDA based on criteria applied on the content of the data for the periods of SSCWeb time-tables 4) Then, these secondary time-tables are exported to detailed analysis softwares or are used inside AMDA for building catalogues, or …

Demo Derived parameters computation (calculator) Automated and semi-automated search Sub-dataset conditional extraction Distant access to complementary data

An example: search of the events when the cross-tail current is “encircled” by CLUSTER Looking for the time periods when 2 s/c are above the neutral sheet and the 2 other ones are below Test: • X 1< -10 Re • BX 1 * BX 2 * BX 3 * BX 4 > 0 • min([BX 1 BX 2 BX 3 BX 4]) < 0 • max([BX 1 BX 2 BX 3 BX 4]) > 0

CL Software for detailed analysis of multi-datasets in space physics Developed by Emmanuel Penou (penou@cesr. fr) Scientific specifications by Jean-André Sauvaud THEMIS SWG meeting SSL Nov. 15 -16, 2006

CL: fact sheet Multi-missions (CLUSTER, VEX, MEX, DEMETER, ACE, STEREO, …) and multi-instrument (B, E, particles, Waves) SOLARIS, LINUX, WINDOWS IDL and C Calibration files L 1 files L 2 files (Internal format CDF, ASCII) Processing Visualisation Computation of derived products Export data (CDF, ASCII) (Conditionnal search)

Demo ØPredefined plots ØBuilding, editing, manipulating plots ØExternal files. CDF files ØHigh level products: -distribution function -spectrograms in PA, 1/V -partial moments ØRemote running

Conclusion: what can be done at CESR and CDPP for the THEMIS community? Mirror site for space THEMIS data Distributing CL, but limited user help Remote running of CL on dedicated machine(s) AMDA for event search, catalogue building, statistics on “simple” THEMIS and complementary data Coupling AMDA event search and CL (and any other softwares or system: CDAWeb, UCB, …) by exporting timetables

Contacts CL: penou@cesr. fr AMDA: jacquey@cesr. fr http: //cdpp. cesr. fr/ forum for feedback on the CDPP services (in french for now)

Example: event search I ele SEE ctr on Old fashion: “paper” search ISEE-data AL IMP PLA 8 P 8 IM AG M AE, AL IMP-8 , AE ISEE MAG Current fashion: web CDAWeb SPDF UCLA CAA CDPP And others Event search takes time and energy

AMDA, fact sheet • Now, very first prototype (construction started in June 2006 after kind of phase-A) • Built on an existing data system (DD_System, A. Fedorov) • Built in IDL, C, FORTRAN, Javascript • Works on standardised and “simple” data, Net. CDF • Includes models (Tsyganenko, Shock, MP, NS, …) • Attempt for using web-services of CDAWeb and the up-coming Vx. Os • Forum for user feedback • Already in use for scientific studies