Digital Packaging Processor Overview Gordon Hurford Nov 7

Digital Packaging Processor - Overview Gordon Hurford Nov 7, 2011 EOVSA Technical Design Meeting - NJIT

Digital Packaging Processor • Role of DPP • Overall assumptions and priorities • Interface Overview • Tasks and algorithms • Hardware/software Implementation ----------------------------- • DPP Interface Details

Data System Approach Fundamental Drivers • • Very limited software resources Non-trivial data rate and volume Automated analysis pipeline for efficient observing Must have science-useable system in place by September 2013 • Data products to be readily useable by broader solar community – Data products with preset parameters – Data products with user-selected parameters – Tools and support for experienced users

Data System Approach Implications • Monomode observing • Calibrated data archived in application-specific databases • Reliance on existing software packages – Miriad package for calibration & mapping – RHESSI Solar. Soft package for user interface and data product display – RHESSI database model • • Err on side of over-rejection of data Limited initial support for ‘nice-to-have’ options Limited initial support for calibration refinements Limited support for non-solar applications

Data System Assumptions • All information required for data analysis is written by the DPP to the Interim Data Base • Engineering data acquisition, archiving and display is the responsibility of the ACC, and is “largely” decoupled from science data.

Nomenclature • Data frame = Interval representing data from one correlator cycle (20 ms, ~4000 channels with 500 MHz range) • Spectral frame = Data corresponding to a complete frequency-agile cycle (nominal 1 second, 10 s to 100’s of ‘science channels, 18 GHz rang) Corresponds to a state frame. • Scan: Observing interval within which target and frequency cycling pattern is unchanged

Role of Digital Packaging Processor • To filter, average, partially calibrate and convert raw correlator output into a Miriad-compatible format that is written to Interim Data Base

DPP Interface Overview Correlator ACC <P>, <P 2>, Correlations Start / End Scan Commands Miriad Scan-independent Calibration Parameters format Interim Data Base DPP Scan Parameters Frame parameters State Frame status report RFI 1 s timing tick 0. 02 s timing tick results Internal RFI Database

DPP Task Timing • Occasional – non operational – Accept, store and preprocess calibration parameters • Scan initiation – Accept, store and preprocess scan-specific parameters • Data frame (20 ms) – filter, and frequency-average correlator output • Spectral frame (1 s) – Assemble, pre-calibrate, reformat and write data to Interim database • TBD – Format results and write to RFI database

DPP – Stage 1 Processing Every data frame (20 ms) • Evaluate kurtosis data to identify RFI-affected subbands as a function of frequency only. • Save RFI statistics • Combine with pre-flagged subbands to generate a “destination vector” for each subband • Apply complex gains at subband level ? ? ? • Average subband data into spectral channels • Save 1 st 3 moments of averages ? ? ?

DPP Stage 2 Processing Every spectral frame (1 s) – Convert antenna-based flags (e. g. slewing) from state frame to baseline-based, frequency-independent flags – Apply time-independent complex gains if available – Apply baseline corrections – Apply non-linearity corrections – Correct for attenuator settings – Correct for spectral simultaneity • Miriad format this is no longer optional – Convert visibility, uv and analysis-relevant state-frame data to Miriad-compatible format – Write spectral frame to IDB – Report DPP status to state frame

DPP - Implementation • Original concept was to follow FASR plan for a cluster-based DPP • Estimate processing requirements for EOVSA DPP at ~100 MIPS = 1/60 of FASR requirements • Implementation will be based on a single multi-core machine • Software organization will be compatible with migration to a cluster if necessary

DPP Software Architecture ACC State Frame IDB Correlator RFI database DPP Coordination Task C 1 I/O, data assembly, no processing per se Pointers within shared memory Parameter Processing Header Processing Stage 1 Processing Stage 2 Processing C 2 C 3, C 4 C 2 Conventional, time-independent processing tasks Cn = core within a quad core processor or nodes in a cluster

DPP Status • Software architecture and tasks identified • Detailed definition of interfaces is underway • EOVSA to Miriad format conversion being tested with FST data – (Fortran 77 for Miriad compatibility) • Next: 1. Complete definition of interfaces 2. Code Stage 1 tasks (GH) • Evaluate timing requirements 3 Code Coordination task (JM). 3. Detailed definition of processing algorithms 4. Code of Stage 2 tasks 5. Machine selection and purchase • Development platform? • Goal: Functional DPP to support prototype testing