Di FX Software Correlation at Swinburne for the
Di. FX: Software Correlation at Swinburne for the LBA Adam Deller Swinburne University/CSIRO Australia Telescope National Facility Supervisors: A/Prof Steven Tingay, Prof Matthew Bailes (Swinburne), Dr John Reynolds (ATNF) 03/05/2006
Outline • History • Correlator architecture • Capabilities • Computing environment and resources • Usage inside and outside LBA • Future improvements 9/6/2007 2
Correlator History • Commenced with my Ph. D thesis (2005) • Preliminary work (XF) undertaken in 2004 by Craig West • First science late 2005 • Conversion to distributed system 2006 • Verification with LBA and VLBA correlators, code released late 2006 http: //astronomy. swin. edu. au/~adeller/software/difx/ 9/6/2007 3
Correlator Architecture Written in C++, MPI for message passing, Intel IPP for vector arithmetic 9/6/2007 4
Capabilities • Supports LBA, Mk 5* (K 5 ->LBA converter) • Pulsar gating, binning or matched filter • Arbitrary time/frequency resolution • Presently, visibilities in RPFITS format • e. VLBI: data can be streamed from network socket directly into correlator • Also used for real-time fringe checks 9/6/2007 *export to linux filesystem 5
The Swinburne supercomputer • Swinburne supercomputer presently consists of ~300 P 4 processors • Typically “reserve” ~30 processors for VLBI after observing run • Next month: complete replacement of cluster with ~140 dual CPU, quad core nodes, >1100 cores total 9/6/2007 6
Computing requirements • Di. FX requires little RAM (except extreme cases): processor speed, cache and SIMD support essential • Performance scales linearly with aggregate bandwidth, near-linear with number of stations • Realtime example: Max sensitivity LBA (6 x 1 Gbps) requires ~25 new nodes 9/6/2007 7
Typical LBA usage • Required to correlate ~5 days of 256 Mbps before next session (2 months) • For each experiment, automatically select and correlate fringe-finders, solve clocks, followed by full correlation. • Using ~30 (old) machines, high data rate experiments take 2 x observe time 9/6/2007 8
Non-LBA applications • New geodetic array in Australia/NZ to use software correlator: modifications to control, output format • VLBA is trialling software correlator to run in parallel with hardware correlator: experiments with specific requirements, testing high data rate 9/6/2007 9
Future improvements • Current version is production: future improvements will be incremental • Direct read from Mk. V • Alternative output formats: AIPS++ MS, FITS-IDI, UVFITS • Support in AIPS: tasks like CVEL currently confuse with LBA S 2 (XF) 9/6/2007 10
9/6/2007 11
Interferometry & Correlators Interferometry: delay signals from two dishes to common reference and multiply Common signal, independent noise: average improves S/N RA Dec Earth rotates, interferometer samples visibility: Fourier transform of sky brightness 9/6/2007 12
Interferometry & Correlators Finite bandwidth, not monochromatic, therefore visibility varies across band To get v( ), XF correlator accumulates lags, then FFTs u ( ) v ( ) FX correlator FFTs segments of baseband data, and crossmultiply/accumulates 9/6/2007 13
Hardware vs software • Software correlator: program running on a • • computer cluster/supercomputer Hardware correlator: ASIC boards, specialised data transport Software is unclocked, could be faster or slower than real-time No channel/integration time restrictions Floating pt vs int calculations 9/6/2007 14
Why software? • Flexibility - you can do things that are • • • impossible with a hardware correlator Rapid (and cheap) development Add-ons MUCH easier in software Compatibility Expandability For me: Allow disk-based correlation, and improved pulsar binning (sensitivity) 9/6/2007 15
Di. FX (Distributed FX) Data. Stream 1 Baseband data Core 1 Data. Stream 2 Source data … Core 2 … Data. Stream N Core M Timerange, destination Visibilities Master Node MPI is used for inter-process communications 9/6/2007 16
Di. FX (Distributed FX) • Configured by text files (like jobscripts) • Delay modelling - CALC 9 (separate) • Output: RPFITS (built on-the-fly) • Arbitrary time/frequency resolution • Arbitrary pulsar binning (incoherent dedispersion) - allows weighted bin sum • Real time LBA @ 1 Gbps: 100 -200 CPUs 9/6/2007 17
Di. FX (Distributed FX) • Verification: recent successful comparisons with LBA and VLBA 9/6/2007 18
LBA Science • All require one or more of flexibility, high time/frequency resolution, or sensitivity • Wide field VLBI (Lenc & Tingay) • Masers (Horiuchi) • CDF radio counterparts (Norris et al. ) • RRATs (Kramer et al. ) • e. VLBI (Phillips et al. ) • Pulsar parallax (me) 9/6/2007 19
Worldwide science • So far, motivated by very high frequency resolution, or flexibility and minimum effort for new system • Pulsar scintillation (Brisken) requires extreme frequency resolution (244 Hz channels over 32 MHz bandwidth) • Geodesy (MPIf. R, Bonn, Germany) t • Geodesy, new Australian array (NCRIS) 9/6/2007 20
Status: Correlator code • “Correlation” code completed and verified - now finishing GUI and packaging for public release • VLBA, MPIf. R and others keen to continue trialling code once released • Should be online within a month • PASP paper submitted simultaneously 9/6/2007 21
Status: Observing and reduction • Four sessions of eight allocated, two observed - applications for 2007/2008 year (four more sessions) soon • Both observed sessions correlated and verified • Starting to work on atmospheric/ ionospheric compensation and pipeline, crucial for efficient, accurate astrometry 9/6/2007 22
Roadmap to submission 1 month Di. FX available online 3 months First four sessions complete 6 months Reduction techniques tested, pipeline started 12 months Pipeline complete, draft begun 18 months Data collection, reduction and thesis draft complete Thesis submission 20 months 9/6/2007 23
Conclusions • Di. FX is a general purpose software correlator, publicly available soon • Already used with success in Australia by the wider VLBI community, and generating interest internationally • Pulsar parallax program is underway, with the bulk of observations to come • Transitioning from code to science 9/6/2007 24
9/6/2007 25
Interferometry & Correlators • Major operations performed by FX software correlator: • • • Delay Per station N Unpack quantized data to float Fringe rotate FFT Correct fractional sample error Cross multiply and accumulate A B Datastream 9/6/2007 Per station N 2 26
- Slides: 26