Recent results from the Swinburne supercomputer VLBI software

Recent results from the Swinburne supercomputer VLBI software correlator(s) Steven Tingay, Craig West, Adam Deller, Shinji Horiuchi http: //astronomy. swin. edu. au Centre for Astrophysics and Supercomputing Swinburne University of Technology 4 th e. VLBI Workshop: July 12 – 14 2005 ATNF Marsfield

Outline of talk • • Goals of project. Why correlate in software? • • Availability of system to proposers. Results for Vela pulsar. Results for J 0006 -0623. Results for PKS 0438 -436 and PKS 0405 -385. Details of the observing/correlator setup. Performance of the software correlator. • OH masers • Wide-field imaging, other pulsars, “Double pulsar” observation on Saturday (ATCA/Parkes) • Summary.

Goals • Use supercomputers to demonstrate SKA-related techniques and technologies: process in software the sampled, digitised output of single dish and interferometric radio telescopes (VLBI/e. VLBI!!); • Develop these techniques and technologies and integrate them into existing radio astronomy National Facilities; • Activities funded by State and Federal bodies and industry partners, in close collaboration with the ATNF, University of Tasmania, the Auckland University of Technology (New Zealand), and the University of Western Australia.

VLBI system Apple Xraid disks ~ few GHz RF electronics Δν = 16 MHz (56 – 72 MHz) Beowulf cluster Sample/ Digitise 101001010… Parkes Multibeam correlator Board (applicationspecific device) Cray XD-1 PC Dell server-class (generic processing device) RF electronics ATCA Δν = 16 MHz (56 – 72 MHz) Sample/ Digitise 101001010…

Why correlate in software? • Moore’s law over the last ten years has made it possible – hopefully Moore holds on a bit longer; • Short development time for code – well known problem, well defined algorithms; • “Embarrassingly parallel” computing problem – suits parallel, multi-processor machines very well – strong cross-fertilisation with HPC and ICT sectors (LOFAR/Blue Gene). Vendor interest in the problem; • The algorithms are very flexible and it is easier to alter the code to reach new regions of parameter space than alter correlators with fundamental hardware constraints; • Software correlators suit small to medium sized arrays.

Details of the VLBI setup – PC-based data recorders (see Chris’s talk) • Dell 1600 SC or similar (now superceded); • Metsahovi Radio Observatory “PC-EVN” VSI boards; • Apple Xraid mass storage (5. 5 TB with latest disks) – Swinburne Beowulf cluster supercomputer • 304 P 4 processors; • Gigabit ethernet; • 6 x Apple Xraid units; – Cray XD-1 chassis • 12 Operton processors; • 6 Xilinx FPGAs (ultimately PGPUs as well) • Co-owned with University of Western Australia and ATNF

Details of the software correlator • Currently runs on Beowulf cluster under MPI; • Uses Intel Performance Primitive (IPP) libraries for optimised crosscorrelation and Fourier transform routines; • First two versions of code implemented as XF correlators – next generation will be FX (see Adam’s talk next); • Complex correlation (multithreaded on single processor PCs); • Post-correlation correction of fractional sample errors on timescales of ~few ms; • Apply system temperature and gain information on the fly, standard FITS output (AIPS, AIPS++, MIRIAD, DIFMAP etc); • Arbitrary correlator integration time (down to microseconds) and number of frequency channels (correlated up to 16000 for narrow band maser observations) – applications for pulsars, wide-field imaging, and super-high spectral resolution masers for example; • Correlator is currently much slower than real-time, depending on the required output (getting much faster – see Adam’s talk); • Trade off unique capabilities with speed performance!

Speed performance of current XF correlator # baselines # polarisation products Bandwidth (MHz) # frequency channels integration time (ms) observation time (h) correlation time (h) # processors X real-time 1 1 16 128 25 5. 5 10 20 1. 8 1* 1* 4* 64* 25* 6* 1. 5* 20* 0. 25* To correlate full Australian LBA: 6 antennas = 15 baselines; 8 x 16 MHz (4 x 16 MHz at RCP; 4 x 16 MHz at LCP); 128 channels; 25 ms integration time; 12 hour observation; All polarisation products; 300 processor cluster; ~180 hours * system is i/o bound. Recent improvement to i/o handling gives ~x 2 speed increase. For next generation FX correlator, see Adam’s talk, next. Faster! Stay tuned for XD-1

Availability of the system • We can now correlate in software VLBI observations at up to x 4 the bandwidth of the S 2 tape-based VLBI system (factor of 2 improved sensitivity): – Aim for 1 Gbps operation (x 8 bandwidth) in real time via optical fibre in 2006 (SKA demonstration) • Data are correlated in software on Swinburne supercomputer – eventually on the end of AARNet; • New facilities are available as part of the VLBI National Facility from June 2005, supported by the MNRF team at Swinburne, the ATNF, and other VLBI partners (first proposals received): See http: //www. atnf. csiro. au/observers/apply/avail. html • Attempting first Trans-Tasman VLBI with Auckland University of Technology facilities; • Fringe-check software (uses software correlator) improves VLBI reliability – see Chris’s talk

Vela pulsar 2. 3 GHz observations, Parkes to Tidbinbilla (test the models and analysis of Gwinn et al. 1997, 2000) Average pulse profile. 10 bins over on-pulse region. Bins of ~1 ms in width. Can generate arbitrary number of bins and bin width. Coherent de-dispersion prior to correlation. Analyse the amplitude statistics in each bin and fit scintillation models to estimate size of pulsar emission region. Work in progress.

Results for J 0006 -0623, PKS 0438 -436, and PKS 0405 -385 2. 3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra (LBAHDR) Hartebeestoek (Mark 5) Kashima (K 5, 256 MHz - digitally filtered in software to match 16 MHz band – Hiroshi Takeuchi, July 2004, NICT News) 2 milliarcsecond resolution Science paper in preparation

Quasar PKS 0438 -436: 2. 3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra, Tidbinbilla (LBAHDR) 10 mas resolution Quasar PKS 0405 -385: Famous intraday variable source (Kedziora-Chudzer et al. 1997) 2. 3 GHz (4 x 16 MHz) ATCA, Parkes, Hobart, Ceduna, Mopra, Tidbinbilla (LBAHDR) 10 mas resolution

OH masers • 4 MHz • 16384 frequency channels = 244 Hz/ch • 1 second of data • LL polarisation • Parkes – ATCA • 1720 MHz

Widefield imaging/other pulsar experiments • Observation of NGC 4945 in May to show that software correlator can be used for wide-field VLBI (2. 3 GHz; 4 x 16 MHz). • Observation of millisecond pulsar scintillation (GBT, Effelsberg, Arecibo; Mark. V – correlated in software at Swinburne (project proposed – pending approval). • Observation of the “double pulsar” on Saturday (this Saturday!: Parkes/ATCA/Hobart; 1. 4 GHz; 8 x 16 MHz – Huygens mode) – – Pulsar “eclipses”; Parallax/proper motion; Pulsar nebula. Phase referenced

Software correlator for realtime fringe checking (see Chris’s talk)

Summary • Swinburne software correlator is operational and available as part of the Australian VLBI National Facility; – Capable for multiple recording formats (LBAHDR; Mark V; K 5 ……. • The next generation FX version will be much faster than our current XF – real-time Gb? ; • Software correlators are: – – Relatively easy to develop – good for niche users; Capable and flexible – good for niche science; Relatively slow – compared to dedicated hardware; Great fun!!