The OSKAR simulator version 2 3 GCII Workshop

The OSKAR simulator (version 2!) 3 GC-II Workshop, 21 st September 2011 Benjamin Mort, Fred Dulwich, Stef Salvini http: //www. oerc. ox. ac. uk/research/oskar 1

What is OSKAR? • Interferometer and beamforming simulator package. – End-to-end simulations of the phase 1 SKA. • Based on a full sky Measurement equation formalism. • High performance library based on NVIDIA CUDA. 2

Motivation for writing OSKAR? • Understanding system noise limits on SKA like AA interferometers. – Dynamic range limits • All sky simulation. • High performance required for large AA interferometer end-to-end simulation. – e. g. ~1 e 6 sources, ~25 stations of ~10, 000 dual polarisation antennas. • Interferometer configuration studies with aperture arrays. 3

What I’m going to talk about! • The OSKAR ME implementation. • OSKAR version 2. 0 • Some example and results. 4

Measurement Equation • The ME as implemented by OSKAR – Scalar version currently being tested, polarised version under development. • • • K – interferometer phase. E – Station beam. G – Antenna element field pattern. P – Propagation term. B – Source brightness. V – Complex visibility. Baseline p, q for all visible sources, s. 5

Sky Model • The global sky model – Equatorial coordinate point source model. • A local sky model is generated for each snapshot – Remove sources below the horizon – Transforming the source Stokes parameters to the horizontal frame. • Emission from bright, extended objects can be included as large collections of point sources. 6

G – Antenna field pattern matrix Embedded element patterns • The average (fully polarized) embedded element pattern for antennas within a station – Factored out if the antennas are sufficiently similar. – otherwise, it is absorbed into the calculation of the E-Jones term. • Requires input from EM simulations (University of Cambridge). – Depends on the station geometry (cross coupling) and element design. • Simple functional responses also possible. 7

Station Beams (E-matrix) • Rotate phase tracking centre (=beam direction) and all sources from equatorial (RA, Dec) to horizontal (azimuth, elevation). • Evaluate station beam response for every source and station, for the direction of interest. • Sources far from the phase centre will be suppressed by station “primary” beams. – (And sources far from the zenith will be suppressed further by antenna element pattern. ) • Obtain complex matrix 8

Station Phases (K-matrix) • K-matrix effectively “phases-up” the array of stations. • Compute phase of each source s at every station a. – Determine station (u, v, w) coordinates by rotating (x, y, z) onto a plane perpendicular to direction of phase centre. 9

“Correlator” • Multiplies appropriate Jones matrices with the source brightness to obtain a complex visibility per source and per baseline, and then collapses the source dimension. • Time-average smearing: each visibility point can be averaged over time. – K is recomputed to include motion of baseline during integration period. – E is allowed to vary throughout the integration at a slower rate than K. • Bandwidth smearing: multiply each visibility by fs, i, j before collapsing the source dimension. 10

The OSKAR Package • OSKAR simulation function library based written in C making extensive use of CUDA • Multiple libraries with simple dependencies – Liboskar • Core CUDA function library. – Liboskar_ms • Interface to casacore for writing simple measurement sets. – Liboskar_apps • Utility library for using OSKAR to write C/C++ applications. – Liboskar_widgets • Set of utility widgets written in Qt 4/Qwt 5. Plotting, gui components. – Liboskar_imaging • FFT imager (CUDA based imager in development) with w-projection. – Liboskar_fits • Interface to cfitsio for writing UVFITS and image fits files. 11

The OSKAR Package • Simple C like interface to the main simulation library using intrinsic types where possible. • Aims to make it possible to quickly construct new C/C++ simulation applications. • Designed to interface easily with other languages – MATLAB either with loadlibrary() or though a MEX interface. – (Python) • MATLAB and C/C++ applications – – Beamforming simulator AA Interferometer simulator DFT imaging Simple image post processing MATLAB scripts e. g. CLEAN. 12

Why CUDA? 13

Why CUDA? • What is CUDA? – CUDA (Compute Unified Device Architecture) is NVIDIA’s – Program development environment based on C/C++ with some extensions. – Compatible with other multi threaded code. – Multiple GPUs can be used to work together for very large problems. • Cost and power effective desktop supercomputing. – SIMT parallelism model. – Requires tens of thousands of threads to be efficient. • Multiple GPUs work together for very large problems 14

Summary of current Features / assumptions Sky Station Interferometer • Equatorial system. • Point source sky model. • Support for large numbers of sources, ~O(10^6) • Use of point source catalogues / image pixels. • Currently only stokes I (polarisation support very soon) • Support for any configuration of antenna elements. • Optimised for large numbers of antennas (e. g. 10, 000+ per station). • Primary beam evaluated for each station. • Antennas currently assumed to be all identical within a station. • EM coupling encapsulated in element pattern. • Any latitude and longitude • Any station positions. • Time-averaging smearing by actual average. • Analytical bandwidth smearing per source and per baseline. • Series of visibility snapshots. 15

Some examples of using OSKAR 16

Source distribution, 2 -degree “hole” at phase centre (49993 sources) 17

Field of view 2 deg across, nearest source is 1. 32 degrees from centre (Telescope at 40 degrees, 480 snapshots, 49993 sources elsewhere) Peak @ ~3 E-3 18

Imperfect source subtraction • Bright interfering source on the flank of the station beam at position X. • A number of other sources scattered over the sky. • Because the source has effectively become highly timevariable, a simple subtraction of its clean-component model leaves large residuals. • limiting the dynamic range of the image. 19

Source removal • Solving for differential gains in Meq. Trees (Ian Heywood) is far more effective. 20

Simulation Example: Observation setup. • Telescope at Faro • Pointing around Cassiopeia. • 24 h observation. 21

Simulation Example: AA station setup • Offset grid geometry. • ~80 m diameter. • ~2600 antennas. 22

Simulation Example: Beam pattern. 23

Simulation Example: Sky model (Simulation with E-Jones disabled) 24

Simulation Example: Telescope setup. 25

Simulation Example: results 26

Simulation Example: Dirty image snapshots 27

Simulation Example: Frequency time source brightness profiles (1) (3) (2) 28

Next Steps • Currently working on – Polarisation. – Efficient modelling of system noise. – Antenna gain and phase errors. – Limited precision numerics (currently floating point). – Scaling up to very large simulations using multiple GPUs • Any suggestions? 29