SCARIe e VLBI Software Correlation Over Dynamic Lambda

SCARIe: e. VLBI Software Correlation Over Dynamic Lambda Grids Damien Marchal, on behalf of the SCARIe team. University of Amsterdam The Netherlands

SCARIe, is a project to make a software correlator for VLBI on top of a grid middleware and infrastructures.

Outline • Introduction: • Radio astronomy • VLBI and e. VLBI • Correlation • The SCARIe project • What is a software correlator • Two mode of operation: Batch or Real-time • Networking challenges • Experiment using DAS-3 and Star. Plane • Conclusions 9/16/2020 Software Correlation Over Dynamic Lambda Grids 3

Outline • Introduction: • Radio astronomy • VLBI and e. VLBI • Correlation • SCARIe project • What is a software correlator • Two mode of operation: Batch or Real-time • Networking challenges • Experiment using DAS-3 and Star. Plane • Conclusions 9/16/2020 Software Correlation Over Dynamic Lambda Grids 4

Radio and optical 408 Mhz optical 1. 4 Ghz 9/16/2020 Software Correlation Over Dynamic Lambda Grids 5

Larger dish: higher resolution Resolution: λ / b Sensitivity: b 2 λ = wavelength b = diameter telescope Arecibo: D = 305 m 9/16/2020 Software Correlation Over Dynamic Lambda Grids 6

Aperture Synthesis Imaging Image Credits: Avruch and Pogrebenko • A technique that uses a number of telescopes to simulate a much larger one. A larger dish, real or simulated, improves image clarity and brightness. This requires coordination between the telescopes and a supercomputer. Consider the examples displaying aperture size, aperture distribution and image quality. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 7

Westerbork / Very Large Array 9/16/2020 Software Correlation Over Dynamic Lambda Grids 8

Even higher resolution: VLBI 9/16/2020 Software Correlation Over Dynamic Lambda Grids 9

Radio-Telescopes participating… 9/16/2020 Software Correlation Over Dynamic Lambda Grids 10

Data Flow Past Today Results Today Soon Today 9/16/2020 Software Correlation Over Dynamic Lambda Grids 11

Data Flow Past Today Results Today Soon Today 9/16/2020 correlation Software Correlation Over Dynamic Lambda Grids 12

Outline • Introduction: • Radio astronomy • VLBI and e. VLBI • Correlation • SCARIe project • What is a software correlator ? • Two mode of operation: Batch or Real-time • Networking challenges • Experiment using DAS-3 and Star. Plane • Conclusions 9/16/2020 Software Correlation Over Dynamic Lambda Grids 13

SCARIe: the project The starting point: The increase of size/performance of a modern grids/clusters and networks make of grid a “to investigate” plate-form for software correlation. Today Results Today 9/16/2020 Software correlator (on a grid) Software Correlation Over Dynamic Lambda Grids 14

SCARIe: the project The starting point: The increase of size/performance of a modern grid/cluster and networks make of grid a “to investigate” plate-form for software correlation. Software Correlator Hardware correlator • High flexibility: • Tracking of spacecrafts • Research on masers • Low flexibility • Share the resource/grid with others to lower the cost • More efficient use of computing resources (Lower energy consumption) • Run everywhere 9/16/2020 • Superior computing power Software Correlation Over Dynamic Lambda Grids 15

� SCARIe: distributed correlation Telescopes Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 16

� SCARIe: distributed correlation Telescopes Input nodes Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 17

� SCARIe: distributed correlation Telescopes Input nodes Correlation nodes Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 18

� SCARIe: distributed correlation Telescopes Input flow = sum(output flows) Input nodes Correlation nodes Output node Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 19

� SCARIe: distributed correlation Telescopes Input nodes � Input flow = sum(output flows) Current experiment at: 4 x 256 Mbps Correlation nodes Output node Correlator Full size experiment: 16 x 1 Gbps Future e. VLBI: 32 x 4 Gbps 9/16/2020 Software Correlation Over Dynamic Lambda Grids 20

SCARIe: distributed correlation We can identify two kinds of data flows. Input flows, link to the RT. In-grid flows. Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 21

SCARIe: distributed correlation We can identify two kinds of data flows. Input flows, link to the Radio-Tel. (point to point, long distance, several Gbps) In-grid flows. (one to all, short distance, low latency). Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 22

SCARIe: operational mode Two kinds of operational mode Batch correlation: • the data are saved on hard-drives and replayed; • the network performance and number of nodes impact the processing speed. Real-time correlation: • the radio-telescopes are streaming the data directly into the software correlator; • the network performance and number of nodes impact the success of the observation. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 23

SCARIe: operational mode We can identify two kind of operational mode… Batch correlation: • the data are saved on hard-drives and replayed; • the network performance and number of nodes impact the processing speed. Fastness ! Real-time correlation: • the radio-telescope are streaming the data directly into the software correlator; • The network performance and number of nodes impact the success of the observation. Reliability! 9/16/2020 Software Correlation Over Dynamic Lambda Grids 24

SCARIe: workflow 9/16/2020 Software Correlation Over Dynamic Lambda Grids 25

SCARIe: workflow 9/16/2020 Software Correlation Over Dynamic Lambda Grids 26

SCARIe: workflow 9/16/2020 Software Correlation Over Dynamic Lambda Grids 27

Outline • Introduction: • Radio astronomy • VLBI and e-VLBI • Correlation • SCARIe project • What is a software correlator • Two mode of operation: Batch or Real-time • Networking challenges • Experiment using DAS-3 and Star. Plane • Conclusions 9/16/2020 Software Correlation Over Dynamic Lambda Grids 28

Experiment on DAS-3 Goals: • evaluate the scalability of a software correlator; • evaluate the DAS-3 network capabilities; • evaluate Bandwidth on Demand service. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 29

What is DAS-3 ? DAS-3 is composed of: • 5 cluster sites, • 1 GE Network, • 10 G Myrinet network, • Interconnexion photonic network called Star. Plane manages: • several 10 Gbps lighpaths • lightpaths have to reserved 9/16/2020 Software Correlation Over Dynamic Lambda Grids 30

What is Star. Plane ? • Key principle: applications have to request to Star. Plane the right to use a certain amount of lightpaths of the fast interconnexion network. • Two ways to acquire a path with Star. Plane: • procedurally get. Path(“source”, by direct calling the “dst”, webservice “time”, “duration”) X • declaratively through a query language like: query( path(X, “leiden”, L 1), path(X, “delft”, L 2), available([L 1, L 2]), reserve([L 1, L 2], Res. Ticket) ) 9/16/2020 Software Correlation Over Dynamic Lambda Grids function: L 1 L 2 leiden delft 31

DAS-3 + Star. Plane Demonstration: Two applications are transferring data between two clusters. Initially they start using the default Internet interconnexion network. The middleware requests path to Star. Plane. When path is acquired the middleware redirect the traffic to make use of it. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 32

DAS-3 + Star. Plane Demonstration: Two applications are transferring data between two clusters. Initially they start using the default Internet interconnexion network. The middleware requests path to Star. Plane. When path is acquired the middleware redirect the traffic to make use of it. 1. 'a' is started and uses the shared 1 GE network; 2. 'b' is started and uses the shared 1 GE network; 3. 'a' has acquired the right to use one of the lighpath. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 33

SCARIe experiment on DAS-3 Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 34

SCARIe experiment on DAS-3 Mapping the application to the resource/service set: Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 35

SCARIe experiment on DAS-3 Mapping the application to the resource/service set: Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 36

SCARIe experiment on DAS-3 Mapping the application to the resource/service set: Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 37

SCARIe experiment on DAS-3 • Node and network resource co-allocation query: query( cluster(“Uv. A”), cluster(C 2), cluster(C 3), C 2 = C 3, NC 1 + NC 2 + NC 3 == 100, nodes(“Uv. A”, NC 1, N 1), nodes(C 2, N 2), nodes(C 3, N 3), path(“Uv. A”, C 2, L 1), path(“Uv. A”, C 3, L 2), reserve([N 1, N 2, N 3, L 1, L 2], Res. Ticket) ) 9/16/2020 Software Correlation Over Dynamic Lambda Grids 38

Results 9/16/2020 Software Correlation Over Dynamic Lambda Grids 39

Results • Input nodes (quad core 2 Ghz CPU) are close to handle 1 Gbps data flow. • We need around 100 correlation nodes to compute a 4 x 256 Mbps observation in real-time. This is much more than our “theoretical estimation”. • Networking speed is not (yet ? ) a bottleneck. • Star. Plane permits a simple, per application Qo. S between the clusters by controlling the access to the medium at a large grain. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 40

Future plan • We need to use a Bo. D service (like GN 2 Autobahn) to establish Qo. S paths between the radio-telescopes and the grid. International Bandwidth on Demand Service (eg. Gn 2 Auto. Bahn) Last 200 m Correlator 9/16/2020 Software Correlation Over Dynamic Lambda Grids 41

Future plan • Integrate the Bo. D service into the “default” Star. Plane network service to submit complex, integrated, jobs to DAS-3. query( path(CINPUT, “Radio. Telescope@Arecibo”, L 1), path(CINPUT, “Radio. Telescope@Dwingeloo”, L 2), cluster(CINPUT), cluster(C 2), cluster(C 3), alldifferent( [CINPUT, C 2, C 3] ), NC 1 + NC 2 + NC 3 == 100, NC 1 >= Num. RT, nodes(CINPUT, NC 1, N 1), nodes(C 2, N 2), nodes(C 3, N 3), path(CINPUT, C 2, L 1), path(CINPUT, C 3, L 2), reserve([N 1, N 2, N 3, L 1, L 2], Res. Ticket) ) 9/16/2020 Software Correlation Over Dynamic Lambda Grids 42

Conclusion Software correlation is challenging : • • • lots of input flows to bring the data; lots of computations that need to be distributed; over multi-domain; for large size (Gbps speed); over several scales: LAN, nation, world; with a “per experiment” dynamicity. Software correlation is possible but it is unclear at what cost ! • In term of computing power: • CPU is slow for a systematic computation like FFT. => using GPU/FPGA for co-processing. • In term of networking: • fat pipes have a cost; • dynamicity (connectivity) has a cost; • Qo. S has a cost. => Selecting the adequate layer to implement the needed service. 9/16/2020 Software Correlation Over Dynamic Lambda Grids 43

Questions/Answers Contact information Nico Kruithof Joint Institute for VLBI in Europe (JIVE) Kruithof@jive. nl Damien Marchal Universiteit van Amsterdam (Uv. A) dmarchal@science. uva. nl SCARIe is made possible through the support of the Netherlands Organisation for Scientific Research (NWO). 9/16/2020 Software Correlation Over Dynamic Lambda Grids 44

Middleware for Star. Plane • Based on linux dlopen to hijack the socket API. Application Starplane Hijacker Socket API 9/16/2020 Hijacker Normal interconnexion Lighpath if available Software Correlation Over Dynamic Lambda Grids Socket API 45