Development Status and Priorities Jeff Kern NRAO Interferometry

Development Status and Priorities Jeff Kern NRAO

Interferometry Progress CASA • Re-architecting of Clean task implementation to t. Clean – MT-MFS with mosaics, AW-Projections, parallelization – Available for use, many “nice” features still to be added – Used in ALMA imaging pipeline • OTF calibration (cal library) – Supported and well tested in plot. MS – Support in tasks implemented, needs to be commissioned – Not yet hooked up to t. Clean • MPI Based parallelization – Fully available and documented for calibration (see talk tomorrow). – Easier integration with most scheduling systems 2015 CASA Users Committee Face to Face 2

Interferometry Progress (2) CASA • MS Transform – Single location for many similar tasks (deprecating old versions) – Fully MMS aware (parallel where available) • Robust statistics algorithms in imstat and ia. statistics – Several algorithms • Channel based weights, weight handling defined, and tools for weight manipulation. • Linear mosaic (lm) tool to mosaic weighted images – Important for VLASS • msuvbin – UV plane accumulation – Designed for CHILES project, allows managing very large data sets 2015 CASA Users Committee Face to Face 3

Single Dish Progress CASA • MS transition (since 2014 Q 4) – All highest priority capabilities, i. e. , ones used in standard manual and pipeline data processing of ALMA-TP, have already been implemented. (But the standard data processing of ALMATP is still based on ASAP) – 21 out of 58 items are implemented in CASA 4. 5 release • Support ALMA-TP data processing: manually processed data started to be delivered in Aug. 2015 Bug fixes and minor enhancements in ASAP to process Cycle 1 and 2 observation data • Support of New multi-beam receiver, FOREST, on Nobeyama 45 -m – Bug fixes in ASAP to support multi-beam data and enhancements for FOREST – Currently, a dataset should be split by beam for proper imaging. 2015 CASA Users Committee Face to Face 4

Pipeline Progress CASA • Successfully supported ALMA Cycle 2 operations. • ALMA SD Pipeline acceptance Cycle 2 • ALMA Imaging Pipeline Processing – Continuum Image Manual Success – Per SPW continuum subtracted cubes Tweaked • VLA Calibration Pipeline in operations • Parallelization – Implemented in pipeline, not yet deployed – See talk tomorrow • ALMA-QA 0 Flagging Implemented Success w/ Additional Flagging – Additional online flags applied for ALMA 2015 CASA Users Committee Face to Face 5

Build and Test Progress CASA • Support for OSX 10. 9 and 10. 10 • Daily automated test versions with automated tests on all four platforms – Stables (those that pass tests) are made available to the community. 89 stable versions available during 4. 5 development cycle • Integration with github: : CASACore project – Unified casacore with rest of community • Normalization and standardization of build process across OSX and Linux • libsakura integration – High performance vector processor library developed at NAOJ • C++11 Support in compilers – Allows removal of custom solutions: container classes, smartpointers 2015 CASA Users Committee Face to Face 6

Misses (Things we didn’t do). CASA • Updates to FFTW (Done, but to late for 4. 5 release) • Complete switch of calibration to new IO infrastructure layer – Full scratchless operation in calibration (done, but needs validation) • Rewrite of visstat – Scope increase (support Single Dish) and new developer. 4. 6 • t. Clean: – Transition to new IO infrastructure (scratchless operation) – Improved image weighting (fixes interpolation issues) – Interactive interface improvements (descoped early) – automasking – Parallelization (Done, but needs validation) 2015 CASA Users Committee Face to Face 7

Misses (Things we didn’t do). CASA • Set. Jy – Full reuse of imaging infrasturucture • Stat. Wt improvements • Enhanced RFI mitigation in RFLAG (requirement mismatch) 2015 CASA Users Committee Face to Face 8

Interferometric Priorities CASA • Reliability and Stability Initiatives – Testing • Enabling and requiring tests at the C++ module level – Technical Debt • Continued migration to standard IO routines • Namespace cleanup, removal of bespoke code – Obsolete Code • Removal of unused or unsupported options • Consolidation of routines to use a single pattern for accomplishing the same thing. 2015 CASA Users Committee Face to Face 9

Interferometric Priorities CASA • Close out of ongoing initiatives – t. Clean – Scratchless operation in Calibration, testing and perfecting • Some “behind the scenes” changes for maintainability – Vis. Stat • Speed, parallelization, and support for large data • Robust estimation – Scratchless operation extended to t. Clean • No more applycal? • New initiatives: – Performance effort: Particularly looking at low level IO access patterns 2015 CASA Users Committee Face to Face 10

Pipeline Priorities CASA • ALMA: – Implement antenna outlier amplitude vs time flagging heuristic – ALMA imaging acceptance (and improvement) – HPC in production – Sessions (? ) Multiple contiguous executions of the same SB • Sharing calibration across exectuions • VLA – VLASS specializations (OTF Mosaic support) – Spectral calibration support • Both – Low signal to noise ratio heuristics – Polarization calibration – Science Ready Data Products 2015 CASA Users Committee Face to Face 11

Single Dish Priorities CASA • Complete the MS transition project Aim finishing migration of most of the items by the end of 2016 • Support ALMA to start using the new single dish module based on MSes The new single dish module is ready for use for ALMA-TP. Need to update manual reduction scripts and pipeline • Support ALMA Cycle 4 TP observations Major new capabilities are fast-scanning observations of the Sun and observations of multiple science targets by an SB • Support Nobeyama start using CASA as the primary data reduction software Nobeyama decided to adopt CASA for new backends Preliminary schedule: 2016 Q 4 - the observatory test use of CASA, 2017 Q 4 - observatory start supporting reduction by CASA 2015 CASA Users Committee Face to Face 12

Build and Test Priorities CASA • Add support for more Oss – OSX 10. 11 (10. 9 going away!) – RHEL 7 – Ubuntu support (Meer. KAT? ) Resources permitting • Transition to Git as version control system for all of CASA – Investigating more integrated tool chain • Automation of new tests under development by development team – Introduce and track test coverage metrics • Addition of tools to enforce Software Quality initiatives • Implementation of framework to support user documentation effort 2015 CASA Users Committee Face to Face 13

CASA 2014 CUC Report Recommendations Since speed and ease-of-use are the main laments of CASA users, we recommend to emphasize improvements in performance and parallelization, and to improve the overall robustness of basic operation. The CASA HPC framework is now in place and available for general use. Parallel imaging is implemented and tested in pipeline context but not yet undergone full validation. Scratchless operation is available in calibration, plot. MS, and MSTransform, but not yet in imaging. Additional attention to setting user expectations may also help ameliorate these issues, such as (crude) time estimators for critical tasks. This has not yet been addressed, it is on the list of remaining imaging re-architecting tasks. 2015 CASA Users Committee Face to Face 14

CASA 2014 CUC Report Recommendations Complete the CLEAN refactor with high priority, as this will bring important capabilities as well as retire some speed and usability issues. The first version of this is now available in CASA 4. 5 Improve and curate better CASA documentation; a small number of well-maintained documents is preferable to many that are not maintained and become out of date. See discussion in talk on documentation Provide updates to CASA users about ongoing and planned developments, as well as prioritized fixes, in a regular and easily digestible form. The first CASA newsletter was published in April, the second is in prepartion and will be out this week or next. 2015 CASA Users Committee Face to Face 15 • Improve feedback to users on helpdesk/ticket disposition, especially

CASA 2014 CUC Report Recommendations Improve feedback to users on helpdesk/ticket disposition, especially where challenging problems may take a long time to address, or will not be addressed. We have created a web page to address this, some of the CASA management consultant recommendations are focused on improving determinism for development. User surveys can uncover problematic issues and help to guide future development; we would like to work with the CASA team to develop and deploy more user surveys. First user survey was conducted and a response has been prepared (see upcoming newsletter). Further details in discussion on user survey. 2015 CASA Users Committee Face to Face 16

CASA 2014 CUC Report Recommendations As there always will be a longer wish list of CASA improvements than available resources can support, we recommend to engage the external community by stabilizing and documenting interfaces and import/export formats to better enable users to contribute and to maintain CASA tasks and scripts. We have addressed a number of known issues with the uvfits import/export tasks. Documentation was discussed earlier. The transition to an enterprise Git implementation will allow more dynamic interaction with external developers. Continue to promote CASA for telescopes other than ALMA and the VLA on a best effort basis; major initiatives formulated by external groups should be undertaken only if the additional resources required are provided. Under these auspices we have entered into a collaboration to support the ACA reduction with CASA. We are working to incorporate resources from other observatories in the NRAO helpdesk to assist in the support of CASA. 2015 CASA Users Committee Face to Face 17