SOLARIS Tango Control System Status Tango Meeting 2018

SOLARIS Tango Control System Status Tango Meeting 2018 MICHAŁ OSTOJA-GAJEWSKI National Synchrotron Radiation Center SOLARIS

National Synchrotron Radiation Center SOLARIS • Operation • Stored 500 m. A at 1. 5 Ge. V energy, operating at 300 m. A @1. 5 Ge. V • Working in 2 operators’ shifts (8: 00 -16: 00, 14: 00 -22: 00) • Beamlines • PEEM/XAS – commissioning / ready for users at 2018. 09 • UARPES – commissioning / ready for users at 2018. 09 • Diagnostic - construction started / X-ray ready at 2018. 08, visible light at 2019. 02 • PHELIX – construction started, commissioning end of 2019 • XMCD – construction started, commissioning end of 2020 • Applying for funding of 4 new beamlines • Cryo. TEM • SOLARIS was chosen to host Cryo. TEM infrastructure • Users • First open call successful – total of 45 applications • 42 applications submitted via DUO (Digital User Office) system • 3 applications via CERIC-ERIC 3

Dealing with Meltdown and Spectre at SOLARIS, 1 of 2 • 2018 started with a shocking information on critical vulnerabilities in modern processors • In December 2017, we upgraded VMware to 6. 5, but this upgrade did not contain Meltdown/Spectre patches • In January 2018 – we applied patches to all servers running VMware ESXi • We have not observed any significant performance degradation 4

Dealing with Meltdown and Spectre at SOLARIS, 2 of 2 • Guest OSes on VMs were not patched in the first run, but are patched systematically when updating linux kernels • All workstations (control room, beamlines) were patched with new Cent. OS kernel • All Windows systems controlled by local domain were patched automatically • Great source of information : https: //meltdownattack. com/ • Linux vulnerability checker : https: //github. com/speed 47/spectremeltdown-checker 5

Control System @ SOLARIS • We have upgraded entire control system to Tango 9. 2. 5 a and observed great improvement of system stability and performance • We observed that Tango is very sensitive to any DNS and Reverse DNS misconfiguration: • Lesson learned: if you see your client application switching to polling mode, be sure that the event system is not working properly and you should check your DNS configuration first. • We moved from Taurus 3. 7 to 4. 1 and it required a lot of extra effort to modify multiple applications for the new API. • Some apps still remain on Taurus 3, but are planned to be adapted for Taurus 4. x soon. 6

Control System @ SOLARIS • Moving out from NFS distribution towards RPM packaging • Created RPMs for all software running on Control Room and Beamlines workstations • Working on packaging of device servers software and configurations • Created and deployed Ansible playbooks for automatic building of control system workstations • Working on Ansible playbooks for automatic configuration of VMs and deployment of device servers software 7

Control System @ SOLARIS • Moving from Jenkins to Gitlab CI/CD and using Docker for builds • Deploying Git. Flow as a standard branching model • Moving from Redmine to You. Track for Agile/Scrum development • Development environment – freedom of choice, but preferred: • Jetbrains Py. Charm, • CLion, • Intelli. J IDEA, • You. Track • Challenge: automatic testing, mockups, automatic devops 8

Control System @ SOLARIS • Started using Valgrind (and Valkyrie) for debugging and memory analysis • Reviewed Python profilers: Yappi, CProfile, VMProf to identify ability to work in multithreaded environment • Very interested in the analysis of Tango kernel with special attention to communication and threading models • In process of identyfing suitable tools (profilers, debuggers) for analysis and visual representation of Tango runtime system 9

Control System @ SOLARIS • Deployed monitoring system based on : • ABRT, • Splunk • Monit See presentation by Michał Gajda • Contributed to Heating Unit Controller project (HUC) • Developed Tango GUI , Device Server and Archiving • Deployed HUC into production in SOLARIS See presentation by Wojciech Kitka • Deployed HDB++ based on Cassandra: • 3 clusters (Machine, BL 04, BL 05) • Each cluster consists of 2 nodes with replication factor 2 • Archived attributes – total : 1800 10

SOLARIS @ Tango Community Statement from SOLARIS • We are eager to contribute to development of Tango kernel or Tango related projects Thank you 11