7 th ACS Workshop ACS The Big Picture

7 th ACS Workshop ACS The Big Picture G. Chiozzi Antofagasta – 15 th to 19 th November 2010

The Modern Observatory • Integrated and distributed architecture (end to end data flow) • Archive and virtual observatory integration. • Feedback systems and distributed control loops. • Big projects and small projects: what are the differences? Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 2

ALMA Software Architecture Executive f. Get science data Observation Archive Researcher d. Notify e. Start of Stop Special Configure Condition 2. Store observing project Preparation 1. Create observing project 9. Get project data g. breakpoint response A r c h i v e Telescope Operator Principal Investigator 3. Get project definition 8. Notify PI Scheduling c. Alter Schedule / Override action 6. Start data reduction 7. 1. Get raw data & meta-data Data Reduction 4. Dispatch scheduling block id 7. 2. Store science results 5. Execute scheduling block Real-time 5. 6. Store calibration results 5. 1. Get SB Control System 5. 4. Store meta-data a. Monitor points Pipeline Calibration Pipeline 5. 2 Setup correlator 5. 5 a. Access raw data & meta-data 5. 3. Store raw data Correlator Quick Look Pipeline Primary functional paths Additional functions ALMA software subsystem external actor 5. 7. Store quick-look results ALMA Common Software Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 3

Distributed System • Requirement: the observatory is a distributed system. Servers and clients are distributed on different machines: – Possibly in different locations – With different purpose and functionality – With different requirements on performance and reliability This actually implies an…. . Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 4

Heterogeneous Distributed System • Requirement: the observatory shall be a heterogeneous distributed system; servers and clients may use different: – Hardware – System software – Programming languages Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 5

Transparent Heterogeneous Distribution • Separation of Concerns: – Developers of clients and servers have to be unaware of the respective architecture – It shall be possible to change the architecture of a server transparently to the client – Ideally the developer of a client should not even know if a server is local or remote. Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 6

Functional + Technical Architecture • Expressing the complexity in SW of a telescope/observatory (our business) is difficult • Having to know also the subfields of computer science associated with distributed object architecture is very challenging • The separation of functional from technical concerns is a strategy for enabling the application developer to concentrate on the “business aspects” Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 7

Functional Architecture A Functional Software Architecture (FSA) is a model that identifies enterprise functions, interactions and corresponding IT needs. – Software components/subsystems • Responsibilities • Interfaces • Primary relationships and interactions – Physics and algorithms – Hardware deployment and distribution • Developed by sub-system responsible Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 8

Technical Architecture The “functional architecture” must be supported by a “technical architecture” that describes (and implements) the technical aspects of the software • • Access remote resources Store and retrieve data (DB technology) Manage security needs Communication mechanisms and networking Software deployment and activation Programming model Provided by the technical team Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 9

Separation of concerns • Functional and technical architecture: two views • Subsystem teams should concentrate on function • Technical architecture provides them with simple and standard ways to, for example: – Access remote resources – Store and retrieve data – Manage security needs • We want to keep the two concerns separate Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 10

Infrastructure Framework An infrastructure framework is the key to the separation between Functional and Technical Architecture Purpose of a framework is to: • provide a programming model • provide common paradigm abstractions • mask heterogeneity • satisfy performance, reliability and security requirements Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 11

Adoption of frameworks Project Framework Keck EPICS, JPL RTC Gemini EPICS ESO Paranal and La Silla observatories VLT CCS ALMA and other projects ACS (CORBA based) Advanced Technology Solar Telescope (ATST) ATSTCS (ICE, CORBA like) Southern Astrophysical Research Telescope (SOAR) Lab. VIEW Large Binocular Telescope (LBT) LBT specific (RPC based) GTC CORBA based Magdalena Ridge Observatory Interferometer JPL RTC, Lab. VIEW Discovery Channel Telescope (DCT) Lab. VIEW Large Synoptic Survey Telescope (LSST) DDS? Lab. VIEW? ATSTCS? ESO Extremely Large Telescope (E-ELT) DDS, Lab. VIEW, ACS? Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 12

The ACS answer • ACS provides the basic services needed for object oriented distributed computing. Among these: – Transparent remote object invocation – Object deployment and location based on a container/component model – Distributed error and alarm handling – Distributed logging – Distributed events • The ACS framework is based on CORBA and built on top of free CORBA implementations. • Model driven development with code generation Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 13

Supported Platforms • Operating system: Linux, RH-E + other flavours (Windows on the way) • Real-time: RTAI and Vx. Works • Languages: C++, JAVA, Python • CORBA middleware: TAO (& ACE) (C++), Jac. ORB (Java), Omniorb (Python), CORBA services. • Embedded ACS Container: PC 104, Debian, 300 Mhz Geode, 256 MB RAM, 256 MB flash (Cosy. LAB micro. IOC) Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 14

LGPL and free software • The strategy to provide common features to our users is: – Use as much as possible open-source tools, instead of implementing things. • • Do not reinvent the wheel Reuse experience of other projects Do not pay for licenses Support from user’s community – Identify the best way to perform a task among the possibilities – Wrap with convenience and unifying APIs • ACS is distributed under LGPL license • Free software has also drawbacks: – Fast lifecycle and support only of the newest – Free/commercial support – Documentation not as good as commercial products Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 15

Separation of roles ACS keeps separate 3 roles/phases: • Development by Developers • Deployment| by Sys. Configurators • Runtime by Operators Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 16

Development Server Component GUI Client ACS container Antofagasta, 15 th to 19 th November 2010 • Developers write Components and GUI clients in Java, C++, or Python. • ACS provides an integrated build environment based on application code modules. • Communication from an application to a component, and among components, uses ACS as middleware. • No thinking about starting and stopping components, or on which machine they should run later. 7 th ALMA Common Software Workshop 17

Deployment Computer 1 GUI Client „container 1“ „container 2“ ACS Configuration Database (CDB) Antofagasta, 15 th to 19 th November 2010 „Comp 3“ Computer 3 „Comp 2“ „Comp 1“ Computer 2 • One or more containers get assigned to each computer. • Components get assigned to containers. • This location information is stored centrally in the Configuration Database (CDB). • Other configuration data for containers and components is also stored in the CDB. • There can be different deployments for unit tests, system tests, and various stages of the productive system. 7 th ALMA Common Software Workshop 18

Runtime Computer 1 GUI Client „container 1“ ACS CDB and ACS Services Antofagasta, 15 th to 19 th November 2010 „Comp 3“ Computer 3 „Comp 2“ „Comp 1“ Computer 2 „container 2“ • ACS containers start and stop components (“lifecycle management”) as needed. • containers provide components and clients with references to other components. • the “manager” is the central intelligence point that keeps the system together. Components never see it directly. • manager, CDB, and other services, are started with the “acs. Start” command. ACS Manager 7 th ALMA Common Software Workshop 19

A key concept: Interfaces vs. Implementations • First step in design: Identify objects – – – Mount CCD Telescope Observation Exposure Interface Object • Second step: Define interfaces – – We need a formal interface definition language Implementation will come later and is independent from interface Deployment is also independent from interface definitions Interfaces shall be kept as stable as possible, but it must be possible to have them evolve when needed. Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 20

Interface Definition Language: IDL • CORBA and DDS both use the same formal interface definitions language called IDL forms a ‘contract’ between client and servant or publisher and subscriber. • IDL reconciles diverse object models and programming languages • Imposes the same object model on all supported languages • Programming language independent means of describing data types and object interfaces – purely descriptive - no procedural components – provides abstraction from implementation – allows multiple language bindings to be defined • A way to integrate and share objects from different object models and languages Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 21

Development Process Using IDL Definition Client Implementation Object Implementation IDL Compiler Client Program Source Stub Source Skeleton Source Java or C++ Compiler Client Program Object Implementation Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop Object Implementation Source

Questions (& Answers) Antofagasta, 15 th to 19 th November 2010 7 th ALMA Common Software Workshop 23