Lessons Learned Enhancing EPICS CA for LANSCE Timed

Lessons Learned Enhancing EPICS CA for LANSCE Timed and Flavored Data Jeffrey O. Hill LANSCE / LANL

Outline Requirements, a Review Design Changes Status, this project and others Lessons Learned Benefits, a Review

Requirements - Subscription Filtering LANSCE timing and flavoring of data Flavoring Selection based on - logical combinatorial of beam gates Timing Selection based on - time sampling within beam pulse Many permutations Too many to, a-priori, install records for all of them Client side tools specify flavoring when subscribing Capabilities to forward timing and beam gate companion data in IOC core Filtering capabilities in the CA server

Requirements - Time Metadata LANSCE timed data requires Array Index Metadata Magnitude of zero-eth element index Floating point Magnitude of one index increment Floating point Units of these magnitudes String

Design - Generic Implementation � Generic interface to site specific subscription update payloads Data Access � What it isn't � A storage format or a linked list of discriminated unions ala GDD and CDev. Data � � What it is � A generic interface for introspecting hierarchical data Data is interfaced in their native (site specific) data structures A more general approach � � � We can wrap GDD with DA, but the converse isn't possible Smart pointer references to site specific payloads Reference counting � � We have done this before - there is a better way IMHO Efficient memory management in device or record support Mutual exclusion, or not, determined by device or record support Filtering expressions specified as strings � Minimal impact to legacy client side tools

Design - Event Queue Upgraded Record Legacy Fixed Event Specific Event Parameter Set Scalar Value Scalar or Vector Time Stamp Alarm Status … Record Upgraded Device Specific Event Parameter Set LANSCE Beam Gate Specification CA Server Event Queue

Design Event Filter Record Event. Circuit TCP Filter Event Queue Event Filter CALC Expression EX 1 beam. Gate. Spec & ( beam. Gate. A | beam. Gate. B ) && ! (beam. Gate. Spec & beam. Gate. C ) EX 2 Value > 100 Discard

Design Changes - Mutal Exlusion Before – Guard based Mutex mutex; Guard guard (mutex ); Service & service = create. Service (guard); Channel & channel = Service. create. Channel (guard, …); After – smart pointer based Ptr < Service > p. Service = create. Service (); Ptr < Channel > p. Channel = p. Service->create. Channel (…);

Design Changes – Smart Pointers For event queue implementation and public CA interfaces we have some special requirements – for flexibility and isolation Data Access Catalog, CA Channel, and others are used by different threads than their creators Runtime polymorphism – requires virtual interface User of Ptr must not know at runtime or compile time which of these are chosen Reference Counted memory management, or not Mutual Exclusion protection for target, or not Intrusive (i. e. within target object) implementation, or not

Design Changes – Smart Pointers � Surveyed popular smart pointers in particular � Boost shared_ptr � Reference counting - always � No mutual exclusion for target only for reference count � Non-intrusive implementation – always Can be less efficient (locked allocation of reference counter) but better code reuse � Modern C++ Design smart pointer (Alexandrescu) � The ultimate in terms of flexibility, and efficiency potential � Complex template parameter based policy selection Pushing the bleeding edge of compiler feature sets Policy selection template parameters can be verbose, intimidating for casual user this is easily ameliorated by ? typedefing runtime polymorphic policy selection ? Implement specializations for runtime polymorphic behavior

Design Changes After some deliberation a custom implementation appeared to be the best choice We need mutual exclusion which boost library’s shared_ptr does not provide The loki library’s smart pointer extensive use of templates is probably not portable enough for EPICS. Legacy vx. Works systems use an old gnu compiler. A custom implementation currently requires 547 lines of code – not a significant maintenance burden

Design Changes – Smart Pointers Internally a smart pointer has a single data member Pointer to a handle Smart pointer same size as ordinary pointer A handle is runtime polymorphic The operator-> is overloaded to return locked smart pointer object class Handle. Intf { virtual Target. Mutex. Pair target. Mutex. Pair () = 0; virtual Handle. Intf & clone () = 0; virtual void release () throw () = 0; };

New Atomic Primitives in R 3. 15 Efficient atomic increment / decrement needed for reference counting RISC architecture requires mutual exclusion Increment requires two (interruptible) instructions SMP architecture requires memory barriers Cache sync / flush operations SMP, the future CPU architecture Atomic increment performance - versus a mutex On typical computer (Intel core 2 duo 3 GHz) 70 μS epics. Mutex lock/unlock 23 μS osi. Atomic. Increment / Decrement Implementations (so far) GCC 4. 1 intrinsic primitives on OS/GNU OS primitives on Solaris, Windows, vx. Works OS, predating SMP, lock interrupts very briefly Legacy vx. Works, RTEMS, …

Status, MAGVIZ, my Full Time Project Prev Year + 2 Months Before Mag. Viz distinguishes potential-threat liquids from harmless shampoos and sodas screened at airport baggage checkpoints

Status, EPICS R 3. 14. 11 Many quality improvement related changes in the portable server library Fixed lockup when flow control started and channel connect requests were pending on the event queue This may be the cause of DB CA links not reconnecting through the gateway at Diamond and SLS Incoming (put) data corruption under heavy CAS load. 2 nd occurrence found Faster string conversions - delays reduced, large arrays Scheduling improvements, large arrays, many others … The portable server library is used by the CA gateway, and application level services

Status, CA Gateway Changes CA Gateway was modified to behave differently for ca put versus ca put callback CA put Caching behavior when client put production rate exceeds put consumption rate of IOC Classic MEDM slider over-pumps the slow motor record situation – is handled well now CA put callback The original behavior is preserved Each and every put request is executed

Status, This Project I am working again on this project since the start of the calendar year Some interruptions this summer due to EPICS R 3. 14. 11, CA gateway changes Guard to smart pointer locking model transition required extensive reorganization of the data structures, but is now close to completion, and worth the delay IMHO The code builds, but there are currently about 40 undefined symbols

Lessons Learned Multithreaded and Object Oriented Designing for both together takes time to master Changing the primary interface’s locking model during the middle of the project can be like pulling on a sweater thread. When designing any modern code SMP must be considered Some additional locking overhead is justified when we consider that N processors can work concurrently Nevertheless, I do worry about the cost of blocking processors while they reload their pipeline due to memory barriers (lesson is pending)

Lessons Learned Working on multiple project simultaneously Positive impacts Returning to a project leads to a fresh object and a desire to review core design decisions Negative impacts Too much context switching Lost momentum

Lessons Learned Breaking large projects up into smaller upgrades is a good idea Incremental upgrades occur more quickly With more frequent releases means better feedback More authors can share the development load

Benefits for LANSCE style dynamic on-the-fly ad-hoc beam flavoring and beam timing experiments But, in homogenous EPICS system Tool based approach to LANSCE applications Applications have abstract model of hardware Incremental upgrades hopefully easier Homogeneous communication model for high level apps might make on-call duties easier Multi-element “Timed” data COTS digitizer Window in time selected

Benefits for EPICS Community Flexible event snapshots Parameters other than alarm status, time stamp, scalar value correlated on event queue Array update bursts buffered on event queue Subscription update filtering Expression (string) based means Project and site independent – tool based approach Minimally invasive for existing client side tools Array index meta data Expanding intersection of EPICS with data acquisition systems