A Dynamically Reconfigurable Data Stream Processing System Jerzy

A Dynamically Reconfigurable Data Stream Processing System Jerzy M. Nogiec Fermilab

Outline Dynamism in Software Systems …. . … EMS Framework ……………. Reconfiguration in EMS ………………. Dynamism using Scripting ………. . A Dynamically Reconfigurable Data Stream Wrap-Up Processing System J. M. Nogiec, Fermilab 2

Dynamism in Software Systems t Open dynamic system Changes at runtime that were not anticipated at build time. dynamism t Closed dynamic system Configuration changes performed at runtime, but planned at build time. t Tailorable system Some aspects of the system can be modified at runtime. t Configurable system Many different configurations can be set up. t Static system System behavior fully described at build time. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 3

Dynamic Reconfiguration Dynamic reconfiguration is a process of modifying/evolving a running application, in which a composition of interacting components changes at runtime. It could be used to: t t Modify behavior of the system why preserving continuity of its execution (mission or safety critical systems) Apply patches to a running system Extend base application functionality with additional nonfunctional services (end user customization and extension) Adapt to changing requirements or environment that are difficult to anticipate (dynamically optimizing system functionality or recovering from errors and failures). J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 4

EMS Framework EMS is a component-based framework for building measurement, data acquisition, and data stream processing systems. The goal of the EMS project was to design, implement, and deploy a system that is extensible, flexible and dynamic. EMS can be used to develop configurable and dynamically reconfigurable systems. *Framework – the skeleton of an application that can be customized and reused **Component – an independently released software module suitable for composition (together with other components) into multiple A Dynamically Reconfigurable Data Stream 5 applications Processing System J. M. Nogiec, Fermilab

Visual Components J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 6

Component Interfaces control data • properties • state • error status exceptiondebug J. M. Nogiec, Fermilab data Components have properties and state. Typical components input, process, and output data. Their behavior depends on their state and property values. Components can be forced to perform certain actions by sending control events to them. Components also output debug and exception information. A Dynamically Reconfigurable Data Stream Processing System 7

Component Connectivity filter splitter merger source sink J. M. Nogiec, Fermilab There are several categories of data processing components: Data sinks Data sources Data filters/processors Data splitters Data mergers A Dynamically Reconfigurable Data Stream Processing System 8

Connecting Components EMS provides for independent “wiring” of components for property, data, exception, control, and debug events. EMS supports unicast, multicast, and broadcast patterns. J. M. Nogiec, Fermilab Source routing (dynamic, run-time) & routing A Dynamically Reconfigurable Data Streamtables (static, Processing System configuration-time). 9

EMS XML Dialect <configuration version="0. 1" title="Display Test XML">  <component id="Producer” class="ems. core. components. Simple. Data. Generator"> <property name="title" value="Data Generator Component"/> </component> <component id="Chart" class="ems. measurement. chart. Chart. Data. Display"> <property name="XPosition" value="0"/> <property name="YPosition" value="200"/> <property name="title" value="Plot Display"/> </component>  <route type="Data" origin="Producer" destination="Chart" />  <control signal="init" destination="!"/> <control signal="start" destination="!"/> </configuration> J. M. Nogiec, Fermilab Producer A Dynamically Reconfigurable Data Stream Processing System Chart 10

Reconfiguration in EMS There is several techniques that can be employed to alter the behavior of the system: t Use of component properties that are modifiable also at runtime t Definition and modification of event paths t Deactivation/activation of components t Deactivation/activation of data paths using gate components t “Coloring” data in data driven processing t Addition/removal of components t Use of externally controllable source routing J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 11

Reconfiguration in EMS: Modification of Component Properties Each component has a set of properties that both control its behavior and exhibit its state. These properties can be introspected and modified externally at runtime. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 12

Reconfiguration in EMS: Modification of Data Paths Data paths between components can be manipulated (added, modified or removed) at runtime. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 13

Reconfiguration in EMS: Deactivation/Activation of Components A component can be made inactive, so it passes the data but does not perform any processing. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 14

Reconfiguration in EMS: Gate Components A gate component passes or stops all data traffic going through it. Therefore it effectively activates or deactivates the data path that originates at it. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 15

Reconfiguration in EMS: Coloring Data A “coloring” component (Marker) is used to mark data. Other components act only on the data marked (“colored”) to be processed by them. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 16

Reconfiguration in EMS: Adding/Removing Components A component can be added to or removed from the system. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 17

Reconfiguration in EMS: Source Routing A component using source routing can send data to a recipient defined in one of its properties that can be manipulated by other (controlling) components. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 18

Reconfiguration Components Separation of concerns between functional code and reconfiguration code can be achieved by introducing a dedicated reconfiguration component (-s). J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 19

Reconfiguration with Scripting allows for automation of processing and for quick construction of new tests or data processing logic. The script control panel controlling execution of a Jython script. J. M. Nogiec, Fermilab Scripting allows for easy implementation of reconfiguration in closed dynamic systems. A Dynamically Reconfigurable Data Stream Processing System 20

Wrap. Up t EMS is a flexible component-based framework for building dynamic systems. t The system has been applied to building measurement and data acquisition systems where continuous data streams are processed in real-time. t EMS allows for high level of reuse. t The architecture supports separation of concerns between reconfiguration and functional aspects. t EMS features allow for agile development of new applications and easy adaptation of existing applications to changing requirements. J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System 21

Resources EMS web site http: //sdsg. fnal. gov/ems web EMS on-line documentation J. M. Nogiec, Fermilab A Dynamically Reconfigurable Data Stream Processing System http: //wwwtsmtf. fnal. gov/ems_javad 22