RealTime Experiment Interface System RTLab Calin A Culianu
Real-Time Experiment Interface System: RTLab Calin A. Culianu and David J. Christini Division of Cardiology Weill Medical College of Cornell University
RT-Lab • Some Background: Ø Ø Ø In 1999 we started as some experimenters who were excited about ‘having our cake and eating it too’: Real-Time capabilities in Linux We wrote several programs to address our experiment needs In 2001 we were given a grant by the National Science Foundation • What we hope for the future: Ø The purpose of our project is to give the Open Source community a generic Real-Time Experiment Interface System
RT-Lab • A Generic Software Experiment Interface System Utilizing either RTAI or RT-Linux or (preferably) both. • Possible software building blocks: Ø Ø Ø RT-Linux 3. 1 on Linux Kernel 2. 4. 4 (http: //www. rtlinux. com) Comedi 0. 7. 60 (http: //stm. lbl. gov/comedi) Qt GUI Toolkit 2. 3 (http: //www. trolltech. com) • Hardware: Ø Ø Commodity IA 32 hardware DAQ Boards (whatever Comedi supports) § Our boards happen to be: • NI AT-MIO 16 e • NI PCI-MIO 16 e • Measurement Computing DAS 1602 -16 PCI
Why RT for experiment interface? • Many biomedical experiments have hard realtime constraints: Ø Ø Ø Cardiac arrhythmia control utilizes electrical stimuli patterns that are a function of heartbeat dynamics. Biomechanical feedback applies electrical or mechanical stimuli to the body as a function of limb or body position. Vestibular stimulation applies electrical stimuli behind the ear to affect the posture of a person with poor balance.
Experiment interface: Why Real-Time Linux? • Possible solutions: Ø Non Multitasking OS (DOS, etc) § Ø Specialty RTOSs § Ø Obsolete, inconvenient Expensive, somewhat inaccessible to scientists Engineer and implement custom solid-state hardware § Difficult and inflexible • Best solution: Ø Real-Time Linux built on standard drivers and API’s § Familiarity of Unix, excellent software selection, ability to get best of both worlds: Great multitasking OS with a Real-Time scheduler.
Project Goals: It would be cool if… • Experimenters didn’t have to “re-invent the wheel” Ø Certain pieces of code are probably going to be common to all experiments • Software tools existed for building custom experiment software Ø Ø Ø Open source Modular, flexible Ability to customize for exact needs of application
History: Programs We’ve Written • DAQ System (http: //www. rtlab. org) Ø Ø Data Acquisition with nice & pretty GUI Data analysis • AV VA Pace (http: //www. rtlab. org) Ø Ø Real-time feedback based on intra-cardiac ECG Signal ‘Chaos-control’ algorithms tweaked in non-Real-Time by UI • Interval Monitor (http: //www. rtlab. org) Ø Ø Variation on a theme: Chaos control problem UI helps illustrate difficult-to-see dynamics
Historical App I: Data Acquisition • Essential for a general Software Experiment System
Historical App II: AV Pace Program • Red tracing represents heartbeat • White tracing represents stimulus • Bottom graphs represent various UI controls for affecting the Real. Time algorithm used to determine characteristics of the white-tracing (the stimulus to the heart)
Some Observations • Hardware needs to be abstracted Ø Application code needs to be sufficiently shielded from implementation details. • Graphing is Ubiquitous Ø Powerful yet adaptable graphing tools/libraries/classes are essential • Experimentalists/Scientists often don’t have the time or the skills to develop software Ø Ø While scientists are quite clever people, in general the most helpful toolkits are the ones that are easiest to install Despite this fact, this does not mean we should compromise versatility for the advanced user in favor of ease-of-use for the beginner. We believe it is possible to be both powerful and obvious at the same time.
Architecture • Kernel Thread: Ø Ø Runs with real-time priority Communicated with user process via shared memory and FIFO’s • User Thread: Ø Ø Responsible to presenting results of kernel thread execution to user (via Graphs, and other visualization mechanisms) Responsible for controlling behavior of Kernel Thread via user-specified controls/parameters
Kernel-User Interaction: AV Pace • RT-Kernel Task and NRT-process communicate via shared memory (1 MB) and FIFOs. • Non-real-time graphing, storage, … • User control through a Qt GUI.
More on our current architecture • “rt_process. o” – an LKM that acts as a bridge between kernel-level Comedi drivers, RT-Linux, and Userland Ø Ø Intelligent understanding of Comedi Present user processes with a very generic and powerful interface to the hardware Goal is to keep user code as hardware-neutral as possible Hot-pluggable code so that “rt_proces. o” can dynamically change its behavior § § Extremely important if one hopes to keep scientific experimenters from needing to learn kernel API Approach is to provide generic interface for defining feedback behavior, using simple API.
rt_process. o: Approach to Feedback • How to specify dynamic feedback control? Ø Ø Ø Each application probably needs its own custom heuristics with respect to dynamic feedback in Real. Time AV VA Pace, for instance, uses some linear algebra tricks to determine how to prevent arrhythmias Some algorithms are stateless from sample to sample, some are stateful
rt_process. o: Approach to Feedback • Conclusion: We cannot easily generalize • Solution: Need to keep architecture of rt_process. o driver itself very open • What we have so far: Ø The venerable C-like approach: Callback Functions! • What we hope to add: Ø Optional ability to specify logic in an interpreted language
rt_process. o: Callback API • “rt_process. o”, after each scan cycle, can optionally call user-created kernel modules, meeting a standard function definition: typedef void (*rtfunction_t)(Shared. Mem. Struct *, Multi. Sample. Struct *); /* registers a function to be run within the rtf loop returns 0 on succes, ENOMEM or EBUSY on error. Specified function won't be run (activated) until rtp_activate_function() is called SIDE Effect: rt_process's use count is incremented */ int rtp_register_function(rtfunction_t function); /* de-registers a function that was previously registered, returns 0 on success, EINVAL or EBUSY on error SIDE Effect: rt_process's use count is decremented */ int rtp_unregister_function(rtfunction_t function); /* de-activates a function that was previously registered, returns 0 on success, EINVAL on error */ int rtp_deactivate_function(rtfunction_t function); /* activates a function that was previously registered, returns 0 on success, EINVAL on error */ int rtp_activate_function(rtfunction_t function);
Going Forward - DAQ: The Next Generation
Other features of our current code: • Hardware Probing/Abstracting C++ classes – smart user classes that talk to Comedi via rt_process. o and abstract-out board capabilities. • Graph C++ classes – powerful extensible classes to address the graphing problem. • Hot-pluggable code so that “rt_proces. o” can dynamically change its behavior with respect to how it acquires data and how it does Real-Time feedback • It comes with a free Data Acquisition App!
Conclusion • RT capabilities are essential for many experiments • Many experiments have a great deal in common • Since the above is true, we, the RT-Lab project, have begun to write a generic experiment toolkit • Software URL: http: //www. rtlab. org
RT-Lab? • We received this somewhat ominous email recently:
- Slides: 20