Tele Presence Microscopy at CSIRO Minerals Colin Mac

Tele. Presence Microscopy at CSIRO Minerals Colin Mac. Rae CSIRO Minerals - Microbeam Laboratory www. minerals. csiro. au/microbeamlab

Talk outline Ø Ø Ø What and why? History of Tele. Presence at CSIRO Minerals 3 rd generation Tele. Presence system l l l Ø System architecture Control interfaces Video streaming technology Security features Administration Modular installation of new functions Conclusions, where to from here? Remote Access and automation Workshop 2005

What is Tele. Presence? Ø Tele-conferencing l Ø Tele-control l Ø Video + Audio Microscope controls On-line data collection + analysis Tele-control is provided by some manufacturers Ideally, Tele. Presence should be: l l Interoperable with many platforms ‘Shrink-wrapped’ software • Looks professional, with easy to use interfaces l l Fast and responsive Stable and secure Remote Access and automation Workshop 2005

Why do we want Tele. Presence? Internal clients (Brisbane, Hobart, Perth) Ø External clients (Melbourne, Sydney, Brisbane) Ø For analyses, our clients must either: Ø l l Ø Tele. Presence can help l Ø Mail / courier samples, and wait for our report (slow) Travel to our laboratory (expensive) Provide results to client in real-time, and also allows instant feedback from client Attract new clients CSIRO Minerals CSIRO BHP-Billiton Minerals, Exploration & Mining CSIRO Comalco & Rio. Tinto Marine Research Remote Access and automation Workshop 2005

What do we want to connect? Ø Ø Ø Scanning Electron Microscopes Electron Microprobes & Proton Microprobe X-ray detectors –EDS & WDS Cathodoluminescence systems Optical microscopes Remote Access and automation Workshop 2005

What do we want to offer Ø Ø Ø Ø Ø Clear fast images On-line microanalysis On-line x-ray spectra Large area optical views Streaming maps Security Voice over IP Data storage Data analysis/presentation software Remote Access and automation Workshop 2005

Ø What do we need from Tele. Presence? Provision of: l l l Real-time video from multiple sources (SEM, room camera) Streaming EDS spectra and other data ( + controls) Control of microscope stage + column (lower priority) ØMinimal hardware requirements l. PC or laptop (no workstations) l. ADSL or cable modem, ~1 Mbit/s (not Internet 2, GRID) Ø‘Shrink-wrap’ software l. Easy to set-up l. Very easy to learn and use, with generic controls/interface l. Very, very stable l. Extraordinary security l. Low maintenance ØMust not affect normal operation of instruments! Remote Access and automation Workshop 2005

Tele. Presence history Tele. Presence Microscopy 1999 Ø l l l Nestor J. Zaluzec, Argonne National Laboratory Web-based, ‘pervasive’ Web based notebook Video with “GTS” streaming technology Remote Access and automation Workshop 2005

Tele. Presence history Tele. Presence Microscopy 1999 Ø l Nestor J. Zaluzec, Argonne National Laboratory Web-based, ‘pervasive’ Java system 2001 l l Platform independent, ‘safe’ code MJPEG video (TCP-IP) Web based notebook with embedded Java applets for Images, Spectra and maps Successful, but limited Remote Access and automation Workshop 2005

Tele. Presence history Tele. Presence Microscopy - 1999 Ø l Nestor J. Zaluzec, Argonne National Laboratory Web-based, ‘pervasive’ Java system - 2001 l l Platform independent, ‘safe’ code MJPEG video Web based notebook with embedded Java applets for Images, Spectra and maps Successful, but limited C++ system – 2004 l l l Widely used language, with broad library support MPEG-4 video compression in software, changeable video size New, secure architecture Remote Access and automation Workshop 2005

Internet protocols: UDP versus TCP Sender Router TCP packet TCP acknowledgement Lost packet? UDP packet UDP TCP = Transport Control Protocol UDP = User Datagram Protocol Remote Access and automation Workshop 2005

Video latency Ø Definition: l Ø What do we want? l l Ø Latency = time displayed – time captured <500 ms for focussing (or better) <200 ms preferred Causes of latency 1. Video compression / decompression • • 2. soon) Router delays (congestion) • • 3. Hardware accelerated rendering (done) Hardware compression (cheap MPEG-4 hardware compression coming Ultra-broadband internet (eg GRID) Reduce stream size (better codec) inter-router delays (speed / distance) • • More bandwidth wont help Use faster internet protocol (i. e. TCP UDP) Remote Access and automation Workshop 2005

Another real life problem : The Firewall Client network Internet Firewall Our network port 8080 (UDP), from specified addresses Client Server Video port 8080 (TCP) Host ‘DMZ’ Remote Access and automation Workshop 2005

Comparison of platforms Web browser Java C++ Functionality Low Moderate High Execution speed Low High The vibe Poor So so Development pain Low High Platform independent? Yes Trusted? Somewhat Yes Remote Access and automation Workshop 2005 Yeah groovy Moderate No Yes, with security

System architecture Password Client (TCP socket) commands & data Host Server Ø Electron Microprobe Server Video (UDP stream) JEOL 8900 R Ø FEG-Environmental SEM Ø Server FEI Quanta Hitachi S-5000 In-Lens Cold FEG SEM Generic Ø PIXE Proton Microprobe interface Instrument specific Drivers Remote Access and automation Workshop 2005

Direct. Show video streaming Timing system (frame-rate) Video capture Network transmission sockets Compression codec ‘Graph. Edit’ component of Microsoft Direct. X software development kit (SDK) Remote Access and automation Workshop 2005

Direct. Show video streaming Very easy to configure and re-configure ‘graphs’ Ø Other stream designs are possible, eg: Ø l l l Ø Saving stream to HDD Motion detection triggering Adaptable to as yet unforseen requirements All components are upgradeable and replaceable l l New codecs can be adopted as they become available ‘Future proof’ We didn’t have to write any of these components (only wire them together) Ø Some Direct. Show components are hardware-accelerated Ø l Fast video rendering Remote Access and automation Workshop 2005

Tele. Presence interfaces Video Ø Camera control Ø Tilt-pan-zoom controls Used in conjunction with video window Ø EDX spectrometers Generic, easy to use interface Fast (2 -5 updates/s) Text streams Ø Miscellaneous others Ø Stage and Electron optical System Ø Remote Access and automation Workshop 2005

Video streaming technology Ø Microsoft Direct. Show l l Ø MPEG-4 encoding l l l Ø Widely supported standard with pre-compiled modules Hardware-accelerated rendering 320 240 at 25 fps is ~500 kbit/s (~ADSL) (~12 Mbit/s by MJPEG) (~59 Mbit/s for raw video) UDP transmission l Lowest possible latency over IP (~200 ms) Variations in humidity cause grass to spiral Themeda Australis and “dig” into the soil Kangaroo grass Remote Access and automation Workshop 2005

Video cameras Canon VC-C 1 and VC-C 3 cameras Ø 450 x 350 pixels at 30 frames/s. Ø ± 90º pan, ± 25º tilt, 10 zoom Ø Good low-light performance Ø Remote Access and automation Workshop 2005

Bench-top camera Ø ‘ELMO’ camera l XGA video, 20 frames/s l www. elmousa. com/presentation Remote Access and automation Workshop 2005

Client feedback Problem: Video stream is one-way. How do clients tell us what they want to see? Ø Verbal feedback (telephone) Ø l l Ø “that thing, over to the left. Up, say, half an inch…” Not very effective Graphical feedback l l l Video overlay marker Visible to all other clients and the operator More tools to follow Remote Access and automation Workshop 2005

Security Ø Firewalls l l Ø Blocks all direct communication with servers and drivers Host and servers are in separate ‘DMZ’ network, not mounted on CSIRO file system Socket encryption: l Sockets are wrapped with ‘Rijndael’ cipher, NIST advanced encryption standard (AES 2001). • Certified for all levels of classification, up to and including ‘top -secret’. l Hackers must break AES to read data, send false data, or issue commands Remote Access and automation Workshop 2005

On-line help Remote Access and automation Workshop 2005

Administration Remote Access and automation Workshop 2005

Web based administration Remote Access and automation Workshop 2005

Software based Administration tools Remote Access and automation Workshop 2005

Administration tools – Modular New instruments can be added and configured Remote Access and automation Workshop 2005

Summary Tele. Presence allows Microscopy facilities providing a alternative service to clients Ø High-quality real-time video is achievable over cable modem using MPEG-4 codec's and UDP transmission Ø Internet security is ensured using firewalls and encryption of communications Ø Architecture is extensible and applicable to any instrument for which DLLs or control code are available Ø Remote Access and automation Workshop 2005

Acknowledgements Ø CSIRO Minerals Aaron Torpy Nick Wilson Ø CSIRO ITS Tony Sanderson Andrew Smith Ø Engineering Firm Cymontkowski Marc Cymontkowski Ø Further information TPM web reference http: //minerals. tpm. csiro. au/software/telepresence/ Remote Access and automation Workshop 2005

Questions? Remote Access and automation Workshop 2005