Developing VR Applications with the in VRs Framework

Developing VR Applications with the in. VRs Framework at IEEE VR’ 10 Part I – Introduction Christoph Anthes Roland Landertshamer

Overview • Introduction • System Core – – User Database World Database Event Manager Transformation Manager • Interfaces – Input Interface – Output Interface • Modules – – – Navigation Interaction Network • Outlook • Acknowledgements 20. 03. 10 Developing VR Applications with the in. VRs 2

Introduction and Architecture Overview 20. 03. 10 Developing VR Applications with the in. VRs 3

Introduction • in. VRs (interactive networked Virtual Reality system) – Pronounced – C++ application framework for Networked Virtual Environments (NVEs) • Publicly available under www. invrs. org (under LGPL) – Uses Trac system for distribution • Alternatively latest stable release as zip • Windows binary available as well – Available for Linux, Windows, Mac OS X, and maybe IRIX • Variety of out-of-the-box Features – – Network communication supporting close coupled collaboration Clear definition for navigation and interaction (with many techniques provided) Designed for reusability of developed components Easy-to-use configuration for immersive devices 20. 03. 10 Developing VR Applications with the in. VRs 4

Introduction • Documentation available on Trac System – – – Several publications (Bi. BTe. X files) Tutorial examples Medieval Town Tutorial and Going Immersive Tutorial Programmers’ Guide (coming soon) Doxygen (http: //doxygen. invrs. org/) • Libraries – Required • Open. SG v 1. 8 (http: //www. opensg. org/) • GMTL distributed with source (http: //ggt. sourceforge. net) • Irr. XML distributed with source (http: //www. ambiera. com/irrxml/) • CMake (http: //www. cmake. org/) – Optional • Open. AL and ALUT (http: //www. openal. org/) • ODE (http: //www. ode. org/) • track. D (with CAVELib), VRPN (http: //www. cs. unc. edu/Research/vrpn/) 20. 03. 10 Developing VR Applications with the in. VRs 5

Introduction • Where to position in. VRs in the development approaches for VR applications – Graphical Editors (using software components) • (e. g. Vir. Tools, Quest 3 D) – VR systems incorporating scripting languages • (e. g. DIVE, ALICE, AVOCADO) – Application Frameworks • (e. g. VRJuggler, instantreality) – Scene Graphs • (e. g. Open. SG, Open. Scene. Graph) – Full development from scratch via C or C++ • Open. GL presents comparably high effort for new development of VR applications 20. 03. 10 Developing VR Applications with the in. VRs 6

Introduction • Application Areas Safety Training Collaborative Work Entertainment Art Scientific Visualization Education Architecture Visualization Product Presentation World Layout 20. 03. 10 Developing VR Applications with the in. VRs 7

Introduction • in. VRs consists of the following components – – – Interfaces (designed for flexible extension) Modules (designed as plug-ins for exchangeability, modularity) System Core (not to be changed in structure, provides many extensible, exchangeable sub-components) • Communication in the diagram happens typically top-down 20. 03. 10 Developing VR Applications with the in. VRs 8

System Core 20. 03. 10 Developing VR Applications with the in. VRs 9

System Core • Acts as glue – Internally between communication units and database units – Externally to interfaces and modules • Core functionality – – – – Data types Configuration handling (parsing and updating of XML configuration) Logging (used for profiling application parts) Timer (common timer for all in. VRs components) Interfaces (interfaces to modules and other in. VRs components) ID management (unique ID handling) Application base (used for simplification of the development process) • Contains subcomponents for database handling and message handling – User and world database are used for VE storage – Event and transformation manager are used for communication 20. 03. 10 Developing VR Applications with the in. VRs 10

System Core User Database 20. 03. 10 Developing VR Applications with the in. VRs 11

System Core - User Database • Stores information about users – Local user – Remote users • Settings for user dependent application behavior – – User Representation Cursor Representation User Transformation Cursor Transformation • Coordinate Systems 1. 2. 3. 4. 5. Origin of the VE Navigated transformation User transformation Hand transformation Head transformation 20. 03. 10 Developing VR Applications with the in. VRs 12

System Core - User Database • Different models for representation are available • User Representation – Simple Avatar • Simple textured geometry not articulated – Avatara Avatar • Use of mesh skinning for animations • Separate control of head, back and hand • More detail in Going Immersive Tutorial • Cursor Representation – Simple Cursor • Basic textured geometry displayed at cursor position – Interaction Cursor • Illustrates the state of interaction (idle, selection, manipulation) • Implemented by switching through three different graphical representations 20. 03. 10 Developing VR Applications with the in. VRs 13

System Core - User Database • Different models for user and cursor behavior are available • User Transformation – Defines the user transformations – Has to be modified e. g. if tracking systems are used • Cursor Transformation – – – Defines behavior of the user cursor Might have to be altered to implement specific interaction technique Models are available for • Virtual Hand • HOMER 20. 03. 10 Developing VR Applications with the in. VRs 14

System Core World Database 20. 03. 10 Developing VR Applications with the in. VRs 15

System Core - World Database • Keeps information about the objects of the virtual world • Initially designed to partition the VE – – – Partitioning idea relates to network scalability concepts Supports as well frustum culling of the underlying scene graph Partitioning concept ideal for 2 D layout of the scenes, usable in 3 D applications as well • Important objects for structuring – Entities and entity types • Objects in a VE and their type – Tiles • Element used to partition the VE – Environment and environment layout • Describes a local coordinate system and the layout of several coordinate systems 20. 03. 10 Developing VR Applications with the in. VRs 16

System Core - World Database • World Database - Scene Graph Representation – – – Data is stored inside STL vectors Mapping of object IDs via hash tables for improved seek performance IDs stored in ID pools 20. 03. 10 Developing VR Applications with the in. VRs 17

System Core Event Manager 20. 03. 10 Developing VR Applications with the in. VRs 18

System Core - Event Manager • Used for handling and distribution of discrete events – – – – – Handles discrete communication Components can register via piping mechanism to the event manager Events are generated by components Sent to an event pipe in the event manager They can be filtered in the pipe (partially implemented) Events are stored in event queues Receiving components poll their queues Events are executed inside the components Depending on the visibility level (LOCAL, REMOTE, GLOBAL) of the messages they are sent to the network module (if available) and are distributed in a reliable manner (typically via TCP) 20. 03. 10 Developing VR Applications with the in. VRs 19

System Core - Event Manager • Conceptual Approach 20. 03. 10 Developing VR Applications with the in. VRs 20

System Core Transformation Manager 20. 03. 10 Developing VR Applications with the in. VRs 21

System Core - Transformation Manager • Used for handling and distribution transformation data – Handles continuous flow of transformation data packets – Transformation Matrices • Common type of data in VEs besides events are transformation matrices (navigation, object manipulation) • Vast amount of transformations are present because of the use of position tracking devices • Transmission not necessarily reliable – Manager has several pipes with source and destination – Every object which is to be transformed in the VE gets it‘s own pipe which is opened at the start of the transformation – Transformations are applied typically on the database of the VE and are (if available) partially transferred via the network module in an unreliable manner • Each pipe can have an arbitrary amount of stages – Each stage contains one transformation modifier – The pipe stages are predefined at application setup • Once per frame all pipe stages on each pipe are computed 20. 03. 10 Developing VR Applications with the in. VRs 22

System Core - Transformation Manager • Pipe stages contain transformation modifiers which are set up by the application developer • If more than one transformation pipe exists for a single object the transformations are processed to one resulting transformation – Implemented as a merger modifier • Transformation Modifiers – Various modules can influence the transformation pipes via the modifiers • Network, interaction, navigation, physics, collision detection, … • Pipe Stages – Different types of modifiers can be applied – Modifiers • Modify pipe content, can write values to other in. VRs components – Writers • Write values to other in. VRs components, do not modify pipe content – Mergers • Used to merge pipes – Interrupters • Pipes have to be interrupted after merging 20. 03. 10 Developing VR Applications with the in. VRs 23

System Core - Transformation Manager • Transformation Manager – Program Flow 20. 03. 10 Developing VR Applications with the in. VRs 24

Interfaces Input Interface 20. 03. 10 Developing VR Applications with the in. VRs 25

Interfaces - Input Interface • Input Interface – Used for the abstraction of input devices – Important since it is not often possible to develop on the VR installation and in case of Networked Virtual Environment seldom similar installations are interconnected – Low-level device drivers or high-level libraries can be used for gathering input 20. 03. 10 Developing VR Applications with the in. VRs 26

Interfaces - Input Interface • Input Interface – Data of the input devices is exposed in a unified form (buttons, sensors, axes) as an abstract controller – This abstract controller can be accessed by in. VRs components like modules or core components 20. 03. 10 Developing VR Applications with the in. VRs 27

Interfaces Output Interface 20. 03. 10 Developing VR Applications with the in. VRs 28

Interfaces - Output Interface • Output Interface – Used for the abstraction of output devices – Interface to common scene graphs (Open. SG, Open. Scene. Graph) and audio library (only partially implemented yet) – Use of Open. SG allows easy rendering on arbitrary multi-display installations 20. 03. 10 Developing VR Applications with the in. VRs 29

Modules Navigation 20. 03. 10 Developing VR Applications with the in. VRs 30

Modules - Navigation • Navigation Module – Architecture – Transformation results from speed, direction and orientation – These three components are individually configurable 20. 03. 10 Developing VR Applications with the in. VRs 31

Modules Interaction 20. 03. 10 Developing VR Applications with the in. VRs 32

Modules - Interaction • Interaction Module – Different interaction techniques exist e. g. • Virtual Hand (aka Natural Interaction) – Most common interaction technique in VR – Direct mapping of cursor transformation for selection and manipulation • HOMER – Combination of ray-casting for selection und Virtual Hand for manipulation at distance • Go. Go – Non-Linear scaling of distance vector between user sensors for determination of the cursor transformation – Interactions techniques can be split up into individual reusable components for different parts of the interaction 20. 03. 10 Developing VR Applications with the in. VRs 33

Modules - Interaction • Interaction Module is implemented as an automaton – Input alphabet Σ defined through components of abstract controller and system states – q = {q 0 (Idle), q 1 (Selection), q 2 (Manipulation)} – Transition functions δ defined through components of interaction techniques – Begin and end state = q 0 20. 03. 10 Developing VR Applications with the in. VRs 34

Modules - Interaction • Collaborative interaction from a scene graph perspective – Collaborative interaction allows concurrent manipulation of an object through several users at the same attribute (typically transformation matrix) – Having more than one parent nodes in a scene graph (if allowed) lead to multiple rendering of an object – Transformation changes initiated by different sources, which refer to the same object are detected as a conflict an resolved with the help of the transformation manager 20. 03. 10 Developing VR Applications with the in. VRs 35

Modules Network 20. 03. 10 Developing VR Applications with the in. VRs 36

Modules - Network • Network Module – General problem with network solutions is their ambivalence • Thus the in. VRs network module is designed for exchangeability – Consists of two layers • High-Level: general interface to the framework • Low-Level: implementation of network communication and topology – Different implementations of the low-level layer exist • Separation of the virtual world on different servers (e. g. via Grid computing) • Standard implementation of the network module uses peer-to-peer communication and topology – Typically events are distributed via TCP (reliable) and transformation data is distributed via UDP (unreliable but fast) 20. 03. 10 Developing VR Applications with the in. VRs 37

Outlook 20. 03. 10 Developing VR Applications with the in. VRs 38

Outlook • Additional Tools Physics Module Joint Interaction Avatars 3 D Menus Particle System Wrapper Height Maps Collision Maps CAVE Scene Manager Free Form Deformation Module 20. 03. 10 Developing VR Applications with the in. VRs 39

Outlook • Novel concepts – – – Clear distinction in the area of communication patterns Events and transformations are handled independently Transformation management allows for concurrent object manipulation Composeable and clearly defined interaction and navigation Automatic data distribution mechanism • Next slide sets – Medieval Town Tutorial – Going Immersive Tutorial 20. 03. 10 Developing VR Applications with the in. VRs 40

Acknowledgements • Contributors – Architecture and Main Development Christoph Anthes, Helmut Bressler, Roland Landertshamer, Marina Lenger – Tools Helmut Garstenauer, Martin Garstenauer, Adrian Haffegee, Marlene Hochrieser, Roland Hopferwieser, Robert Owen, Stephan Reiter, Thomas Weberndorfer, Christian Wressnegger, Johannes Zarl 20. 03. 10 Developing VR Applications with the in. VRs 41

Questions ?

Bibliography • Christoph Anthes A Collaborative Interaction Framework for Networked Virtual Environments. Ph. D Thesis at Institute of Graphics and Parallel Processing at JKU Linz, Austria, August 2009. • Christoph Anthes, Philipp Aumayr, Clemens Birklbauer, Roland Hackl, Marlene Hochrieser, Roland Hopferwieser, Simon Opelt, Christoph Payrhuber, Mika Satomi, Stefan Simmer, Georg Stevenson, Roland Landertshamer, Bernhard Lehner, Marina Lenger, Martin Lenzelbauer, Clemens Mock, and Alexander Wilhelm. Space Trash – Development of a Networked Immersive Virtual Reality Installation. Technical Report at Institute of Graphics and Parallel Processing, Johannes Kepler University Linz, June 2009. • Christoph Anthes, Mika Satomi, Alexander Wilhelm, Christa Sommerer, and Jens Volkert. Space Trash – An Interactive Networked Virtual Reality Installation. In Proceedings of the 11 th Virtual Reality International Conference (VRIC ’ 09), pages 107 -118, Laval, France, April 2009. • Christoph Anthes, Mika Satomi, Alexander Wilhelm, Christa Sommerer, and Jens Volkert. Design and Setup of Space Trash – A Collaborative Virtual Reality Installation using Physical Interfaces. Installation description in Proceedings of the 11 th Virtual Reality International Conference (VRIC ’ 09) Re. Volution, pages 355 -362, Laval, France, April 2009. 20. 03. 10 Developing VR Applications with the in. VRs 43

Bibliography • Christoph Anthes, Roland Landertshamer, Helmut Bressler, and Jens Volkert. Developing VR Applications for the Grid. In 1 st International Workshop on Real–Time Online Interactive Applications (ROIA) on the GRID at the 14 th International Euro-Par Conference European Conference on Parallel and Distributed Computing (Euro-Par ’ 08), pages 343– 351, Las Palmas de Gran Canaria, Spain, August 2008. • Eva Pajorová, Ladislav Hluchý, and Christoph Anthes. 3 D Geovisualization Service for Grid-oriented applications of Natural Disasters. Poster in the 16 th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG '08), Plzen, Czech Republic, February 2008. • Christoph Anthes, Alexander Wilhelm, Roland Landertshamer, Helmut Bressler, and Jens Volkert. Net’O’Drom – An Example for the Development of Networked Immersive VR Applications. In International Conference on Computational Science (ICCS '07), pages 752 -759, Beijing, China, May 2007. • Christoph Anthes, Roland Landertshamer, and Jens Volkert. Physically-Based Interaction for Networked Virtual Environments. In International Conference on Computational Science (ICCS '07), pages 776 -783, Beijing, China, May 2007. 20. 03. 10 Developing VR Applications with the in. VRs 44

Bibliography • Christoph Anthes, Roland Landertshamer, Helmut Bressler, and Jens Volkert. Managing Transformations and Events in Networked Virtual Environments. In ACM Multimedia Modeling (MMM '07), pages 722 -729, Singapore, January 2007. • Helmut Bressler, Roland Landertshamer, Christoph Anthes, and Jens Volkert. An efficient Physics Engine for Virtual Worlds. In medi@terra, pages 152 -158, Athens, Greece, October 2006. • Christoph Anthes and Jens Volkert. in. VRs - A Framework for Building Interactive Networked Virtual Reality Systems. In International Conference on High Performance Computing and Communications (HPCC '06), pages 894 -904, Munich, Germany, September 2006. • Christoph Anthes, Adrian Haffegee, Paul Heinzlreiter, and Jens Volkert. A Scalable Network Architecture for Closely Coupled Collaboration. In Journal of Computing and Informatics, 24(1): 31 -51, August 2005. • Christoph Anthes and Jens Volkert. A Toolbox Supporting Collaboration in Networked Virtual Environments. In International Conference on Computational Science (ICCS '05), pages 383 -390, Atlanta, GA, USA, May 2005. 20. 03. 10 Developing VR Applications with the in. VRs 45

Bibliography • Adrian Haffegee, Ronan Jamieson, Christoph Anthes, and Vassil N. Alexandrov. Tools For Collaborative VR Application Development. In International Conference on Computational Science (ICCS '05), pages 350 -358, Atlanta, GA, USA, May 2005. • Christoph Anthes, Paul Heinzlreiter, Adrian Haffegee, and Jens Volkert. Message Traffic in a Distributed Virtual Environment for Close-coupled Collaboration. In Parallel and Distributed Computing and Systems (PDCS '04), pages 484 -490, San Francisco, CA, USA, September 2004. • Christoph Anthes, Paul Heinzlreiter, Gerhard Kurka, and Jens Volkert. Navigation models for a flexible, multi-mode VR Navigation Framework. In ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry (VRCAI '04), pages 476 -479, Singapore, June 2004. • Christoph Anthes, Paul Heinzlreiter, and Jens Volkert. An adaptive Network Architecture for Close-coupled Collaboration in Distributed Virtual Environments. In ACM SIGGRAPH International Conference on Virtual Reality Continuum and its Applications in Industry (VRCAI '04), pages 382 -385, Singapore, June 2004. 20. 03. 10 Developing VR Applications with the in. VRs 46