VR Inputs and Outputs Output Devices n n

VR Inputs and Outputs

Output Devices n n n Visual Auditory Haptic

Visual Devices n Dimensions • Stereoscopic/monoscopic • Image resolution • Field of view • Display tech • Ergonomic factors • Cost

Human Eye n n 126 million photoreceptors Eye-gaze technology not useful yet • Calibration • Error • Overlaying control n IPD • 53 -75 mm

Head-tracked Displays n n Displays in which the user’s head is tracked and the image display screen is located at a fixed location in physical space. Examples? • CRT • Virtual Workbench or Immersa. Desk • CAVE • Many large screen displays

CRT HTD (Fishtank VR) Stationary Display 3 D Glasses Head Tracker

Stereoscopic Display Standard Display Stereoscopic Display Screen User C C A B

Virtual Workbench 3 D Glasses 3 D Display 3 D Object

CAVE

Any Projected Large Stereoscopic Screen

Characteristics n n n Large Projection-Based (except for Fishtank VR) Stereoscopic Head Tracked Stationary Display Screen(s) Let’s try to identify the pros and cons and application domains

3 D Glasses 3 D Display 3 D Object

Large Screen Projection n Infrastructure • Front Projection (user occlusion) • Back Projection (takes up space)

Large Screen Projection n n Advantages Viewer not isolated Collaboration Minimal physical gear Good resolution Large field of view n n Disadvantages One person “sweetspot” Real objects may occlude virtual objects Synchronization Light • Reflection • Bleeding n n Equipment footprint Alignment

Head-Mounted Display

Visually Coupled Systems n n Operator uses natural visual and motor skills for system control Basic Components • An immersive visual display (HMD, large screen projection (CAVE), dome projection) • Tracking system for head and/or eye motion

HMD Optical System n Image Source • CRT or Flat Panel (LCD) • Video, See–Through or Video-Pass Through

HMD Optical System n Image Source • CRT or Flat Panel (LCD) • Video, See–Through or Video -Pass Through

Head-Mounted Displays Optical System n Mounting Apparatus • What are some factors? n n Eyeglasses Weight Earphones Trackers

Field of View Monocular FOV is the angular subtense (usually expressed in degrees) of the displayed image as measured from the pupil of one eye. Total FOV is the total angular size of the displayed image visible to both eyes. Binocular(or stereoscopic) FOV refers to the part of the displayed image visible to both eyes. FOV may be measured horizontally, vertically or diagonally.

Focal Length & Diopter Focal Length - The distance from the surface of a lens at which rays of light converge. Diopter - The power of a lens. Equal to 1/(focal length of the lens measured in meters)

Ocularity n Monocular - HMD image goes to only one eye. n Biocular - Same HMD image to both eyes. n Binocular (stereoscopic) Different but matched images to each eye.

IPD Interpupillary Distance (IPD) n IPD is the horizontal distance between a user's eyes. n IPD is the distance between the two optical axes in a binocular view system.

Vignetting and Eye Relief Vignetting n The blocking or redirecting of light rays as they pass through the optical system. Eye Relief Distance n Distance from the last optical surface in the HMD optical system to the front surface of the eye.

Basic Eye Cornea Crystalline Lens Fovea Retina Optic Nerve

Properties of the Eye n Approximate Field of View • 120 • 150 • 200 n degrees vertical horizontal (one eye) (both eyes) Acuity • 30 cycles per degree (20/20 Snellen acuity).

Simple Formulas Visual Resolution in Cycles per degree (Vres) = Number of pixels /2 (Fo. V in degrees) Example: (1024 pixels per line)/(2*40 degrees) = Horizontal resolution of 12. 8 cycles per degree n To convert to Snellen acuity (as in 20/xx) Vres = 600/xx (20/47) n

Optical System n n Move image to a distance that can be easily accommodated by the eye. Magnify the image

Simple Magnifier HMD Design q p f Eyepiece (one or more lenses) Image Display (Image Source) 1/p + 1/q = 1/f where p = object distance (distance from image source to eyepiece) q = image distance (distance of image from the lens) f = focal length of the lens

Virtual Image Lens Display

Resolution n n (low) 160 x 120 color pixels per eye (current low) 640 x 480 – 1024 x 768 (high) 1000+ x 1000+ Note that resolution and FOV are independent Another important factor: pixel density • Pixels per degree of FOV n How can we use the make up of the eye to better leverage resolution?

Basic Eye Cornea Crystalline Lens Fovea Retina Optic Nerve

LEEP Optics n n Large Expanse Extra Perspective Very wide FOV for stereoscopic images • Higher resolution in the middle of FOV • Lower resolution on the periphery n Pincushion distortion

Fresnel Lens n n n A lens consisting of a concentric series of simple lens sections Result is a thin lens with a short focal length and large diameter More even resolution distribution Less distortion from lanternroom. com

Distortion in LEEP Optics A rectangle How would you correct this? Maps to this

To correct for distortion n n n Predistort image This is a pixelbased distortion Graphics rendering uses linear interpolation! Too slow on most systems Pixel shaders! Render to Texture

Distorted Field of View n n n Your computational model (computer graphics) assumes some field of view. Scan converter may over or underscan, not all of your graphics image may appear on the screen. Are the display screens aligned perpendicular to your optical axis?

Distorted Fo. V (cont. ) Distance along z-axis

Compound Microscope HMD Design Relay lens produces a real image of the display image source (screen) at some intermediate location in the optical train. The eyepiece is then used to produce an observable virtual image of this intermediate image. Exit Pupil Intermediate Real Image Eyepiece Image Relay Lens

Exit Pupil n n n The area in back of the optics from which the entire image can be seen. Important if IPD not adjustable. Compound microscope optical systems have a real exit pupil. Simple magnifier optical systems do not have an exit pupil. Exit Pupil Intermediate Real Image Eyepiece Image Relay Lens

Virtual Research V 8 HMD n Display • Dual 1. 3” diagonal Active Matrix LCD • Resolution per eye: 640 x 480 • focal length = 1 m n Optical • Field of view: 60° diagonal n Solve • What is the cycles per degree? • What is its horizontal and vertical field of view? n Pros/Cons

Characteristics of HMDs n Immersive • You are inside the computer world • Can interact with real world (mouse, keyboard, people) • Mask out real world (including body) n Ergonomics • • • Headborn weight Length of use Cue conflict n n accommodation vs. parallax Perspective • Resolution and field of view • Tethered • Avatars

Multi-tile and see-through HMDs

Hand Mounted Displays n Binoculars • 1280 x 1024 n $19900

Floor Supported Displays n Articulated mechanical arm • • n n n Offload weight 0. 2 ms Latency High accuracy 0. 1 degree orientation error Pole in the way Limited Space (6’ diam, 3’ hgt) Good FOV Good resolution (1280 x 1024) Fakespace Boom 3 C Windows. VR • Differences n n Stereo Tracking • Advantages

Desk Supported Displays n n n Fishtank VR Autostereoscopic Displays Addresses weight fatigue Tracking? DTI 2018 XL Virtual Window Elsa Ecomo 4 D

Monitor Large Volume Displays n n Multiple people Monitor Based LVD • Fishtank VR • With limited FOV, what is a possible solution? n n n Exaggerate tracking More monitors (synch, bezel) Most use active stereo (shutterglasses) • 29 -32% transmittance • $100 -800 • Can be used for prolonged periods

Projector LVD n n Workbench CAVE • $300 -500 k with SGI • $100 k with PC clusters • Issues? n Large wall displays

Sound Displays n n What are good goals for VR Sound Displays Compare importance to: • Movies • Video Games n What role does sound play? • Interactivity • Immersion • Perceived image quality(!) n Dimensions • Mono/Stereo/”virtual sound” n Tracker • Occlusion • What does it take to uniquely place a sound source?

3 D Sound n Azimuth Cues • Interaural Time Difference n head radius/(speed of sound(θ+sin θ)) • Interaural Intensity Difference n n n High frequency Head shadow effect Elevation Cues • • • How are ears modeled? If simple holes what is the problem? Pinna effects sound propagation n Frequency attenuation and amplification

n Range Cues 3 D Sound • Intensity • Motion Parallax n n As the user moves his/her head, the changes in azimuth Head Response Transfer Functions • Two microphones at the ears • Record sounds (HR impulse responses) • Corresponding Fourier transforms (HRTFs) • Depends on sound frequency • No two people are exactly the same n Given HRTFs • Take sound • Apply filter • Get two signals that should replicate the sound to the user n However what’s the effect of using another person’s HRTF?

Sound Display Hardware n n Offload compuational load Convolvotron • Crystal River • First virtual 3 D sound Generator • NASA 1988 • Real-time DSP • Calculate new HRTFs given n n Sound Head position (tracker) Audigy Soundblaster EAX/ Aureal A 3 D • Simple geometry • Occlusion, reflection n n Tracking? High-end SDKs - $14 k

Speaker Layouts n n Stereo Quad speakers 5. 1 surround sound (6. 1/7. 1) Multiple users?

Haptic Feedback n n n Hapthai – touch Greatly improves realism When is it needed? • Other cues occluded/obstructed • Required for task performance • High bandwidth! n Why are hands and wrist the most important? • High density of touch receptors • Full body are still in research and not very usable (back, body) n Two kinds of feedback • Touch Feedback – information on surface geometry, roughness, temperature, etc. Does not resist user contact • Force Feedback – information on compliance, weight, and inertia. Actively resists contact motion

Passive Haptics n n Not controlled by system Pros • • • n Cheap Large scale Accurate Cons • Not dynamic • Limited use

Active Haptics n n Actively resists contact motion Dimensions? • • Force resistance Frequency Response Degrees of Freedom Latency Intrusiveness Safety Comfort Portability

Human Haptic System Sensors n Meissner and Pacinian • Quick (high frequency) sensors (50 -300 Hz) • Vibration, acceleration • Low resolution n Merkel and Ruffini - Slow (low frequency) adapting (0 -10 Hz) • Constant, surfaces, edges • High resolution n n Thermoreceptors – temperature Proprioceptors/Kinesthesia • Perception of own body motion n Page 96

Tactile Feedback Interfaces n n Goal: Stimulate skin tactile receptors How? • Air bellows • Jets • Actuators (commercial) • Micropin arrays • Electrical (research) • Neuromuscular stimulations (research)

Tactile Mouse n n Logitch i. Feel Mouse Electrical Actuator • Shakes up and down (do not disturb XY motion) • Mouse over buttons n n Haptic Bump Rumble Pack

Cyber. Touch Glove n n n Cyber. Glove Immersion Corporation Six Vibrotactile actuators • Back of finger • Palm n n Off-centered actuator motor Rotation speed=frequency of vibration (0 -125 Hz) When tracked virtual hand intersects with virtual object, send signal to glove to vibrate $15000

Force Feedback Devices n n n Attempts to stop the user’s motion if necessary Larger Actuators Grounded Mechanical bandwidth - perceived frequency of a haptic interaction (in Hz) Control bandwidth – system control bandwidth to device

Force Feedback Joysticks n n n Wing. Man Force 3 D Inexpensive ($60) Actuators that can move the joystick given system commands Max 3. 3 N of force Force feedback driving wheel

Sens. Able PHANTo. M Arm n n n n n 3 and 6 DOF force feedback device Most popular (1000+) in VR research 3 DC motors Little interitia/friction 6. 4 N (1. 7 continuous) Control loop = 1000 Hz $1500 – Omni $16000 – standard Desktop $56000 – 6 DOF high end

Other Force Feedback n Large Forces • Haptic Master Arm n 250 N • Sarcos Arm • http: //www. princeton. e du/~jmelli/papers/ch 4 haptics. pdf n Gloves • Cyber. Grasp n $39000 • Cyber. Force n $56000

Why is it hard? n n n n Control bandwidth Mechanical bandwidth Forces Hygiene Gender differences Portability Encumbrance Solutions? • Specific engineering