Raskar Camera Culture MIT Media Lab Camera Culture

Raskar, Camera Culture, MIT Media Lab Camera Culture Ramesh Raskar Camera Culture Associate Professor, MIT Media Lab

Capturing and Sharing the Experience Ramesh Raskar, MIT Media Lab • Computational Cinematography and Display – Future cameras for Movies and News – 6 D displays • Morphable Studios – Illumination and appearance on proxy geometry • Programmable Movies – Enriching with Meta-data • Performance Capture – ‘On-set’ natural motion capture, ‘Second Skin’

Captured Blurred Photo

Refocused on Person

6 D = light sensitive 4 D display One Pixel of a 6 D Display = 4 D Display

Morphable Studios Desired Virtual Model © Andrei State Shader. Lamps and Being. There with UNC Chapel Hill

Towards ‘on-set’ performance capture Traditional: High-speed IR Camera + Body markers Second Skin: High-speed LED emitters+ Photosensing Body markers • • 500 Hz with Id for each Marker Tag Visually imperceptible tags + Natural lighting Unlimited Number of Tags Base station and tags only a few 10’s $

Imperceptible Tags under clothing, tracked under ambient light

Capturing and Sharing the Experience Ramesh Raskar, MIT Media Lab • Computational Cinematography and Display – Future cameras for Movies and News – 6 D displays • Morphable Studios – Illumination and appearance on proxy geometry • Programmable Movies – Enriching with Meta-data • Performance Capture – ‘On-set’ natural motion capture, ‘Second Skin’

Inverse Optical Mo-Cap Mitsubishi Electric Research Laboratories Special Effects in the Real World Device High Speed Projector + Photosensing Markers Params Location, Orientation, Illum Natural Settings Ambient Light Settings Outdoors, Stage lighting Imperceptible tags Raskar 2006 Traditional High Speed Camera + Reflecting/Emitting Markers Location Controlled Lighting Visible, High contrast Markers Hidden under wardrobe #of Tags Speed Cost Unlimited Space Labeling Unique Id Virtually unlimited Optical comm comps Low Open-loop projectors Current: Projector/Tag=$100 Limited No Unique Id Marker swapping Limited Special high fps camera High bandwidth camera Current Camera: $10 K

Coded Aperture Camera The aperture of a 100 mm lens is modified Insert a coded mask with chosen binary pattern Rest of the camera is unmodified

Capturing and Sharing the Experience • • Morphable Studios – Illumination and appearance on proxy geometry – Separating content from physical proxy Programmable Movies – Cliplet aggregation – Storytelling cameras via meta-data – Making cameras and world intelligent – Long-distance bar-codes Performance Capture – ‘On-set’ motion capture, ‘Second Skin’ – Natural environments – Lightweight technologies Computational Cinematography and Display – Future cameras for Movies and News – Universal software platform – 4 D and 6 D displays