Immersive actor feedback system What sees the actor
Immersive actor feedback system • What sees the actor on set ? ( Answer! ) Research & Innovation 1
Immersive actor feedback system • Viewpoint dependent projection – Virtual reality: CAVE system • Integration into Chroma-keying environment – Retro-reflective cloth (BBC patent) Research & Innovation 2
Immersive actor feedback system • Retro-reflective cloth – “truematte” developed by BBC LED off Camera with LED ring Research & Innovation LED on 3
Immersive actor feedback system Mask generation Projector Head position Viewpoint Renderer Research & Innovation 4 3 D model of virtual scene 3 D model of Actor
Immersive actor feedback system • View dependent projection The Ambassadors (1533), by Hans Holbein the Younger. National Gallery London Research & Innovation 5
Immersive actor feedback • Head tracking Fast template matching filter on voxel data Research & Innovation 6
Production Visualisation Tools Director’s View Planning, On-set 3 D model of real scene Monitor Actor Feedback On-set 3 D model of virtual scene • real-time, low-quality Research & Innovation 7
Immersive actor feedback system Research & Innovation 8
Bamzooki • BBC Children Production Research & Innovation 9
Free-viewpoint video for sports • ‘Virtual Replay’ of football scenes – Allows viewer interactive review – Manually modelled offline – Not realistic Research & Innovation 10
Free-viewpoint video for Sport visualisation • Piero • Overcome limitations: use multi-camera system, iview project Research & Innovation 11
Applying multi-camera techniques to sport scenarios • Test shoot England-Wales in Old Trafford 9 Oct 2004 Research & Innovation 12
Where to put the cameras? • Broadcast coverage cameras • Pro: – Works without extra setup • Cons: – Unreliable framing Research & Innovation 13
Applying multi-camera techniques to sport scenarios • Calibration with wand • Problems: – Cameras knocked overnight – Cameras not rigidly mounted Research & Innovation 14
Line-based calibration • Colour-based edge detection • Line-fitting • Calibration of camera parameters (T, R, f) against dimensions of the pitch • Runs at video frame rate Research & Innovation 15
Processing - Segmentation • Chroma-key against green Research & Innovation 16
Processing – 3 D Reconstruction • 3 D representations – Billboard (flat polygon, sprite) – Visual hull (Shape from silhouette) Research & Innovation 17
Results From 16 cameras, visual hull + graph-cut shape optimisation (results from iview project, jointly with University of Surrey) Research & Innovation 18
Results Research & Innovation 19
Summary • Computer vision techniques are important building blocks for the automated generation of visual effects • For the integration of video and graphics several optical phenomena have to be harmonised. In descending order of importance: – Camera perspective, occlusions, shadows, . . • Visual feedback systems integrated into the production pipeline are important to allow interaction with virtual objects • Free-viewpoint video allows new insights – Goal: Stream 3 D to the viewer at home Research & Innovation 20
Research & Innovation 21
3 D Modelling • Graph-cut • Details: A Bayesian Framework for Simultaneous Matting and 3 D Reconstruction, J. -Y. Guillemaut, A. Hilton, J. Starck, J. Kilner, and O. Grau, 3 DIM’ 07 Research & Innovation 22
Results • Video from ORIGAMI demo production Research & Innovation 23
Applications: Requirements Shape classes: IV 2 -D 2 1/2 -D III I II 3 -D coarse III rich detailed IV c d II b I a 1 2 3 e 4 Optical phenomenon: Applications: 1) camera perspective 2) occlusions 3) shadows 4) reflections a) virtual studio b) On-set Visualisation c) Pre-Visualisation d) Interactive e) Digital effects Research & Innovation 24
Introduction • Conventional work flow in (feature) TV or film productions Animations / Virtual Set Story board Planning Phase On-set Phase Post Production Final Programme expensive Research & Innovation 25
Introduction • ORIGAMI approach – Mixed-reality techniques for visualisation through the production Animations / Virtual Set Story board Planning Phase Research & Innovation On-set Phase 26 Post Production Final Programme
- Slides: 26