Vision Video and Virtual Reality Omnidirectional Vision CSC

Vision, Video and Virtual Reality Omnidirectional Vision CSC 59866 CD Fall 2004 Lecture 6 Omnidirectional Cameras Zhigang Zhu, NAC 8/203 A http: //www-cs. engr. ccny. cuny. edu/~zhu/ Capstone 2004/Capstone_Sequence 2004. html

Vision, Video and Virtual Reality n Lecture Outline Applications l l l Robot navigation, Surveillance, Smart rooms Video-conferencing/ Tele-presence Multimedia/Visualization n Page of Omnidirectional Vision (Many universities and companies…. ) l http: //www. cis. upenn. edu/~kostas/omni. html n Design Requirements l l n Several Important Designs l l l n 360 degree FOV, or semi-sphere or full sphere in one snapshot Single effective viewpoint Image Resolutions – one or more cameras? Image Sharpness – optics as well as geometry Catadioptric imaging : mirror (reflection) + lens ( refraction) Mirrors: Planar, Conic, Spherical, Hyperboloidal, Ellipsoidal, Paraboloidal Systematic design ( S. Nayar’s group) Calibrations l Harder or simpler?

Vision, Video and Virtual Reality n Catadioptric imaging : l mirror (reflection) + lens ( refraction) l Theory of Catadioptric Image Formation ( S. Nayar’s group) n "A Theory of Single-Viewpoint Catadioptric Image Formation" , Simon Baker and Shree K. Nayar , International Journal of Computer Vision, 1999. n Mirrors l Planar l Conic, Spherical l Hyperboloidal, Ellipsoidal l Paraboloidal n Cameras (Lens) l l n Sensor Design Perspective (pinhole) or orthogonal (tele-centric lens) projection One or more? Implementations l l Compactness - size, support, and installation Optics – Image sharpness, reflection, etc.

Vision, Video and Virtual Reality n Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa 96) has a single effective viewpoint Mirror pyramid 6 cameras 4 camera design and 6 camera prototype: Full. View - Lucent Technology http: //www. fullview. com/

Vision, Video and Virtual Reality n Planar Mirror Panoramic camera system using a pyramid with four (or more) planar mirrors and four (or more) cameras (Nalwa 96) has a single effective viewpoint Geometry of 4 camera approach: four separate cameras in 4 viewpoints can generate images with a single effective viewpoint

Vision, Video and Virtual Reality Planar Mirror Approach n A single effective viewpoint n More than one cameras High image resolution n

Vision, Video and Virtual Reality Planar Mirror Approach n A single effective viewpoint n More than one cameras High image resolution n

Vision, Video and Virtual Reality n n Conic Mirror Viewpoints on a circle semispherical view except occlusion Perspective projection in each direction Robot Navigation (Yagi 90, Zhu 96/98) viewpoint pinhole

Vision, Video and Virtual Reality Spherical Mirror n Viewpoints on a spherical-like surface n Easy to construct (Hong 91 -UMass ) Locus of viewpoints Intersection of incoming rays are along this line

Vision, Video and Virtual Reality n Single Viewpoint l n Hyperboloidal Mirror if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the hyperboloid Semi-spherical view except the self occlusion viewpoint Rotation of the hyperbolic curve generates a hyperboloid P 1 P 2 pinhole

Vision, Video and Virtual Reality n Hyperboloidal Mirror ACCOWLE Co. , LTD, A Spin-off at Kyoto University l http: //www. pluto. dti. ne. jp/~accowle 1/ n n n Spherical Mirror Hyperbolic Mirror Image: High res. in the top

Vision, Video and Virtual Reality n Single Viewpoint l n Ellipsoidal Mirror if the pinhole of the real camera and the virtual viewpoint are located at the two loci of the ellipsoid Semi-spherical view except the self occlusion pinhole P 2 P 1 viewpoint

Vision, Video and Virtual Reality Panoramic Annular Lens - geometric mathematical model for image transform & calibration P 1 panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 * single view point (O) P B Hyperboloidal mirror O pinhol C e p p 1 Ellipsoidal mirror

Vision, Video and Virtual Reality panoramic annular lens (PAL) - invented by P. Greguss * 40 mm in diameter, C-mount * view: H: 360, V: -15 ~ +20 • single view point (O) Panoramic Annular Lens Image: High res. In the bottom

Vision, Video and Virtual Reality Cylindrical panoramic un-warping Two Steps: (1). Center determination (2) Distortion rectification 2 -order polynomial approximation

Vision, Video Paraboloidal Mirror and Virtual Reality n n n Semi-spherical view except the self occlusion Single Viewpoint at the locus of the paraboloid, if l Tele-lens - orthographic projection is used Mapping between image, mirror and the world invariant to translation of the mirror. This greatly simplifies calibration and the computation of perspective images from paraboloidal images viewpoint P 1 P 2 tele-lens

Vision, Video and Virtual Reality n n Paraboloidal Mirror Remote Reality – A Spin-off at Columbia University http: //www. remotereality. com/ Camcorder Web Camera Back to Back : Full Spherical View

Vision, Video and Virtual Reality n n Paraboloidal Mirror Remote Reality – A Spin-off at Columbia University http: //www. remotereality. com/

Vision, Video and Virtual Reality n Omnidirectional Camera Calibration – Harder or Easier? l l n Catadioptric Camera Calibration In general, the reflection by the 2 nd order surface makes the calibration procedure harder However, 360 view may be helpful Paraboloidal mirror + orthogonal projection l Mapping between image, mirror and the world invariant to translation of the mirror. l Projections of two sets of parallel lines suffice for intrinsic calibration from one view n C. Geyer and K. Daniilidis, "Catadioptric Camera calibration", In Proc. Int. Conf. on Computer Vision, Kerkyra, Greece, Sep. 22 -25, pp. 398 -404, 1999.

Vision, Video and Virtual Reality Image Properties of Paraboloid System (Assuming aspect ratio = 1) n The Image of a Line l l n Dual Vanishing Points l n There are two VPs for each set of parallel lines, which are the intersections of the corresponding circles Collinear Centers l n is a circular arc if the line is not parallel to the optical axis Is projected on a (radial) line otherwise The center of the circles for a set of parallel lines are collinear Vanishing Circle l The vanishing points of lines with coplanar directions* lie on a circle ( all the lines parallel to a common plane)

Vision, Video and Virtual Reality Image Properties of Paraboloid System (with aspect ratio) n The Image Center l l n Projection of a Line with unknown aspect ratio l n Is an elliptical arc in the general case The Aspect Ratio l n Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines Can be determined by two sets of parallel lines Is determined by the ratio of the lone-short axes of the ellipse corresponding to a line Intrinsic Calibration l l Estimate aspect ratio by the ratio of ellipse Estimate the image center by the intersection of vanishing lines of two sets of parallel lines in 3 -D space

Vision, Video and Virtual Reality. Calibration n of Paraboloid System The Image Center l l Is on the (“vanishing”) line connecting the dual vanishing points of each set of parallel lines Can be determined by two sets of parallel lines

Vision, Video and Virtual Reality. Calibration n n of Paraboloid System The Image Center l Yellow “vanishing” line of horizontal set of parallel lines l Pink “vanishing” line of vertical set of parallel lines The Vanishing Circle (Red dotted) l The vanishing points of lines with coplanar directions ( on a plane in this example) Projected to the plane of the calibration pattern

Vision, Video Next and Virtual Reality n Next: Features END