Putting Objects in Perspective Derek Hoiem Alexei A
![Putting Objects in Perspective Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Putting Objects in Perspective Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University](https://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-1.jpg)
Putting Objects in Perspective Derek Hoiem Alexei A. Efros Martial Hebert Carnegie Mellon University Robotics Institute
![Understanding an Image Understanding an Image](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-2.jpg)
Understanding an Image
![Today: Local and Independent Today: Local and Independent](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-3.jpg)
Today: Local and Independent
![What the Detector Sees What the Detector Sees](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-4.jpg)
What the Detector Sees
![Local Object Detection True Detection False Detections Missed True Detections Local Detector: [Dalal-Triggs 2005] Local Object Detection True Detection False Detections Missed True Detections Local Detector: [Dalal-Triggs 2005]](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-5.jpg)
Local Object Detection True Detection False Detections Missed True Detections Local Detector: [Dalal-Triggs 2005]
![Work in Context • Image understanding in the 70’s Guzman (SEE) 1968 Brooks (ACRONYM) Work in Context • Image understanding in the 70’s Guzman (SEE) 1968 Brooks (ACRONYM)](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-6.jpg)
Work in Context • Image understanding in the 70’s Guzman (SEE) 1968 Brooks (ACRONYM) 1979 Hansen & Riseman (VISIONS) 1978 Marr 1982 Barrow & Tenenbaum 1978 Ohta & Kanade 1973 Yakimovsky & Feldman 1973 • Recent work in 2 D context Kumar & Hebert 2005 He, Zemel, Cerreira-Perpiñán 2004 Torralba, Murphy, Freeman 2004 Carbonetto, Freitas, Banard 2004 Fink & Perona 2003 Winn & Shotton 2006
![Real Relationships are 3 D Close Not Close Real Relationships are 3 D Close Not Close](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-7.jpg)
Real Relationships are 3 D Close Not Close
![Recent Work in 3 D [Han & Zu 2003] [Oliva & Torralba 2001] [Torralba, Recent Work in 3 D [Han & Zu 2003] [Oliva & Torralba 2001] [Torralba,](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-8.jpg)
Recent Work in 3 D [Han & Zu 2003] [Oliva & Torralba 2001] [Torralba, Murphy & Freeman 2003] [Han & Zu 2005]
![Objects and Scenes Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among Objects Objects and Scenes Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among Objects](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-9.jpg)
Objects and Scenes Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among Objects in a Well-Formed Scene (1981): – Support – Size – Position – Interposition – Likelihood of Appearance
![Contribution of this Paper Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among Contribution of this Paper Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-10.jpg)
Contribution of this Paper Hock, Romanski, Galie, & Williams 1978 • Biederman’s Relations among Objects in a Well-Formed Scene (1981): – Support – Size – Position – Interposition – Likelihood of Appearance
![Object Support Object Support](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-11.jpg)
Object Support
![Surface Estimation Image V-Left Support Vertical V-Center V-Right Sky V-Porous V-Solid Object Surface? [Hoiem, Surface Estimation Image V-Left Support Vertical V-Center V-Right Sky V-Porous V-Solid Object Surface? [Hoiem,](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-12.jpg)
Surface Estimation Image V-Left Support Vertical V-Center V-Right Sky V-Porous V-Solid Object Surface? [Hoiem, Efros, Hebert ICCV 2005] Support? Software available online
![Object Size in the Image World Object Size in the Image World](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-13.jpg)
Object Size in the Image World
![Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-14.jpg)
Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior
![Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-15.jpg)
Object Size ↔ Camera Viewpoint Input Image Loose Viewpoint Prior
![Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-16.jpg)
Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint
![Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-17.jpg)
Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint
![Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-18.jpg)
Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint
![Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-19.jpg)
Object Size ↔ Camera Viewpoint Object Position/Sizes Viewpoint
![What does surface and viewpoint say about objects? Image P(surfaces) P(viewpoint) P(object | surfaces) What does surface and viewpoint say about objects? Image P(surfaces) P(viewpoint) P(object | surfaces)](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-20.jpg)
What does surface and viewpoint say about objects? Image P(surfaces) P(viewpoint) P(object | surfaces) P(object | viewpoint)
![What does surface and viewpoint say about objects? Image P(object) P(surfaces) P(viewpoint) P(object | What does surface and viewpoint say about objects? Image P(object) P(surfaces) P(viewpoint) P(object |](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-21.jpg)
What does surface and viewpoint say about objects? Image P(object) P(surfaces) P(viewpoint) P(object | surfaces, viewpoint)
![Scene Parts Are All Interconnected Objects Camera Viewpoint 3 D Surfaces Scene Parts Are All Interconnected Objects Camera Viewpoint 3 D Surfaces](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-22.jpg)
Scene Parts Are All Interconnected Objects Camera Viewpoint 3 D Surfaces
![Input to Our Algorithm Object Detection Surface Estimates Viewpoint Prior Local Car Detector Local Input to Our Algorithm Object Detection Surface Estimates Viewpoint Prior Local Car Detector Local](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-23.jpg)
Input to Our Algorithm Object Detection Surface Estimates Viewpoint Prior Local Car Detector Local Ped Detector Local Detector: [Dalal-Triggs 2005] Surfaces: [Hoiem-Efros-Hebert 2005]
![Scene Parts Are All Interconnected Objects Viewpoint 3 D Surfaces Scene Parts Are All Interconnected Objects Viewpoint 3 D Surfaces](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-24.jpg)
Scene Parts Are All Interconnected Objects Viewpoint 3 D Surfaces
![Our Approximate Model Objects Viewpoint 3 D Surfaces Our Approximate Model Objects Viewpoint 3 D Surfaces](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-25.jpg)
Our Approximate Model Objects Viewpoint 3 D Surfaces
![Inference over Tree Easy with BP Viewpoint θ Local Object Evidence Objects o 1 Inference over Tree Easy with BP Viewpoint θ Local Object Evidence Objects o 1](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-26.jpg)
Inference over Tree Easy with BP Viewpoint θ Local Object Evidence Objects o 1 Local Surface Evidence . . . on Local Surface Evidence Local Surfaces s 1 … sn
![Viewpoint estimation Viewpoint Final Likelihood Viewpoint Prior Height Horizon Viewpoint estimation Viewpoint Final Likelihood Viewpoint Prior Height Horizon](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-27.jpg)
Viewpoint estimation Viewpoint Final Likelihood Viewpoint Prior Height Horizon
![Object detection Car: TP / FP Initial (Local) Final (Global) 4 TP / 2 Object detection Car: TP / FP Initial (Local) Final (Global) 4 TP / 2](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-28.jpg)
Object detection Car: TP / FP Initial (Local) Final (Global) 4 TP / 2 FP 4 TP / 1 FP Ped: TP / FP Car Detection Ped Detection 3 TP / 2 FP 4 TP / 0 FP Local Detector: [Dalal-Triggs 2005]
![Experiments on Label. Me Dataset • Testing with Label. Me dataset: 422 images – Experiments on Label. Me Dataset • Testing with Label. Me dataset: 422 images –](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-29.jpg)
Experiments on Label. Me Dataset • Testing with Label. Me dataset: 422 images – 923 Cars at least 14 pixels tall – 720 Peds at least 36 pixels tall
![Each piece of evidence improves performance Car Detection Pedestrian Detection Local Detector from [Murphy-Torralba-Freeman Each piece of evidence improves performance Car Detection Pedestrian Detection Local Detector from [Murphy-Torralba-Freeman](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-30.jpg)
Each piece of evidence improves performance Car Detection Pedestrian Detection Local Detector from [Murphy-Torralba-Freeman 2003]
![Can be used with any detector that outputs confidences Car Detection Pedestrian Detection Local Can be used with any detector that outputs confidences Car Detection Pedestrian Detection Local](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-31.jpg)
Can be used with any detector that outputs confidences Car Detection Pedestrian Detection Local Detector: [Dalal-Triggs 2005] (SVM-based)
![Accurate Horizon Estimation Median Error: 90% Bound: Horizon Prior [Murphy-Torralba. Freeman 2003] [Dalal. Triggs Accurate Horizon Estimation Median Error: 90% Bound: Horizon Prior [Murphy-Torralba. Freeman 2003] [Dalal. Triggs](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-32.jpg)
Accurate Horizon Estimation Median Error: 90% Bound: Horizon Prior [Murphy-Torralba. Freeman 2003] [Dalal. Triggs 2005] 8. 5% 4. 5% 3. 0%
![Qualitative Results Car: TP / FP Ped: TP / FP Initial: 2 TP / Qualitative Results Car: TP / FP Ped: TP / FP Initial: 2 TP /](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-33.jpg)
Qualitative Results Car: TP / FP Ped: TP / FP Initial: 2 TP / 3 FP Final: 7 TP / 4 FP Local Detector from [Murphy-Torralba-Freeman 2003]
![Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP / Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP /](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-34.jpg)
Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP / 14 FP Final: 3 TP / 5 FP Local Detector from [Murphy-Torralba-Freeman 2003]
![Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP / Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP /](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-35.jpg)
Qualitative Results Car: TP / FP Ped: TP / FP Initial: 1 TP / 23 FP Final: 0 TP / 10 FP Local Detector from [Murphy-Torralba-Freeman 2003]
![Qualitative Results Car: TP / FP Ped: TP / FP Initial: 0 TP / Qualitative Results Car: TP / FP Ped: TP / FP Initial: 0 TP /](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-36.jpg)
Qualitative Results Car: TP / FP Ped: TP / FP Initial: 0 TP / 6 FP Final: 4 TP / 3 FP Local Detector from [Murphy-Torralba-Freeman 2003]
![meters Summary & Future Work Ped Car Reasoning in 3 D: • Object to meters Summary & Future Work Ped Car Reasoning in 3 D: • Object to](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-37.jpg)
meters Summary & Future Work Ped Car Reasoning in 3 D: • Object to object • Scene label • Object segmentation meters
![Conclusion • Image understanding is a 3 D problem – Must be solved jointly Conclusion • Image understanding is a 3 D problem – Must be solved jointly](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-38.jpg)
Conclusion • Image understanding is a 3 D problem – Must be solved jointly • This paper is a small step – Much remains to be done
![Thank you Thank you](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-39.jpg)
Thank you
![](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-40.jpg)
![A Return to Scene Understanding [Ohta & Kanade 1978] • Guzman (SEE), 1968 • A Return to Scene Understanding [Ohta & Kanade 1978] • Guzman (SEE), 1968 •](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-41.jpg)
A Return to Scene Understanding [Ohta & Kanade 1978] • Guzman (SEE), 1968 • Hansen & Riseman (VISIONS), 1978 • Barrow & Tenenbaum 1978 • • Brooks (ACRONYM), 1979 Marr, 1982 Ohta & Kanade, 1978 Yakimovsky & Feldman, 1973
![Images Images](http://slidetodoc.com/presentation_image/2b3e6609d1784ffa106c7abbc92010a1/image-42.jpg)
Images
- Slides: 42