panoramic viewfinder patrick baudisch desney tan drew steedly

panoramic viewfinder patrick baudisch desney tan, drew steedly, eric rudolph, matt uyttendaele, chris pal, and richard szeliski microsoft research OZCHI 2005 Preview: overall area covered so far Cropping frame: this area will survive cropping Viewfinder: what the camera sees right now

summary preview cropping frame ultra portable PC + web cam viewfinder

stitching photos into a panorama from photos [Teodosio & Bender, 1993] from video [Irani and Anandan, 1998]

stitching: allows taking wide-angle pictures

. . . getting hi-res photos

but sometimes it goes wrong…

but sometimes it goes wrong… • missing content (65% respondents) • tricky because of perspective projections • ghosting (88% respondents) • hard to detect • stitching failed (38% respondents) • lack of overlap, of texture, of focus • when users notice flaws it is too late

panoramic viewfinder

user interface Preview: overall area covered so far Cropping frame: this area will survive cropping Viewfinder: what the camera sees right now

walkthrough: goal building 115 building 114

walkthrough

post-processing • restitching • upload photo to PC • restitch using our high-quality offline stitcher maximum image quality

post-processing • auto cropping to rectangular shape • • • all desired content is preserved cropping frame guarantees this or manual cropping

live demo…

dealing with stitching failure • if user moves camera too quickly (or lack of texture) • viewfinder remains stationary • but continues to update • turns red, error sound • to fix the problem • pan camera back halfways inside panorama • and take an additional shot • or zoom out • alert ends when next match is found

the real-time cropping frame. . .

…is the key component 1. shoot the desired scene elements 2. fill the bounding box around these elements cropping frame tells users when they are done it is the focus of the user’s attention (traditional: viewfinder is focus of attention)

interaction • requirements • • • goal: make sure we have enough content too much content is ok resulting design: control cropping frame indirectly by adding content • • • no resize handles! users can focus on pointing camera interaction transfers to consumer camera

what it does not do • for artistic reasons, user may want to crop • cannot crop panorama while shooting • crop offline, with all other photos

related work

related work • stitch arbitrary order and arrangement [Szeliski & Shum ‘ 97] • real-time [Peleg and Herman, 1997]

our focus • user interface • preview to help users create successful panorama • match the interaction model of existing digital cameras allow port to existing camera interaction mechanisms

stitch assist canon powershot S 230 only horizontal HP Photosmart R 707 stitch in camera layback mode still need to retake panorama we made 4 -way assist can’t fill area take content in any order can add content post-hoc

related work on interaction tiltable interfaces [Rekimoto 1996] chameleon [Fitzmaurice 1993] peep-hole displays [Yee 2003] paintable interfaces [Baudisch 1998]

implementation

computing panorama • • written in C, frontend GDI+ and Direct. X 9 feature-based stitching [Brown et al 92] • uses same libraries as off-line stitcher 1. 2. 3. 4. • extracting multi-scale oriented patches matching with previous image estimating camera orientation warping image Difference: keep existing panorama unchanged • • benefit: faster (~ constant time) drawback: more accumulation error

capture modes • “video mode” • accumulates calibration error (Sawney et al. ’ 98) • “auto mode” • snap picture only when necessary • “manual mode” • best control, avoid motion blur

computing cropping frame • naive: O(n 4), but we do it in ~ O(n 2) • step 1: downsample image by factor 4 • step 2: for each pixel • compute hor and vert spans • step 3: for each pixel • upper bound = hor span x vert span • if (upper bound < current max) move on • else traverse down length of horizontal span starting at this pixel

future work & conclusion

future: user study. . . • as soon as we have a more reliable stitcher • consider mocking-up stitcher to simulate 100% reliability

• Losing track breaks the flow • Even with perfect stitcher there will be stitching failures • We need to be able to lose track without breaking the flow

code editing vs. stitching 1. Batch compiler 1. Offline stitcher 2. Forced syntax 2. Panoramic viewfinder 3. Curly underlines 3. Asynchronous stitcher

Panorama: largest sub panorama created so far Last frame added to the panorama 2 1 Viewfinder: what the camera sees right now Burst buffer: # pictures taken, but not processed Recycle buffer: # of frames not yet matched

future: form factor…

future • start thinking about using cameras in mobile phones as pointing devices

conclusions • one step closer to making process of taking panoramic pictures similar to process of taking normal photos

read more & try out patrickbaudisch. com/projects

END