Outdoors Augmented Reality on Mobile Phone using LoxelBased

Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization Gabriel Takacs, Vijay Chandrasekhar, Thanos Bismpigiannis, Bernd Girod Stanford University Radek Grzeszczuk, Natasha Gelfand, Wei-Chao Chen, Yingen Xiong, Kari Pulli Nokia Research Center, Palo Alto

Video Demonstration Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 2

Outline • System Overview – Image matching on the cell phone • Data Organization – Server groups images by location • Data Reduction – Server clusters, prunes and compresses descriptors • Image Matching Results – Qualitative and quantitative • System Timing – Low latency image matching on the cell phone Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 3

System Overview GPS Server Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 4

Image Matching Query Image Affine. Test SURF-64 Ratio Descriptors Matching RANSAC Prefetched Database Images Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 5

Data Organization Kernel Loxel Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 6

System Block Diagram Group Images by Loxel Extract Features Cluster Features Geometric Consistency Check Match Images Prune Features Compress Descriptors Loxel-Based Feature Store Extract Features Compute Feature Matches Camera Image Geometric Consistency Check Feature Cache Server Network Geo-Tagged Images Device Location ANN Display Info for Top Ranked Image Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 7

Feature Descriptor Clustering • • Match all images in loxel Form graph on features Cut graph into clusters Create representative meta-features meta-feature Example Feature Cluster Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 8

Feature Descriptor Clustering Images of the same landmark meta-feature single feature Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 9

Database Feature Pruning • • Rank images by number of meta-features Allocate equal budget for each landmark Fill budget with meta-features by rank Fill any remaining budget with single features 4 x Reduction Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 10

Feature Descriptor Pruning Budget: 500 100 200 All Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 11

Feature Compression • Quantization – Uniform, equal step-size – 6 bits per dimension • Entropy coding – Huffman tables – 12 different tables S dx S dy S|dx| S|dy| Original Feature Compressed Feature Quantization Entropy Coding • 64 -dimensional SURF – 256 bytes uncompressed – 37 bytes with compression Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 12

Zu. Bu. D Dataset (~1000 Images) Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 13

Stanford Dataset (~2500 Images) Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 14

Matching Results Query Rank 1 Rank 2 Rank 3 Rank 4 Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 15

Matching Performance True Matches False Matches ~10 images / loxel ~1000 images / loxel Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 16

Timing Analysis Nokia N 95 332 MHz ARM 64 MB RAM 100 KByte JPEG over 60 Kbps Uplink Downloads Upload Geometric Consistency Feature Matching Extract Features All on Phone Extract Features All on Server on Phone Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 17

Conclusions • Image matching on mobile phone • Use loxels to reduce search space • 27 x reduction in data sent to phone – Clustering – Pruning – Compression • ~3 seconds for image matching on N 95 Outdoors Augmented Reality on Mobile Phone using Loxel-Based Visual Feature Organization 18

Questions