Business Identification Local Neighborhood Alexander Darino Outline Where

Business Identification: Local Neighborhood Alexander Darino

Outline • Where Am I? project obtains geolocation of camera from image • Objective: Obtain the geolocation and address of Businesses in image – Assume Business is nearby, eg. < 100 m from camera – Compare methods of obtaining this information

Outline Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Name Matching Image OCR Detected Text Business Identification

Outline • This Week: – Finding Local Businesses via Geocode Search – Finding Local Addresses via Reverse Geocoding – Extracting Identifying Text (ie. store names) via Optical Character Recognition (OCR) – Matching OCR text to Business Names • Next Steps/Weekend Objectives • Acknowledgements

Obtaining Business Names ✓

Local Businesses: Geocode Search • Used Three Place-Search APIs: – Yelp API - detailed yellow page-type results – Google Places API - "Skinny" + Reference to more information – City. Grid API - minimal yellow page-type results • Used by Yellow Pages, Super Pages • At present, only interested in business names • Aggregated names from all three APIs • Example (next slide)

Local Businesses: Geocode Search 40. 441127247181797 -80. 002821624487595 Denham & Company Salon Ullrich's Shoe Repairing Nicholas Coffee Co Bella Sera On the Square A & J Ribs Starbucks Coffee Jenny Lee Bakery Galardi's 30 Minute Cleaners Jimmy John's Gourmet Sandwiches Charley's Grilled Subs Fresh Corner Lagondola Pizzeria & Restaurant Camera Repair Service Inc Pittsburgh Cigar Bar Original Oyster House Mix. Stirs 1902 Tavern Costanzo's Pittsburgh Silver Llc Graeme St Galardi's 30 Minute Cleaners Denham & Co Salon Bruegger's Bagel Bakery Nicholas Coffee Co Market Square Fat Tommy's Pizzeria Mixstirs Cafe Giggles Rycon Construction Inc Garbera, Dennis C, Dds - Emmert Dental Assoc Bella Sera on the Square Mancini's Bread Co Las Velas Ciao Baby Washington Reprographics Inc Highmark Life Insurance Co Fischer, Donald R, Md - Highmark Life Insurance Co Jimmy John's Lynx Energy Partners Inc Emmert Dental Assoc

Local Businesses: Geocode Search Results: 12 Success, 3 Partial Q 9: First Presbyterian Church [turns out it wasn't a cathedral] (SUCCESS) Q 28: Moe's (SUCCESS) Q 34: Bruegger's Bagels (SUCCESS) Q 35: Breuggers, Tavern, Nicholas (SUCCESS) Q 42: Tavern, Nicholas, Constanzo's [in distance] (SUCCESS) Q 57: Tambellini (SUCCESS) Q 63: Benedum Center (SUCCESS) Q 141: Roberts/7 -Eleven (PARTIAL - misses Roberts) Q 200: Goodyear (SUCCESS) Q 238: Far from Bruegger's, Tavern, Nicholas (PARTIAL - misses Tavern) Q 246: Some theater (can't read it) (SUCCESS) Q 249: George Aikens (SUCCESS) Q 260: Dogs Dun Wright, Cherrie's diner (SUCCESS) Q 300: Giggles, Bruegger's, Tavern (in distance) (SUCCESS) Q 318: Fifth Avenue Place, Wines & Spirits (PARTIAL - misses Wine & Spirits)

Local Businesses: Geocode Search • Strengths – Aggregated results almost always found Business of interest • Weaknesses – Each API limits query result set size - this is why we aggregate – Contacted Yelp, Google, City. Grid for extended API Access. • Heard back from City. Grid; conference call next week. – Only businesses listed – Not all businesses listed • All but one "Partial" result were for unlisted businesses • Limitations – Have only tested for 15 Pittsburgh images - unknown result quality for rural areas.

Local Businesses: Geocode Search ✓ ✓* ✓ * Implicitly verified: APIs can search by latitude/longitude OR address

Local Addresses: Reverse Geocoding • Used Two Reverse-Geocoding APIs – Google: provides a range of addresses on the same road • Usually the road is correct, but sometime's it's slightly off • Sometimes the road is correct, but the actual address number is not in the range • Bing: provides one or two proximate addresses – Rates it's own confidence. Even 'Medium' confidences are very accurate – Address is never exact, but is almost always adjacent to correct address – Results returned never consistent: always returns one or the other or both of the two addresses regardless of confidence level

Local Addresses: Reverse Geocoding • Intent: Get up to ~500 nearby addresses • No Address Search API Available ✓ ✓* ✗ ✓

Extracting Identifying Text: OCR Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Name Matching Image OCR Detected Text Business Identification

Extracting Identifying Text: OCR • Given: – List of nearby businesses (names, addresses, etc) – Image containing businesses with visible names • Objective: – Extract name of businesses from image – Identify businesses located in image • Match names extracted from image to names in business list

Extracting Identifying Text: OCR • Used Two OCR APIs: – GNU OCR (Ocrad) – GOCR • OCR APIs highly sensitive to: – Font (only works well with roman font) – Perspective – Scale – Binarization Threshold – Dark on Light vs. Light on Dark (inversion)

Extracting Identifying Text: OCR • OCR API evaluations – Ocrad - could not yield any meaningful data across over 200 scale/threshold/inversion combinations – GOCR - produced good results across 10 scales with and without inversion using threshold automatically determined by Otsu's method • Examples of GOCR output (next slides)

Extracting Identifying Text: OCR

Extracting Identifying Text: OCR n. c. . . . o. a. . . u. . . o. U. . D. oa. . e. . . _Ru. EGGE. . KERy. . J. . . w. . . L. . . . M. II. . . c. . . i. . . . l. . J. t. . . llt. . . l SHA. P. It. . tllt. . _. l. . . J y. _. c _. . _tt. . _. r. . t. t_t. . _. l. . J. r. r. I.

Extracting Identifying Text: OCR

Extracting Identifying Text: OCR u. . . _nq. . . eo. R. E. l. e. . . í. . e. . . n. e. R. E. . . e. . o. _. . E. R. E. IKE. . . . I. ltl. O. . r. E. . o. . . E. . . I. K. E. o. . . J. n. . c. . . E. R. E. I. E. . . . M. . E. R. E. . . a J. . . Gu. ge E. F. . _. . . E. . g. E. D. . . f. Ul. I. . lll. IIi. l. . Xl. .

Extracting Identifying Text: OCR

Extracting Identifying Text: OCR

Extracting Identifying Text: OCR __. ncu_. l. . . _J. . . ne. . . . _n. _. . v. . . ra. . . d_. . . . i. . n. . U ll REs. T. un. AN. . . r. c. . . r. . . T t. r. Jll. . . m. . . c. . . n. . J n. I. . c. . . r. r ESTAU. ANT. r. O. . c. cc. Note: Even though "Tambellini" is a roman font, it is too stretched to be picked up by GOCR

Extracting Identifying Text: OCR • Strengths – Applicable to expected input of orthogonal images – Output can be run through word similarity matching algorithms • Weaknesses – Only works well(-ish) for strictly roman font • Limitations – Will perform poorly for artistic fonts and business signs • Conclusion – By itself, OCR is not the best approach towards Business identification (poor recognition, franchises, perspective, etc) – OCR could be used as part of Business identification voting scheme

Matching OCR Text to Business Names Latitude Longitude Geocoding Reverse Geocoding Nearby Businesses Business Name Matching Image OCR Detected Text Business Identification

Matching OCR Text to Business Names • Fuzzy String Matching: TRE Package – Approximate Regular Expression Matching – Returns edit-distance of matched text • Filter OCR text – Trimming – Chunking – Uselessness (ie. Less than two letters) • Developing algorithm to rate confidence of business name appearing in image

Matching OCR Text to Business Names •

Matching OCR Text to Business Names

Next Steps/Weekend Objectives • Implement ‘chunking’ to OCR output • Evaluate and refine algorithm against multiple inputs • Detect location of text in image

Acknowledgements • Subh – Directed us to the Ocrad and GOCR packages – Provided feedback on how to calibrate OCR packages to extract meaningful text (eg. scaling, inversion, etc)

Thank You