blind Bike an assistive bike navigation system for

blind. Bike: an assistive bike navigation system for low vision persons Lynne Grewea , William Overella and Christopher Lagallia a. Computer Science, California State University East Bay, 25800 Carlos Bee Boulevard, Hayward, CA USA, 94542

Assistive Biking for Low Vision • Can we develop a mobile application that assists Low Vision with task of biking • Needs: road following, intersection detection & crossing, obstacle detection, route planning

Previous Work • NONE in biking (one unique echolocation system a blind person created for themselves not safe, not readily adaptable) …. beeps when close to something (handles some kinds of obstacle detection but, NOT road following, intersection detection • Closest = self-driving cars • • • Physically more stable On-board computers 3 D + more sensing Expensive Too heavy for bikes • Low Vision= closest is street crossing using mobile apps (zebra walk detection)

Biking more challenging than driving • Bike can tilt and almost nothing is visible • Jittery less stable movement

Operating Conditions • VIDEO: 1920 x 1080 320 x 240 for faster processing • DAYLIGHT: Assumes daylight conditions with no extreme weather (snow, sleet, heavy rain) • TILT: no more than 30 degrees • HEADING: no more than 30 degrees off directs user to correct heading • POSITIONING: goal is for the biker to remain on the right side of the road.

Safety is paramount • • Goal: Assistive technology not autonomous No control of bike steering or pedaling Responsive to biker Use audio and speech recognition, magnification, simple interface “TURN LEFT IN 320 METERS”

Self Driving Cars – Previous Work • Proposed (NOT available) – communications solution • Some rely on 3 D and HEAVY computational power Wont work for a bike

Complexity shift highway driving city/road driving • Way more occlusion of view (from Google)

Computation and our Initial Goal • Want to create free software • Try to do on device only processing we anticipated would not be enough frame rate (unfortunately true) • Next iteration Cloud + leverage Deep Learning Services

Sensors? • Simple Lane Detectors • Complex Multi Sensors 3 D, Radar, Sonar, 2 D, audio ++++

Our Sensors • We cant fit these things on Gramma Cadence sensor Bike mounted Ant+/Bluetooth

blind. Bike Current System Overview

blind. Bike Route Planning & Navigation • Uses on phone Location Services (GPS) • Uses Map. Quest free tier to perform Route Planning

Route Tracking • Route broken up into set of points where maneuvers or road variations can occur Details of shape points indicate: changes in road orientation as well as intersections and other maneuver points • Track where we are on our set of shape points in route and when we move past a shape point in correct heading means we are on next leg of our route

On / Off route and heading • If we are outside of our route must redirect user determine if off route or going in wrong direction • Calculate current heading using gyroscope if wrong heading (compare to heading of current road segment biker is on) THEN must ell user and redirect. This can be result of bike tilt or heading in wrong direction so look for repeatable # of frames off heading before warn

blind. Bike Road Following • Purpose: follow right side of road

First --- Example partial run Running in debug mode Specifying local coffee shop Biking and first turn instruction coming up “TURN LEFT IN 320 METERS” Biking and wandered too far to right in road “Lane Departure Warning: Merge Left” Biking and back in the correct location so no audio After turning left. . further down in navigation we are approaching a right turn “Turn right in 160 meters” Arrived “You have arrived at your desti- nation”

Road Following – Preprocessing • Color space -> HIS Hue Saturation Intensity For speed no perspective correction • Crop out top of image = correspond to sky (reject too severe tilt)

Road Following – Classification • Initial Classes: road, sky, white line, “other” • Final classes: road, dark road, sky, white line, sidewalk, “other”

Classification • OPTION 1: Thresholding ü Fast Ø Not as effective as GMM, range overlaps issue • OPTION 2: GMM ü More accurate Ø Slower, especially when use more than one mode in GMM q. So, currently for frame rate restrict to single mode –some issues with overlap (hence conversion to 5 classes for better separation)

GMM classification results • Overall Good Results Grey = road Purple = dark road Yellow = sidewalk Green = other White = white line

GMM classification results • Ground cover variation examples Grey = road Purple = dark road Yellow = sidewalk Green = other White = white line

2 D is NOT enough • Obstacles not road material or road like are okay BAD OK Shows limitation of only using 2 D not 3 D **** Grey = road Purple = dark road Yellow = sidewalk Green = other White = white line

Classification need improvement Class % Labeled Correctly % non-class wrongly labeled as this class Sky - - All Road 88. 84% 0. 92% White-line 11. 24% 0. 00% Sidewalk 23. 20% 0. 06% • HOWEVER: Road=72. 36%, Dark-road= 46. 31%, some misclassification of road-> dark-road and vice versa. • Classification is biggest problem in system • Solutions • Go to multi-modal GMM and move to Cloud to recover from computational burden • Add texture feature • As 3 D becomes available on phones add 3 D sensing- but, increases processing

Road Following – Blob Selection Group the labeled pixels into Blob Select “best” Blobs corresponding to Roads • Eliminate size, location (looking for right)- know heading

Road Following – Blob Selection GOOD results – no errors introduced. Future with going to cloud (+computation)= • Merging • Adaptive • Special Adaptive Shadow Future with 3 D • Introduction of 3 D in merging

Road Following – Line Detection Goal: to find ridge edge of road Idea: approximate with line from edges of blobs Problem: we don’t have perfect or even good linear edges Some problems from classification errors – some obstacles. Future with cloud(+computation) = smoothing of boundaries, 3 D elimination

Line Detection • OPTION 1: number of Hough based algorithms • OPTION 2: simple line fitting algorithm bottom right portion

Line Detection –Modified Hough • Using gradient to direct voting • Smoothing in Hough Space 1. Hough Transform with Elimination by Gradient • 2. Don’t allow vote if not near heading direction of road Hough Transform with Magnification by Gradient • 3. Magnify vote by gradient value match Hough Transform with Reduction by Gradient • Only allow to vote for lines near gradient value

Must select best line of Hough Lines Series of algorithms testes that focus on slope, location of line and its distance to right at different areas of image furthest right at bottom and middle of image yielded best results Slope & Right Right&Bottom intersection Mid Row Intersection Highest Vote only

Option 2: Vertical Line “fitting” • Solve the problems of Hough –do not have linear edges • IDEA: fit best vertical line but, only to lower right portion of blobs • Reasoning: estimating small portion of a road edge as vertical is not terrible + we really only want to know the location of the road edge nearest the bike

Which option? Close call. . . both can do well Hough Vertical

Which option? Future need to merge the results – look at irregularity of boundary in lower right and if too much use Hough otherwise use Vertical Line fitting. Current measure certainty/goodness of vertical line fit could use this metric easily to switch to Hough Line – but, +++ computational costs

Road Following Decision Using distance range at bottom of image On Track Merge. Left Merge. Right

Road Following Decision Results • • • 50 video test runs ( frames analyzed) NOT Good enough Biggest problem is first stage miss-classification ALSO – scenarios not handled (colored bike lanes) As expected Vertical best • Merge could do better but, still problems with miss-classification go to cloud for more computationally expensive modeling + 3 D as released Algorithm Accuracy when correct line Accuracy regardless of line correction Vertical Line 100% 84% Hough 100% 66. 1%

Demo

Intersection Detection Intersection Prediction Shape Point Following/ Tracking Route Segment Extraction Intersection Shape Point Searching Distance to Intersection Estimation Search Area Calculation Intersection Light Detection Pixel Level Classification Light Blob Extraction Blob Size & Shape Filtering Final Light Decision

Intersection Prediction Shape Point Following/ Tracking Route Segment Extraction Intersection Shape Point Searching Distance to Intersection Estimation Search Area Calculation • Route broken up into set of points where maneuvers or road variations can occur Details of shape points indicate: changes in road orientation as well as intersections and other maneuver points

Route Tracking & Prediction • Track location on set of shape points in route and when move past a shape point in correct heading on next leg of our route • When close enough to potential intersection trigger Intersection Detection predicted location based on distance to intersection range and range of light heights. Lets Start looking for Light here

Intersection Detection Intersection Light Detection Pixel Level Classificati on Light Blob Extraction Blob Size & Shape Filtering Final Light Decision

Pixel Classification • Hue, Saturation, Intensity Space

Pixel Classification - Results Note: red is picking up some of the building –it is in range but, it won’t cause problems due to later stages Red Yellow Green

Pixel Classification - Results Note: red is picking up some of the red curb paint –it is in range but, it won’t cause problems due to later stages Red Yellow Green

Pixel Classification - Results Note: red of cone –it is in range but, it won’t cause problems due to later stages Note: red of tail lights –it is in range This is a special issue we handle later Red Yellow Green

Pixel Level Classification Results • RED = average detection of 89% for red light pixels with the highest being 95% in an image • GREEN = average detection of 84% with the highest per image rate again being 95% Explanation = some green lights can run “hot” and center is almost white. • YELLOW = average detection of 96%

Blob Detection & Filtering • Group for Red, Yellow, Green binary images separately Create Binary Images for Red, Green and Yellow light values Extract Contours from the Binary Images SHAPE 1: Filtering of blobs based on complexity SHAPE 2: Filtering of blobs based on compactness Filtering Contours by Blob Size

Filtering - size

Filtering - compactness

Filtering - complexity

Light Selection If Red or Yellow only are found If Green only is found Find the Biggest Red blob Find the Biggest Green blob Calculate distance from the blob based on size Make a decision to Proceed across the intersection Make a decision to stop or slow down based on the distance

Radius Distance as function of blob size Distance

Intersection Detection Examples TOO FAR to stop yet SLOW DOWN –Red detected in distance

Intersection Detection Examples STOP

Green light case

ASSISTED Intersection Crossing • Safety Key – current design to be assistive/informative • Does not use ANY vision • Quick but, relies on user walk the bike Wahoo Bluetooth Speed and Cadence Sensor Distance Monitoring Min/ Max Intersection Distance calculation Alerts to User via TTS & Speech-to. Text

Intersection Crossing Example

No errors from crossing • BUT – noise can be an issue with TTS being heard and Speech recognition • SOLUTION: Bluetooth headset

Future • Handle traffic signs • Detect crosswalks for navigation during crossing using vision • Obstacle detection for crossing

WE MUST GO TO CLOUD Speed –too slow Overall framerate = 4. 5 seconds per frame (need 1 frame per second for bike speeds) for Road Following. Runs at. 5 seconds per frame for Intersection light detection –that is sufficient. • BIGGEST computational needs pixel level classification replace with Deep Learning to improve both accuracy and speed.

Other Future work • Recognizing colored bike lanes • Handling obstacles !!!!!

Many problems not considered • Moving smaller obstacles with erratic behavior

Conclusion • Tested mainly with sighted person and one low vision person in constrained/safe environment • Concept: good but NOT ACCURATE ENOUGH for SAFETY • Algorithm / Speed tradeoff not good • Need more features, go to cloud for speed and to do more elaborate model. • Currently using Deep Learning in Seeing Eye Drone technology and i. Sight –next step apply here for road following module. • Current Beta for Tensor. Flow Lite— current development for SSD and NN Hardware Acceleration not yet supported (but near future? )