A Road Sign Recognition System Based on a

A Road Sign Recognition System Based on a Dynamic Visual Model C. Y. Fang Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. S. Fuh Department of Computer Science and Information Engineering National Taiwan University, Taipei, Taiwan, R. O. C. S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University, Taipei, Taiwan, R. O. C. P. S. Yen Department of Information and Computer Education National Taiwan Normal University, Taipei, Taiwan, R. O. C. National Taiwan University violet@ice. ntnu. edu. tw 1

Outline n n n Introduction Dynamic visual model (DVM) Neural modules Road sign recognition system Experimental Results Conclusions National Taiwan University violet@ice. ntnu. edu. tw 2

Introduction -- DAS n Driver assistance systems (DAS) n The method to improve driving safety n n n Passive methods: seat-belts, airbags, anti-lock braking systems, and so on. Active methods: DAS Driving is a sophisticated process n The better the environmental information a driver receives, the more appropriate his/her expectations will be. National Taiwan University violet@ice. ntnu. edu. tw 3

Introduction -- VDAS n n Vision-based driver assistance systems (VDAS) Advantages: n n n High resolution Rich information Road border detection or lane marking detection Road sign recognition Difficulties of VDAS n n n Weather and illumination Daytime and nighttime Vehicle motion and camera vibration National Taiwan University violet@ice. ntnu. edu. tw 4

Subsystems of VDAS n n n n Road sign recognition system System to detect changes in driving environments System to detect motion of nearby vehicles Lane marking detection Obstacle recognition Drowsy driver detection …… National Taiwan University violet@ice. ntnu. edu. tw 5

Introduction -- DVM n DVM: dynamic visual model n n Two ways to develop a visual model n n n A computational model for visual analysis using video sequence as input data Biological principles Engineering principles Artificial neural networks National Taiwan University violet@ice. ntnu. edu. tw 6

Dynamic Visual Model Video images Data transduction Episodic Memory Sensory component Information acquisition Spatialtemporal information Perceptual component STA neural module Focuses of attention Yes Feature detection No Categorical features CART neural module Conceptual component Category Pattern extraction Patterns CHAM neural module Action National Taiwan University violet@ice. ntnu. edu. tw 7

Human Visual Process Physical stimuli Transducer Data compression Sensory analyzer Low-level feature extraction Perceptual analyzer High-level feature extraction Conceptual analyzer Classification and recognition Class of input stimuli National Taiwan University violet@ice. ntnu. edu. tw 8

Neural Modules n n n Spatial-temporal attention (STA) neural module Configurable adaptive resonance theory (CART) neural module Configurable heteroassociative memory (CHAM) neural module National Taiwan University violet@ice. ntnu. edu. tw 9

STA Neural Network (1) ai Output layer (Attention layer) nk ni Inhibitory connection Excitatory connection ak wij xj nj Input layer National Taiwan University violet@ice. ntnu. edu. tw 10

STA Neural Network (2) n The input to attention neuron ni due to input stimuli x: Gaussian function G Attention layer nk rk wkj ni corresponding neurons nj Input neuron The linking strengths between the input and the attention layers National Taiwan University violet@ice. ntnu. edu. tw 11

STA Neural Network (3) n The input to attention neuron ni due to lateral interaction: Interaction + Lateral distance “Mexican-hat” function of lateral interaction National Taiwan University violet@ice. ntnu. edu. tw 12

STA Neural Network (4) n The net input to attention neuron ni : : a threshold to limit the effects of noise where 1< d <0 National Taiwan University violet@ice. ntnu. edu. tw 13

STA Neural Network (5) stimulus activation t 1 1 p pd The activation of an attention neuron in response to a stimulus. National Taiwan University violet@ice. ntnu. edu. tw 14

ART 2 Neural Network (1) Orienting subsystem Signal generator S Reset ＋ signal Attentional subsystem Category representation field F 2 y Input representation field F 1 ＋ q ＋ r ＋ － ＋ G G ＋ － CART p ＋ ＋ ＋ G － ＋ G ＋ v ＋ u ＋ ＋ x － ＋ w ＋ G ＋ Input vector i National Taiwan University violet@ice. ntnu. edu. tw 15

ART 2 Neural Network (2) n The activities on each of the six sublayers on F 1: where I is an input pattern where the J th node on F 2 is the winner National Taiwan University violet@ice. ntnu. edu. tw 16

ART 2 Neural Network (3) n n Initial weights: n Top-down weights: n Bottom-up weights: Parameters: National Taiwan University violet@ice. ntnu. edu. tw 17

HAM Neural Network (1) v 1 v 2 vi i Excitatory connection CHAM vn Output layer (Competitive layer) wij xj j Input layer National Taiwan University violet@ice. ntnu. edu. tw 18

HAM Neural Network (2) n The input to neuron ni due to input stimuli x: nc: the winner after the competition National Taiwan University violet@ice. ntnu. edu. tw 19

Road Sign Recognition System n Objective n Get information about road n n n Warn drivers Enhance traffic safety Support other subsystems National Taiwan University violet@ice. ntnu. edu. tw 20

Problems n n contrary light side by side shaking occlusion National Taiwan University violet@ice. ntnu. edu. tw 21

Information Acquisition n Color information n n Example: Red color Shape information n Example: Red color edge National Taiwan University violet@ice. ntnu. edu. tw 22

Results of STA Neural Module— Adding Pre-attention National Taiwan University violet@ice. ntnu. edu. tw 23

Locate Road Signs — Connected Component National Taiwan University violet@ice. ntnu. edu. tw 24

Categorical Feature Extraction n Normalization: 50 X 50 pixels Remove the background pixels Features: n n n Red color horizontal projection: 50 elements Green color horizontal projection: 50 elements Blue color horizontal projection: 50 elements Orange color horizontal projection: 50 elements White and black color horizontal projection: 50 elements Total: 250 elements in a feature vector National Taiwan University violet@ice. ntnu. edu. tw 25

Conceptual Component— Classification results of the CART Training Set Test Set National Taiwan University violet@ice. ntnu. edu. tw 26

Conceptual Component— Training and Test Patterns for the CHAM National Taiwan University violet@ice. ntnu. edu. tw 27

Conceptual Component— Training and Test Patterns for the CHAM National Taiwan University violet@ice. ntnu. edu. tw 28

Conceptual Component— Another Training Patterns for the CHAM National Taiwan University violet@ice. ntnu. edu. tw 29

Experimental Results of the CHAM National Taiwan University violet@ice. ntnu. edu. tw 30

Experimental Results National Taiwan University violet@ice. ntnu. edu. tw 31

National Taiwan University violet@ice. ntnu. edu. tw 32

Other Examples National Taiwan University violet@ice. ntnu. edu. tw 33

Discussion n n Vehicle and camcorder vibration Incorrect recognitions Input patterns Recognition results Correct patterns National Taiwan University violet@ice. ntnu. edu. tw 34

Conclusions (1) n Test data: 21 sequences n n Detection rate (CART): 99% Misdetection: 1% (11 frames) Recognition rate (CHAM): 85% of detected road signs Since our system only outputs a result for each input sequence, this ratio is enough for our system to recognize road signs correctly. National Taiwan University violet@ice. ntnu. edu. tw 35

Conclusions (2) n A neural-based dynamic visual model n n Three major components: sensory, perceptual and conceptual component Future Researches n n n Potential applications Improvement of the DVM structure DVM implementation National Taiwan University violet@ice. ntnu. edu. tw 36