University of Southern California Sign Classification using Boosted

University of Southern California Sign Classification using Boosted Cascade of Classifiers Computer Vision group: Thang Dinh thang. dinh@usc. edu Eunyoung Kim eunyoung. kim@usc. edu Li Zhang li. zhang@usc. edu Yuping Lin yuping. lin@usc. edu

Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

Sign classification: Applications & Challenges Applications Robot Control Sign language Sign Classification ……. . Jan 2007 Virtual Reality

Sign classification: Applications & Challenges 1. Detection Rate An interpreting system that interprets wrongly, a robot that always misunderstands commands… can not be employed. 2. Computation Time Robots that need 30 s to understand a command telling them to do something immediately, Systems that need 30 s to interpret each word of speakers… will not be employed either. need a Fast and Robust system Jan 2007

Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

Problem statement Human Gesture: • Dynamic Gesture: requires motion of hand, body… • Static Gesture: static pose of hand, body… • Finger Spelling Jan 2007 (ASL System)

Problem statement Implemented system: Challenges: • Detection rate, Computation time • Many similar signs Jan 2007 ….

Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

Why Boosted Cascade of Classifiers • Cascade of boosted classifiers with Haar wavelet features (Viola and Jones) is currently ‘The state of the art’ in face detection. • Cascade was brought to field of hand detection by Eng-Jon with impressive results. Jan 2007

Why Boosted Cascade of Classifiers Face detection system of Viola and Jones is: • 15 times faster than Rowley’s (double layer Neural Network) • 600 times faster than Schneiderman-Kanade’s (Statistics) Cascade of boosted classifiers seems to be a good approach for the Sign classification problem. Jan 2007

Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

Haar wavelet features • Haar wavelet features (Papageorgiou ) • Feature set was enlarged by Lien Hart to 45º rotated. • We proposed Double-L for Sign detection Jan 2007

Integral Image – Fast calculation of Haar features • There are millions of features that need processing and each feature itself needs time to be calculated makes the training process time-consuming • Viola and Jones proposed Integral Image in order to reduce the feature calculation complexity, which results in less computation time Definition of intergral image at point (x, y) Then: Sum(D) = 1 + 4 – (2 + 3) Jan 2007

Adaboost • Proposed by Freund and Schapire • Combine many ‘weak’ hypothesis to form a ‘strong’ one • ‘weak’ classifier ht only need to be better than chance Jan 2007

Adaboost with Haar wavelet features Advantages: • Adaboost adjusts adaptively the errors of the weak hypotheses • Weak Hypothesis learnt from this algorithm can run very fast because of simple calculation of Haar feature, which speeds up the whole system Disadvantages: • Require large amount of training samples • The training process is rather time-consuming because the algorithm needs to check through millions of features extracted from thousands of samples. Jan 2007

Cascade of classifiers • Cascade of boosted classifiers is a tree of classifiers where classifier lying at each stage is better than the last • Only those input patterns having passed through all the layers are considered objects Simple backgrounds can be easily rejected by one-feature classifier Jan 2007

Sign Classifiers • Each Sign Detectors Di is a cascade of boosted classifier evaluating an input image to give out value vi which is then compared to the threshold of it • 24 Sign Detectors are combined to form a Sign Classifier C = signi | Di = signi and g(Di)=max{g(Dj)|j=1, 2, … 24} Where g(Di) = |vi – i| Jan 2007

Sign Classifiers Jan 2007

Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

Experimental Results Samples Collection • Collect raw image • Find the ‘key’ point (which make it different from others) • Cut and Align the image (base on ‘key’ point) • Aligned images are finally made greyscale Image B - Raw Jan 2007 Image B - Focus Image B - Aligned

Experimental Results Training Process • First stage: • Positive: Thousands of aligned sign images • Negative: Background only – Buildings, paintings, trees, other parts of human body… (also non-sign) Fast eliminate simple background • Second stage: • Positive: also thousands of aligned sign images • Negative: aligned images of other signs Distinguish each sign from others Jan 2007

Experimental Results 1. Detectors • We have trained 24 detectors for 24 signs • Average number of stages: 14 • Total features: approx 123 • Detection rate: 90% - 100% • Test Images were divided into 2 groups • Simple background Jan 2007 • Complex background DR = 100% DR = … FA = 0% FA = …