Week 6 Fatemeh Yazdiananari Feature Extraction 75 Validation

Week 6 Fatemeh Yazdiananari

Feature Extraction • 75 Validation Videos • Were sent to the cluster for DTF feature extraction • The extracted features are large in size • Max size: 152 GB • Min size: 540 MB • Total size: 1. 6 TB • Programing • Since the files are large • Textscan, fgetl, dlmread • Ran each feature and saved all the information into a mat file • Used the mat files to run the rest of the codes.

Histogram code • For each video • Obtain 4 feature matrix • Per feature find the closest codebook term and save the index • Using the saved indices run a histogram function • Save : video name, first 10 elements of features, Indices of Tr, Hof, Hog, and Mbh, and histogram of Tr, Hof, Hog, and Mbh.

Over. View 15 Validation Videos • Holistic • Histograms of 15 videos • Binary SVM • Sliding Windows (10 sec long windows) • Histograms of the Sliding window • Binary SVM

Histograms 15 validation videos § Normalized histograms HOF histogram for one video HOG histogram for one video

Histograms (cont. ) MBH histogram for one video TR histogram for one video

Action recognition steps • Binary SVM • Trained using the UCF 101 train splits (split 1) • obtained models • Tested using the 15 validation videos

SVM & Results Added two UCF 101 features §Class 1 and class 2 Ran Binary SVM Classification results (right) § Accurately predicted class 1 and 2 of UCF 101 (11. 76%)

Sliding window steps • Obtain the frame rate of the video (videoreader) • 10 secs worth of frames: multiply to frame rate and get the frame number (window size) • Load the histograms of the 15 videos • Using the frame number we read in the histograms and save them into a structure • After all the videos are divided into their windows and saved we load them • run normalization • binary SVM • Trained on UCF 101 split 1 • Tested on all the windows

Sliding Window Histogram • 15 Validation Videos • Divided into sliding windows per 10 secs • Normalized histograms Hof histogram for one sliding window Hog histogram for one sliding window

Sliding Window Histogram (cont. ) Mbh histogram of one sliding window Tr histogram of one sliding window

SVM & Results Added two UCF 101 features §Class 1 and class 2 Ran Binary SVM Classification results § Classified class 1 and 2 of UCF 101 accurately § The sliding windows were misclassified as class 38

Conclusion • Action recognition using the existing methods on temporally untrimmed videos were done. • The videos are long You. Tube videos (THUMOS’ 14) • The approach based on one Bag-of-words histogram per videos (state-of-the-art) obviously failed as expected. • The Sliding Window approach failed since: • The histograms extracted from the untrimmed clips may not include any particular action; • such histograms happen to be similar to typical histograms of Class 38 misclassification as class 38. • The effect of not having the exact boundaries of the shots/clips have a nontrivial impact on the formed histograms.

Next Steps 1. Visual comparison of the class 38 histograms with the untrimmed histograms a visual similarities should be observed. 2. Extracting the histograms from UCF 101 videos and classifying them using the same code used for the untrimmed videos sanity check. 3. Manually identifying the boundaries of the action in the untrimmed videos and selecting the windows accordingly investigating the impact of unknown boundaries and partial windows.