Fast Time Series Classification Using Numerosity Reduction Xiaopeng

Fast Time Series Classification Using Numerosity Reduction Xiaopeng Xi Eamonn Keogh Christian Shelton Li Wei Chotirat Ann Ratanamahatana xxi@cs. ucr. edu Department of Computer Science and Engineering University of California, Riverside June 28, 2006 ICML 2006, CMU, Pittsburgh 1

Overview n n n Background Motivation Naïve rank reduction Adaptive warping window in DTW Experimental results Conclusions 2

Time Series ECG Heartbeat A B 0 200 400 C 600 800 1000 1200 1400 Stock Flat-tailed Horned Lizard 0 200 400 600 800 1000 1200 1400 3

Time Series Classification n Applications n n Insect species, heart beat, etc 1 -Nearest Neighbor Classifier n Distance measures: Euclidean, LCSS, DTW… 4

Dynamic Time Warping Euclidean warping window r Warping path w Sakoe-Chiba Band Dynamic Time Warping 5

Flat-tailed Horned Lizard Alignment by Dynamic Time Warping Texas Horned Lizard 6

Observation I n 1 -Nearest Neighbor with DTW distance is hard to beat in time series classification 7

Comparison of classification error rate (%) 6 5 4 3 2 1 0 DT W er ep log tro ic Su nn ru pe le eu r- k r a ern ln etw al fu or sio k ns ch M em ult ipl e M ec ult las i-s sif ca ier le his tog ram Decision Tree (4. 9) First Order logic rule (3. 6) Multi layer Grammar-guided perceptron neural feature network (1. 9) extraction Super-kernel fusion (13. 22) scheme (0. 79) Multiple classifier (7. 2) Multi-scale histogram (6. 0) 7 rd 1 NN-DTW (9. 09) to 1 NN-DTW (0. 33) 8 Pe rc 1 NN-DTW (0. 0) Control Chart 1 NN FORTE-2 Fi rs Control Chart error rate (%) Two Patterns 8

Observation I n 1 -NN DTW achieves high accuracy but slow n n n 1 -NN needs O(N 2), N = dataset size DTW is computationally expensive Can we speed 1 -NN DTW up? 9

Observation II n As the data size decreases, larger warping window achieves higher accuracy - Numerosity Reduction C n Q The accuracy peaks very early with small warping window - Accelerate DTW Computation Band width d 10

100 95 100 instances 90 50 instances Accuracy(%) 85 24 instances 80 12 instances 75 70 6 instances 65 60 0 10 20 30 40 50 60 70 80 90 100 Warping Window r(%) Gun-Point 11

DTW gives better accuracy than Euclidean distance, peaks very early with small warping window Dataset DTW(%) Warping Window (%) Euclidean(%) Gun-Point 99. 00 3 94. 50 Trace 100. 00 3 89. 00 Two Patterns 100. 00 3 98. 96 CBF 100. 00 1 97. 67 Control Chart 99. 67 8 92. 50 Face 96. 43 3 94. 64 Leaf 96. 15 8 66. 74 Ecg 90. 00 1 90. 00 Pulse 99. 00 1 99. 00 Lighting(FORTE-2) 90. 91 5 75. 21 Wafer 99. 93 1 99. 90 Word Spotting 80 3 70. 06 Haptic. X 65 15 60 12

Speed Up 1 -NN DTW n Numerosity reduction (data editing) n n Heuristic order searching, prune worst examplar first – Naive Rank Dynamically adjust warping window C Q Band width d 13

Naïve Rank Reduction n n Assign rank to each instance Prune the one with lowest rank Reassign rank for other instances Repeat above steps until stop criterion 14

Naïve Rank Reduction + - P + + + break ties by Rank( P ) = 1 + 1 - 2 = 1 15

Adaptive Warping Window n Basic ideas: n n Adjust warping window dynamically during the numerosity reduction Prune instances one at a time, increase the warping band by 1% if necessary C 100 95 100 instances 90 50 50 instances 85 Q 24 24 instances 80 12 instances 75 70 instances 66 instances 65 60 0 10 20 30 40 50 60 70 80 90 100 Band width d 16

warping window searching (%) 2 3 99% 999 98% 998 97% 98% data pruning 1000 1 97% 4 99% 997 97% 96% 996 95% 96% 995 : : : 94% 5 99% …. . . 15 92% 96% 17

Speeding-up DTW classification n Solution: n n LB_Keogh lower bounding, amortized cost O(n) Store DTW distance matrix and nearest neighbor matrix, update dynamically Compute accuracy by looking up matrices 4 or 5 orders of magnitude faster 18

Experimental Results Two Patterns, on training set DTW Euclidean 100 90 accuracy(%) 80 70 Random, Euclidean 1 -NN Euclidean Random, Fixed 1 -NN DTW, Fixed fix window Naive. Rank 1 -NN DTW, Adaptive adaptive Naive. Rank 60 50 500 600 700 800 4% 400 4% 300 5% 7% 200 8% 100 9% 10% 11% 13% 0 6% 12% 30 14% 40 900 1000 data instances 19

Experiments Results Two Patterns, on test set Swedish Leaf, on test set 90 100 80 70 80 accuracy(%) 90 70 1 -NN Euclidean RT 1 RT 2 RT 3 60 50 800 900 1000 20 0 50 100 3% data instances 700 4% 600 5% 6% 500 4% 400 1 -NN Euclidean RT 1 RT 2 RT 3 1 -NN DTW 30 5% 7% 300 6% 8% 200 9% 100 10% 11% 13% 0 50 40 1 -NN DTW 12% 30 14% 40 60 150 200 250 300 350 400 450 500 data instances RT algorithms are introduced in Wilson, D. R. & Martinez, T. R. (1997). Instance Pruning Techniques. ICML’ 97 20

Conclusions n n n 1 -NN DTW is very competitive in time series classification We show novel observations of relationship between warping window size and dataset size We produce an extremely fast accurate classifier 21

Thank You! 22

x 108 brute force Two Patterns, 1, 000 training / 4, 000 test Fast DTW LB_Keogh Brute force DTW computation (during process) 2 x 103 3 x 108 7 x 108 Process Time (sec) 1 x 101 LB_Keogh fast DTW x 105 1 x 106 brute force 3 x 106 LB_Keogh fast DTW 23