Perceptually Consistent Examplebased Human Motion Retrieval Zhigang Deng

Perceptually Consistent Example-based Human Motion Retrieval Zhigang Deng*, Qin Gu, Qing Li University of Houston

Introduction l l Popularization of human motion capture data in animation and gaming applications Efficient retrieval of similar motions from a large data repository l Fundamental basis for many motion data based applications e. g. CMU motion capture library. http: //mocap. cs. cmu. edu. 2605 trials in 6 categories and 23 subcategories.

Related Work – Motion retrieval l Transform original high-dimensional human motion data to a reduced representation [Agrawal et al. 1993; Faloutsos et al. 1994; Chan and Fu 1999; Liu et al. 2003; Chiu et al. 2004; Baciu 2006; Lin 2006]. l Match webs [Kovar and Gleicher 2004] l l Describe potential subsequence matches between any pair of motion sequences. Semantics-based motion retrieval [Muller et al. 2005; Muller and Roder 2006] l Users provide a query motion as a set of time-varying geometric feature relationships.

Our Approach Pipeline Motion Data Preprocessing l Human hierarchy construction l Motion segmentation and normalization l Motion pattern detection and indexing Runtime Motion Query l Hierarchical pattern matching l Search result ranking

Data Preprocessing - Motion Hierarchy Construction l Decompose human motion into a hierarchical structure [Gu et al. 08] l l l Local control granularity Correlations among different human parts are embedded in different layers 4 layers, 18 parts are used in this work.

Data Preprocessing - Motion Segmentation and Normalization l l Existing human motion segmentation techniques l Angular acceleration [Zhao 01, Fod et al. 02, Kim et al. 03], SVM classifier [Li et al. 07], weighted sum of marker velocities [Gu et al. 08], PCA/PPCA [Barbic et al. 04]. Probabilistic PCA [Barbic et al. 04] is used to segment motion into short motion segments for each body part in the hierarchy. Parts Head LHand LArm RArm Ave Frm + Var 18. 32 + 2. 32 8. 43 + 6. 43 11. 39 + 6. 54 12. 75 + 7. 34 Parts Torso RLeg LFoot RFoot Ave Frm + Var 13. 43 + 5. 65 11. 24 + 5. 12 6. 75 + 5. 35 6. 05 + 5. 88 Average Frame Information of segments

Data Preprocessing - Clustering l Motion Pattern for each body part l l l Adaptive K-Means clustering l l A representative motion segment for a node (i. e. , a body part) in the constructed human hierarchy Normalization of motion segments Increase K when the clustering error metric is larger than a threshold Resulting data structures l (1) Motion Pattern Library, (2) Pattern Index Lists, (3) Pattern Dissimilarity Maps.

Review of Motion Preprocessing

Runtime Motion Query l Query motion transformation l l l Map the query motion into a motion pattern index list for each hierarchy node Fast (no clustering, just database matching) Motion similarity score computing l l Local motion similarity between two index lists l Extended Knuth-Morris-Pratt (KMP) string matching algorithm [Knuth et al. 77] Global motion similarity computing and ranking l Hierarchical propagation

Local Motion Similarity l Similarity between two pattern index lists l l l Different length of index lists Matching of two integer lists Extended KMP String match algorithm l l l Introducing “Quasi-Match” based on the pre-constructed pattern dissimilarity maps l Large numbers of different motion segments l Distance is less than a threshold Update matching score l If the number of consecutive quasi-matches is larger than a threshold, otherwise decrease. . Score normalization based on the length of index lists

Global Motion Similarity l Hierarchical Score Propagation l l High local motion similarity does not mean global motion similarity Nodes in the upper levels encode more global motion information From bottom to top Ranking of the final scores at the root node

Review of Runtime Motion Retrieval

Results and Evaluation l l Time and Storage Search Accuracy Search Quality Perceptual Consistency Experiment

Results and Evaluations – Time and Storage l l l We tested our method on four datasets with different sizes The test computer with a Intel Duo Core 2 GHz CPU and 2 GB memory. The average duration of used query motions is 10 seconds. 56 MB, 170 motions, 68, 293 frames 456 MB, 396 motions, 556, 097 frames 976 MB, 542 motions, 1, 190, 243 frames 1452 MB, 941 motions, 1, 770, 731 frames

Results and Evaluations – Search Accuracy l Accuracy evaluation scheme [Kovar and Gleicher 04] l Two different types of datasets: single-type motion datasets (prelabeled dataset with the same semantic category, walking) – Ground truth, mixed motion dataset (unlabeled, mixed of various types). l True-positive accuracy ratio is defined top N (=20) results from mixed motion datasets are in the correct/expected single-type motion dataset. l 56 M test dataset: 170 sequences, 68, 293 frames, five categories – walking, running, jumping, kicking, basket-playing.

Results and Evaluation – Comparative User Studies l Compare our approach with match-webs approach [Kovar and Gleicher 04], piecewise linear space [Liu et al. 05], weighted PCA [Forbes and Fiume 05]. l l Semantic-based motion retrieval [Muller et al. 05] was not chosen, because of significant differences in input requirements. Two usability questions l l (a) Perceptual Consistency: Retrieved results (motions) are ranked in a perceptually consistent order? (b) Search Quality: Motion similarities of retrieved results?

Results and Evaluation – Comparative User Studies l l Perceptual-consistency l Computer algorithms rank motions in a certain order, C. l Humans rank these (the same) motions in another order, H. l Relationship/consistency between C and H? Study Experiments l 3 query motions (walking, running, basketball-playing), Top-ranked N (=6) results for query, 4 approaches, total 72 = 3*6*4 results. l Side-by-side comparison and user rating (one is a searched motion, the other is the query motion), in a random order. l Rating is from 1 (“completely different”) to 10 (“identical”). l 24 experiment participants

Results and Evaluation – Comparative User Studies l Quality of searched motions l l Compute average similar ratings and standard deviation Higher the average similar rating is, the better quality of search it achieves.

Results and Evaluation – Comparative User Studies l Perceptual-Consistency l l l Plot human-rankings vs computerrankings in a 2 D space. Ideal consistency is shown as a straight line. Walking Canonical Correlation Analysis l Scale-invariant optimum linear framework CCA Coefficient results Running Basketball-playing

Review of User Studies

Conclusions l l An efficient, example-based human motion retrieval technique Major distinctions of our approach l Efficiency l l Flexible search query l l Linear to the size of query motion and database size A human motion subsequence, or a hybrid of multiple motion sequences Perceptually consistent search outcomes l Comparative user studies to find out the correlations between the resultranking by computer algorithms and the result-ranking by humans

Discussion and Limitations l l Current approach does not consider the path/motion trajectory of the root of the human in the retrieval algorithm. The search results may enclose different paths/trajectories. Current approach can only search for singlecharacter motion sequences.

Future Work l A number of empirical parameters of current approach may critically affect the search accuracy and outcomes. l l Establish quantitative correlations between “parameter setting” and “search accuracy and outcomes”. Graphics hardware accelerated, motion query processing.

Thank You! Questions?