An Infant Facial Expression Recognition System Based on

An Infant Facial Expression Recognition System Based on Moment Feature Extraction C. Y. Fang, H. W. Lin, S. W. Chen Department of Computer Science and Information Engineering National Taiwan Normal University Taipei, Taiwan

Outline n Introduction n System Flowchart n Infant Face Detection n Feature Extraction n Correlation Coefficient Calculation n Infant Facial Expression Classification n Experimental Results n Conclusions and Future Work 1

Introduction n Infants can not protect themselves generally. n Vision-based surveillance systems can be used for infant care. n Warn the baby sitter n Avoid dangerous situations n This paper presents a vision-based infant facial expression recognition system for infant safety surveillance. camera 2

The classes of infant expressions n Five infant facial expressions: n crying, gazing, laughing, yawning and vomiting n Three poses of the infant head: n front, turn left and turn right n Total classes: 15 classes gazing crying front laughing turn left yawning vomiting 3

System Flowchart n Infant face detection: n to remove the noises and to reduce the effects of lights and shadows n to segment the image based on the skin color information n Feature extraction: n to extract three types of moments as features, including Hu moments, R moments, and Zernike moments n Feature correlation calculation: n to calculate the correlation coefficients between two moments of the same type for each 15 -frame sequence n Classification: n to construct the decision trees to classify the infant facial expressions 4

Infant Face Detection n Lighting compensation n To make the skin color detection correctly n Infant face extraction Lighting Step 1: Skin color detection compensation n Using three bands n n n S of HSI Cb of YCr. Cb U of LUX Skin color detection Step 2: Noise reduction n Using 10 x 10 median filter Step 3: Infant face identification n Using temporal information Noise reduction 5

Infant Face Detection n Step 3: Infant face identification 6

Moments n To calculate three types of moments n Hu moment [Hu 1962] n R moment [Liu 2008] n Zernike moment [Zhi 2008] n Given an image I and let f be an image function. The digital (p, q)th moment of I is given by n The central (p, q)th moments of I can be defined as where and n The normalized central moments of I where 7

Hu Moment • Hu moments are translation, scale, and rotation invariant. normalized central moments 8

Example: Hu Moments crying 9

Example: Hu Moments yawning 10

Example: Hu Moments crying yawning • If the infant facial expressions are different then the values of Hu moments are also different. 11

R Moment • Liu (2008) proposed ten R moments which can improve the scale invariability of Hu moments 12

Example: R Moments crying Hu moments • R moments and Hu moments may have different properties. 13

Zernike Moment n Zernike moments of order p with repetition q for an image function f is where real part imaginary part To simplify the index, we use Z 1, Z 2, …, Z 10 to represent Z 80, Z 82, …, Z 99, respectively. 14

Example: Zernike Moments crying 15

Correlation Coefficients • A facial expression is a sequential change of the values of the moments. • The correlation coefficients of two moments may be used to represent the facial expressions. • Let Ai = , i = 1, 2, …, m, indicates the ith moment Ai of the frame Ik, k = 1, 2, …, n. The correlation coefficients between Ai and Aj can be defined as where and : the mean of the elements in Ai 16

Correlation Coefficients • The correlation coefficients between seven Hu moment sequences. yawning H 1 H 2 H 3 H 4 H 5 H 6 H 7 H 1 1 H 2 0. 8778 1 H 3 0. 9481 0. 9474 1 H 4 -0. 033 0. 1887 0. 1410 1 H 5 -0. 571 -0. 4389 -0. 6336 0. 0568 1 H 6 H 7 -0. 8052 0. 8907 -0. 8749 0. 9241 -0. 9044 0. 9719 -0. 3431 0. 2995 0. 7138 -0. 6869 1 -0. 9727 1 17

Decision Tree • Decision trees are used to classify the infant facial expressions. H 1 H 2 H 1 H 3 H 2 H 3 - + + + + + - - - + - - - + correlation coefficients H 1 H 3>0 18

Decision Tree • The correlation coefficients between two attributes Ai and Aj are used to split the training instances. • Let the training instances in S be split into two subsets S 1 and S 2 by the correlation coefficient, then the measure function is • The best correlation coefficient selected by the system is 19

Decision tree construction Step 1: Initially, put all the training instances into the root SR, regard SR as an internal decision node and input SR into a decision node queue. Step 2: Select an internal decision node S from the decision node queue calculate the entropy of node S. If the entropy of node S larger then a threshold Ts, then goto Step 3, else label node S as a leaf node, goto Step 4. Step 3: Find the best correlation coefficient to split the training instances in node S. Split the training instances in S into two nodes S 1 and S 2 by correlation coefficients and add S 1, S 2 into the decision node queue. Goto Step 2. Step 4: If the queue is not empty, then goto Step 2, else stop the algorithm. 20

Experimental Results n n n Training: 59 sequences Testing: 30 sequences Five infant facial expressions: crying, laughing, dazing, yawning, vomiting Three different poses of infant head: front, turn left, and turn right Fifteen classes are classified. crying laughing dazing yawning vomiting Turn left Front Turn right 21

no yes crying yes no no yes vomiting no laughing yes crying no yawning no yes laughing no yes Feature type: Hu moments Internal nodes: 16 Leaf nodes: 17 Height: 8 no no vomiting yes yawning yes no dazing yes dazing no crying no yes no dazing laughing

Experimental Results • The classification results of the Hu-moment decision tree Testing sequences Classification results laughing dazing laughing vomiting 23

yes Feature type: R moments Internal nodes: 15 Leaf nodes: 17 Height: 10 no vomiting yes yes crying no dazing no yawning no yes no laughing no yes vomiting no yes yes dazing no no crying laughing dazing yes crying no yes no no no yes vomiting yes dazing yes crying no laughing 24

Experimental Results • The classification results of the R-moment decision tree Testing sequences Classification results vomiting yawning dazing 25

yes yes no no yes crying no no dazing no no crying yes vomiting no laughing yawning no dazing yes no yes laughing yes crying yes dazing no yawning no no no dazing yes yawning no vomiting yes Feature type: Zernike moments Internal nodes: 19 Leaf nodes: 20 Height: 7 no yes vomiting crying no yes yes laughing yes no laughing no dazing

Experimental Results • The classification results of the Zernike-moment decision tree Testing sequences Classification results crying vomiting crying 27

Conclusions n The comparison of the results Height of the decision tree Number of nodes Number of training sequences Number of testing sequences Classification Rate Hu moments 8 16+17 59 30 90% R moments 10 15+17 59 30 80% Zernike moments 7 19+20 59 30 87% n The correlation coefficients of the moments are useful attributes to classify the infant facial expressions. n The classification tree created by the Hu moments has less height and number of node, but higher classification rate. 28

Conclusions and Future Work n Conclusion n A vision-based infant facial expression recognition system n n Infant face detection Moment features extraction Correlation coefficient calculation Decision tree classification n Future work n n To collect more experimental data To fuzzify the decision tree n n Binary decision trees may have less noise tolerant ability. If the correlation coefficients are close to zero, the noises will greatly affect the classification results. 29

30