So nification of the facial expressio ns Guo

  • Slides: 23
Download presentation
So nification of the facial expressio ns Guo Zhiguo Ioulia Guizatdinova Zhiguo. Guo@uta. fi

So nification of the facial expressio ns Guo Zhiguo Ioulia Guizatdinova Zhiguo. Guo@uta. fi ioulia_gizatdinova@pochtamt. ru New Interaction Techniques Department of Computer Science University of Tampere 27 February 2003

Contents • Introduction to the problem • Background & related research • Objectives •

Contents • Introduction to the problem • Background & related research • Objectives • Methods • Resources & experimental setup • Preliminary results • Prospectives p 02_23 27. 02. 03 Sonification of the facial expressions

Introduction to the problem Sonification is the use of non-speech audio to convey information

Introduction to the problem Sonification is the use of non-speech audio to convey information more specifically sonification is the transformation of data relations into perceived relations in an acoustic signal for the purposes of facilitating communication or interpretation [1] p 03_23 27. 02. 03 Sonification of the facial expressions

Introduction to the problem The main question for sonification of the facial expression is:

Introduction to the problem The main question for sonification of the facial expression is: How to convey the emotional expressions/facial features/ of some person presented like any visual imaging into the sounds (e. g. earcons, music chords) or whatever ? p 04_23 27. 02. 03 Sonification of the facial expressions

Introduction to the problem The sonification system for facial expressions should allow the transformation

Introduction to the problem The sonification system for facial expressions should allow the transformation of the visual information (attributes specifically describing emotional expression/state) into sound patterns This kind of sounds (“e. ARmoticons”) should evoke similar emotional feelings of the listener that is, to provide an emotional access through “audio snapshot” when visual perception is blocked or imaging is hidden for observer p 05_23 27. 02. 03 Sonification of the facial expressions

Background & Related Research Some facial expressions can hardly be transformed by sound (e.

Background & Related Research Some facial expressions can hardly be transformed by sound (e. g. , congenital or non-classified features) Blind people have a wide experience with real environmental sounds, while the congenital blinded can have not any association to visual facial imaging; only tactile memory of some facial features can be insufficient or non-adequate p 06_23 27. 02. 03 Sonification of the facial expressions

Background & Related Research There is not signs’ system/auditory or tactile/specifically & briefly describing

Background & Related Research There is not signs’ system/auditory or tactile/specifically & briefly describing facial features/expressions. Only some general emotional states are presented like artificial set of static visual/tactile symbols [2, 11]. Happiness Sadness : ( : ) p 07_23 27. 02. 03 Frustration Madness : / >: -( Sonification of the facial expressions

Background & Related Research There have been a lot of approaches to data sonification

Background & Related Research There have been a lot of approaches to data sonification of like math functions, simple graphics, diagrams, … The v. OICe Java Applet http: //www. visualprosthesis. com/javoice. htm p 08_23 27. 02. 03 Sonification of the facial expressions

Background & Related Research Relevantly, there a lot of earcons used in MS Windows

Background & Related Research Relevantly, there a lot of earcons used in MS Windows and other software, sound signaling for special equipment and application… sound p 09_23 sound 27. 02. 03 sound Sonification of the facial expressions

Objectives The goal of this project is designing and usability investigation of the sonification

Objectives The goal of this project is designing and usability investigation of the sonification system for attributes specifically describing facial expression to provide an emotional access through e. ARmoticons when visual perception is blocked or imaging is hidden for observer p 10_23 27. 02. 03 Sonification of the facial expressions

Schedule Project’s activity Weeks Expected results Overview on the topic Definition of the interaction

Schedule Project’s activity Weeks Expected results Overview on the topic Definition of the interaction scenarios & research methods 1 Sonification & visual attributes Scenarios Design of software prototype Design & collection of sound files 1 Software prototype Sound database Classification of the facial expressions by sound Revision of software 1 Sonification System for facial attributes Revised software Usability exploration of the new interaction techniques 2 Data analysis Report writing 1 2 Project paper p 11_23 27. 02. 03 Sonification of the facial expressions

Research Methods • constructive research to design and simulate the new techniques, software tools

Research Methods • constructive research to design and simulate the new techniques, software tools and doing usability evaluation of novel sound objects – e. ARmoticons • empirical research is based on crossmodal information transfer and perception of intermodal events p 12_23 27. 02. 03 Sonification of the facial expressions

Preliminary Modelling Facial expression Recognition of the sound patterns to facial expression (image, picture,

Preliminary Modelling Facial expression Recognition of the sound patterns to facial expression (image, picture, video, etc) Video processing Facial landmarks Facial expression-to-sound mapping transformation system sounds As a whole Sound Database p 13_23 27. 02. 03 Sonification of the facial expressions

Resources & Experimental Setup • Subjects (people with normal vision) • Stimuli – e.

Resources & Experimental Setup • Subjects (people with normal vision) • Stimuli – e. ARmoticons – earcons specifically & briefly describing facial features/expressions FACS – Facial Action Coding System [ 7, 8] a prototype of the sonification system for facial expression transformation p 14_23 27. 02. 03 Sonification of the facial expressions

Resources & Experimental Setup Experimental procedure p 15_23 27. 02. 03 Sonification of the

Resources & Experimental Setup Experimental procedure p 15_23 27. 02. 03 Sonification of the facial expressions

Resources & Experimental Setup Data analysis: reaction time accuracy = error number p 16_23

Resources & Experimental Setup Data analysis: reaction time accuracy = error number p 16_23 27. 02. 03 Sonification of the facial expressions

Preliminary results Classification of the emotions by Ekman & Friesen (1975) NEUTRAL NEGATIVE POSITIVE

Preliminary results Classification of the emotions by Ekman & Friesen (1975) NEUTRAL NEGATIVE POSITIVE surprise happiness sadness fear disgust anger http: //www. paulekman. com/ p 17_23 27. 02. 03 Sonification of the facial expressions

Preliminary results Facial landmarks. Eye & eyebrow Inner brow raiser Lid tightener Outer brow

Preliminary results Facial landmarks. Eye & eyebrow Inner brow raiser Lid tightener Outer brow raiser Lid droop Brow lowerer Slit Upper lid raiser Eyes closed Cheek raiser Squint p 18_23 27. 02. 03 Sonification of the facial expressions

Preliminary results Facial landmark. Mouth Lip suck Lip corner puller Cheek puffer Lip tightener

Preliminary results Facial landmark. Mouth Lip suck Lip corner puller Cheek puffer Lip tightener Lip pressor Dimpler Lip corner depressor Lips part Lower lip depressor Jaw drop Lip puckerer Mouth stretch Lip stretcher Chin raiser p 19_23 Lip funneler 27. 02. 03 Sonification of the facial expressions

Preliminary results Facial landmark. Nose wrinkler Upperupper lip raiser Nasolabial deepener p 20_23 27.

Preliminary results Facial landmark. Nose wrinkler Upperupper lip raiser Nasolabial deepener p 20_23 27. 02. 03 Sonification of the facial expressions

Prospectives The sonification system of facial expressions which includes new sound mapping techniques to

Prospectives The sonification system of facial expressions which includes new sound mapping techniques to display facial features and expressions p 21_23 27. 02. 03 Sonification of the facial expressions

references 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.

references 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. http: //www. icad. org/website. V 2. 0/Resources/glossary. html#S http: //www. tsbvi. edu/Education/vmi/tactile_symbols. htm#EMOTIONS http: //www. mcsdtechcenter. org/multimedia/sounds-samp. htm http: //www. matthawkins. co. uk/html/sound_fx_index. htm http: //kin-dza. etel. ru/ http: //www. soundobject. org/ http: //viswiz. gmd. de/~eckel/publications/eckel 98 a. html http: //www-2. cs. cmu. edu/afs/cs/project/face/www/facs. htm http: //www. paulekman. com/ http: //www. ifeelpixel. com/download/whatsnew. htm http: //www. symbols. net/emoticons. htm http: //www. fingertek. com/smilies. htm http: //www-unix. mcs. anl. gov/~kaper/Sonification/ http: //www-pablo. cs. uiuc. edu/Project/Pablo. Sonification. htm p 22_23 27. 02. 03 Sonification of the facial expressions

Thank yo u for yo ur attentio n! 27. 02. 03 Sonification of the

Thank yo u for yo ur attentio n! 27. 02. 03 Sonification of the facial expressions

Expressio ns of Politeness and Idiomatic versus LiteralExpressio ns of Politeness and Idiomatic versus Literal
Zhong guo China Chinese fa guo France FrenchZhong guo China Chinese fa guo France French
Facial Expression Recognition Shane Steward Facial Action CodingFacial Expression Recognition Shane Steward Facial Action Coding
MASAJUL FACIAL MANUAL Masajul facial si ce rezultateMASAJUL FACIAL MANUAL Masajul facial si ce rezultate
MUSCLES OF FACIAL EXPRESSION The muscles of facialMUSCLES OF FACIAL EXPRESSION The muscles of facial
LE NERF FACIAL Dr Soussa le nerf facialLE NERF FACIAL Dr Soussa le nerf facial
Best Facial Chicago magazine August 2012 Best facialBest Facial Chicago magazine August 2012 Best facial
Tratamiento facial 5 Sesiones de limpieza facial microdermoabrasionTratamiento facial 5 Sesiones de limpieza facial microdermoabrasion
Complications of Facial Paralysis n Facial paralysis severelyComplications of Facial Paralysis n Facial paralysis severely
FACIAL EXPRESSIONS Why are Facial Expressions Important 90FACIAL EXPRESSIONS Why are Facial Expressions Important 90
Week 6 Facial Bone Nasal Bone Anatomy FacialWeek 6 Facial Bone Nasal Bone Anatomy Facial
PERFILOMETRIA FACIAL PERFILOMETRIA FACIAL Conceptos ES LA PROPORCIONPERFILOMETRIA FACIAL PERFILOMETRIA FACIAL Conceptos ES LA PROPORCION
THE FACIAL NERVE FACIAL NERVE FIBERS Motor toTHE FACIAL NERVE FACIAL NERVE FIBERS Motor to
Facial trauma Facial trauma is any injury ofFacial trauma Facial trauma is any injury of
IPL Facial Fotorejuvenecimiento IPL Facial G Introduccin alIPL Facial Fotorejuvenecimiento IPL Facial G Introduccin al
EL ARCO FACIAL ARCO FACIAL DENTATUS 1 ArcoEL ARCO FACIAL ARCO FACIAL DENTATUS 1 Arco
KECEDERAAN FACIAL A PENGENALAN 1 KETERUKAN KECEDERAAN FACIALKECEDERAAN FACIAL A PENGENALAN 1 KETERUKAN KECEDERAAN FACIAL
NONVERBAL COMMUNICATION 1 Facial Expression Facial expressions areNONVERBAL COMMUNICATION 1 Facial Expression Facial expressions are
Facial Expression 1 Facial Expression Analysis CVPR 0708Facial Expression 1 Facial Expression Analysis CVPR 0708
Facial Anatomy Injuries Facial Anatomy There are 18Facial Anatomy Injuries Facial Anatomy There are 18
Control of facial expressions Domina Petric MD FacialControl of facial expressions Domina Petric MD Facial
Complications of Facial Paralysis n Facial paralysis severelyComplications of Facial Paralysis n Facial paralysis severely