The FAME Project Acronym Facilitating Agent for Multicultural

  • Slides: 14
Download presentation
The FAME Project • Acronym: Facilitating Agent for Multicultural Exchange • Partners: Universität Karlsruhe,

The FAME Project • Acronym: Facilitating Agent for Multicultural Exchange • Partners: Universität Karlsruhe, UJF Grenoble, UPC Bacelona, ATLAS Barcelona • Project volume: • Duration: • More info: INPG Grenoble, ITC-irst Trento, SONY Europe, Stuttgart 5. 5 M Euro 40 months, started October 2001 http: //www. fame-project. org Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 1

The FAME Projekt Facilitating Agent for Multicultural Exchange • Volume: • Duration: 40 months

The FAME Projekt Facilitating Agent for Multicultural Exchange • Volume: • Duration: 40 months (since October 2001) • Financial Volume: 5, 5 Mio. € • currently approx. 30 scientists www. fame-project. o • Partners: Uni Karlsruhe , INPG Grenoble , UJF Grenoble, UPC Barcelona , ITC-irst Trento , Stuttgart, Barcelona Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 2

Project Goals • long term vision: facilitate communication between humans • reduce the workload

Project Goals • long term vision: facilitate communication between humans • reduce the workload on the users of technical equipment • observe humans and their activities in an intelligent room and serve as a context-aware information butler • FAME project goal: provide and integrate core technologies (video and speech perception, augmented reality, translation, information retrieval) to show feasibility of the concept • demonstrate system at fair • scenario 1 (lecture scenario): one person is giving a talk or lecture or presentation • scenario 2 (meeting scenario): several people are discussing / working on a common task Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 3

The FAME Showcases scenario 1 (presentation) • use A/V equipment • intelligent cameraman •

The FAME Showcases scenario 1 (presentation) • use A/V equipment • intelligent cameraman • presentation tracking • summarisation + archiving • translation, crosslingual IR scenario 2 (meeting) • augmented reality • video-based activity tracking • topic spotting • information butler • service: planning of fair visit Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 4

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) FAME outside view meeting

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) FAME outside view meeting inside people mention topics Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 reception by FAME-guy room gives information about spotted topics 5

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) at the phicon wall

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) at the phicon wall gestures multimodal input on table interactive visit planning output also on the wall Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 the projection table borrow a camera for photographs of the visit 6

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) Back from the visit

The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) Back from the visit in the FAME room: dowload. . . record testimony . . . and look at photos select, print, save photos using phicon interaction intelligent cameraman, presentation tracker take home photos and information about FAME Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 7

Important Components • multimodal environment • context-aware intelligent camera-man automatically track people and their

Important Components • multimodal environment • context-aware intelligent camera-man automatically track people and their activities • augmented reality environment move physical icons (phicons) on table/wall, and interact with projection on table/wall • spontaneous speech recognition (with distant microphones) • translation and crosslingual information retrieval in European-English, Catalan, and Spanish • dialog and context model Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 8

Multimodal Environment at UKA Smartboard as Projection Wall Livingroomg Microphon-Array (Speaker Lokalization) Audio Signals

Multimodal Environment at UKA Smartboard as Projection Wall Livingroomg Microphon-Array (Speaker Lokalization) Audio Signals IR-Remote Control. X-10 Illumination Loudspeakers Microphone Several Beamers 4 Cameras TV/Video Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 9

Augmented Reality Table • project virtual reality on real table • move around physical

Augmented Reality Table • project virtual reality on real table • move around physical icons (multiple users) • interact with projection • select, move, rotate, resize, delete, change color • write on table, pass notes to others, point to items Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 10

Intelligent Camera Man • follow speaker while talking and moving around • detect interaction

Intelligent Camera Man • follow speaker while talking and moving around • detect interaction from audience • zoom on area of interest e. g. when pointing somewhere or showing something Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 11

Lecture Supporter • track lecture or presentation • operate FAME room equipment by speech

Lecture Supporter • track lecture or presentation • operate FAME room equipment by speech commands • automatically switch slides during presentation • automatically create transcript of lecture • create summary, translate to other languages • record and store all lectures in searchable database • retrieve and browse through previously recorded lectures Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 12

Adaptation Overview 60 k vocabulary HUB-4 corpus estimate trigrammmodel P(wn|wn-1, wn 2) 72 classes

Adaptation Overview 60 k vocabulary HUB-4 corpus estimate trigrammmodel P(wn|wn-1, wn 2) 72 classes P(Cn|wn-1, wn -2) · P(wn|Cn) tf-idf most frequent 40 k words least frequent 20 k words add to classes 20% fewer errors wichtige important Wörter words presentationslides ± 2 contexts 100 Links 100 links scores top n CLASS 32 CLASS 14 CLASS 57 CLASS 6 CLASS 70 TO THE RECOGNITION OF THE perplexity. CONTINUOUS SPEECH RECOGNITION IN NOISY AND PATTERN RECOGNITION NOT IN SECURING DUE RECOGNITION AND RES Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 13

Welcome in Barcelona in Summer 2004 Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003

Welcome in Barcelona in Summer 2004 Seminar „Multimodale Räume“ Uni Karlsruhe, 14. 5. 2003 14