Language and Speech Technology Introduction Jan Odijk January

Language and Speech Technology: Introduction Jan Odijk January 2011 LOT Winter School 2011 1

Overview • What is language and speech technology (LST)? (3 -7) • Major Subfields of LST (8 -25) • Characterization of the last 30 years (26 -27) – 80 s (28 -36), 90 s (37 -49), 00 s (50 -56) – Current Status (57 -69) • CLARIN infrastructure (70 -75) • This week’s programme (76) 2

Language Technology • Language Technology is the study of computational systems that process natural language • Alternative names: – Human Language Technology (HLT) – Natural Language Processing (NLP) 3

Speech Technology • Speech Technology is the study of computational systems that process speech • Is a part of Language Technology • Often – Term “Language technology” reserved for the study of computational systems that process written language 4

Computational Linguistics • Computational Linguistics (CL) is the study of language from a computational perspective • Often used interchangeably with language technology • Often grouped under Artificial Intelligence (AI) , although CL predates AI – AI: the study and design of intelligent systems 5

Computational Systems • Computational systems to process natural language do not exist naturally (except in the human brain) – They must be designed, implemented, and evaluated – Therefore it is a kind of engineering 6

Computational Systems • LST is NOT • the study of processing of natural language by humans in – cognition, – (cognitive) psychology, – (psycho)linguistics – phonetics 7

Language Technology Subfields • Orthographic processing – Text = sequence of characters – Tokenization • Text => sequence of tokens • Token= occurrence of a word form • Relatively simple for languages that uses interpunction (space, dot, comma, etc. ) for separating tokens • More difficult for languages such as Chinese, Thai, etc. 8

Language Technology Subfields • Orthographic processing – Orthographic normalization – Token => (token, normalized token) – Normalized token = canonical orthographic representation for a set of orthographic variants – Examples: • • Contemporary spelling variants: aktie => actie Older spelling variants: vleesch => vlees Typos: actei => actie OCR errors: raarn => raam 9

Language Technology Subfields • Morphological processing – Lemmatization: token => (token, lemma) • Lemma = canonical orthographic representation for an inflectional paradigm • Often ambiguities • Examples – lemma(walked) = walk; Lemma(men) = man – Lemma (graven) = {graf, graven} (Dutch) 10

Language Technology Subfields • Morphological processing – Inflection analysis/generation • Word form (lemma, inflectional features) • Examples: – – graven (graf, Po. S=Noun, number=plural) graven (graven, Po. S=Verb, form=infinitive) graven (graven, Po. S=Verb, form= indicative, tense=present, number = plural) 11

Language Technology Subfields • Morphological processing – Compound processing – word form ((word form, affix? )+, word form) – lemma ((word form, affix? )+, lemma) – Example: – Vleeskoeienhouders ([vlees, koeien], houders) ‘meat cow farmers’ – gebiedsbepaling ([(gebied, s)], bepaling) 12

Language Technology Subfields • Morphological processing – Derivational morphology processing – word form (prefix*, lemma, suffix*) – Example: • Characterization ([], characterize, [ation]) 13

Language Technology Subfields • (Po. S-)tagging – Assignment of a grammatical tag to a token in context (tag=label for grammatical properties) – Token => (token, tag) in context – Usually assignment of Po. S-tags – Often more detailed grammatical (inflectional) tags 14

Language Technology Subfields • (Po. S-)tagging – Context: usually: • Some words and/or tags preceding • Some words following – Examples: • (graven, Zij __ een graf) => Vindprespl • (graven, De __ zijn boos) => Npl 15

Language Technology Subfields • Chunking – identifying major phrases in a sentence – Example • The man bought a present for his wife => • [NP The man] bought [NP a present] [PP for his wife] 16

Language Technology Subfields • Parsing – Assign a syntactic structure to a sentence – Example: The man bought a present for his wife => [S [subj/NP The man] [pred/VP bought [obj/NP a present] [pobj/PP for [obj/NP his wife]] ] ] 17

Language Technology Subfields • Machine Translation – Automatic translation of an input text – Example • The man bought a present for his wife => • L’homme a acheté un cadeau pour sa femme 18

Language Technology Subfields • Content extraction and processing – Named entity recognition – Question-answering – Information retrieval – Information extraction – Sentiment/ opinion mining – Reasoning/Inference on semantic representation –… 19

Speech Technology Subfields • Speech Synthesis – Artificial production of human speech – Text => speech – Often called Text-To-Speech (TTS) – TTS system usually contains two components • Grapheme to Phoneme (G 2 P) component – Text => symbolic speech representation (phonetic representation) • Speech Synthesis component – Symbolic speech representation => speech 20

Speech Technology Subfields • Speech Synthesis (cont. ) – Term Speech Synthesis often reserved for this second component – Meaning => speech – Usually called Speech Generation, or Concept. To-Speech, or Data-to-Speech 21

Speech Technology Subfields • Speech Recognition – Recognition of human speech – Audio containing speech => text – Often called automatic speech recognition (ASR) • Speech Understanding – Understanding of human speech – Audio containing speech => meaning or action 22

Speech Technology Subfields • Speaker Recognition – Recognition of a speaker given a speech signal – Speech => person identity • Speaker Verification – Verification of the identity of a person – Speech + claimed identity => Boolean 23

Speech Technology Subfields • Speech Compression – Reduction of the size of speech representations (speech encoding), or – Time-compression of speech representations (so that they sound faster to the listener) 24

Related fields • Speech often used in dialogues – Study of spoken dialogues (human-human, human-machine) • Speech often combined with other modalities – Study of Multimodal Interaction • Speech part of an man-machine interface – Study of Human - Machine Interaction 25

Introduction • Three decades: – “ 80 s”= 1980 -1994 – “ 90 s”= 1990 -2005 – “ 00 s” = 2000 -2011 26

Overview • • • 80 s: Language Technology 80 s: Speech Technology 90 s Language and Speech Technology 90 s Commercial Activity 90 s Importance of Data 00 s Language and Speech Technology 27

80 s: Language Technology • Focus on MT (in Europe) – Eurotra (Europe) – Rosetta (Philips, Netherlands) – Distributed Translation (BSO, Netherlands) 28

80 s: Language Technology • Linguistic “Research Approach” • Focus on Research – not/less on Technology Development • Knowledge-based approach – hand-crafted lexicons and rules – based on a theory / grammatical formalism • Focus on linguistically interesting complex phenomena – less on phenomena that occur often – not strongly data-driven 29

80 s: Language Technology • Focus on an idealized language – not on actual language use – no focus on robustness • Computational approach seen (in research) as a way to gain insight into language, grammar and grammar formalisms – no focus on developing a working system – no pragmatic solutions 30

80 s: Language Technology • Little formal (quantitative) evaluation – only with test suites • constructed sentences illustrating linguistic phenomena • E. g. the HP Test Suite (Flickinger et al. 1987) • computational linguistics rather than language technology 31

80 s: Language Technology Major Problems (from a technology point of view): • Ambiguity – Real – Temporary • Computational Complexity – computation-intensive grammar formalisms • Complexity of language – handcrafting lexicons and rules • requires linguistic and computational expertise • requires a lot of effort and time 32

80 s: Language Technology • Major problems (cont. ): • Idealized Language v. actual Language Use • Require large and rich lexicons, suited to the application domain: difficult/ large effort to make them, and to tune (adapt) to specific domains 33

80 s: Speech Technology • • Automatic Speech Recognition (ASR) Statistical “Engineering Approach” approach based on Noisy Channel Model derive acoustic models from a lot of annotated speech examples • derive statistical language models from large text corpora (n-gram probabilities) 34

80 s: Speech Technology • Focus on making (small) working systems • Statistical approach: system uses probabilities derived from data • Focus initially on limited, “simple” tasks (e. g. digit recognition), and increasingly on more complex tasks 35

80 s: Speech Technology • Focus on real language use under realistic conditions • Progress made by making concrete systems and evaluating them rigorously 36

90 s: Language Technology • Statistical MT – derive language models from monolingual corpora (probabilities of word ( sequence)s – align “sentences” with their translations – derive translation model from parallel corpora: • estimate translation probabilities for words and word sequences from the aligned “sentences” • use these probabilities to compute translations for new “sentences” 37

90: Language Technology • Ambiguity: resolved by probabilities based on statistics • Computational Complexity – computationally feasible formalisms – proven in speech recognition • Complexity of language – language and translation model automatically derived from data • Strong focus on actual language use – Highly data driven • Lexicons can be simpler and are derived automatically from the data; adaptation to specific domains easy once the data are available 38

90 s: Language Technology • Rise of Internet • increasing need for information retrieval • approximated by search for word and word sequence strings • Information Retrieval – strongly statistically based – Limited linguistics – formal evaluation (recall, precision, F-score) 39

90 s: Language Technology • Resulted in – strongly data-driven approach in language technology – increasing use of machine learning techniques – explicit focus on formal, esp. quantative evaluation – re-examination of simpler/computationally less intensive formalisms (finite-state) for syntax 40

90 s: Speech Technology • Continued working under the established paradigm • increasingly improving performance and extending environments and application areas 41

90 s: Companies • many companies active in Speech technology – IBM, Microsoft, Siemens, Nokia, Philips, Motorola, Matra Nortel, . . – Dragon, Kurzweil, Lernout & Hauspie, Speech. Works, Nuance, Babel, Loquendo, Rhetorical, Vocalis, Telisma, Elan, . . . 42

90 s: Companies • many companies in Language technology – IBM, Microsoft, INSO, Novell, . . . – GMS, Apptek, Globalink, Lernout & Hauspie, Systran, LANT (Xplanation), . . . 43

90 s: Companies • MT systems: – knowledge based systems, – developed under an engineering approach • grammatical formalism simple or pruning in search space – to reduce ambiguity – to reduce computational resource requirements – to reduce hand-crafting of rules 44

90 s: Companies • resulted in low quality MT systems – still useful in many circumstances • Differentiating factors – rapid adaptation to (multi-word) terms / vocabulary of new domain – good performance on named entity recognition 45

90 s: Data • Knowledge Based NLP realized cooperation on lexicons was required • ASR Methodology requires a lot of data: – “There is no data like more data” • This led to – Data creation projects – Set-up of data distribution centers – Projects for developing standards for data 46

90 s: Data • Projects – Lexicon projects • • • Multilex, Genelex Acquilex Parole Word. Net, Euro. Word. Net – Speech. Dat projects • Speech. Dat, Speech. Dat-Car, Speech. Dat-East, SPEECON, Orientel – National / Local projects • Spoken Dutch Corpus (Netherlands and Flanders) 47

90 s: Data • Data distribution Centers are set up – LDC (1993) – ELRA (1995) • Standards: – TEI for text corpora • CES, XCES – Eagles, ISLE for grammatical properties 48

Automating Data Production • Usually existing (imperfect) tools are used to create data (semi-)automatically – G 2 P for creating phonetic dictionaries – Po. S-tagging for Po. S-tagged text corpora – Parsers for treebanks • For bootstrapping annotations – Faster and more consistent results • Followed by (partial) manual correction 49

00 s • Early 00 s – Many data and research initiatives, nationally – Netherlands • IMIX 2001 -2008 • STEVIN 2004 -2011 • TST-Centrale (HLT Agency) 2005 -. . – France • EVALDA • Technolangue 50

00 s • Early 00 s – International • • TREC CLEF TC-STAR 2004 -2007 Euro. Matrix 2006 -2009 EUROMATRIXPlus 2009 -2012 ECESS PASCAL / PASCAL 2 ACE 51

00 s • Early 00 s – International • • TAC US DUC US GALE US NTCIR Japan RTE Sem. Eval Sens. Eval 52

00 s • More recent projects • FLa. Re. Net • META-NET 53

00 s • Companies offer services via the internet and via mobile (smart) phones – Search: Google, Bing, Yahoo!, etc. – Social networks: Face. Book, Linked. In, Youtube – Cloud Computing: Amazon, Google, Salesforce • Companies gain access to huge amounts of data (text, pictures, movies, etc, ) including user behavior 54

00 s • Data are used – to improve existing services – To create new services – To personalize services and advertisements 55

00 s • New Services relevant for LST – Google: Translation, search by voice, open platform for mobile devices (Android) – Amazon: Mechanical Turk • Allows large scale distribution of work, e. g. on manual annotation of language resources – Apple: several i. Phone Apps • Dragon Dictate (for SMS, e-mail) • Jibbigo – Re. Captcha: transcription of (hand-written) documents (now part of Google) 56

Current Status • Language and Speech Technology in 2011: – Exciting area! • A lot of commercial activity, and expanding • A large and active research community • A lot of interesting topics are open for research 57

Commercial Activity • many companies in Language technology – Google, Yahoo!, IBM, Microsoft, . . . – Apptek, Linguatec, Systran, Knowledge Concepts, Q-go, . . . • applications – MT, content management, information retrieval, dealing with customer questions, sentiment and opinion mining, . . . 58

Commercial Activity • many companies in Speech technology – Google, IBM, Microsoft, Motorola, Nokia, . . . – Nuance, Loquendo, Acapela, SVOX, Telisma, . . . • even more in application development and system integration 59

Commercial Activity • applications – Network IVR applications (Call centers, banking, information services, . . . ) – Embedded applications • in-car applications, e. g. voice activated dialing, navigation (voice destination entry) • mobile phone/PDA applications – multimodal output e. g. for navigation – command control – (SMS) dictation coming soon 60

Commercial Activity • applications – Office Applications • Dictation, horizontal and vertical (medical, legal) • Language learning – Audiomining • information retrieval from recorded speech (possibly incl. other modalities): Radio/TV-broadcasts, parliamentary sessions, . . . 61

Research Topics? • Speech Technology (Recognition) – new paradigms? • cf. FLAVOR project http: //www. esat. kuleuven. be/psi/spraak/projects/FLa. Vo. R/ – Combination with other modalities • AMI http: //www. amiproject. org • CHIL http: //chil. server. de/servlet/is/101/ • IMIX (Interactive Multimodal Information e. Xtraction) 62

Research Topics? • Speech Technology (Recognition) – robustness against noise and other speakers • increasing use in car and in public places on PDAs and mobile phones • MIDAS project – pronunciation of names • Autonomata I and TOO (incl. Nuance, Ghent, Nijmegen and Utrecht) 63

Research Topics? • Speech technology (Text-to-Speech) – better control over prosody in corpus-based TTS? – Combination with other modalities 64

Research Topics? • Language Technology – Semantic Lexical databases created – Word. Net and Euro. Word. Net – Cornetto 65

Research Topics? • Language Technology – Focus now on Semantic Annotation of Corpora • Onto. Notes http: //www. isi. edu/naturallanguage/people/hovy/papers/06 HLT-NAACLOnto. Notes-short. pdf • STEVIN D-COI and SONAR • Dutch. Sem. Cor – How to use this semantic annotation in practical systems? 66

Research Topics? • Language Technology – (Semi-)automatic lexicon creation/adaptation – Sophisticated information retrieval • Information extraction, summarization and merging, opinion and sentiment mining, 67

Research Topics? • Language And Speech Technology – Speech to Speech Translation • TC-STAR http: //www. tc-star. org/ 68

Research Topics? • Dutch-Flemish STEVIN programme – running from 2004 -2011 – 11. 4 M€ budget • • resources research applications demonstration projects – Most projects finished – some projects are still running – http: //www. taalunieversum. nl/stevin 69

CLARIN • aims to design, construct, validate, and exploit – a research infrastructure that is needed to provide a sustainable and persistent e. Science working environment – for researchers in the Social Sciences & Humanities – who want to make use of language data and tools 70

CLARIN • Make data and tools on different locations easily accessible – via web interfaces and services – CLARIN-portal(s) with intelligent searching, browsing, viewing and querying services) • make it possible for non-technical researchers to extract / combine/ enrich data (supported by dissemination and training) 71

CLARIN • Will make available interoperable data and tools based on existing standards and best practices – Formal interoperability and – Semantic interoperability 72

CLARIN • For researchers that work with language data and tools – Humanities and Social Sciences • • • Linguistics (broadly construed) Literary and Theatrical Studies Media en Culture History Political Sciences … 73

CLARIN • Preparatory Project (CLARIN-prep) – – Funded by EU 2008 -2011 >33 partners from >23 countries Goals • Get commitments from EU countries to contribute to the CLARIN infrastructure after CLARIN-prep • Investigate needs, requirements • Make initial specification (and prototype implementations) 74

CLARIN • Current Status – Most countries in the process – CLARIN infrastructure to start in Mid 2011 – Netherlands committed and has leading role • CLARIN-NL – – Funded by NWO 2009 -2015 Many subprojects running Focus on Humanities 75

This week’s Programme • Tuesday: Parsing • Wednesday: Machine Learning • Thursday: Speech Recognition – Guest lecturer: Arjan van Hessen • Friday: Machine Translation 76

Thanks for Your Attention! 77

References • Flickinger D. , Nerbonne J. , Sag I. , Wasow T. , "Toward Evaluation of NLP Systems", Hewlett-Packard Laboratories, Palo Alto, CA, 1987. 78