Chapter 10 Natural Language Processing Xiujun GONG Ph

Chapter 10 Natural Language Processing Xiu-jun GONG (Ph. D) School of Computer Science and Technology, Tianjin University gongxj@tju. edu. cn http: //cs. tju. edu. cn/faculties/gongxj/course/ai/

Turing's test p Is a machine is intelligent or not? If a machine could carry on a conversation (over a teletype) that was indistinguishable from a conversation with a human being, then the machine could be called "intelligent. " Understand the language is the essential step for conservation

Language p We can define a language formally as: n n p a set of symbols (“alphabet”) a set of rules to combine those symbols This mathematical definition covers many classes of languages, not just human language.

Java: An artificial (formal) language p fixed set of basic symbols: n p public, static, for, while, {, }… fixed syntax for symbol combination public static void main (String[] args) { for(int i = 0; i < args. length; i++) { … } }

Natural language p Often much more complicated than an artificial language. n p It does conform to the formal definition: n n p Some people view NL as a special kind of formal language as well. there are symbols there are modes of combination However, there are many levels at which these symbols and rules are defined.

Levels of analysis in Natural language (1) p Acoustic properties (phonetics: 语音学) n n p defines a basic set of sounds in terms of their features studies the combination of these phonemes Higher-order acoustic features (phonology: 语 音体系) n how combinations of phonemes combine into larger units, with supra-segmental features such as intonation.

Levels of analysis in Natural language (2) p Word formation (morphology: 形态学) n p Combination into longer units in a structuredependent way (syntax) n n p “legal” word combinations in a language recursive phrasal combination Interpretation (semantics): n n p combines morphemes into words of words (lexical semantics) of longer units (sentential/propositional semantics) Interpretation in context (pragmatics: 语用论)

Natural Language Processing p Studies language at all its levels. n n n p phonology, morphology, syntax, semantics… focuses on process computational methods to understand generate human language Often, NLP is called computational linguistics

Task Language Modeling p Machine Translation p Information Extraction/Retrieval p Named entity recognition p Automatic summarization p Speech recognition p Question Answering p

Tools for NLP p Open. NLP n p Stanford NLP n p http: //nlp. stanford. edu/links/statnlp. html NLP Toolsuite n p http: //opennlp. sourceforge. net/ http: //www. julielab. de/Resources/Software/NL P_Tools. html Resource maintained by Kita n http: //www-a 2 k. is. tokushimau. ac. jp/member/kita/NLP/nlp_tools. html