CPSC 503 Computational Linguistics Natural Language Generation Lecture

CPSC 503 Computational Linguistics Natural Language Generation Lecture 20 Giuseppe Carenini 12/15/2021 CPSC 503 Spring 2004 1

Un d e r s t a n d i n g Knowledge-Formalisms Map Intended meaning Pragmatics Discourse and Dialogue Semantics Syntax Morphology Ge n e r a t i o n AI planners Logical formalisms (First-Order Logics) Rule systems (features and unification) State Machines Discourse (English) 12/15/2021 CPSC 503 Spring 2004 2

NLG Systems (see handout) • Communicative Goals • Domain Knowledge • Context Knowledge Examples NLG System Text • FOG – Input: numerical data about future. Output: textual wheatear forecasts • IDAS – Input: KB describing a machinery (e. g. , bike), user’s level of expertise Output: hypertext help messages • Model. Explainer – Input: OO model. Output: textual description of information on aspects of the model • STOP – Input: user history and attitudes toward smoking 12/15/2021 CPSC 503 Springcessation 2004 3 Output: personalize smoking letter

GEA: the Generator of Evaluative Arguments 12/15/2021 CPSC 503 Spring 2004 4

Four Basic Types of Arguments • Factual Argument (e. g. , Canada is the only country outside of Asia to record SARS-related deaths) • Causal Argument (e. g. , Travelers from Honk Kong brought SARS to Toronto…. ) • Recommendation (e. g. , You should not go to China in the next few weeks…. . ) • Evaluative Argument (e. g. , Some Asian governments were inefficient in stopping the SARS outbreak…) 12/15/2021 CPSC 503 Spring 2004 5

Sample Textual Evaluative Arguments Single entity House-A is great! Although it is somewhat old, the house is spacious and is in an excellent location. Comparison Vancouver is better than Seattle. There is less crime. Also, social services are more accessible. 12/15/2021 CPSC 503 Spring 2004 6

Evaluative Arguments: Importance Natural Language Generation Theory: model of argument type which is pervasive in natural human communication. Applications: – Advisor, Personal assistants – Recommendation systems – Critiquing systems 12/15/2021 CPSC 503 Spring 2004 7

Limitations of Previous Research [Ardissono and Goy 99] [Chu-Carroll and Carberry 1998] [Elhadad 95] [Kolln 95] [Klein 94] [Morik 89] • Focus on specific aspects of generation – Selection of content – Realization of content into language • Lack of systematic evaluation – proof-of-concept system – analyzed on a few examples 12/15/2021 CPSC 503 Spring 2004 8

Methodology • Develop evaluative argument generator – complete – integrate and extend previous work • Develop evaluation framework • Perform experiment within framework to test generator 12/15/2021 CPSC 503 Spring 2004 9

Outline • Generator of Evaluative Arguments (GEA) • Evaluation Framework • Experiment 12/15/2021 CPSC 503 Spring 2004 10

Text Generator Architecture Content Selection and Organization Communicative Strategies Communicative (User (dis)likes entity degree) Goals Text Planner Knowledge Sources: - User Model - Domain Model Text Plan Content Realization Text Micro-planner Sentence Generator Linguistic Knowledge Sources: - Lexicon - Grammar English 12/15/2021 CPSC 503 Spring 2004 11

GEA User Model Argumentation Theory tells us [Miller 96, Mayberry 96] User Model must – Represent values and preferences of user – Supporting (opposing) evidence depends on values and preferences of audience – Enable identification of supporting and opposing evidence – Evidence arranged according to importance (i. e. , strength of support or opposition) – Concise: only important evidence included 12/15/2021 – Provide measure of evidence importance … and can be elicited in practice. . . CPSC 503 Spring 2004 12

Model of User’s Preferences Additive Multi-attribute Value Function (AMVF) • Decision Theory and Psychology (Consumer’s Behavior) • Can be elicited in practice [Edwards and Barron 1994] User-1 OBJECTIVES Location House Value 0. 7 0. 4 0. 6 0. 3 Amenities 0. 8 0. 2 12/15/2021 COMPONENT VALUE FUNCTIONS Neighborhood Park-Distance Deck-Size Porch-Size CPSC 503 Spring 2004 13

AMVF application User-1 OBJECTIVES 0. 78+ 0. 4 COMPONENT VALUE FUNCTIONS Neighborhood 0. 6 + Location House Value 0. 64+ 0. 7 0. 6 Westend Park-Distance 0. 9+ 0. 3 Amenities 0. 8 _ _ 0. 2 0. 5 km 20 m 2 Deck-Size 0. 32 House-A 0. 25 36 m 2 Porch-Size 0. 6+ + _ 12/15/2021 Likes it Does not like it CPSC 503 Spring 2004 14

Supporting and Opposing Evidence User-1 0. 4 Location + 0. 78 0. 7 + House Value + 0. 64 0. 3 + Neighborhood + 0. 6 + House-A Park-Distance n 2 0. 9+ _ 0. 5 km Amenities 0. 32 0. 8 _ 0. 2 + Deck-Size 0. 25 _ 20 m 2 _ 36 m 2 Porch-Size + 0. 6 + _ Likes it Does not like it o Parent(o) + _ 12/15/2021 relation + _ _ supporting + opposing supporting + _ Supporting Opposing opposing CPSC 503 Spring 2004 15

Measure of Importance [Klein 94] 1 User-1 0. 4 Location 0. 55 0. 7 + House Value + 0. 64 0. 3 + 0. 78 + Neighborhood 0. 24 0. 6 + 0. 5 1 + 0. 6 House-A Park-Distance 0. 54 _ 0 0. 9+ n 2 0. 5 km Amenities 0. 2 0. 32 0. 8 _ 0. 2 + _ Deck-Size 0. 6 0. 25 _ 20 m 2 36 m 2 Porch-Size 0. 12 + _ Likes it Does not like it 12/15/2021 + _ + 0. 6 Supporting Opposing CPSC 503 Spring 2004 16 vo

Why AMVF? - summary An AMVF • Represents user’s values and preferences • Enables identification of supporting and opposing evidence • Provides measure of evidence importance – Evidence arranged according to importance – Concise arguments can be generated • Can be elicited in practice 12/15/2021 CPSC 503 Spring 2004 17

GEA Architecture Content Selection and Organization Communicative Strategies Communicative (User (dis)likes entity degree) Goals Text Planner Knowledge Sources: - User Model AMVF - Domain Model Text Plan Content Realization Text Micro-planner Sentence Generator Linguistic Knowledge Sources: - Lexicon - Grammar English 12/15/2021 CPSC 503 Spring 2004 18

Argumentative Strategy [Carenini and Moore INLG-2000] Based on guidelines from argumentation theory [Miller 96, Mayberry 96] Selection: include only “important” evidence (i. e. , above threshold on z-scores of measure of importance) Organization: (1) Main Claim (e. g. , “This house is interesting”) (2) Opposing evidence (3) Most important supporting evidence (4) Further supporting evidence -- ordered by importance with strongest last Strategy applied recursively on supporting evidence 12/15/2021 CPSC 503 Spring 2004 19

Sample GEA Text Plan EVALUATIVE ARGUMENT MAIN-CLAIM SUPPORTING EVIDENCE (VALUE (House-A) 0. 72) SUB-CLAIM OPPOSING EVIDENCE SUPPORTING EVIDENCE (VALUE (Location) 0. 7) (VALUE (distance-from-park 1. 8 m) 0. 3) decomposition 12/15/2021 ordering (VALUE (distance-from-work 1 mi) (distance-from-rap-trans 0. 5 mi) 0. 75) rhetorical relations CPSC 503 Spring 2004 20

GEA Architecture Content Selection and Organization Argumentative Communicative Strategy Strategies Communicative (User (dis)likes entity degree) Goals Text Planner Knowledge Sources: - User Model AMVF - Domain Model Text Plan Content Realization Text Micro-planner Sentence Generator Linguistic Knowledge Sources: - Lexicon - Grammar English 12/15/2021 CPSC 503 Spring 2004 21

Text Micro-Planner • Aggregation: combining multiple propositions in one single sentence [Shaw 98] • Scalar Adjectives (e. g. , nice, far, convenient) [Elhadad 93] • Discourse cues (e. g. , although, because, in fact) [Knott 96; Di Eugenio, Moore and Paolucci 97] • Pronominalization: deciding whether to use a pronoun to refer to an entity (centering [Grosz, Joshi and Weinstein 95]) 12/15/2021 CPSC 503 Spring 2004 22

Aggregation (Logical Forms) • Conjunction via shared participants “House B-11 is far from a shopping area” + “House B-11 is far from public transportation” = “House B-11 is far from a shopping area and public transportation”. • Syntactic embedding “House B-11 offers a nice view” + “House B-11 offers a view on the river” = “House B-11 offers a nice view on the river”. 12/15/2021 CPSC 503 Spring 2004 23

Scalar Adjectives Selection Value > 0. 8 The house has an excellent location 0. 65 < Value < 0. 8 …aaconvenient… … 0. 5 < Value < 0. 65 …a reasonable … 0. 35 < Value < 0. 5 …an anaverage… HOUSE-LOCATION HAS_PARK_DISTANCE HAS_COMMUTING_DISTANCE 0. 2 < Value < 0. 35 Value < 0. 2 …aabad bad… … …aaterrible… … HAS_SHOPPING_DISTANCE HOUSE-AMENITIES. . . 12/15/2021 CPSC 503 Spring 2004 24

Discourse Cues Selection Rel-type Type-ofnesting CONCESSION ROOT Typed-ordering ("CORE" "CONCESSION" "EVIDENCE") or …. Discourse cue Although (placed on contributor) EVIDENCE ("CORE" "CONCESSION" "EVIDENCE") EVIDENCE Even though (placed on contributor) SEQUENCE 12/15/2021 CPSC 503 Spring 2004 25

Pronominalization Centering tells us: entity providing link preferentially realized as pronoun (within a discourse segment) • Successive references always pronoun • First reference in segment pronoun only if both conditions hold: – Segment boundary explicitly marked by discourse cue – No pronoun was used in previous sentence 12/15/2021 CPSC 503 Spring 2004 26

Output of Micro. Planning Lexicalized Functional Descriptions (LFDs) Example: “House-B 11 is close to shops and reasonably close to work” ((CAT CLAUSE) (PROCESS ((TYPE ASCRIPTIVE) (MODE ATTRIBUTIVE)((POLARITY POSITIVE(EPISTEMIC-MODALITY NONE))) (PARTICIPANTS ((CARRIER ((CAT NP)(COMPLEX APPOSITION) (RESTRICTIVE YES) (DISTINCT ((AND ((CAT COMMON)(DENOTATION ZERO-ARTICLE-THING)(HEAD ((LEX " house")))) ((CAT PROPER) (LEX " B-11")))(CDR NONE)))) (ATTRIBUTE (AND((CAT AP)(HEAD ((CAT ADJ)(LEX " close"))) (QUALIFIER ((CAT PP) (PREP ((CAT PREP) (LEX "to"))) (NP((CAT COMMON) (NUMBER PLURAL)(DEFINITE NO) (HEAD ((CAT NOUN) (LEX " shop"))))) ((CAT AP)(HEAD ((CAT ADJ)(LEX " reasonably close"))) (QUALIFIER ((CAT PP) (PREP ((CAT PREP) (LEX " to"))) (NP ((CAT COMMON)(DEFINITE NO) (HEAD ((CAT NOUN)(LEX " work"))))))) 12/15/2021 CPSC 503 Spring 2004 27

Last Step: Sentence Generator • Unify LFDs with large grammar of English (FUF/SURGE [Elhadad 93, Robin 94]) – fill in syntactic constraints (e. g. , agreement, ordering) – choose closed class words (e. g. , prepositions, articles) • Apply morphology • Linearize as English sentences 12/15/2021 CPSC 503 Spring 2004 28

GEA Highlights • GEA implements a computational model of generating evaluative arguments • All aspects covered in a principled way: – argumentation theory – decision theory – computational linguistics 12/15/2021 CPSC 503 Spring 2004 29

Outline • Generator of Evaluative Arguments (GEA) • Evaluation Framework • Experiment 12/15/2021 CPSC 503 Spring 2004 30

Evaluation Framework: [Carenini INLG-2000] Task Efficacy Hot List Subtask 1 User presented with info about set of alternatives 1 st best - Select preferred N alternatives - Order them by preference 2 nd best …. . nth best Subtask 2 User Model has been elicited Hot List Where? YES New. Instance is created 12/15/2021 User presented with Evaluative argument about New. Instance CPSC 503 Spring 2004 1 st best 2 nd best. . . Include? nth best NO End Fill-out final 31 questionnaire

Selection Task in Real-Estate • Why Real-Estate? – No background or expertise – But still presents challenging decision task • Instructions – Move to new town – Buy house – Use system for data exploration 12/15/2021 CPSC 503 Spring 2004 32

Data Exploration System 2 -13 12/15/2021 CPSC 503 Spring 2004 33

Argument is presented… 2 -13 12/15/2021 CPSC 503 Spring 2004 34

Measures of Effectiveness • Behavior and Attitude change SAMPLE SELF-REPORT – Record of user actions • Whether or not adopts new instance • Position in Hotthe Listnew house? How would you judge Satisfaction Z-score – Final Questionnaire The more • you thelikes house closer Howlike much newthe instance you should put amuch cross to “good choice”in Hot • How likes the instances List X : ___: good choice bad choice : ___ : ___ • Others (Final questionnaire) – Decision Confidence – Decision Rationale 12/15/2021 CPSC 503 Spring 2004 35

Outline • Generator of Evaluative Arguments (GEA) • Evaluation Framework • Experiment 12/15/2021 CPSC 503 Spring 2004 36

Two Empirical Questions [Carenini and Moore IJCAI-2001, ACL-2000] • Argument content, structure and phrasing tailored to user-specific AMVF, but. . . Does this tailoring actually contribute to argument effectiveness? • Arguments should be concise. Conciseness can be varied, but…. What is the optimal level of conciseness? 12/15/2021 CPSC 503 Spring 2004 37

Experimental Conditions • Tailored-Concise (~ 50% of objectives) • Tailored-Verbose (~ 80% of objectives) • Non-Tailored-Concise (~ 50% of objectives) • No-Argument 12/15/2021 CPSC 503 Spring 2004 38

Experimental Hypotheses Tailored-Verbose > Tailored-Concise ? ? > Non-Tailored-Concise ? > No-Argument 12/15/2021 CPSC 503 Spring 2004 39

Experimental Procedure 40 subjects (10 for each condition) PHASE 1 Online questionnaire to acquire preferences (AMVF - 19 objectives, 3 layers) [Edwards and Barron 1994] PHASE 2 - randomly assigned to condition -interacts with evaluation framework 12/15/2021 - fill-out questionnaire CPSC 503 Spring 2004 40

Experiment Results Satisfaction Z-score Decision Confidence Decision Rationale 12/15/2021 CPSC 503 Spring 2004 41

Results Satisfaction Z-score 0. 05 p=0. 02 e. s. =0. 8 Tailored-Verbose > Tailored-Concise 0. 88 > > p=0. 03 e. s. =0. 8 ? 0. 31 p=0. 04 e. s. =0. 9 ? 0. 18 Non-Tailored-Concise ? 0. 33 0. 31 No-Argument 0. 25 12/15/2021 CPSC 503 Spring 2004 42

Summary Generator of Evaluative Argument (GEA): generates concise arguments tailored to a model of the user’s preferences (AMVF) Evaluation Framework – Basic decision tasks – Evaluate wide range of generation techniques Experiment – Tailoring to AMVF is effective – Differences in conciseness influence effectiveness 12/15/2021 CPSC 503 Spring 2004 43

Future Work (in 2001) Argument Generator – More Complex Textual Arguments – Speech AT&T MATCH system – Other domains – Other languages – Arguments combining text and graphics More Experiments to test all extensions 12/15/2021 CPSC 503 Spring 2004 44

Multimodal Access to City Help (MATCH) (AT&T Johnston, Ehlen, Bangalore, Walker, Stent, Maloor and Whittaker 2002) Multimodal interface • Portable Fujitsu tablet • Input: Pen for deictic gestures and Speech input • Output: Text, Speech and graphics 12/15/2021 CPSC 503 Spring 2004 45

MATCH Example: User: “Show me Italian restaurants in the West Village” User: “Recommend/Compare” MATCH generates responses using techniques inspired by GEA • Evaluation (Lab - argument quality judgments) – Users prefer tailored responses • Future: Field Study 12/15/2021 CPSC 503 Spring 2004 46

Next Time (Wed 8: 30 sharp!) Project update - 5 min presentation in class • Brief description of the research problem you are targeting. • Describe your original research plan • Describe/Justify any change to your original plan • Describe what part of your (new) plan you have: – completed, – currently working – left to be done • For the part of the plan you still have to work on give an estimate of how much time each step will take. • Any other info you feel appropriate. . . 12/15/2021 CPSC 503 Spring 2004 47