Automatic Text Summarization A Solid Base Martijn B

Automatic Text Summarization: A Solid Base Martijn B. Wieling, Rijksuniversiteit Groningen November, 25 th 2004

Outline • • • Why should we bother at all? (a. k. a. Introduction) A frequency based ATS [Luhn, 1958] An ATS based on multiple features [Edmundson, 1969] Automatically combining the features (1) [Kupiec et al, 1995] Automatically combining the features (2) [Teufel & Moens, 1997] Why should we still bother? (a. k. a. Conclusion) 0000001 ATS: A Solid Base

Why should we bother at all? • Time saving • Large scale application possible, e. g. – ‘Google-xtract’ – Extract translation • Abstracts will be consistent and objective 0000010 ATS: A Solid Base

And in the beginning there was … • Hans Peter Luhn (“father of Information Retrieval”): The Automatic Creation of Literature Abstracts - 1958 0000011 Image: Courtesy IBM ATS: A Solid Base

Luhn’s method: basic idea • Target documents: technical literature • The method is based on the following assumptions: – Frequency of word occurrence in an article is a useful measurement of word significance – Relative position of these significant words within a sentence is also a useful measurement of word significance • Based on limited capabilities of machines (IBM 704) no semantic information 0000100 IBM 704 - Courtesy IBM ATS: A Solid Base

Why word frequency? • Important words are repeated throughout the text – examples are given in favor of a certain principle – arguments are given for a certain principle – Technical literature one word: one notion • Simple and straightforward algorithm cheap to implement (processing time is costly) – Note that different forms of the same word are counted as the same word 0000101 ATS: A Solid Base

When significant? • Too low frequent words are not significant • Too high frequent words are also not significant (e. g. “the”, “and”) • Removing low frequent words is easy – set a minimum frequency-threshold • Removing common (high frequent) words: – Setting a maximum frequency threshold (statistically obtained) – Comparing to a common-word list 0000110 Figure 1 from [Luhn, 1958] ATS: A Solid Base

Using relative position • Where greatest number of high-frequent words are found closest together probability very high that representative information is given • Based on the characteristic that an explanation of a certain idea is represented by words closely together (e. g. sentences – paragraphs - chapters) 0000111 ATS: A Solid Base

The significance factor • The “significance factor” of a sentence reflects the number of occurrences of significant words within a sentence and the linear distance between them due to non-significant words in between • Only consider portion of sentence bracketed by significant words with maximum of 5 non-significant words in between, e. g. “ (*) - - - [ * - * * - - * ] - - (*) “ • Significance factor formula: (Σ[*])2 / |[. ]| (2. 5 in the above example) 0001000 ATS: A Solid Base

Generating the abstract • For every sentence the significance factor is calculated • The sentences with a significance factor higher than a certain cut-off value are returned (alternatively the N highest-valued sentences can be returned) • For large texts, it can also be applied to subdivisions of the text • No evaluation of the results present in the journal paper! 0001001 ATS: A Solid Base

A new method by Edmundson • H. P. Edmundson: New methods in Automatic Extracting - 1969 IBM 7090 - Courtesy IBM 0001010 ATS: A Solid Base

Four methods for weighting • Weighting methods: – – Cue Method Key Method Title Method Location Method • The weight of a sentence is a linear combination of the weights obtained with the above four methods • The highest weighing sentences are included in the abstract • Target documents: technical literature 0001011 ATS: A Solid Base

Cue Method • Based on the hypothesis that the probable relevance of a sentence is affected by presence of pragmatic words (e. g. “Significant”, “Greatest”, Impossible”, “Hardly”) • Three types of Cue words: – Bonus words: positively affecting the relevance of a sentence (e. g. “Significant”, “Greatest”) – Stigma words: negatively affecting the relevance of a sentence (e. g. “Impossible”, “Hardly”) – Null words: irrelevant 0001100 ATS: A Solid Base

Obtaining Cue words • The lists were obtained by statistical analyses of 100 documents: – Dispersion (λ): number of documents in which the word occurred – Selection ratio (η): ratio of number of occurrences in extractor-selected sentences to number of occurrences in all sentences • Bonus words: η > thighη • Stigma words: η < tlowη • Null words: λ > tλ and tlowη< η < thighη 0001101 ATS: A Solid Base

Resulting Cue lists • Bonus list (783): comparatives, superlatives, adverbs of conclusion, value terms, etc. • Stigma list (73): anaphoric expressions, belittling expressions, etc. • Null list (139): ordinals, cardinals, the verb “to be”, prepositions, pronouns, etc. 0001110 ATS: A Solid Base

Cue weight of sentence • Tag all Bonus words with weight b > 0, all Stigma words with weight s < 0, all Null words with weight n = 0 • Cue weight of sentence: Σ (Cue weight of each word in sentence) 0001111 ATS: A Solid Base

Key Method • Principle based on [Luhn], counting the frequency of words. • Algorithm differs: – Create key glossary of all non-Cue words in the document which have a frequency larger than a certain threshold – Weight of each key word in the key glossary is set to the frequency it occurs in the document – Assign key weight to each word which can be found in the key glossary – If word is not in key glossary, key weight: 0 – No relative position is used ([Luhn]) • Key weight of sentence: Σ (Key weight of each word in sentence) 0010000 ATS: A Solid Base

Title Method • Based on the hypothesis that an author conceives title as circumscribing the subject matter of the document (similarly for headings vs. paragraphs) • Create title glossary consisting of all non-Null words in the title, subtitle and headings of the document • Words are given a positive title weight if they appear in this glossary • Title words are given a larger weight than heading words • Title weight of sentence: Σ (Title weight of each word in sentence) 0010001 ATS: A Solid Base

Location Method • Based on the hypothesis that: – Sentences occurring under certain headings are positively relevant – Topic sentences tend to occur very early or very late in a document and its paragraphs • Global idea: – Give each sentence below his heading the same weight as the heading itself (note that this is independent from the Title Method) – Heading weight – Give each sentence a certain weight based on its position - Ordinal weight – Location weight of sentence: Ordinal weight of sentence + Heading weight of sentence 0010010 ATS: A Solid Base

Location Method: Heading weight • Compare each word in a heading with the pre-stored Heading dictionary • If the word occurs in this dictionary, assign it a weight equal to the weight it has in the dictionary • Heading weight of a heading: Σ (heading weight of each word in heading) • Heading weight of a sentence = Heading weight of its heading 0010011 ATS: A Solid Base

Creating the Heading dictionary • The Heading dictionary was created by listing all words in the headings of 120 documents and calculating the selection ratio for each word: – Selection ratio (η): ratio of number of occurrences in extractor-selected sentences to number of occurrences in all headings • Deletions from this list were made on the basis of low frequency and unrelatedness to the desired information types (subject, purpose, conclusion, etc. ) • Weights were given to the words in the Heading dictionary proportional to the selection ratio • The resulting Heading dictionary contained 90 words 0010100 ATS: A Solid Base

Location Method: Ordinal weight • • Sentences of the first paragraph are tagged with weight O 1 Sentences of the last paragraph are tagged with weight O 2 The first sentence of a paragraph is tagged with weight O 3 The last sentence of a paragraph is tagged with weight O 4 • Ordinal weight of sentence: O 1 + O 2 + O 3 + O 4 0010101 ATS: A Solid Base

Generating the abstract • Calculate the weight of a sentence: a. C + b. K + c. T + d. L, with a, b, c, d constant positive integers, C: Cue Weight, K: Key weight, T: Title weight, L: Location weight • The values of a, b, c and d were obtained by manually comparing the generated automatic abstracts with the desired (human made) abstract • Return the highest N sentences under their proper headings as the abstract (including title) – N is calculated by taking a percentage of the size of the original documents, in this journal paper 25% is used 0010110 ATS: A Solid Base

Which combination is best? • • • All combinations of C, K, T and L were tried to see which result had (on average) the most overlap with the handmade extract As can be seen in the figure below (only the interesting results are shown), the Key method was omitted and only C, T and L are used to create the best abstract Surprising result! (Luhn used only keywords to create the abstract) Figure 4 from [Edmundson, 1969] 0010111 ATS: A Solid Base

Evaluation • Evaluation was done on unseen data (40 technical documents), comparison with handmade abstracts – Result: 44% of the sentences co-selected, 66% similarity between abstracts (human judge) – Random ‘abstract’: 25% of the sentences co-selected, 34% similarity between abstracts • Another evaluation criterion: ‘extract-worthiness’ – Result: 84% of the sentences selected is extract-worthy – Therefore: for one document many possible abstracts (differing in length and content) 0011000 ATS: A Solid Base

Comments • [Goldstein e. a. , 1999]: Not good to base length of abstract on length of document – Summary length is independent of document length – The longer the document, the smaller the compression ratio ( |doc. | / |abstract| ) – Better to use constant summary length • [Rath e. a. , 1961] Human selection of sentences in abstracts is very variable – 6 abstracts of 20 sentences: only 32% overlap between 5 subjects (6: 8%) – Abstracting the same document 2 times by the same person with 8 weeks in between: only 55% overlap (average for 6 subjects) • Perhaps the Key Method algorithm used here is not that good (Luhn’s algorithm could be better) 0011001 ATS: A Solid Base

Time and cost of this system • Speed of extracting: 7800 words/minute • Cost: $ 0, 015 / word – Including keypunching costs: $ 0. 01 / word – Used corpus of 29, 500 words $ 442. 50 total cost – CPI 2003: $ 2798. 00 total cost 0011010 ATS: A Solid Base

A jump in time • • • 1969: First man on the moon 1972: Watergate scandal 1980: John Lennon killed 1981: First identification of AIDS & Birth of me 1986: Space Shuttle Challenger explodes after launch 1989: Fall of Berlin Wall 1990: Start Gulf War & Introduction WWW 1991: Soviet Union breaks up 1992: Formal end of Cold War 1993: Creation of European Union (“Verdrag van Maastricht”) 1994: Nelson Mandela president of South Africa 0011011 ATS: A Solid Base

1995: Trained summarization • Julian Kupiec, Jan Pedersen and Francine Chen: A Trainable Document Summarizer - 1995 0011100 ATS: A Solid Base

Trained weighting • Edmundson used subjective weighting of the features (Cue, Key, Title, Location) to create an abstract • In this journal paper generating the abstract is approached as a statistical classification problem – Given a training set of documents with handmade abstracts: – Develop a classification function that estimates the probability a given sentence is included in the abstract • This requires a training corpus of documents with abstracts • Target documents: technical literature 0011101 ATS: A Solid Base

Features • Five features were used: – – – Sentence Length Cut-off Feature Fixed Phrase Feature Paragraph Feature Thematic Word Feature Uppercase Word Feature • The above features were chosen by experimentation 0011110 ATS: A Solid Base

Sentence Length Cut-off Feature • Based on the principle that short sentences are often not included in abstracts • Given a threshold (e. g. 5 words): – SLC-value is true for sentences longer than the threshold – SLC-value is false otherwise • Note that this feature is not similar to any of the features Edmundson used 0011111 ATS: A Solid Base

Fixed-Phrase Feature • Based on the hypothesis that: – sentences containing any of a list of fixed phrases (mostly 2 words long) are likely to be in the abstract (e. g. “in conclusion”, “this result” – total: 26 elements) – Sentences following a heading containing a certain keyword are more likely to be in the abstract (e. g. , “conclusions”, “results”, “summary”) • FP-value is true for sentences in the above situations, false otherwise • Note that this feature is a combination of Edmundson’s Location Method and Cue Method, though in reduced form 0100000 ATS: A Solid Base

Paragraph Feature • Each sentence in the first ten and last five paragraphs is tagged based on it’s location – Paragraph-initial – Paragraph-final (|P| > 1 sentence) – Paragraph-medial (|P| > 2 sentences) • Note that this feature is a reduced form of Edmundson’s Location Method 0100001 ATS: A Solid Base

Thematic Word Feature • The most frequent words in a document are defined as thematic words • A small number of thematic words is selected and each sentence is scored as a function of frequency of these thematic words • TW-value is true if it is one of the highest scoring sentences • TW-value is false otherwise • Note that this feature is an adapted version of Edmundson’s Key Method 0100010 ATS: A Solid Base

Uppercase Word Feature • Based on the hypothesis that proper names often are important, since it is the explanatory text for acronyms (e. g. “… the ISO (International Standards Organization) …”) • Count the frequency of each proper name – Constraint: the uppercase thematic word is not sentence initial and begins with a capital letter – The word must occur several times and may not be an abbreviated measurement unit • Score each sentence based on the number of frequent proper names in each sentence – The score of a sentence in which the frequent proper name appears first is twice as high as later occurrences • UW-value is true if it is one of the highest scoring sentences, false otherwise • Note that this feature is a bit similar to Edmundson’s Key Method 0100011 ATS: A Solid Base

Classification • For each sentence s the probability P is calculated that it will be included in the summary S given the k features (Bayes’ rule): • Assuming statistical independence of the features: • is constant, and can be estimated directly from the training set by counting occurrences • This function assigns for each s a score which can be used to select sentences for inclusion in the abstract 0100100 ATS: A Solid Base

The training material • 188 documents with professionally created abstracts from the scientific/technical domain, the average length of the abstracts is 3 sentences (3. 5% of the total size of the document) • Sentences from the abstract were matched to the original document: – 79% direct sentence matches – 3% direct joins (2 sentences combined) – 18% no direct match or join possible • Therefore the maximum performance of the automatic system is 82% 0100101 ATS: A Solid Base

Evaluation (1) • Too little material Cross-validation used to evaluate • Two evaluation measures – Fraction of manually selected sentences which were reproduced correctly: average result: 35% – Fraction of the matchable selected sentences which were reproduced correctly: average result: 42% • Performance of features (2 nd measure): Feature Individual % sentences correct Cumulative % sentences correct Paragraph 33 33 Fixed Phrases 29 42 Length Cut-off 24 44 Thematic Word 20 42 Uppercase Word 20 42 0100110 ATS: A Solid Base

Evaluation (2) • Best combination is: Paragraph + Fixed Phrase + Length Cut-off (44% performance) • Addition of frequency keyword features results in a slight decrease of performance (44% 42%) – Note that Edmundson in this case also reports a decrease in performance • In final implementation frequency keyword features are retained in favor of robustness • Baseline used in this experiment: Selecting N sentences from the beginning (Length Cut-off, thus positively biased) • Full feature set has an improvement of 74% over baseline (24% 42%) 0100111 ATS: A Solid Base

Evaluation (3) • If the size of the generated abstract is increased to 25%, the performance improves to 84% • Edmundson ‘only’ had a performance of 44% 0101000 ATS: A Solid Base

Comments • The features used in this paper were chosen by experimentation – No results/discussions of these experiments are given in the paper, so the reason for the choices remain unclear… • The comparison to Edmundson is not very fair – Handmade reference abstracts of Edmundson had a size of 25% (here 3. 5%) • Also the comments which were given about [Edmundson] apply here: – Not good to base length of abstract on length of document – Human selection of sentences in abstracts is very variable – Perhaps the Key Method algorithm used here is too simple (Luhn’s algorithm could be better) 0101001 ATS: A Solid Base

Revisited: [Kupiec e. a. , 1995] • Simone Teufel and Marc Moens: Sentence extraction as a classification task - 1997 0101010 ATS: A Solid Base

Main research questions • Could Kupiec e. a. ’s methodology (training a model with a corpus) be used for another evaluation criterion? • What was the difference in extracting performance of both evaluation criterions for different types of documents? • Note that another set of features is used here than Kupiec e. a. used 0101011 ATS: A Solid Base

Another evaluation method • Kupiec e. a. used the ‘match sentences’ evaluation criterion • Here the training and test set abstracts are created by the authors themselves (as opposed to Kupiec e. a. ) • Hence less alignable sentences are available in the document – 32% on average vs. 79% in Kupiec e. a. • This does not mean there are less ‘extract-worthy’ sentences in the document another evaluation method is chosen • Evaluation: ask human to identify abstract-worthy non-matchable sentences in the original document 0101100 ATS: A Solid Base

Features • The features used here are different from Kupiec e. a. – – – 0101101 Cue Phrase Method (1670 cue phrases): Location Method Sentence Length Method Thematic Word Method Title Method ATS: A Solid Base

Cue Phrase Method • Similarly as in Edmundson, with some differences: – A 5 -point scale (-1 … +3) is used instead of 3 (Bonus, Null, Stigma) – Cue phrases are used instead of Cue words – If a phrase was entered into the list, also syntactically and semantically similar phrases were manually included in the list – A sentence gets the score of it’s maximum-scored Cue phrase, if no Cue phrases are present it gets a score of 0 • The list was manually created by inspecting extracted sentences – Also based on relative frequency in abstract and relative frequency in document • Sentences occurring directly after headings like ‘Introduction’ or ‘Conclusion’ are given a prior score of +2 (in Edmundson this is part of the Location Method) 0101110 ATS: A Solid Base

Location Method • As in Edmundson, with the exception of the sentences directly after headings previously mentioned • Sensitive for certain headings (e. g. “Introduction”); if such headings cannot be found: only the sentences of the first 7 and last 3 paragraphs are tagged (initial, medial, final) 0101111 ATS: A Solid Base

Sentence Length Method • As in Kupiec e. a. • The threshold is set to 15 tokens (including punctuation) 0110000 ATS: A Solid Base

Thematic Word Method • As in Kupiec e. a. , with a few differences: • Selecting (non-Cue) words which occur frequently in this document, but rarely in the overall collection of documents • For each (non-Cue) word the term-frequency*inverse-documentfrequency value is calculated: • score(w) = floc * log (100*N / fglob) – with N: total number of documents, floc: frequency of word w in document, fglob: number of documents containing word w • Top 10 scoring words are defined as thematic words • Top 40 sentences based on the frequency of thematic words (meaned by sentence length) are given a TW-value of 1, all others 0 0110001 ATS: A Solid Base

Title Method • As in Edmundson, with the difference that: • The Title score of the sentence is the mean frequency of Title word occurrences in the sentence (in Edmundson each Title word was given the same score and the scores were summed) • Headings are not taken into account here (by experimentation) • The 18 top-scoring sentences receive a Title-value of 1, the others 0 0110010 ATS: A Solid Base

The experiment • Training set: a corpus of 124 documents from different areas of computational linguistics with summaries written by the authors • A human judge marked additional abstract-worthy sentences in each document • 32% alignable sentences in the abstracts • Two evaluation methods (‘alignable’ and ‘abstract-worthy’) which were also combined 0110011 ATS: A Solid Base

Summary of results • • ‘Alignability’ ‘Abstract-worthy’ Combined Best single feature: Cue Method 23. 2% 46. 7% 55. 2% All features 31. 6% 57. 2% 68. 4% Baseline: 28% (obtained in a similar fashion as Kupiec e. a. ) Bad performance of 31. 6% for alignability can be explained because there are less alignable sentences to train on Short abstracts were generated (2 – 5% of size original document) If abstract size would be increased to 25%, performance would increase to: – ‘Alignability’: 96% (Kupiec e. a. : 84%) – ‘Abstract-worthy’: 98% – Combined: 97. 3% • Therefore compression makes the difference, not the evaluation criterion 0110100 ATS: A Solid Base

Conclusions of this experiment • The method proposed by Kupiec e. a. of classificatory sentence selection is not restricted to texts which have high-quality handmade abstracts • A higher alignability of the handmade abstract is therefore not necessary for the purpose of sentence extraction – compression rate is the factor which influences the result • However, if more flexible abstracts should be generated, the addition of other training and evaluation criterions is useful • Increased training did not improve results, improvement can be obtained in the extraction methods themselves 0110101 ATS: A Solid Base

Comments • The features used in this paper were different from Kupiec e. a. – No motivation was given why for instance the Uppercase Word feature was omitted, and why adapted versions of Edmundson were chosen instead of the versions Kupiec e. a. used • Also comments which were given about [Edmundson] apply here: – Not good to base length of abstract on length of document – Human selection of ‘abstract-worthy’ sentences in abstracts is very variable 0110110 ATS: A Solid Base

Why should we still bother … • In the discussed methods no attention is given to: – Cohesion of the abstract: filtering anaphors out of an abstract (e. g. ‘it’, ‘that’) – Filtering out repetition in the abstract – The semantics of the document • • • Cohesion: an attempt is made by using Lexical Chains Repetition: an attempt is made by using Maximum Marginal Relevance Semantics: this can still not be done for the general case, but an attempt is made by using Rhetorical Tree Structures • • Interested about these problems? Wicher will explain extraction methods which will address repetition and semantics problems in his presentation Terrence will explain Lexical Chains in his presentation • 0110111 ATS: A Solid Base

References • • The Automatic Creation of Literature Abstracts, H. P. Luhn, 1958 New Methods in Automatic Extracting, H. P. Edmundson, 1969 A Trainable Document Summarizer, J. Kupiec e. a. , 1995 Sentence Extraction as a Classification Task, S. Teufel and M. Moens, 1997 The Formation of Abstracts by the Selection of Sentences, G. J. Rath e. a. , 1961 Constructing Literature Abstracts by Computer: Techniques and Prospects, C. D. Paice, 1990 Summarizing Text Documents: Sentence Selection and Evaluation Metrics, Goldstein e. a. , 1999 0111000 ATS: A Solid Base

Any questions? 0111001 ATS: A Solid Base