Learning Semantic Subgraphs for Document Summarization Jure Leskovec

Learning Semantic Subgraphs for Document Summarization Jure Leskovec, Marko Grobelnik Jozef Stefan Institute, Slovenia Natasa Milic-Frayling Microsoft Research, Cambridge, UK

What is summarization? The task is to produce shorter summary version of an original document n Two main approaches to the problem: n ¨ Selection based – summary is a selection of sentences from an original document ¨ Knowledge rich – performing semantic We do this analysis, representing the meaning and generating the text satisfying length restriction

Selection based summarization n Three main phases: Analyzing the source text ¨ Ranking units of the text by importance for the summary ¨ Synthesizing an appropriate output ¨ n Ranking is based on a location of the unit in the text ¨ presence of cue phrases (example: It is important that…) ¨ additional statistical attributes ¨ …lots of heuristics and tuning of parameters (also with ML) ¨ n …output consists from most highly ranked text units (keywords, sentences, paragraphs)

Microsoft Word’s sentence ranking summarizer

Knowledge rich summarization n n To generate ‘true’ summary of a document we need to (at least partially) ‘understand’ the document text Why is knowledge rich summarization difficult? ¨ One document gives us a little of useful information ¨ Can not count of statistical properties of the text (lack of data) ¨ Must rely on syntactical and logical structure of the document (sentences)

Our Approach n n The task is to produce shorter version of an original document by selecting sentences from the text Approach: ¨ Learn a machine learning model for selecting sentences ¨ Use information about semantic structure of the document (concepts and relations among concepts) Hypothesis: Extracted summaries should capture prominent concepts and relations in the text. Structure of the semantic graph of a document could help identify the key concepts and relations for summarization.

Original document Cracks Appear in U. N. Trade Embargo Against Iraq. Cracks appeared Tuesday in the U. N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan, meanwhile, announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose Baghdad, Japan said up to $2 billion in aid may be sent to nations most affected by the U. N. embargo on Iraq. President Bush on Tuesday night promised a joint session of Congress and a nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must stand up to aggression, and we will, '' said Bush, who added that the U. S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how long it will take to convince Iraq to withdraw from Kuwait, '' Bush said. More than 150, 000 U. S. troops have been sent to the Persian Gulf region to deter a possible Iraqi invasion of Saudi Arabia. Bush's aides said the president would follow his address to Congress with a televised message for the Iraqi people, declaring the world is united against their government's invasion of Kuwait. Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations to respond to an offer Monday by Saddam of free oil _ in exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as a none-too-subtle attempt to bypass the U. N. embargo, in effect since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their tankers in Iraq. But according to a State Department survey, Cuba and Romania have struck oil deals with Iraq and companies elsewhere are trying to continue trade with Baghdad, all in defiance of U. N. sanctions. Romania denies the allegation. The report, made available to The Associated Press, said some Eastern European countries also are trying to maintain their military sales to Iraq. A well-informed source in Tehran told The Associated Press that Iran has agreed to an Iraqi request to exchange food and medicine for up to 200, 000 barrels of refined oil a day and cash payments. There was no official comment from Tehran or Baghdad on the reported food-for-oil deal. But the source, who requested anonymity, said the deal was struck during Iraqi Foreign Minister. Tariq Aziz's visit Sunday to Tehran, the first by a senior Iraqi official since the 1980 -88 gulf war. After the visit, the two countries announced they would resume diplomatic relations. Well-informed oil industry sources in the region, contacted by The AP, said that although Iran is a major oil exporter itself, it currently has to import about 150, 000 barrels of refined oil a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News reported that following. Aziz's visit, Iraq is apparently prepared to give Iran all the oil it wants to make up for the damage Iraq inflicted on Iran during their conflict. Secretary of State James A. Baker III, meanwhile, met in Moscow with Soviet Foreign Minister Eduard Shevardnadze, two days after the U. S. -Soviet summit that produced a joint demand that Iraq withdraw from Kuwait. During the summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they remain to fulfill contracts. Shevardnadze told the Soviet parliament Tuesday the specialists had not reneged on those contracts for fear it would jeopardize the 5, 800 Soviet citizens in Iraq. In his speech, Bush said his heart went out to the families of the hundreds of Americans held hostage by Iraq, but he declared, ``Our policy cannot change, and it will not change. America and the world will not be blackmailed. '' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally trying to wipe a country off the face of the Earth. '' In other developments: _A U. S. diplomat in Baghdad said Tuesday up to 800 Americans and Britons will fly out of Iraqi-occupied Kuwait this week, most of them women and children leaving their husbands behind. Saddam has said he is keeping foreign men as human shields against attack. On Monday, a planeload of 164 Westerners arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups of young people suspected of involvement in the resistance. ``There is no law and order, '' said Thuraya, 19, who would not give her last name. ``A soldier can rape a father's daughter in front of him and he can't do anything about it. '' _The State Department said Iraq had told U. S. officials that American males residing in Iraq and Kuwait who were born in Arab countries will be allowed to leave. Iraq generally has not let American males leave. It was not known how many men the Iraqi move could affect. _A Pentagon spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said there was little indication hostilities are imminent. Defense Secretary Dick Cheney said the cost of the U. S. military buildup in the Middle East was rising above the $1 billion-a-month estimate generally used by government officials. He said the total cost _ if no shooting war breaks out _ could total $15 billion in the next fiscal year beginning Oct. 1. Cheney promised disgruntled lawmakers ``a significant increase'' in help from Arab nations and other U. S. allies for Operation Desert Shield. Japan, which has been accused of responding too slowly to the crisis in the gulf, said Tuesday it may give $2 billion to Egypt, Jordan and Turkey, hit hardest by the U. N. prohibition on trade with Iraq. ``The pressure from abroad is getting so strong, '' said Hiroyasu. Horio, an official with the Ministry of International Trade and Industry. Local news reports said the aid would be extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as mid-September. On Friday, Treasury Secretary Nicholas Brady visited Tokyo on a world tour seeking $10. 5 billion to help Egypt, Jordan and Turkey. Japan has already promised a $1 billion aid package for multinational peacekeeping forces in Saudi Arabia, including food, water, vehicles and prefabricated housing for non-military uses. But critics in the United States have said Japan should do more because its economy depends heavily on oil from the Middle East. Japan imports 99 percent of its oil. Japan's constitution bans the use of force in settling international disputes and Japanese law restricts the military to Japanese territory, except for ceremonial occasions. On Monday, Saddam offered developing nations free oil if they would send their tankers to pick it up. The first two countries to respond Tuesday _ the Philippines and Namibia _ said no. Manila said it had already fulfilled its oil requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a ``propaganda ploy. '' Venezuela, an OPEC member, has led a drive among oil-producing nations to boost production to make up for the shortfall caused by the loss of Iraqi and Kuwaiti oil from the world market. Their oil makes up 20 percent of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State Department, Cuba, which faces an oil deficit because of reduced Soviet deliveries, has received a shipment of Iraqi petroleum since U. N. sanctions were imposed five weeks ago. And Romania, it said, expects to receive oil indirectly from Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday, calling it ``absolutely false and without foundation. ''. Our approach to summarization Automatically generated document summary Linguistic processing and Creation of semantic graph Automatic summarization by selecting relevant parts of the graph We use machine learning to learn selection model Cracks appeared in the U. N. trade embargo against Iraq. The State Department reports that Cuba and Romania have struck oil deals with Iraq as others attempt to trade with Baghdad in defiance of the sanctions. Iran has agreed to exchange food and medicine for Iraqi oil. Saddam has offered developing nations free oil if they send their tankers to pick it up. Thus far, none has accepted. Japan, accused of responding too slowly to the Gulf crisis, has promised $2 billion in aid to countries hit hardest by the Iraqi trade embargo. President Bush has promised that Saddam's aggression will not succeed. Natural language generation

Detailed Summarization Procedure Linguistic analysis of the text - Deep parsing of sentences Refinement of the text parse - Named-entity consolidation Determine that ’George Bush’ = ‘U. S. president’ - Anaphora resolution Link pronouns with name-entities Extract Subject–Predicate–Object triples Compose a graph from triples Describe each triple with a set of features for learning Learn a model to classify triples into the summary Generate a summary graph Use summary graph to generate textual document summary Tom Sawyer went to town. He met a friend. Tom was happy. … Tom Sawyer went to town. He [Tom Sawyer] met a friend. Tom [Tom Sawyer] was happy. … Tom go town Tom meet friend Tom is happy

Detailed Summarization Procedure Linguistic analysis of the text - Deep parsing of sentences Refinement of the text parse - Named-entity consolidation Determine that ’George Bush’ = ‘U. S. president’ - Anaphora resolution Link pronouns with name-entities Extract Subject–Predicate–Object triples Compose a graph from triples Describe each triple with a set of features for learning Learn a model to classify triples into the summary Generate a summary graph Use summary graph to generate textual document summary Tom Sawyer went to town. He met a friend. Tom was happy. … Tom Sawyer went to town. He [Tom Sawyer] met a friend. Tom [Tom Sawyer] was happy. … Tom go town Tom meet friend Tom is happy

Linguistic analysis n n We use Microsoft’s NLPWin linguistic processing tool (deep parser) NLPWin uses various external resources: ¨ knowledge n n n base Mind. Net, dictionary, thesaurus Input to NLPWin: a sentence Output from NLPWin: a parse tree of syntactical and logical form of the sentence NLPWin parse tree is the input to procedures for anaphora resolution, name-entity consolidation and extraction of triples

Linguistic processing tool NLPWin Syntactic tree for the sentence “Jure sent Marko a letter” Logical form the sentence “Jure sent Marko a letter” Past – Past Tense for ‘send’ Sing – Singular Prpr. N – Proper Name Pers 3 – Third person singular

Detailed Summarization Procedure Linguistic analysis of the text - Deep parsing of sentences Refinement of the text parse - Named-entity consolidation Determine that ’George Bush’ = ‘U. S. president’ - Anaphora resolution Link pronouns with name-entities Extract Subject–Predicate–Object triples Compose a graph from triples Describe each triple with a set of features for learning Learn a model to classify triples into the summary Generate a summary graph Use summary graph to generate textual document summary Tom Sawyer went to town. He met a friend. Tom was happy. … Tom Sawyer went to town. He [Tom Sawyer] met a friend. Tom [Tom Sawyer] was happy. … Tom go town Tom meet friend Tom is happy

Named entities consolidation (1) n n n Named entities are names of people, places, companies, … Pronouns refer (link) to named entities For each named entity determine the type of pronoun that can refer to it Named entity types: He, She, He-or-She, They, Name … types are ordered into the hierarchy: He is more specific than He-or-She … for example: a name entity of type He has to be masculine, singular and human ¨ n Consolidation procedure: ¨ ¨ Determine name entity type Split each entity into words Remove all stop words and common words Try to match with previously seen entities Common words: mr, ms, co, inc, federal, international, school, national, group, …

Named entities consolidation (2) n Examples of names which are found to refer to the same named entity: ¨ Hillary Rodham Clinton, Hillary Rodham, Mrs. Clinton n n Accuracy around 90% Typical errors: ¨ President Kennedy ≠ Kennedy School of Government n After common words removal, only word Kennedy is left of companies Reuter(s), Levi(‘s) and Lloyd(‘s) are mistaken for a person ¨ We do not distinguish between first and last name ¨ Names n n … in some cases brothers can become one named entity Michael Milken = Robert Milken

Pronomial anaphora resolution n We link pronouns with their references Mary likes Paul. She went to buy him a present. Mary likes Paul. She [Mary] went to buy him [Paul] a present. n Method: ¨ ¨ ¨ We restrict anaphora resolution to 5 pronouns: she, who, I, they. From the pronoun, traverse the text searching for candidate references and assign a score The score is based on the distance from the pronoun and semantic information Note we assume that pronouns refer only to named entities found in the document Problem: n One passenger in King's car said they had been drinking liquor.

Anaphora resolution (2) n What we don’t find: ¨ ¨ ¨ n n Error: he == John Quoted speech: John said: “He is sick. ” he == I But: “I hope so, " he replies after a pause. Error: him == Tom wrote a letter to Bill. He told him … The relationship between active volcanoes and the communities that surround them is not always confrontational. We don’t link them Jordan's King Hussein and Yasser Arafat's open sympathy for Iraq has strained their relations with the U. S. Can’t link they Mistakes are not so common The most fatal case is when we wrongly resolve first occurrence of a pronoun and then follow many sentences using only the pronoun to refer to a person.

Anaphora resolution evaluation n We manually labeled 91 articles ¨ Containing 1506 pronouns ¨ 1024 (68%) pronouns are he, she, I, they, who n We try to link all of them ¨ Other 482 (32%) pronouns are: it, you, we, what, …

Anaphora resolution evaluation Pronoun Frequency [%] Accuracy [%] He 681 45. 22 86. 9 They 244 16. 20 67. 2 It 204 13. 55 I 64 4. 25 You 50 3. 32 We 44 2. 92 That 44 2. 92 What 27 1. 79 She 24 1. 59 This 22 1. 46 Who 11 0. 73 63. 6 1506 100 81. 2 82. 8 62. 5 … Total Accuracy on 5 selected 81. 2% (55. 2% if counting all pronouns)

Detailed Summarization Procedure Linguistic analysis of the text - Deep parsing of sentences Refinement of the text parse - Named-entity consolidation Determine that ’George Bush’ = ‘U. S. president’ - Anaphora resolution Link pronouns with name-entities Extract Subject–Predicate–Object triples Compose a graph from triples Describe each triple with a set of features for learning Learn a model to classify triples into the summary Generate a summary graph Use summary graph to generate textual document summary Tom Sawyer went to town. He met a friend. Tom was happy. … Tom Sawyer went to town. He [Tom Sawyer] met a friend. Tom [Tom Sawyer] was happy. … Tom go town Tom meet friend Tom is happy

Extracting triples n Enhanced parse tree is traversed to identify Subject–Predicate–Object triples n Example: “Conservatives embraced the nomination while liberals were cautious or hostile” Resulting triples: conservative embrace nomination liberal is cautious liberal is hostile

Detailed Summarization Procedure Linguistic analysis of the text - Deep parsing of sentences Refinement of the text parse - Named-entity consolidation Determine that ’George Bush’ = ‘U. S. president’ - Anaphora resolution Link pronouns with name-entities Extract Subject–Predicate–Object triples Compose a graph from triples Describe each triple with a set of features for learning Learn a model to classify triples into the summary Generate a summary graph Use summary graph to generate textual document summary Tom Sawyer went to town. He met a friend. Tom was happy. … Tom Sawyer went to town. He [Tom Sawyer] met a friend. Tom [Tom Sawyer] was happy. … Tom go town Tom meet friend Tom is happy

Learning: Feature construction n Graph consists of nodes, referred as concepts, which can be subjects or objects and edges which are predicates and capture relations among concepts. n We use Word net to identify and compact synonym nodes – as they correspond to the same concepts.

To create semantic graph find Subjects (Objects) with the same meaning Word. Net Concept Relation Concept 26 types of relations 115, 000 concepts

Constructing semantic graph (2) Two words (Subjects, Objects) are synonyms if they are synonyms in any of their meanings in Word. Net n Examples: Using Word. Net we find n ¨ watcher that the following triples follow moon have the same meaning spectator watch moon ¨ governor Patten change position governor Patten modify attitude ¨ earthquake hit Northern Iran quake strike Northern Iran

Experiments We would like learn how to select relevant parts of document semantic graph into the summary n For learning we will use summaries created by humans n Each Subject-Predicate-Object triple will be a learning example n Each triple will be labeled whether it is interesting for the summary or not n

Example of automatic summary Cracks Appear in U. N. Trade Embargo Against Iraq. Human extracted summary Cracks appeared Tuesday in the U. N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan, meanwhile, announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose Baghdad, Japan said up to $2 billion in aid may be sent to nations most affected by the U. N. embargo on Iraq. President Bush on Tuesday night promised a joint session of Congress and a nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must stand up to aggression, and we will, '' said Bush, who added that the U. S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how long it will take to convince Iraq to withdraw from Kuwait, '' Bush said. More than 150, 000 U. S. troops have been sent to the Persian Gulf region to deter a possible Iraqi invasion of Saudi Arabia. Bush's aides said the president would follow his address to Congress with a televised message for the Iraqi people, declaring the world is united against their government's invasion of Kuwait. Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations to respond to an offer Monday by Saddam of free oil _ in exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as a none-too-subtle attempt to bypass the U. N. embargo, in effect since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their tankers in Iraq. But according to a State Department survey, Cuba and Romania have struck oil deals with Iraq and companies elsewhere are trying to continue trade with Baghdad, all in defiance of U. N. sanctions. Romania denies the allegation. The report, made available to The Associated Press, said some Eastern European countries also are trying to maintain their military sales to Iraq. A well-informed source in Tehran told The Associated Press that Iran has agreed to an Iraqi request to exchange food medicine for upafter to 200, 000 barrels ofscare refinedincited oil a day and cash payments. There was nogloom official comment from Tehran or Baghdad on the n and Eight years volcano anger and during economic in Sierra reported food-for-oil deal. But the source, a who requested anonymity, saidfear, the deal was struck Iraqi Foreign Minister Tariq Aziz's visit Sunday to Tehran, the resorts, residents aregulf nonchalant about renewed underground lava movement that isrelations. Well-informed oil first by a senior Iraqi official since the 1980 -88 war. After the visit, the two countries announced they would resume diplomatic industry sources in triggering the region, contacted by Theof. AP, saidearthquakes. that although Iran is a major oil exporter itself, it currently has to import about 150, 000 barrels of refined oil thousands tiny a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News reported that following Aziz's visit, Iraq is apparently prepared n the oil The resort town's 4, 700 permanent residents intheir Long Valley, a 19 -mile-long, 9 -mileto give Iran all it wants to make up for the damage Iraq inflicted on Iran live during conflict. Secretary of State James A. Baker III, meanwhile, met in Moscow with Soviet Foreign wide Minister Eduard Shevardnadze, two days the U. S. -Soviet summit that produced a joint demand that Iraq withdraw from Kuwait. During the volcanic crater known as aafter caldera. summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they remain to fulfill contracts. Shevardnadze told the Soviet n The Earth's had crust beingonstretched apart in the region, allowing molten to infill. Iraq. halfparliament Tuesday the specialists not is reneged those contracts for fear it would jeopardize the 5, 800 Sovietrock citizens In his speech, Bush said his heart went out to the families of the hundreds of Americans by Iraq, but he declared, ``Our policy cannot change, and it will not change. America and the mile-wide chambers underheld thehostage caldera. world will not be blackmailed. '' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally trying to wipe a country off the face of the n developments: The valley_Awas createdin 730, 000 years agoupby one. Americans of Earth's eruptions, a Kuwait this week, most of Earth. '' In other U. S. diplomat Baghdad said Tuesday to 800 and most Britons powerful will fly out of Iraqi-occupied them women and children leaving their husbands behind. more Saddammaterial has said he is keeping foreign men as human shields against. St. attack. On Monday, a planeload of blast that spewed 600 times than the May 1980 eruption of Mount 164 Westerners arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups of young people suspected of Helens in``There Washington state. involvement in the resistance. is no law and order, '' said Thuraya, 19, who would not give her last name. ``A soldier can rape a father's daughter in front of him and he can't anything about it. '' _The activity, State Department said Iraq hadoftold U. S. officials that American residingeruption in Iraq and Kuwait who were born in n do. Despite the current the probability a major earthquake or amales volcanic Arab countries will be allowed to leave. Iraq generally has not leteach American malessaid leave. David It was Hill, not known how many men the Iraqi move could affect. _A Pentagon in the area is “less than 1 percent year, ” the U. S. spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said there was little indication hostilities are imminent. Secretary Dick Cheney said the cost of in thecharge U. S. military buildup in the was rising above the $1 billion-a-month estimate generally n Defense Geological Survey geophysicist of research at. Middle Long. East Valley. Mono County used by government officials. He said the total cost _ if no shooting war breaks out _ could total $15 billion in the next fiscal year beginning Oct. 1. Cheney promised Sheriff Martin Strelneck called estimates “a U. S. scientific game, ” and said disgruntled lawmakers ``a significant increase'' in help fromsuch Arab nations and other allies forguessing Operation Desert Shield. Japan, which has been accused of area residents rarely discuss latest earthquakes, which hit started responding too slowly to the crisis in the gulf, said Tuesdaythe it may give swarm $2 billion of to Egypt, Jordan and Turkey, hardestin by. May the U. N. prohibition on trade with Iraq. ``The pressure from abroad is getting so strong, '' said Hiroyasu Horio, an official with the Ministry of International Trade and Industry. Local news reports said the 1989. aid would be extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as mid-September. On Friday, Treasury n As a visited result, theon Geological issued of potential volcanic hazard” for promised a $1 billion aid package Secretary Nicholas Brady Tokyo a world tour Survey seeking $10. 5 billionato“notice help Egypt, Jordan and Turkey. Japan has already for multinational peacekeeping forces Saudi 1982. Arabia, including food, water, vehicles and prefabricated housing for non-military uses. But critics in the United States Long Valley inin. May have said Japan should do more because its economy depends heavily on oil from the Middle East. Japan imports 99 percent of its oil. Japan's constitution bans the That warning, disputes coupled with jarring earthquakes, tourismexcept and for aggravated a use of force n in settling international and Japanese law restricts the military damaged to Japanese territory, ceremonial occasions. On Monday, Saddam offered developing recession nations free oilinif they once-booming would send their tankers to pick it market. up. The first two countries to respond Tuesday _ the Philippines and Namibia _ said no. real estate Manila said it had already fulfilled its oil requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a ``propaganda ploy. '' Venezuela, an OPEC member, has led a drive among oil-producing nations to boost production to make up for the shortfall caused by the loss of Iraqi and Kuwaiti oil from the world market. Their oil makes up 20 percent of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State Department, Cuba, which faces an oil deficit because of reduced Soviet deliveries, has received a shipment of Iraqi petroleum since U. N. sanctions were imposed five weeks ago. And Romania, it said, expects to receive oil indirectly from Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday, calling it ``absolutely false and without foundation. ''. 7800 characters, 1300 words

Human written summary n California's Long Valley is a 19 -mile caldera created 730, 000 years ago by an eruption 600 times larger than Mount St. Helens. In May 1989, new underground lava movement began triggering thousands of tiny earthquakes and raising the valley floor. Residents refuse to heed warnings, remembering a 1982 false alarm that damaged tourism and aggravated a recession. Afterward, journalists were accused of sensationalism and scientists of scaring people to get more funding. Currently, 5 -10 small quakes happen daily as the Earth's crust is stretched apart and magma fills half-mile-wide chambers 4 miles under the caldera, but the probability of eruption is less than one percent. Human sentence selection summary n n n n Eight years after a volcano scare incited fear, anger and economic gloom in Sierra resorts, residents are nonchalant about renewed underground lava movement that is triggering thousands of tiny earthquakes. The resort town's 4, 700 permanent residents live in Long Valley, a 19 -mile-long, 9 -mile-wide volcanic crater known as a caldera. The Earth's crust is being stretched apart in the region, allowing molten rock to fill half-mile-wide chambers under the caldera. The valley was created 730, 000 years ago by one of Earth's most powerful eruptions, a blast that spewed 600 times more material than the May 1980 eruption of Mount St. Helens in Washington state. Despite the current activity, the probability of a major earthquake or a volcanic eruption in the area is “less than 1 percent each year, ” said David Hill, the U. S. Geological Survey geophysicist in charge of research at Long Valley. Mono County Sheriff Martin Strelneck called such estimates “a scientific guessing game, ” and said area residents rarely discuss the latest swarm of earthquakes, which started in May 1989. As a result, the Geological Survey issued a “notice of potential volcanic hazard” for Long Valley in May 1982. That warning, coupled with jarring earthquakes, damaged tourism and aggravated a recession in the once-booming real estate market.

Full document semantic graph

Full document semantic graph

Automatically generated summary

Experimental settings n n We use Support Vector Machines machine learning algorithm with linear kernel Binary classification problem: ¨ Positive examples are triples from the sentences which humans selected into the summary ¨ Negative examples are all other triples n Treat each document as a unit: ¨ Take all triples from the document for either testing or learning n Report microaveraged values of precision, recall and F 1

DUC 2001 dataset Document understanding conference n 300 newspaper articles on 30 different topics: people, natural disasters, events, … n Each topic has about 10 articles Learn if a triple comes n For each article we have… n … hand made summary … sentence selection summary n from the sentence which was selected into the summary Articles are long (1000 words, 50 sentences, each Only about 30% of the triples from the human written abstract has 22 words) can be found in the document

Triple attributes n Features used in the learning process include triples described by the following attributes: ¨ Positional information n Of the sentence from which the triple was derived relative to the document text n Of the triple relative to the beginning of the sentence ¨ NLPWin linguistic attributes of the nodes in the triple: n 18 syntactic attributes n 100 semantic attributes ¨ 14 graph attributes: Page. Rank, In/Out Degree, reachable neighbours, etc. On our dataset this yield: TOTAL of 466 attributes On average 72 non-zero attributes per triple.

Performance for various attribute sets Attribute set Training Set Test Set Precision Recall F 1 Sentence Position + Terms 65. 87 92. 48 76. 94 28. 87 37. 08 32. 46 only Position (triple + sentence) 31. 21 52. 49 39. 15 31. 05 52. 58 39. 05 only Graph 27. 78 57. 46 37. 46 27. 25 56. 90 36. 85 only Linguistic 29. 77 61. 79 40. 18 22. 29 47. 52 30. 29 Position + Linguistic 31. 16 67. 00 42. 54 28. 67 62. 57 39. 33 Position + Graph 33. 51 63. 85 43. 95 42. 71 63. 02 43. 07 Position + Graph + Linguistic 35. 82 72. 69 47. 99 31. 41 64. 88 42. 33

Baseline performance (sentence position + selected terms from the Performance for various attribute sets sentence) F 1=32. 46 is lower than in any of the other runs, except for ‘only linguistic’ attributes (F 1=30. 29). Training Set Test Set ‘only set linguistic’ run includes only generic syntactic and semantic Attribute Precision Recall F 1 Precision Recall labels - not expected to be good discriminators on their own. F 1 Sentence Position + Terms 65. 87 92. 48 76. 94 28. 87 37. 08 32. 46 only Position (triple + sentence) 31. 21 52. 49 39. 15 31. 05 52. 58 39. 05 only Graph 27. 78 57. 46 37. 46 27. 25 56. 90 36. 85 only Linguistic 29. 77 61. 79 40. 18 22. 29 47. 52 30. 29 Position + Linguistic 31. 16 67. 00 42. 54 28. 67 62. 57 39. 33 Position + Graph 33. 51 63. 85 43. 95 42. 71 63. 02 43. 07 Position + Graph + Linguistic 35. 82 72. 69 47. 99 31. 41 64. 88 42. 33

Adding generic linguistic attributes reduces precision Performance various attribute sets Position of triples andfor sentences P=31. 05 Adding linguistic attributes P=28. 67 Training Set but consistently Attribute set increases recall Test Set Precision Recall F 1 Sentence Position + Terms 65. 87 92. 48 76. 94 28. 87 37. 08 32. 46 only Position (triple + sentence) 31. 21 52. 49 39. 15 31. 05 52. 58 39. 05 only Graph 27. 78 57. 46 37. 46 27. 25 56. 90 36. 85 only Linguistic 29. 77 61. 79 40. 18 22. 29 47. 52 30. 29 Position + Linguistic 31. 16 67. 00 42. 54 28. 67 62. 57 39. 33 Position + Graph 33. 51 63. 85 43. 95 32. 71 63. 02 43. 07 Position + Graph + Linguistic 35. 82 72. 69 47. 99 31. 41 64. 88 42. 33

Performance various attribute sets Information for about the graph structure helps Position of triples and sentences F 1=39. 05 Training Set Test Set Attribute set Adding structure information F 1=43. 07 Precision Recall F 1 Sentence Position + Terms 65. 87 92. 48 76. 94 28. 87 37. 08 32. 46 only Position (triple + sentence) 31. 21 52. 49 39. 15 31. 05 52. 58 39. 05 only Graph 27. 78 57. 46 37. 46 27. 25 56. 90 36. 85 only Linguistic 29. 77 61. 79 40. 18 22. 29 47. 52 30. 29 Position + Linguistic 31. 16 67. 00 42. 54 28. 67 62. 57 39. 33 Position + Graph 33. 51 63. 85 43. 95 42. 71 63. 02 43. 07 Position + Graph + Linguistic 35. 82 72. 69 47. 99 31. 41 64. 88 42. 33

Insights n We determine the median and quartiles of the ranks across 10 runs. Most highly ranked features in SVM normal: Attribute 1 st quartile Median 3 rd quartile Object – Authority weight 1 1 2 Object – size of weakly connected component 2 2. 5 3 Object – degree of a node 2 3 3 Object – is name of a country 4 5 5 Subject – size of weakly connected component 6 7 9 Subject – degree of a node 6 10. 5 12 Object – Page. Rank weight 6 11 12 Object – is name of a geographical location 8 13 16 Subject – Authority weight 13 18. 5 23

Insights n We determine the median and quartiles of the ranks across 10 runs. Most highly ranked features in SVM normal for nonsummary triples: Attribute 1 st quartile Median 3 rd quartile Subject – 1 st person 1 1 1 Predicate – quoted speech 2 2 2 Predicate – position of a word in the sentence 3 3 3 Object – quoted speech 4 4. 5 6 Object – feminine gender 5 5 7 Object – position of a word in the sentence 8 10 14 Subject – position of a sentence in the document 14 16 19

Learning cross-topic summarization n 10 fold cross validation for different sample size of training data: Training Set Test Set Learning documents Precision Recall F 1 10 33. 48 88. 44 48. 58 23. 05 64. 67 33. 99 20 31. 26 86. 45 45. 92 24. 49 68. 69 36. 11 50 29. 42 82. 53 43. 37 25. 75 72. 81 38. 04 100 28. 64 79. 91 42. 16 26. 25 73. 22 38. 64 Learn if a triple comes from the sentence which was selected into the summary Precision = TP / (TP + FP) Recall = TP / (TP + FN) F 1 = 2 Prec Rec/(Prec+Rec)

Within-topic summarization n Leave one out cross validation averaged over 30 different topics: Learning using samples of 5 Training Set Test Set Precision Recall F 1 Within-topic 36. 49 90. 63 52. 03 23. 60 60. 05 33. 89 Cross-topic 36. 59 92. 23 52. 40 20. 73 60. 28 30. 85 documents

Further work n Additional experiments: no-structure vs. structure (graph vs. no-graph) ¨ different levels of linguistic processing ¨ n n n Use Word. Net less naively for graph construction, abstraction New sets of attributes: Word. Net Study the differences between the triples from human written abstracts and sentence selection abstracs

Conclusion n Experiments on the dataset used show: ¨ n Attributes that characterize the document semantic graph improve selection of triples for summarization. This results need to be verified on additional data sets Need to perform comparison with additional summarization methods Explore various strategies for extracting and generating summaries based on extracted triples. We observe: ¨ No combination of features that we examined lead to good separation of positive and negative triples in the feature space Opportunity for further investigations and improvements.

Example of automatic summary Cracks Appear in U. N. Trade Embargo Against Iraq. Human written summary Cracks appeared Tuesday in the U. N. trade embargo against Iraq as Saddam Hussein sought to circumvent the economic noose around his country. Japan, meanwhile, announced it would increase its aid to countries hardest hit by enforcing the sanctions. Hoping to defuse criticism that it is not doing its share to oppose Baghdad, Japan said up to $2 billion in aid may be sent to nations most affected by the U. N. embargo on Iraq. President Bush on Tuesday night promised a joint session of Congress and a nationwide radio and television audience that ``Saddam Hussein will fail'' to make his conquest of Kuwait permanent. ``America must stand up to aggression, and we will, '' said Bush, who added that the U. S. military may remain in the Saudi Arabian desert indefinitely. ``I cannot predict just how long it will take to convince Iraq to withdraw from Kuwait, '' Bush said. More than 150, 000 U. S. troops have been sent to the Persian Gulf region to deter a possible Iraqi invasion of Saudi Arabia. Bush's aides said the president would follow his address to Congress with a televised message for the Iraqi people, declaring the world is united against their government's invasion of Kuwait. Saddam had offered Bush time on Iraqi TV. The Philippines and Namibia, the first of the developing nations to respond to an offer Monday by Saddam of free oil _ in exchange for sending their own tankers to get it _ said no to the Iraqi leader. Saddam's offer was seen as a none-too-subtle attempt to bypass the U. N. embargo, in effect since four days after Iraq's Aug. 2 invasion of Kuwait, by getting poor countries to dock their tankers in Iraq. But according to a State Department survey, Cuba and Romania have struck oil deals with Iraq and companies elsewhere are trying to continue trade with Baghdad, all in defiance of U. N. sanctions. Romania denies the allegation. The report, made available to The Associated Press, said some Eastern European countries also are trying to maintain their military sales to Iraq. A well-informed source in Tehran told The Associated Press that Iran has agreed to an Iraqi request to exchange food and medicine for up to 200, 000 barrels of refined oil a day and cash payments. There was no official comment from Tehran or Baghdad on the reported food-for-oil deal. But the source, who requested anonymity, said the deal was struck during Iraqi Foreign Minister Tariq Aziz's visit Sunday to Tehran, the first by a senior Iraqi official since the 1980 -88 gulf war. After the visit, the two countries announced they would resume diplomatic relations. Well-informed oil industry sources in the region, contacted by The AP, said that although Iran is a major oil exporter itself, it currently has to import about 150, 000 barrels of refined oil a day for domestic use because of damages to refineries in the gulf war. Along similar lines, ABC News reported that following Aziz's visit, Iraq is apparently prepared to give Iran all the oil it wants to make up for the damage Iraq inflicted on Iran during their conflict. Secretary of State James A. Baker III, meanwhile, met in Moscow with Soviet Foreign Minister Eduard Shevardnadze, two days after the U. S. -Soviet summit that produced a joint demand that Iraq withdraw from Kuwait. During the summit, Bush encouraged Mikhail Gorbachev to withdraw 190 Soviet military specialists from Iraq, where they remain to fulfill contracts. Shevardnadze told the Soviet parliament Tuesday the specialists had not reneged on those contracts for fear it would jeopardize the 5, 800 Soviet citizens in Iraq. In his speech, Bush said his heart went out to the families of the hundreds of Americans held hostage by Iraq, but he declared, ``Our policy cannot change, and it will not change. America and the world will not be blackmailed. '' The president added: ``Vital issues of principle are at stake. Saddam Hussein is literally trying to wipe a country off the face of the Earth. '' In other developments: _A U. S. diplomat in Baghdad said Tuesday up to 800 Americans and Britons will fly out of Iraqi-occupied Kuwait this week, most of them women and children leaving their husbands behind. Saddam has said he is keeping foreign men as human shields against attack. On Monday, a planeload of 164 Westerners arrived in Baltimore from Iraq. Evacuees spoke of food shortages in Kuwait, nighttime gunfire and Iraqi roundups of young people suspected of involvement in the resistance. ``There is no law and order, '' said Thuraya, 19, who would not give her last name. ``A soldier can rape a father's daughter in front of him and he can't do anything about it. '' _The State Department said Iraq had told U. S. officials that American males residing in Iraq and Kuwait who were born in Arab countries will be allowed to leave. Iraq generally has not let American males leave. It was not known how many men the Iraqi move could affect. _A Pentagon spokesman said ``some increase in military activity'' had been detected inside Iraq near its borders with Turkey and Syria. He said there was little indication hostilities are imminent. Defense Secretary Dick Cheney said the cost of the U. S. military buildup in the Middle East was rising above the $1 billion-a-month estimate generally used by government officials. He said the total cost _ if no shooting war breaks out _ could total $15 billion in the next fiscal year beginning Oct. 1. Cheney promised disgruntled lawmakers ``a significant increase'' in help from Arab nations and other U. S. allies for Operation Desert Shield. Japan, which has been accused of responding too slowly to the crisis in the gulf, said Tuesday it may give $2 billion to Egypt, Jordan and Turkey, hit hardest by the U. N. prohibition on trade with Iraq. ``The pressure from abroad is getting so strong, '' said Hiroyasu Horio, an official with the Ministry of International Trade and Industry. Local news reports said the aid would be extended through the World Bank and International Monetary Fund, and $600 million would be sent as early as mid-September. On Friday, Treasury Secretary Nicholas Brady visited Tokyo on a world tour seeking $10. 5 billion to help Egypt, Jordan and Turkey. Japan has already promised a $1 billion aid package for multinational peacekeeping forces in Saudi Arabia, including food, water, vehicles and prefabricated housing for non-military uses. But critics in the United States have said Japan should do more because its economy depends heavily on oil from the Middle East. Japan imports 99 percent of its oil. Japan's constitution bans the use of force in settling international disputes and Japanese law restricts the military to Japanese territory, except for ceremonial occasions. On Monday, Saddam offered developing nations free oil if they would send their tankers to pick it up. The first two countries to respond Tuesday _ the Philippines and Namibia _ said no. Manila said it had already fulfilled its oil requirements, and Namibia said it would not ``sell its sovereignty'' for Iraqi oil. Venezuelan President Carlos Andres Perez dismissed Saddam's offer of free oil as a ``propaganda ploy. '' Venezuela, an OPEC member, has led a drive among oil-producing nations to boost production to make up for the shortfall caused by the loss of Iraqi and Kuwaiti oil from the world market. Their oil makes up 20 percent of the world's oil reserves. Only Saudi Arabia has higher reserves. But according to the State Department, Cuba, which faces an oil deficit because of reduced Soviet deliveries, has received a shipment of Iraqi petroleum since U. N. sanctions were imposed five weeks ago. And Romania, it said, expects to receive oil indirectly from Iraq. Romania's ambassador to the United States, Virgil Constantinescu, denied that claim Tuesday, calling it ``absolutely false and without foundation. ''. Cracks appeared in the U. N. trade embargo against Iraq. The State Department reports that Cuba and Romania have struck oil deals with Iraq as others attempt to trade with Baghdad in defiance of the sanctions. Iran has agreed to exchange food and medicine for Iraqi oil. Saddam has offered developing nations free oil if they send their tankers to pick it up. Thus far, none has accepted. Japan, accused of responding too slowly to the Gulf crisis, has promised $2 billion in aid to countries hit hardest by the Iraqi trade embargo. President Bush has promised that Saddam's aggression will not succeed. 7800 characters, 1300 words

Iraq

More examples