Relation Extraction for Academic Collaboration 10 709 Project

Relation Extraction for Academic Collaboration 10 -709 Project Presentation Justin Betteridge, Matthew Bilotti, Simon Fung, Sophie Wang February 16, 2006

Academic Collaboration l When two academic researchers work together. . . on a proposal l by co-authoring a paper l by co-chairing a committee l in the same project or research group l ¡ This is evidence of Academic Collaboration l Binary, symmetric relation l Arguments are of type <person>

Motivation l Why might we be interested in extracting Academic Collaboration relations? ¡ Social Networking ¡ Explore the transitivity of the relation ¡ Proof-of-concept for extending relation extraction machinery to other types of relations

Architectural Overview Query Formulator Pattern Bank Query Formulator Relation Bank IR Pattern Extractor Relation Extractor

Co-training Algorithm l Do until termination condition is reached ¡ For l Generate an IR query and send it to the IR engine getting back a set of documents ¡ For l each pattern in the pattern bank each document in the set extract relations ¡ Score all relations (new and old) ¡ Remove relations below threshold

Co-training Algorithm II ¡ For l Generate an IR query and send it to the IR engine getting back a set of documents ¡ For l each relation in the relation bank each document in the set, extract context strings for patterns ¡ Score all patterns (new and old) ¡ Remove patterns below threshold l Loop

Extraction Pattern Formalism l From the proposal: ¡ “left” <x> “between” <y> “right” ¡ Arguments extracted with respect to context l Current status quo: ¡ “context string” <y> ¡ <x> argument extracted from page title l Extracts the relation ¡ Collaborates. With( <x>, <y> )

Detecting Argument Types l Collaborates. With( ¡ <x>, <y> ) and <y> must be of type <person> l Essential to weed out low quality relations produced by noisy patterns such as “in collaboration with” l Heuristics currently encoded as regular expressions

Measuring Confidence with Coverage l Confidence for an Extraction Pattern ¡ Intuitively, relations “vote” for patterns ¡ Query each relation, try to extract the pattern ¡ score = proportion of successful relations l Confidence ¡ Query for a Relation each pattern, try to extract the relation ¡ Score = proportion of successful patterns

Issues with Coverage as Confidence l Seed relations and pattern must co-occur l Very little tolerance for “new” information ¡ It is difficult for a new pattern that broadens the scope of the relations extracted to gain enough confidence to surpass the threshold l Scores tend to zero as pools grow ¡ However, ad-hoc methods of confidence method combination from one iteration to the next introduces a new problem: there is no way to oust bad relations or patterns once extracted

Example Seed Data for Co-Training l Extraction Patterns ¡ <x> “in collaboration with” <y> ¡ <x> “my advisor is” <y> l Relations ¡ Collaborates. With( Tom, Roni ) ¡ Collaborates. With( William, Ken )

Extraction Pattern Examples Query: “my advisor is” site: cs. cmu. edu

Extracted Relations "Miroslav Dudik""Rob Schapire" 0. 33333333 "Personal""Prof. Sanjeev" 0. 33333333 "Research""Professors Jonathan" 0. 33333333 "Sharon Whiteman""Mary Vernon" 0. 33333333 "Sudhakar""Prof. Edward" 0. 33333333 "Ting""Professor Andrew" 0. 33333333 "Adriana Karagiozova""Moses Charikar" 0. 66666666 "Akash Lal""Tom Reps" 0. 66666666 "Amy Karlson""Benjamin B. Bederson" 0. 66666666 "Aravind Kalaiah""Dr. Amitabh" 0. 66666666 "Chi Zhang""Randolph Y. Wang" 0. 66666666 "Gaurav Shah""Matt Blaze" 0. 66666666 "Jennifer Beckmann""Jeff Naughton" 0. 66666666 "Lucja Kot""Dexter Kozen" 0. 66666666 "Mark Sandler""Jon Kleinberg" 0. 66666666 "Nina""Prof. Avrim" 0. 66666666 "Patrick Ng""Uri Keich" 0. 6666666666666666 "Pavlos Papageorgiou""Prof. Michael" 0. 6666666666666666 "Pratyusa Manadhata""Jeannette M. Wing" 0. 66666666 "Sudipta""Marc Pollefeys" 0. 66666666 "Sven Koenig""Reid Simmons" 0. 66666666 "Yan Liu""Jaime Carbonell" 0. 66666666

Learned Patterns l “My advisor is” <y> 0. 6 l Near misses (hard to assess confidence): ¡ “I work with” 0. 4 ¡ “Together with” 0. 0667 ¡ “Languages Research under” 0. 0333 ¡ “Computer Science advisor” 0. 0333 ¡ “Languages under Prof” 0. 0 ¡ “Study under Prof” 0. 0 ¡ “currently working with” 0. 0 ¡ “user studies with” 0. 0

Bad Patterns l From citations: ¡ ¡ l From professional titles: ¡ l “Professor”, “Professor of Mathematics”, etc. From course web pages: ¡ l “Amit Agarwal and”, etc. (other authors) “L 1 Norm with” (part of a title) “courses cs 686 2003 sp” Other: ¡ “be addressed to”

Software and Datasets Used l Indri retrieval engine l Locally crawled collection of pages from CS departments of universities ¡ Using a local collection greatly improved the development experience by shortening the debugging cycle, and relieved us from the Google API query quota ¡ No features of Indri that Google does not support were used so that Google could be substituted for Indri in the future

Future Work l Different methods of combining confidence scores ¡ including weighting of votes during scoring Different confidence metrics, e. g. , PMI l Additional useful sources of information: l ¡ bibliographies, anchor text and link structure: advisor-advisee cross-refs, department or lab organization Better argument type checking l Tuning of the threshold l Termination condition l Integration with citations group l Integrate with Google l Make code run faster l