CS 276 A Text Information Retrieval Mining and

CS 276 A Text Information Retrieval, Mining, and Exploitation Lecture 15 26 Nov 2002

Recap: Web Anatomy …/~newbie/ www. ibm. com /…/…/leaf. htm

Recap: Size of the Web n Capture – Recapture technique n Assumes engines get independent random subsets of the Web E 2 contains x% of E 1. Assume, E 2 contains x% of the Web as well Knowing size of E 2 compute size of the Web Size of the Web = 100*E 2/x E 2 E 1 WEB Bharat & Broder: 200 M (Nov 97), 275 M (Mar 98) Lawrence & Giles: 320 M (Dec 97)

Recent Measurements Source: http: //www. searchengineshowdown. com/stats/change. shtml

Today’s Topics n n Web IR infrastructure Search deployment XML intro XML indexing and search

Web IR Infrastructure n Connectivity Server n n Fast access to links to support link analysis Term Vector Database n Fast access to document vectors to augment link analysis

Connectivity Server [CS 1: Bhar 98 b, CS 2 & 3: Rand 01] n n Fast web graph access to support connectivity analysis Stores mappings in memory from n n URL to outlinks, URL to inlinks Applications n n HITS, Pagerank computations Crawl simulation Graph algorithms: web connectivity, diameter etc. Visualizations

Usage Input Graph algorithm + URLs + Values URLs to FPs to IDs Execution Output Graph algorithm runs in memory URLs + Values IDs to URLs Translation Tables on Disk URL text: 9 bytes/URL (compressed from ~80 bytes ) FP(64 b) -> ID(32 b): 5 bytes ID(32 b) -> FP(64 b): 8 bytes ID(32 b) -> URLs: 0. 5 bytes

ID assignment n Partition URLs into 3 sets, sorted lexicographically n n High: Max degree > 254 Medium: 254 > Max degree > 24 Low: remaining (75%) IDs assigned in sequence (densely) E. g. , HIGH IDs: Max(indegree , outdegree) > 254 ID URL … 9891 www. amazon. com/ 9912 www. amazon. com/jobs/ … 9821878 www. geocities. com/ … Adjacency lists § In memory tables for Outlinks, Inlinks § List index maps from an ID to start of adjacency list 40930030 www. google. com/ … 85903590 www. yahoo. com/

Adjacency List Compression - I … 104 105 106 … List Index … 98 132 153 98 147 153 … Sequence of Adjacency Lists … 104 105 106 … List Index … -6 34 21 -8 49 6 … Delta Encoded Adjacency Lists • Adjacency List: - Smaller delta values are exponentially more frequent (80% to same host) - Compress deltas with variable length encoding (e. g. , Huffman) • List Index pointers: 32 b for high, Base+16 b for med, Base+8 b for low - Avg = 12 b per pointer

List Index Pointers ID offset Base (4 bytes) URL Info LC: TID Offsets For 16 IDs … Adjacency LC: TID lists FRQ: RL … FRQ: RL ID to adjacency list lookup

Adjacency List Compression - II n Inter List Compression n Basis: Similar URLs may share links n n Close in ID space => adjacency lists may overlap Approach n Define a representative adjacency list for a block of IDs n n Adjacency list of a reference ID Union of adjacency lists in the block Represent adjacency list in terms of deletions and additions when it is cheaper to do so Measurements n n Intra List + Starts: 8 -11 bits per link (580 M pages/16 GB RAM) Inter List: 5. 4 -5. 7 bits per link (870 M pages/16 GB RAM. )

Term Vector Database [Stat 00] n Fast access to 50 word term vectors for web pages n Term Selection: n n n Term Weighting: n n Deferred till run-time (can be based on term freq, doc length) Applications n n Restricted to middle 1/3 rd of lexicon by document frequency Top 50 words in document by TF. IDF. Content + Connectivity analysis (e. g. , Topic Distillation) Topic specific crawls Document classification Performance n n Storage: 33 GB for 272 M term vectors Speed: 17 ms/vector on Alpha. Server 4100 (latency to read a disk block)

Architecture URLid * 64 /480 offset URL Info Base (4 bytes) LC: TID Terms LC: TID … Bit vector For 480 URLids LC: TID FRQ: RL Freq FRQ: RL … FRQ: RL URLid to Term Vector Lookup 128 Byte TV Record

Search Deployment n n Web IR is just one (very specific) type of IR Commercially most important IR application: n n n Enterprise search (large corporations) Problem different from Web IR Peer-2 -Peer (P 2 P) search n Another search deployment strategy

Enterprise Search Deployment Search Boxes Markets Content Location Sources E-Commerce Proprietary content Database Web Portals Enterprises Company Web Site Public content Corporate Network Content Management Groupware World Wide Web

Evolution of Enterprise Search 1 st Generation: 2 nd Generation: 3 rd Generation: Classic Information Retrieval Driven by WWW Discovery (Text Mining) User: Trained specialist Everyone Scope: Small, closed collections Intranet/Extranet Everyone and software agents Structured, semistructured and unstructured information Technology: Pattern/string matching 1985 - 1993 Pattern/string matching and external factors for relevance ranking + categorization 1994 - 1999 Introduction of linguistic and semantic processing 2000+

Enterprise IR is a lot more than search … Security n Cannot search what you should not read n. Content organization & creation n Automatic classification n Taxonomy generation n Support for multiple languages, multiple formats n Conduits into databases and other content management -- homes for “valuable” content n n n Information processing tools n Annotation n Range searches n Custom ranking criteria n Cross lingual tools, … Individual preferences n Personalization n Notification, …

Peer-To-Peer (P 2 P) Search n n n No central index Each node in a network builds and maintains own index Each node has “servent” software n n n On booting, servent pings ~4 other hosts Connects to those that respond Initiates, propagates and serves requests

Which hosts to connect to? n n The ones you connected to last time Random hosts you know of Request suggestions from central (or hierarchical) nameservers All govern system’s shape and efficiency

Serving P 2 P search requests n n Send your request to your neighbors They send it to their neighbors n n n decrement “time to live” for query dies when ttl = 0 Send search matches back along requesting path

Some P 2 P Networks n n n Gnutella Kazaa Bearshare Aimster Grokster Morpheus

P 2 P: Information Retrieval Issues n Why is this more difficult than centralized IR?

P 2 P: Information Retrieval Issues n n n Selection of nodes to query Merging of results Spam

What is XML? n n n e. Xtensible Markup Language A framework for defining markup languages No fixed collection of markup tags Each XML language targeted for application All XML languages share features Enables building of generic tools

Basic Structure n n An XML document is an ordered, labeled tree character data leaf nodes contain the actual data (text strings) n n data nodes must be non-empty and nonadjacent to other character data nodes element nodes, are each labeled with n a name (often called the element type), and n n a set of attributes, each consisting of a name and a value, can have child nodes

XML Example

XML Example <chapter id="cmds"> <chaptitle>File. Cab</chaptitle> <para>This chapter describes the commands that manage the <tm>File. Cab</tm>inet application. </para> </chapter>

Elements n n n Elements are denoted by markup tags <foo attr 1=“value” … > thetext </foo> Element start tag: foo Attribute: attr 1 The character data: thetext Matching element end tag: </foo>

XML vs HTML n Relationship?

XML vs HTML n n n HTML is a markup language for a specific purpose (display in browsers) XML is a framework for defining markup languages HTML can be formalized as an XML language (XHTML) XML defines logical structure only HTML: same intention, but has evolved into a presentation language

XML: Design Goals n n Separate syntax from semantics to provide a common framework for structuring information Allow tailor-made markup for any imaginable application domain Support internationalization (Unicode) and platform independence Be the future of (semi)structured information (do some of the work now done by databases)

Why Use XML? n n Represent semi-structured data (data that are structured, but don’t fit relational model) XML is more flexible than DBs XML is more structured than simple IR You get a massive infrastructure for free

Applications of XML n n XHTML CML – chemical markup language WML – wireless markup language Th. ML – theological markup language n <h 3 class="s 05" id="One. 2. p 0. 2">Having a Humble Opinion of Self</h 3> <p class="First" id="One. 2. p 0. 3">EVERY man naturally desires knowledge <note place="foot" id="One. 2. p 0. 4"> <p class="Footnote" id="One. 2. p 0. 5"><added id="One. 2. p 0. 6"> <name id="One. 2. p 0. 7">Aristotle</name>, Metaphysics, i. 1. </added></p> </note>; but what good is knowledge without fear of God? Indeed a humble rustic who serves God is better than a proud intellectual who neglects his soul to study the course of the stars. <added id="One. 2. p 0. 8"><note place="foot" id="One. 2. p 0. 9"> <p class="Footnote" id="One. 2. p 0. 10"> Augustine, Confessions V. 4. </p> </note></added> </p>

XML Schemas n n n Schema = syntax definition of XML language Schema language = formal language for expressing XML schemas Examples n n n DTD XML Schema (W 3 C) Relevance for XML IR n Our job is much easier if we have a (one) schema

XML Tutorial n n n http: //www. brics. dk/~amoeller/XML/index. h tml (Anders Møller and Michael Schwartzbach) Previous (and some following) slides are based on their tutorial

XML Indexing and Search

Native XML Database n n Uses XML document as logical unit Should support n n n Elements Attributes PCDATA (parsed character data) Document order Contrast with n n DB modified for XML Generic IR system modified for XML

XML Indexing and Search n Most native XML databases have taken an DB approach n n Exact match Evaluate path expressions No IR type relevance ranking Only a few that focus on relevance ranking

Timber: XML as DB extension n DB: search tuples Timber: search trees Main focus n n n Complex and variable structure of trees (vs. tuples) Ordering XML query optimization vs relational optimization

To. Xin n n Native XML database Exploits overall path structure n n Supports any general path query Query evaluation in three stages n n n Preselection stage Selection stage Postselection stage

To. Xin: Motivation n Strawman: n n n Index all paths occurring in database Does not allow backward navigation Example query: n find all the titles of articles from 1990

Query Evaluation Stages n Pre-selection n n Selection n n First navigation down the tree Value selection according to filter Post-selection n Navigation up and down again

To. Xin

Factors Impacting Performance n Data source specific n n n Document size Number of XML nodes and values Path complexity (degree of nesting) Average value size Query specific n n n Selectiveness of path constraint Size of query answer Number of elements selected by filter

Benchmark Parameters

Query Classification

Evaluation

To. Xin: Summary n n Efficient native XML database All paths are indexed (not just from root) Path index linear in corpus size Shortcomings n n Order of nodes ignored Semantics of IDRefs ignored What is missing?

IR/Relevance Ranking for XML n Why is this difficult?

IR XML Challenge 1: Term Statistics n n There is no document unit in XML How do we compute tf and idf? Global tf/idf over all text context is useless Indexing granularity

IR XML Challenge 2: Fragments n n n IR systems don’t store content (only index) Need to go to document for displaying fragment Easier in DB framework

Relevance Ranking for XML n Will revisit next week

Querying XML n n n Semistructured queries XPath XQuery

Types of (Semi)Structured Queries n n Location/position (“chapter no. 3”) Simple attribute/value n n Path queries n n title contains “hamlet” /play//title contains “hamlet” Complex graphs n n /play/title contains “hamlet” Employees with two managers All of the above: mixed structure/content Subsumes: hyperlinks

XPath n Declarative language for n n n Location path n n Addressing (used in XLink/XPointer and in XSLT) Pattern matching (used in XSLT and in XQuery) a sequence of location steps separated by / Example: n child: : section[position()<6] / descendant: : cite / attribute: : href

Axes in XPath n ancestor, ancestor-or-self, attribute, child, descendent-or-self, following-sibling, namespace, parent, preceding-sibling, self

Location steps n n n A single location step has the form: n axis : : node-test [ predicate ] The axis selects a rough set of candidate nodes (e. g. the child nodes of the context node). The node-test performs an initial filtration of the candidates based on their n types (chardata node, processing instruction, n n n etc. ), or names (e. g. element name). The predicates (zero or more) cause a further, potentially more complex, filtration child: : section[position()<6]

XQuery n n SQL for XML Usage scenarios n n Relies on n n Human-readable documents Data-oriented documents Mixed documents (e. g. , patient records) XPath XML Schema datatypes Turing complete XQuery is still a working draft. More than a hundred open issues as of 2002. 11. 10

XQuery n The principal forms of XQuery expressions are: n n n n path expressions element constructors FLWR ("flower") expressions list expressions conditional expressions quantified expressions datatype expressions Evaluated with respect to a context

FLWR n n n FOR $p IN document("bib. xml")//publisher LET $b : = document("bib. xml”)//book[publisher = $p] WHERE count($b) > 100 RETURN $p FOR generates an ordered list of bindings of publisher names to $p LET associates to each binding a further binding of the list of book elements with that publisher to $b at this stage, we have an ordered list of tuples of bindings: ($p, $b) WHERE filters that list to retain only the desired tuples RETURN constructs for each tuple a resulting value

XQuery vs SQL n Order matters! n n document("zoo. xml")//chapter[2]//figure[cap tion = "Tree Frogs"] XQuery is turing complete, SQL is not.

XQuery Example Møller and Schwartzbach

XQuery Standard on Ranking (2. 3. 1) Document order defines a total ordering among all the nodes seen by the language processor. Within a given document, the document node is the first node, followed by element nodes, text nodes, comment nodes, and processing instruction nodes in the order of their representation in the XML form of the document (after expansion of entities). Element nodes occur before their children. The namespace nodes of an element immediately follow the element node, in implementation-defined order. The attribute nodes of an element immediately follow its namespace nodes, and are also in implementationdefined order. n The relative order of nodes in distinct documents is implementation-defined but stable within a given query or transformation. In other words, given two distinct documents A and B, if a node in document A is before a node in document B, then every node in document A is before every node in document B. The relative order

Next Week (12/3) n n n XML indexing and search II Metadata indexing and search Dublin Core, RDF, DAML+OIL