Indexing Indexing The goal of indexing is to
- Slides: 30
Indexing
Indexing • The goal of indexing is to speed up online querying – Retrieval needs to be performed in milliseconds – Without an index, retrieval would require streaming through the collection • Computing static (query independent) ranking features – Example: Page. Rank • Building specialized data structures (indexes)
DB vs. IR-style indexing • The type of indexes you need depend on the query language to support – SPARQL is a highly-expressive SQL-like query language for experts – End-users are accustomed to keyword queries with very limited structure (see Pound et al. WWW 2010) • DB-style indexing – Support for SPARQL queries – Optimized for Read/Write workload • IR-style indexing – Keyword-queries – Read workload
DB-style indexing • Index RDF as data • Triple stores: repositories for RDF data – Sesame, Mulgara, Virtuoso, Redland etc. – Most triple stores can rely on a traditional database backend e. g. My. SQL • Enjoy all the features of a mature database (query optimizations, compression, security, logging, replication etc. ) • But worse performance than native implementations • Difference to databases – Schema may not be known in advance and subject to change – Heterogeneous data – Sparsity
Table layouts • Schema-aware layouts – aka property tables • Triple stores • Vertical partitioning
Schema aware layout (a) – – Classical database layout One table per class and relation Good choice if the schema is known in advance and stable Lot’s of NULLs if the data is sparse Person id name age Pid 1 “Alice” “ 21” Pid 2 “Joe” “ 63” Person id Publication id Pid 1 Publication id title year Pid 2 Pub 1 2005 Pid 2 Pub 2 “The life of chimps” Pub 2 “My life” 2002
Schema aware layout (b) • Same as I(a) but one generic triples table for relations – Implemented in Jena, Oracle Person id name age Pid 1 “Alice” “ 21” Pid 2 “Joe” “ 63” Subject Predicate Object Pid 1 published Pub 1 Pid 2 published Pub 2 Publication id title year Pub 1 “The life of chimps” 2005 Pub 2 “My life” 2002
Triple stores • One table for all triples – Expensive self-joins • Performance can be improved by – Indexing all or some combinations of columns – Dictionary encoding of URIs and strings Subject Predicate Object Pid 1 name “Alice” Pid 1 age “ 21 Pid 1 type Person Pid 2 name “Joe” Pid 2 age “ 63” Pid 1 published Pub 1 Pid 2 type Person Pid 2 published Pub 1 Pid 2 published Pub 2 Pub 1 title “The life of Chimps Pub 1 year 2005
Resolving queries in a triple store SELECT ? name SPARQL query WHERE { ? x name Person. ? x name ? name } SQL query SELECT B. object FROM triples AS A, triples as B WHERE A. subject= B. subject AND A. property= “type” AND A. object= “Person” AND B. predicate= “name”;
Vertical partitioning • • One table per property Nulls are not stored Easy to handle multi-valued properties Only need to read relevant properties Joins are mostly merge-joins Materializing paths can speed up frequent path patterns see Abadi, VLDB 2007 name age published Subject Object Pid 1 “Alice” Pid 1 “ 21 Pid 1 Pub 1 Pid 2 “Joe” Pid 2 “ 63” Pid 2 Pub 1 Pid 2 Pub 2
IR-style indexing • Index data as text – Create virtual documents from data – One virtual document per subgraph, resource or triple • typically: resource • Key differences to Text Retrieval – RDF data is structured – Minimally, queries on property values are required
Virtual documents • Single index – All words – Post-fixing • Multiple indexes – Horizontal indexing – Vertical indexing
All words • Treat the RDF document simply as a sequence of tokens • Works quite well but no support for querying values for a particular property http: //www. example. org/peter http: //xmlns. com/foaf/0. 1/name “Peter Mika”. http: //www. example. org/peter http: //xmlns. com/foaf/0. 1/age “ 32”. http: //www. example. org/peter http: //www. w 3. org/2006/vcard/ns# “Barcelona”. peter xmlns foaf name Peter Mika peter xmlns foaf age 32 peter w 3 2006 vcard ns Barcelona
Post-fixing • This can be achieved without any special index structure by ”post-fixing” – Instead of the term Peter index the term Peter#foaf_name – Prefix queries are needed to search only for Peter ✗ Dictionary is number of unique terms per property – It works well when the number of properties are small • Example: NER indexing with a small number of types – RDF has large number of properties: dictionary explodes • the_name, the_title, the_address, the_org, ….
Horizontal index structure • Two fields (indices): one for terms, one for properties • For each term, store the property on the same position in the property index – Positions are required even without phrase queries • Query engine needs to support the alignment operator ✓ Dictionary is number of unique terms + number of properties ✓ Occurrences is number of tokens * 2
Vertical index structure • One field (index) per property • Positions are not required – But useful for phrase queries • Query engine needs to support fields ✓ Dictionary is number of unique terms ✓ Occurrences is number of tokens ✗ Number of fields is a problem for merging, query performance
Inverted index construction 1. Crawling 2. Data extraction, integration and reasoning 3. Collection parsing – language detection – tokenization – stemming, – stop word removal
Inverted index construction II. 4. For each partition of documents to be indexed – Build inverted-index in memory – Write out each partition 5. Merge blocks – Merging sorted lists • Encoding and compression techniques reduce the size of the dictionary and posting lists – Dictionary needs to fit in memory – Minimize read time from disk
Indexing using Map. Reduce • Map. Reduce is the perfect model for building inverted indices – Map creates (term, {doc 1}) pairs – Reduce collects all docs for the same term: (term, {doc 1, doc 2…} – Sub-indices are merged separately • Term-partitioned indices • Build indices using any IR library – Katta project for Lucene
Implementation • Mika. Distributed Indexing for Semantic Search. Sem. Search 2010. • Reality is more complex than the textbooks… – Hashed subject URIs using MG 4 J’s Minimal Perfect Hash • hash function occupies only 307 MB • Fits in memory for now, only required for map – Implemented fields, positions: keys are (field, term) pairs, values are (docid, position) pairs – For each term, documents need to be indexed in increasing order of docid • Secondary sort by making value part of the key • Increased amount of data
Implementation • Document frequency needs to be known when starting to write out the posting list – Introduced dummy occurrences, one for each document – Set docid to -1 to make sure dummy occurrences come first • Memory problems with unbalanced executions (too many documents for a single term) • Memory problems with large number of indices (index caching) • Trade-offs between how much memory is left for our job vs. memory for the system itself
Example: Semplore • Zhang et al. (IBM China and Shanghai Jiao. Tong Univ. ): Semplore: An IR Approach to Scalable Hybrid Query of Semantic Web Data • Hybrid query capability – Added: virtual “keyword” concepts that can be combined with other formal concepts in ontology – Limited to tree-shaped queries with a single query target • Efficient indexing by extending an IR engine (Lucene) – Transform RDF triples to artificial documents with fields and terms – Optimized by the IR engine (e. g. index compression)
Hybrid Query Capability • Example: Find directors who have directed films that are American films and are about war, and have directed at least one romantic comedy starring both a Best Actor and a Best Actress winner. Best Actor Academy Award Winner “romantic” directs starring y 1 starring Film. Director y 2 Best Actress Academy Award Winner y 3 x directs American. Film AND“war” y 4 © 2007 IBM Corporation
Query Evaluation • Example Best Actor s 3=(type, Best. Actor) Academy Award Winner ⋈ (s 2, starring, s 3) “romantic” s 2=(text, “romantic”) starring ⋈ (s 1, directs, s 2) directs y 1 starring s 1=(type, Film. Director) Film. Director y 2 s 2 = ⋈ (s 3, starring¯, s 2) Best Actress Academy Award Winner y 3 x directs American. Film u “war” y 4 DFS the tree and conduct the basic operations on AIS. © 2007 IBM Corporation
Index Structure: The Virtual Documents Document Concept C Relation R Individual i Field Term sub. Con. Of Super concepts of C super. Con. Of Sub concepts of C text Terms in textual properties of C sub. Rel. Of Super relations of R super. Rel. Of Sub relations of R text Terms in textual properties of R type Concepts that i belongs to subj. Of All relations R that (i, R, ? ) exists obj. Of All relations R that (? , R, i) exists text Terms in textual properties of i © 2007 IBM Corporation
Indexing the Artificial Documents • Feed the artificial documents to existing IR engine • Reuse existing IR engine to index the data • The result index structure (logical) For a given subject and a relation, all the objects are stored as the subject’s artificial document’s positions. © 2007 IBM Corporation
Example: Sindice • Oren et al. : Sindice. com: A Document-oriented Lookup Index for Open Linked Data • Crawling Linked Data and extracting microformats, RDFa from well-known sources – – Implements politeness policies Uses Yahoo BOSS to discover additional sources Accepts pings through Ping. The. Semantic. Web. com Supports Semantic Sitemaps • Limited reasoning during indexing – ‘Sandboxed’ reasoning: the ontologies of each document are loaded separately into the reasoner – Inverse-Functional Properties (IFPs) such as email addresses are identified and indexed in a separate index
Sindice • Siren indexing engine – Open source extension to Lucene – New field type “tuples” • Index (s, p, o) + (o, rdfs: label, l) as (p, o, l) • Tokenize URIs – Expressivity: queries where the subject is the variable, e. g. – (*, name, "renaud delbru") AND (*, workplace, deri) – Ranking “comparable to TF-IDF” without global features (such as Page. Rank) • Efficiency – Example: 1. 5 B triples, 4 nodes, ~22 hours of indexing
Demo
Example: Sindice and Sig. ma • Sindice is a Semantic Web Search engine – Indexing Linked Data and some microformat/RDFa data • Uses Yahoo’s BOSS API to find more at query time – Lucene-based engine (Siren) • Sig. ma: interactive interface for Sindice – Clean up search results by removing irrelevant results and untrustworthy sources – Share cleaned-up search result pages
- Precoordinate
- Contoh pengindeksan manual
- Gấu đi như thế nào
- Thiếu nhi thế giới liên hoan
- Phối cảnh
- điện thế nghỉ
- Một số thể thơ truyền thống
- Thế nào là hệ số cao nhất
- Ng-html
- Sơ đồ cơ thể người
- Số nguyên tố là
- đặc điểm cơ thể của người tối cổ
- Mật thư anh em như thể tay chân
- Các châu lục và đại dương trên thế giới
- Chụp tư thế worms-breton
- ưu thế lai là gì
- Thẻ vin
- Tư thế ngồi viết
- Cái miệng bé xinh thế chỉ nói điều hay thôi
- Các châu lục và đại dương trên thế giới
- Từ ngữ thể hiện lòng nhân hậu
- Bổ thể
- Tư thế ngồi viết
- V cc
- Thể thơ truyền thống
- Chúa yêu trần thế
- Khi nào hổ mẹ dạy hổ con săn mồi
- đại từ thay thế
- Diễn thế sinh thái là
- Vẽ hình chiếu vuông góc của vật thể sau
- Làm thế nào để 102-1=99