Instant Indexing Greg Lindahl CTO Blekko October 21

Instant Indexing Greg Lindahl CTO, Blekko October 21, 2010 - BCS Search Solutions 2010

Blekko Who? • Founded in 2007, $24 m in funding • Whole-web search engine • Currently in invite-only beta – 3 B page crawl – innovative UI • … but this talk is abut indexing

What whole-web search was • Sort by relevance only • News and blog search done with separate engines • Main index updated slowly with a batch process • Months to weeks update cycle

What web-scale search is now Relevance and date sorting Everything in a single index Incremental updating Live-crawled pages should appear in the main index in seconds • All data stored as tables • •

Instant Search Indexing • /date screnshot

Another Example

Google’s take on the issue • Daniel Peng and Frank Dabek, Large Scale Incremental Processing Using Distributed Transactions and Notifications • “Databases do not meet the storage or throughput requirements of these tasks… Map. Reduce and other batch-processing systems cannot process small updates individually as they rely on creating large batches for efficiency. ”

Percolator details • ACID, with multi-row transactions • triggers ("observers"), can be cascaded • crawler is a cascade of triggers: – – – Map. Reduce writes new documents into bigtable trigger parses and extracts links cascaded trigger does clustering cascaded trigger exports changed clusters 10 triggers total in indexing system • max 1 observer per column for complexity reasons • message collapsing when there are multiple updates to a column

Blekko’s take on this • We want to run the same code in a mapjob or in an incremental crawler/indexer • Our bigtable-like thingie shouldn’t need a percolator-sized addition to do it • Needs to be more efficient than other approaches • OK with non-ACID, relaxed eventual consistentcy, etc

Combinators • Task: gather incoming links and anchortext • Each crawled webpage has dozens of outlinks • Crawler wants to write into dozens of inlists, each in a separate cell in a table • Top. N combinator: list of N highest-ranked items • If a cell is frequently written, writes can be combined before hitting disk

Combining combinators • Combine within the writing process • Combine within the local write daemon • Combine within the 3 disk daemons, and the ram daemon – highly contented cells result in 1 disk transaction per 30 seconds • Combinators are represented as strings and can be used without the database • Using combinators seems to be a significant reduction of RPCs over Percolator, but I have no idea what the relative performance is.

Top. N example • table: /index/32/url row: pbm. com/~lindahl/ column: inlinks – a list of: rank, key, data – 1000, www. disney. com, “great website” – 540, britishmuseum. com/dance, “ 16 th century dance manuals in facsimile” – 1, www. ehow. com/dance, “renaissance dance”

Map. Reduce from a combinator perspective • Map. Reduce is really map, shuffle, reduce • input: a file/table, output: a file/table • An incremental job to do the same Map. Reduce looks completely different; you have to implement the shuffle+reduce • Could write into Big. Table cells…

Map. Jobs+Combinators • Map function runs on shards • All output is done by writing into a table, using combinators • The same map function can also be run incrementally on individual inputs • The shuffle+reduce is still there, it’s just done by the database+combinators

Combinator types • • top. N last. N = top. N, using time as the rank sum, avg, eavg, min, max counting things – logcount: +- 50% count of strings in 16 bytes • set -- everything is a combinator • Cells in our tables are native Perl/Python data structures • hence: atomic updates on a sub-cell level

Combinators for indexing • The basic data structure for search is the posting list: – for each term, a list with rows • docid, rank • Sounds like a custom top. N to us – rank = rank or date or … – lists heavily compressed • Each posting list has N shards

Combinators for crawling • Pick a site, crawl the most important uncrawled pages – that’s stored as a top. N • (the “livecrawl” uses other criteria) • Crawl, parse, and spew writes – outlinks into inlinks cells – page ip/geo into incoming ips, geos – page hashes into duptext detection table – count everything under the sun – 100 s of writes total

Instant index step • Crawler does the indexing • Decides which terms to index based on page contents and incoming anchortext • Writes into posting lists – if indexed before, use list of previously indexed terms to delete any obsolete terms • Heavily-contented posting lists are not a problem due to combining -- that’s how a naked [/date] query works.

Supporting date queries • /date queries fetch about 3 X the posting lists of a relevance query • to support [/health /date], we keep a posting list of the most recent dated pages for each website • date needs some relevance; every date-sorted posting list has a companion date-sorted lists of only highly-relevant articles

Example: [obama /date] • The term posting list for ‘obama’ has overflowed -- moderately relevant dated queries are probably smushed out • The date posting list for ‘obama’ has overflowed • The date posting list for highly-relevant dated ‘obama’ is not full

To Sum Up • There’s more than one way to do it – yes, we use Perl • I don’t think Blekko’s scheme is better or worse than Google’s, but at least it’s very different • See me if you’d like an invite to our beta-test