New Generation Database Systems IR Systems and the

New Generation Database Systems: IR Systems and the Grid/Cloud University of California, Berkeley School of Information IS 257: Database Management IS 257 – Fall 2010. 11. 16 - SLIDE 1

Lecture Outline • XML and DBMS – Cheshire II as XML Database • The Grid and DBMS – The Grid – Data Grids – Grid-based DBMS IS 257 – Fall 2010. 11. 16 - SLIDE 2

Lecture Outline • XML and DBMS – Cheshire II as XML Database • The Grid and DBMS – The Grid – Data Grids – Grid-based DBMS IS 257 – Fall 2010. 11. 16 - SLIDE 3

Standards: XML/SQL • That table can be mapped to: <EMPLOYEE> <row><EMPNO>000020</EMPNO> <FIRSTNAME>John</FIRSTNAME> <LASTNAME>Smith</LASTNAME> <BIRTHDATE>1955 -08 -21</BIRTHDATE> <SALARY>52300. 00</SALARY> </row> <row> … etc. … IS 257 – Fall 2010. 11. 16 - SLIDE 4

The following slides are adapted from: XML to Relational Database Mapping Bhavin Kansara Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 5

Introduction • XML/relational mapping means data transformation between XML and relational data models • XML documents can be transformed to relational data models or vice versa. • Mapping method is the way the mapping is done Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 6

DTD graph Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 7

Inlined DTD graph • Given a DTD graph, a node is inlinable if and only if it has exactly one incoming edge and that edge is a normal edge. Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 8

Inlined DTD graph Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 9

Generated Database Schema Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 10

Data Mapping • XML file is used to insert data into generated database schema • Parser is used to fetch data from XML file. Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 11

Summary • • Simplify DTD Create DTD graph from simplified DTD Create inlined DTD graph from DTD graph Use inlined DTD graph to generate database schema • Insert values from XML file into generated tables Slide from Bhavin Kansara IS 257 – Fall 2010. 11. 16 - SLIDE 12

Issues • So, we can convert the XML to a relational database, but can we then export as an XML document? – This is equally challenging • But MOSTLY involves just re-joining the tables • How do you store and put back the wrapping tags for sets of subelements? • Since the decomposition of the DTD was approximate, the output MAY not be identical to the input IS 257 – Fall 2010. 11. 16 - SLIDE 13

Anatomy of a Native XML database • The next set of slides (available on the class web site) come from George Feinberg of Sleepy. Cat Software – Sleepy. Cat is now part of Oracle IS 257 – Fall 2010. 11. 16 - SLIDE 14

Further comments on NXD • Native XML databases are most often used for storing “document-centric” XML document – I. e. the unit of retrieval would typically be the entire document and not a particular node or subelement • This supports query languages like Xquery – Able to ask for “all documents where third chapter contains a page that has boldfaced word” – Very difficult to do that kind of query in SQL IS 257 – Fall 2010. 11. 16 - SLIDE 15

XML-Based IR - Cheshire II • I thought I would take a little time to talk about how the Cheshire system (that I have been working for nearly 20 years) uses XML, since it has some similarities (and many differences) to XML database systems • Cheshire II (and Cheshire 3) are document-centric and involve parsing the XML for the purposes of indexing (and sometimes for retrieval of partial documents) IS 257 – Fall 2010. 11. 16 - SLIDE 16

Cheshire II SGML/XML Support • Underlying native format for all data is SGML or XML • The DTD defines the file format for each file • Full SGML/XML parsing • SGML/XML Format Configuration Files define the database • USMARC DTD and MARC to SGML conversion (and back again) • Access to full-text via special SGML/XML tags 2010. 11. 16 - SLIDE 17

SGML/XML Support • Example XML record for a DL document <ELIB-BIB> <BIB-VERSION>ELIB-v 1. 0</BIB-VERSION> <ID>756</ID> <ENTRY>June 12, 1996</ENTRY> <DATE>June 1996</DATE> <TITLE>Cumulative Watershed Effects: Applicability of Available Methodologies to the Sierra Nevada</TITLE> <ORGANIZATION>University of California</ORGANIZATION> <TYPE>report</TYPE> <AUTHOR-INSTITUTIONAL>USDA Forest Service</AUTHOR-INSTITUTIONAL> <AUTHOR-PERSONAL>Neil H. Berg</AUTHOR-PERSONAL> <AUTHOR-PERSONAL>Ken B. Roby</AUTHOR-PERSONAL> <AUTHOR-PERSONAL>Bruce J. Mc. Gurk</AUTHOR-PERSONAL> <PROJECT>SNEP</PROJECT> <SERIES>Vol 3</SERIES> <PAGES>40</PAGES> <TEXT-REF>/elib/data/docs/0700/756/HYPEROCR/hyperocr. html</TEXT-REF> <PAGED-REF>/elib/data/docs/0700/756/OCR-ASCII-NOZONE</PAGED-REF> </ELIB-BIB> IS 257 – Fall 2010. 11. 16 - SLIDE 18

SGML Support • Example SGML/MARC Record <USMARC Material="BK" ID="00000003"><leader><LRL>00722</LRL>< Rec. Stat>n</Rec. Stat> <Rec. Type>a</Rec. Type><Bib. Level>m</Bib. Level><UCP></UCP><Ind. Count>2</Ind. Count> <SFCount>2</SFCount><Base. Addr>00229</Base. Addr><Enc. Level> </Enc. Level> <Dsc. Cat. Fm></Dsc. Cat. Fm><Link. Rec></Link. Rec><Entry. Map><FLength>4</Flength><SChar. Pos> 5</SChar. Pos><IDLength>0</IDLength><EMUCP></Entry. Map></Leader> <Directry>001001400000005001700014008004100031010001400072035002000086035001700106100001900123 2450105001422500011002472600032002583000033002905040050003236500036003737000022004097000022004 31950003200453998000700485< /Directry><Var. Flds> <Var. CFlds><Fld 001>CUBGGLAD 1282 B</Fld 001><Fld 005>19940414143202. 0</Fld 005> <Fld 008>830810 1983 nyu eng u</Fld 008></Var. CFlds> <Var. DFlds><Numb. Code><Fld 010 I 1="Blank" I 2="Blnk"><a>82019962 </a></Fld 010> <Fld 035 I 1="Blank" I 2="Blnk"><a>(CU)ocm 08866667</a></Fld 035><Fld 035 I 1="Blank" I 2="Blnk"><a>(CU)GLAD 1282</a></Fld 035>< /Numb. Code><Main. Enty><Fld 100 Name. Type="Single" I 2=""><a>Burch, John G. </a></Fld 100>< /Main. Enty><Titles><Fld 245 Add. Enty="Yes" NFChars="0"><a>Information systems : </a><b>theory and practice /</b><c>John G. Burch, Jr. , Felix R. Strater, Gary Grudnitski</c></Fld 245></Titles><Ed. Imprnt><Fld 250 I 1="Blank" I 2="Blnk"><a>3 rd ed</a></Fld 250><Fld 260 I 1="" I 2="Blnk"><a>New York : </a><b>J. Wiley, </b><c>1983</c></Fld 260>< /Ed. Imprnt><Phys. Desc><Fld 300 I 1="Blank" I 2="Blnk"><a>xvi, 632 p. : </a><b>ill. ; </b><c>24 cm</c></Fld 300>< /Phys. Desc><Series></Series><Notes><Fld 504 I 1="Blank" I 2="Blnk"><a>Includes bibliographical references and index</a></Fld 504></Notes>< Subj. Accs><Fld 650 Subj. Lvl="No. Info" Subj. Sys="LCSH" ><a>Management information systems. </a></Fld 650>. . . 2010. 11. 16 - SLIDE 19

SGML Support • Mini-TREC document… <DOC> <DOCNO>FT 931 -3566</DOCNO> <PROFILE>_AN-DCPCCAA 3 FT</PROFILE> <DATE>930316 </DATE> <HEADLINE> FT 16 MAR 93 / Italy's Corruption Scandal: Magistrates hold key to unlocking Tangentopoli - They will set the investigation agenda </HEADLINE> <BYLINE> By ROBERT GRAHAM </BYLINE> <TEXT> OVER the weekend the Italian media felt obliged to comment on a non-event. No new arrests had taken place in any of the country's ever more numerous corruption scandals which centre on the illicit funding of political parties. . . </TEXT> <XX> … IS 257 – Fall 2010. 11. 16 - SLIDE 20

… Companies: </XX> <CO>Ente Nazionale Idrocarburi. Ente Nazionale per L'Energia Electtrica. Ente Partecipazioni E Finanziamento Industria Manifatturiera. IRI Istituto per La Ricostruzione Industriale. </CO> <XX> Countries: </XX> <CN>ITZ Italy, EC. </CN> <XX> Industries: </XX> <IN>P 9222 Legal Counsel and Prosecution. P 91 Executive, Legislative and General Government. P 13 Oil and Gas Extraction. P 9631 Regulation, Administration of Utilities. P 6719 Holding Companies, NEC. </IN> <XX> Types: </XX> … IS 257 – Fall 2010. 11. 16 - SLIDE 21

… <TP>CMMT Comment & Analysis. GOVT Legal issues. </TP> <PUB>The Financial Times </PUB> <PAGE> London Page 4 </PAGE> </DOC> IS 257 – Fall 2010. 11. 16 - SLIDE 22

SGML/XML Support • Configuration files for the Server are also SGML/XML: – They include tags describing all of the data files and indexes for the database. – They also include instructions on how data is to be extracted for indexing and how Z 39. 50 attributes map to the indexes for a given database. 2010. 11. 16 - SLIDE 23

Cheshire Configuration Files     <DBCONFIG> <DBENV>/projects/is 240/Group. X/indexes </DBENV>   <FILEDEF TYPE=SGML> <DEFAULTPATH>/projects/is 240/Group. X </DEFAULTPATH>  <FILETAG> trec </FILETAG>  <FILENAME> /projects/is 240/ft </FILENAME> <CONTINCLUDE> /projects/is 240/ft. CONT </CONTINCLUDE>  <FILEDTD> /projects/is 240/TREC. FT. DTD </FILEDTD>  <ASSOCFIL> ft. assoc </ASSOCFIL>  <HISTORY> cheshire_index/TESTDATA. history </HISTORY> … IS 257 – Fall 2010. 11. 16 - SLIDE 24

<INDEXES>     <INDEXDEF ACCESS=BTREE EXTRACT=KEYWORD NORMAL=NONE PRIMARYKEY=IGNORE> <INDXNAME> cheshire_index/trec. docno. index </INDXNAME> <INDXTAG> docno </INDXTAG> <INDXMAP> <USE> 12 </USE><struct> 1 </struct> </INDXMAP> <USE> 12 </USE><struct> 2 </struct> </INDXMAP> <USE> 12 </USE><struct> 6 </struct> </INDXMAP> <INDXKEY> <TAGSPEC> <FTAG>DOCNO </FTAG> </TAGSPEC> </INDXKEY> </INDEXDEF> … IS 257 – Fall 2010. 11. 16 - SLIDE 25

<INDEXDEF ACCESS=BTREE EXTRACT=KEYWORD_PROXIMITY NORMAL=STEM> <INDXNAME> cheshire_index/trec. topic. index </INDXNAME> <INDXTAG> topic </INDXTAG> <INDXMAP> <USE> 29 </USE><POSIT> 3 </posit> <struct> 6 </struct> </INDXMAP> <USE> 29 </USE><RELAT> 102 </RELAT><POSIT> 3 </posit> <struct> 6 </struct> </INDXMAP> … <STOPLIST> cheshire_index/topicstoplist </STOPLIST> <INDXKEY> <TAGSPEC> <FTAG>HEADLINE </FTAG> <FTAG>DATELINE </FTAG> <FTAG>BYLINE </FTAG> <FTAG>TEXT </FTAG> </TAGSPEC> </INDXKEY> </INDEXDEF> IS 257 – Fall 2010. 11. 16 - SLIDE 26

Cluster Definitions   <CLUSTER> <clusname> classcluster </clusname> <cluskey normal=CLASSCLUS> <tagspec> <FTAG>FLD 950 </FTAG> <s> â </s> </tagspec> </cluskey> <stoplist> /usr 3/cheshire 2/data 2/clasclusstoplist </stoplist> <clusmap> <from> <tagspec> <ftag>FLD 245</ftag><s>^[ab]</s> <ftag>FLD 440</ftag><s>â</s> <ftag>FLD 490</ftag><s>â</s> <ftag>FLD 830</ftag><s>â</s> <ftag>FLD 740</ftag><s>â</s> </tagspec></from> <to> <tagspec> <ftag>titles</ftag> </tagspec></to> <from> <tagspec> <ftag>FLD 6. . </ftag><s>^[abcdxyz]</s> </tagspec></from> <to> <tagspec> <ftag>subjects</ftag> </tagspec></to> <summarize> <maxnum> 5 </maxnum> <tagspec> <ftag>subjsum</ftag> </tagspec></summarize> </clusmap> </CLUSTER> IS 257 – Fall 2010. 11. 16 - SLIDE 27

Component Definitions <COMPONENTS> <COMPONENTDEF> <COMPONENTNAME> TESTDATA/COMPONENT_DB 1 </COMPONENTNAME> <COMPONENTNORM>NONE</COMPONENTNORM> <COMPSTARTTAG> <TAGSPEC> <FTAG>mainenty </FTAG> <FTAG>titles </FTAG> </TAGSPEC> </COMPSTARTTAG> <COMPENDTAG> <TAGSPEC><FTAG>Fld 300 </FTAG></TAGSPEC> </COMPENDTAG> <COMPONENTINDEXES>  <INDEXDEF ACCESS=BTREE EXTRACT=KEYWORD NORMAL=NONE> <INDXNAME> TESTDATA/comp 1 index 1. author … </INDEXDEF> </COMPONENTS> IS 257 – Fall 2010. 11. 16 - SLIDE 28

Result Formatting (Display) <DISPOPTIONS> KEEP_ENTITIES </DISPOPTIONS> <DISPLAY> <FORMAT NAME="B" OID="1. 2. 840. 10003. 5. 105" DEFAULT> <convert function="TAGSET-G"> <clusmap> <from> <tagspec> <ftag>DOCNO</ftag> </tagspec></from> <to> <tagspec> <ftag>28</ftag> </tagspec></to> <from> <tagspec> <ftag>#DOCID#</ftag> </tagspec></from> <to> <tagspec> <ftag>5</ftag> </tagspec></to> </clusmap> </convert> </FORMAT> </DISPLAY> IS 257 – Fall 2010. 11. 16 - SLIDE 29

Indexing • Any SGML/XML tagged field or attribute can be indexed: – B-Tree and Hash access via Berkeley DB (Sleepycat) – Stemming, keyword, exact keys and “special keys” – Mapping from any Z 39. 50 Attribute combination to a specific index – Underlying postings information includes term frequency for probabilistic searching. – SGML may include address of full-text for indexing • New indexes can be easily added, or old ones deleted 2010. 11. 16 - SLIDE 30

Database Storage • All data stored as SGML/XML flat text files or in a Berkeley. DB database • File format is defined though SGML/XML DTD (also flat text file) or XML Schema • “Associator” files provide indexed direct access to each record in SGML/XML files. – Contain offset and record length for each “record” – Associators can be built to index any conformant document in a directory sub-tree 2010. 11. 16 - SLIDE 31

Database Storage Index File Prox data File Page Data File Postings File Index File Associator File Index File Remote RDBMS Config File History File Associator File SGML /XML File Cluster File DTD File 2010. 11. 16 - SLIDE 32

Client/Server Architecture • Server Supports: – – – Database storage Indexing Z 39. 50 access to local data Boolean and Probabilistic Searching Relevance Feedback External SQL database support • Client Supports: – Programmable (Tcl/Tk – Python in C 3) Graphical User Interface – Z 39. 50 access to remote servers – SGML & MARC formatting • Combined Client/Server CGI scripting via Web. Cheshire 2010. 11. 16 - SLIDE 33

Z 39. 50 Overview UI Map Query Map Results Map Query Search Engine Internet Map Query Map Results UI Map Results IS 257 – Fall 2010. 11. 16 - SLIDE 34

Lecture Outline • XML and DBMS • The Grid and DBMS – The Grid – Data Grids – Grid-based DBMS IS 257 – Fall 2010. 11. 16 - SLIDE 35

Grid-based Digital Libraries • • • So what’s this Grid thing anyhow? Data Grids and Distributed Storage Grid-Based IR Grid-Based Digital Libraries Grid vs “Cloud” This lecture borrows heavily from presentations by Ian Foster (Argonne National Laboratory & University of Chicago), Reagan Moore and others from San Diego Supercomputer Center IS 257 – Fall 2010. 11. 16 - SLIDE 36

Quality, economies of scale The Grid: On-Demand Access to Electricity Source: Ian Foster IS 257 – Fall 2010 Time 2010. 11. 16 - SLIDE 37

By Analogy, A Computing Grid • Decouples production and consumption – Enable on-demand access – Achieve economies of scale – Enhance consumer flexibility – Enable new devices • On a variety of scales – Department – Campus – Enterprise – Internet IS 257 – Fall 2010 Source: Ian Foster 2010. 11. 16 - SLIDE 38

What is the Grid? “The short answer is that, whereas the Web is a service for sharing information over the Internet, the Grid is a service for sharing computer power and data storage capacity over the Internet. The Grid goes well beyond simple communication between computers, and aims ultimately to turn the global network of computers into one vast computational resource. ” Source: The Global Grid Forum IS 257 – Fall 2010. 11. 16 - SLIDE 39

Not Exactly a New Idea … • “The time-sharing computer system can unite a group of investigators …. one can conceive of such a facility as an … intellectual public utility. ” – Fernando Corbato and Robert Fano , 1966 • “We will perhaps see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country. ” Len Kleinrock, 1967 Source: Ian Foster IS 257 – Fall 2010. 11. 16 - SLIDE 40

But, Things are Different Now • Networks are far faster (and cheaper) – Faster than computer backplanes • “Computing” is very different than pre-Net – Our “computers” have already disintegrated – E-commerce increases size of demand peaks – Entirely new applications & social structures • We’ve learned a few things about software Source: Ian Foster IS 257 – Fall 2010. 11. 16 - SLIDE 41

Computing isn’t Really Like Electricity • I import electricity but must export data • “Computing” is not interchangeable but highly heterogeneous: data, sensors, services, … • This complicates things; but also means that the sum can be greater than the parts – Real opportunity: Construct new capabilities dynamically from distributed services • Raises three fundamental questions – Can I really achieve economies of scale? – Can I achieve Qo. S across distributed services? – Can I identify apps that exploit synergies? Source: Ian Foster IS 257 – Fall 2010. 11. 16 - SLIDE 42

Why the Grid? (1) Revolution in Science • Pre-Internet – Theorize &/or experiment, alone or in small teams; publish paper • Post-Internet – Construct and mine large databases of observational or simulation data – Develop simulations & analyses – Access specialized devices remotely – Exchange information within distributed multidisciplinary teams Source: Ian Foster IS 257 – Fall 2010. 11. 16 - SLIDE 43

Why the Grid? (2) Revolution in Business • Pre-Internet – Central data processing facility • Post-Internet – Enterprise computing is highly distributed, heterogeneous, inter-enterprise (B 2 B) – Business processes increasingly computing- & data-rich – Outsourcing becomes feasible => service providers of various sorts Source: Ian Foster IS 257 – Fall 2010. 11. 16 - SLIDE 44

The Information Grid Imagine a web of data • Machine Readable – Search, Aggregate, Transform, Report On, Mine Data – using more computers, and less humans • Scalable – Machines are cheap – can buy 50 machines with 100 Gb or memory and 100 TB disk for under $100 K, and dropping – Network is now faster than disk • Flexible – Move data around without breaking the apps Source: IS 257 – Fall 2010 S. Banerjee, O. Alonso, M. Drake - ORACLE 2010. 11. 16 - SLIDE 45

The Foundations are Being Laid Edinburgh Glasgow DL Belfast Newcastle Manchester Cambridge Oxford Cardiff RAL Hinxton London Soton Tier 0/1 facility Tier 2 facility Tier 3 facility 10 Gbps link 2. 5 Gbps link 622 Mbps link Other link IS 257 – Fall 2010. 11. 16 - SLIDE 46

Data Grid Problem • “Enable a geographically distributed community [of thousands] to pool their resources in order to perform sophisticated, computationally intensive analyses on Petabytes of data” • Note that this problem: – Is common to many areas of science – Overlaps strongly with other Grid problems IS 257 – Fall 2010. 11. 16 - SLIDE 47

Data Grids for High Energy Physics ~PBytes/sec Online System ~100 MBytes/sec ~20 TIPS There are 100 “triggers” per second Each triggered event is ~1 MByte in size ~622 Mbits/sec or Air Freight (deprecated) France Regional Centre Spec. Int 95 equivalents Offline Processor Farm There is a “bunch crossing” every 25 nsecs. Tier 1 1 TIPS is approximately 25, 000 Tier 0 Germany Regional Centre Italy Regional Centre ~100 MBytes/sec CERN Computer Centre Fermi. Lab ~4 TIPS ~622 Mbits/sec Tier 2 ~622 Mbits/sec Institute ~0. 25 TIPS Physics data cache Institute ~1 MBytes/sec Tier 4 Caltech ~1 TIPS Tier 2 Centre ~1 TIPS Physicists work on analysis “channels”. Each institute will have ~10 physicists working on one or more channels; data for these channels should be cached by the institute server Physicist workstations Image courtesy Harvey Newman, Caltech IS 257 – Fall 2010. 11. 16 - SLIDE 48

Increased functionality, standardization Grids and Open Standards App-specific Services Open Grid Services Arch Web services X. 509, LDAP, FTP, … Custom solutions Globus Toolkit GGF: OGSI, … (+ OASIS, W 3 C) Multiple implementations, including Globus Toolkit Defacto standards GGF: Grid. FTP, GSI Time IS 257 – Fall 2010. 11. 16 - SLIDE 49

The Grid as Enabler of 21 st Century Science • Entirely new approaches to enquiry based on – Deep analysis of huge quantities of data – Interdisciplinary collaboration – Large-scale simulation – Smart instrumentation • Enabled by an infrastructure that enables access to, and integration of, resources & services without regard for location IS 257 – Fall 2010. 11. 16 - SLIDE 50

Not only Science… • The Database world is moving to the Grid for large-scale applications • Oracle 10 g is specifically designed to exploit clustered/grid computing using RACs (Real Application Clusters) • An example from the Information/Publishing world… – Presentation from Oracle about Thomson Legal’s use of Oracle 10 g and RACs IS 257 – Fall 2010. 11. 16 - SLIDE 51