MOCHA A SelfExtensible Database Middleware System for Distributed

Motivation Data Sources are distributed and heterogeneous: Fact of Life. . . Client Internet Oracle 8 i SIGMOD 2000 Informix XML Data Text Data M. Rodriguez-Martinez – N. Roussopoulos 2

Client-Server Connectivity 2 -tier architecture means FAT Clients Client Not a Good Idea Internet Oracle 8 i SIGMOD 2000 Informix XML Data Text Data M. Rodriguez-Martinez – N. Roussopoulos 3

Middleware Integration Service Middleware is a 3 -tier connectivity solution – Thin Clients Client Integration Server Catalog Internet Translator Oracle 8 i Informix SIGMOD 2000 Translator XML Data Text Data M. Rodriguez-Martinez – N. Roussopoulos 4

Problem 1: Code Deployment • User-defined types and functions – Polygon – Composite() – image aggregation • Porting and manual installation of code – Operating system – Hardware platform • Expensive Software Maintenance – Updates – Version management • Security – Software certification SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 5

Problem 1: Code Deployment Not Scalable – Expensive System Growth Client Integration Server Catalog Internet Translator Oracle 8 i Informix SIGMOD 2000 Translator XML Data Text Data M. Rodriguez-Martinez – N. Roussopoulos 6

Problem 2: Query Processing • Operator placement options – Limited by site-dependent software • Composite() – got to have it before using it! • Most processing at Integration Server – Powerful Data Servers are under-utilized • I/O Nodes – Excessive data movement over the network • Network bottleneck • Unfeasible in WANs, Internet SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 7

Problem 2: Query Processing Not Scalable – Inefficient evaluation of queries Client Integration Server Catalog 100 MB Internet 100 MB Translator XML Data Text Data 100 MB Oracle 8 i SIGMOD 2000 Informix M. Rodriguez-Martinez – N. Roussopoulos 8

MOCHA Solution: Ship Code! Oracle Texas Client SIGMOD 2000 Informix QPC DAP Virginia Code Repository Catalog Virginia DAP Maryland Internet Select location, Composite(image) From Rasters Where week BETWEEN t 1 and t 2 Group By location M. Rodriguez-Martinez – N. Roussopoulos 9

MOCHA Solution: Filter Data! 200 MB 100 MB tuples Code Repository Catalog Oracle DAP 150 KB results Texas 150 KB results 150 KBQPC 200 KB results 350 KB results Virginia Informix tuples 200 KB DAP results 200 KB Maryland Internet results 350 KB results Client Virginia SIGMOD 2000 Select location, Composite(image) From Rasters Where week BETWEEN t 1 and t 2 Group By location M. Rodriguez-Martinez – N. Roussopoulos 10

MOCHA Goals ü Automatic Deployment of Code (self-extensible) – QPC ships compiled Java classes • User-defined types and functions – XML for their metadata (easy exchange) ü Data processing at data source sites – Utilize powerful machines • On-site data distillation ü Processing based on data movement reduction – “Filter” data at the data sources – “Expand” data near the clients SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 11

The MOCHA Architecture Coordination Thread Client Execution Thread QPC Code Repository DAP Informix SIGMOD 2000 Oracle Catalog • Multi-threaded • Distributed Objects M. Rodriguez-Martinez – N. Roussopoulos 12

QPC: The Integration Server QPC Controls and Coordinates Query Execution Client API Query Parser Query Optimizer XML Catalog Code Repository Catalog Manager Execution Engine Proc. Interface SQL & XML Code Loader DAP Access API SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos DAP 13

DAP: The Facilitator of Data DAP Provides QPC with Remote Access to the Data DAP Access API 150 KB Control Module results Execution Engine 100 MB Proc. Interface SQL & Code tuples XML 100 MBLoader Data Source Access Layer tuples JDBC I/O API DOM JNI 100 MB Data Source tuples 100 MB SIGMOD 2000 tuples M. Rodriguez-Martinez – N. Roussopoulos 14

Road Map ü Introduction ü Problem Definition ü MOCHA Architecture • Query Processing • Experiments • Summary SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 15

Processing The Queries • Issue 1: Placement and deployment of operators – Which operators go to QPC, and which go to the DAPs? • Issue 2: How to determine this placement? – Dynamic programming [SAC+79], [ML 86] – But search space is enormous • Placement of UDF, joins, execution sites … • Plenty of “bad” plans In MOCHA: Query Optimization based on heuristics – Network usually is the critical factor optimize for it first – CPU and I/O are cheaper optimize for them later – Quickly converge to a “good” plan SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 16

Operator Placement • Data-Reducing Operators – “Filter” the data – Aggregates, predicates, projections, semi-joins • Composite(), Overlaps() , Avg. Energy() ü Push to the DAPs – Code Shipping policy (Unique to MOCHA) – Only send back distilled results + Less data movement • Cost: – Computation cost – Transfer of filtered results SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 17

Operator Placement • Data-Inflating Operators – “Expand” the data – projections, image processing, some joins … • Double. Resolution(), Rotate. Solid() ü Pull to the QPC – Data Shipping policy [FJK 96] – Only send back raw arguments + Less data movement • Cost: – Computation cost – Transfer of raw argument values SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 18

Placement Metric: VRF Volume Reduction Factor: Given operator and relation R, then • VDT - volume of data transmitted after applying to R • VDA - volume of data originally present in R is Data-Reducing VRF < 1 Composite() SIGMOD 2000 is Data-Inflating VRF 1 Double. Res() M. Rodriguez-Martinez – N. Roussopoulos 19

Goal: Plans with small CVRF Cumulative Volume Reduction Factor: Given a plan P to solve query Q over relations R 1, …, Rn • CVDT - volume of data transmitted by applying all operators in P to R 1, …, Rn • CVDA- volume of data originally present in R 1, …, Rn Search Space Optimizer searches for plans that move minimal amount of data. CVRF(Plan) [0, 1] SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 20

Performance Evaluation • Goals of this study: – Measure how good code shipping can be – Validate heuristics being proposed • VRF • CVRF – Guide implementation of the optimizer • Configured MOCHA with plans that place operators based on heuristics. SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 21

Experimental Environment • Sequoia 2000 Benchmark – scientific data - points, polygons, satellite images – Distributed applications • Software and Hardware: – – JDK 1. 2 QPC - Sun Ultra 60, Solaris 2. 6 DAPs - Sun Ultra 1, Sun Ultra 5, Solaris 2. 6 Data Sources • 2 Informix IUS 9. 12 Server – 10 Mpbs Ethernet SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 22

Reducing vs. Inflating • Query classes ü Performance gains – composites • 99% data reduction • 4 -1 better performance – clipping and expansion • 80% data reduction • 3 -1 better performance ü Validates heuristics SIGMOD 2000 DAP Runnning Time (secs) – Composite of all images – Clipping and sub-setting – Double resolution of images QPC QPC DAP Q 1 Q 2 Q 3 Query Class M. Rodriguez-Martinez – N. Roussopoulos 23

• Consider 50% selectivity – DAP CVRF = 0. 01 – QPC CVRF = 1 SIGMOD 2000 0 . 25 . 50 . 75 DAP QPC • Need to also consider size of arguments for predicates! DAP 5 Selectivity [HS 93] and cardinality [HKWY 97] are not enough for distributed predicate placement QPC • Select graphs identifiers based on number of vertices and arc length Runnning Time (secs) VRF vs Selectivity 1 Selectivity ü VRF is a better metric M. Rodriguez-Martinez – N. Roussopoulos 24

Implementation Status ü Operational System – SIGMOD 2000 Demo ü Experimental deployment of MOCHA – NASA Earth Scientists (ESIP Federation) – Goddard Space Flight Center – NCSA ü Land Cover Visualization Tool SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 25

Summary and Conclusions • Proposed a new Middleware Architecture: MOCHA – Automatic Code Deployment (self-extensible) • Shipping Java classes – Query processing based on data movement reduction • Proposed VRF metric for placement of functions – Better than selectivity and result cardinality • Future work – Deployment of MOCHA for NASA ESIP Federation – Full implementation of MOCHA Optimizer • More Info: – http: //mocha. umiacs. umd. edu/ SIGMOD 2000 M. Rodriguez-Martinez – N. Roussopoulos 26

Problem 2: Query Processing Not Scalable – Inefficient evaluation of queries Client Integration Server Client Catalog 100 MB 200 MB 100 MB Translator 100 MB Oracle 8 i SIGMOD 2000 Internet 200 MB Translator XML Data Text Data 200 MB Informix M. Rodriguez-Martinez – N. Roussopoulos 27