Introduction to Query Processing and Query Optimization Techniques

  • Slides: 77
Download presentation
 Introduction to Query Processing and Query Optimization Techniques 1

Introduction to Query Processing and Query Optimization Techniques 1

 Introduction to Query Processing m Query optimization: q The process of choosing a

Introduction to Query Processing m Query optimization: q The process of choosing a suitable execution strategy for processing a query. q Amongst all equivalent evaluation plans choose the one with lowest cost. q Cost is estimated using statistical information from the database catalog Ø e. g. number of tuples in each relation, size of tuples m Two internal representations of a query: q Query Tree q Query Graph 2

 Basic Steps in Query Processing 3

Basic Steps in Query Processing 3

 Basic Steps in Query Processing m Select balance From account where balance <

Basic Steps in Query Processing m Select balance From account where balance < 2500 q balance 2500( balance(account)) is equivalent to q m balance( balance 2500(account)) Annotated relational algebra expression specifying detailed evaluation strategy is called an evaluation-plan. q E. g. , can use an index on balance to find accounts with balance 2500, q or can perform complete relation scan and discard accounts with balance 2500 4

 Measures of Query Cost m Cost is generally measured as total elapsed time

Measures of Query Cost m Cost is generally measured as total elapsed time for answering query. Many factors contribute to time cost q disk accesses, CPU, or even network communication m Typically disk access is the predominant cost, and is also relatively easy to estimate. m For simplicity we just use number of block transfers from disk as the cost measure m Costs depends on the size of the buffer in main memory q Having more memory reduces need for disk access 5

Translating SQL Queries into Relational Algebra m m Query Block: q The basic unit

Translating SQL Queries into Relational Algebra m m Query Block: q The basic unit that can be translated into the algebraic operators and optimized. A query block contains a single SELECT-FROMWHERE expression, as well as GROUP BY and HAVING clause if these are part of the block. Nested queries within a query are identified as separate query blocks Aggregate operators in SQL must be included in the extended algebra. 6

 Translating SQL Queries into Relational Algebra SELECT FROM WHERE LNAME, FNAME EMPLOYEE SALARY

Translating SQL Queries into Relational Algebra SELECT FROM WHERE LNAME, FNAME EMPLOYEE SALARY > ( SELECT FROM WHERE LNAME, FNAME EMPLOYEE SALARY > C πLNAME, FNAME (σSALARY>C(EMPLOYEE)) MAX (SALARY) EMPLOYEE DNO = 5); MAX (SALARY) EMPLOYEE DNO = 5 ℱMAX SALARY (σDNO=5 (EMPLOYEE)) 7

 Algorithms for External Sorting m Sorting is needed in q Order by, join,

Algorithms for External Sorting m Sorting is needed in q Order by, join, union, intersection, distinct, … m For relations that fit in memory, techniques like quicksort can be used. m External sorting: q Refers to sorting algorithms that are suitable for large files of records stored on disk that do not fit entirely in main memory, such as most database files. m For relations that don’t fit in memory, external sort-merge is a good choice 8

 External Sort-Merge m m External sort-merge algorithm has two steps: q Partial sort

External Sort-Merge m m External sort-merge algorithm has two steps: q Partial sort step, called runs. q Merge step, merges the sorted runs. Sort-Merge strategy: q Starts by sorting small subfiles (runs) of the main file and then merges the sorted runs, creating larger sorted subfiles that are merged in turn. Sorting phase: q Sorts n. B pages at a time Ø n. B = # of main memory pages buffer q creates n. R = b/n. B initial sorted runs on disk Ø b = # of file blocks (pages) to be sorted Sorting Cost = read b blocks + write b blocks = 2 b 9

 External Sort-Merge m Example: q n. B = 5 blocks and file size

External Sort-Merge m Example: q n. B = 5 blocks and file size b = 1024 blocks, then q n. R = (b/n. B) = 1024/5 = 205 initial sorted runs each of size 5 bocks (except the last run which will have 4 blocks) n. B = 2, b = 7, n. R = b/n. B = 4 run 10

 External Sort-Merge m Sort Phase: creates n. R sorted runs. i = 0

External Sort-Merge m Sort Phase: creates n. R sorted runs. i = 0 Repeat Read next n. B blocks into RAM Sort the in-memory blocks Write sorted data to run file Ri i = i + 1 Until the end of the relation n. R = i 11

 External Sort-Merge m Merging phase: q The sorted runs are merged during one

External Sort-Merge m Merging phase: q The sorted runs are merged during one or more passes. q The degree of merging (d. M) is the number of runs that can be merged in each pass. q d. M = Min (n. B-1, n. R) q n. P = (logd. M(n. R)) Ø n. P: number of passes. m In each pass, q One buffer block is needed to hold one block from each of the runs being merged, and q One block is needed for containing one block of the merged result. 12

 External Sort-Merge m m m We assume (for now) that n. R n.

External Sort-Merge m m m We assume (for now) that n. R n. B. Merge the runs (n. R -way merge). q Use n. R blocks of memory to buffer input runs, and 1 block to buffer output. Merge n. R Runs Step Read 1 st block of each n. R runs Ri into its buffer page Repeat Select 1 st record (sort order) among n. R buffer pages Write the record to the output buffer. If the output buffer is full write it to disk Delete the record from its input buffer page If the buffer page becomes empty then read the next block (if any) of the run into the buffer Until all input buffer pages are empty 13

 External Sort-Merge m m Merge n. B - 1 Runs Step If n.

External Sort-Merge m m Merge n. B - 1 Runs Step If n. R n. B, several merge passes are required. q merge a group of contiguous n. B - 1 runs using one buffer for output q A pass reduces the number of runs by a factor of n. B - 1, and creates runs longer by the same factor. Ø E. g. If n. B = 11, and there are 90 runs, one pass reduces the number of runs to 9, each 10 times the size of the initial runs q Repeated passes are performed till all runs have been merged into one 14

 External Sort-Merge m Degree of merging (d. M) q # of runs that

External Sort-Merge m Degree of merging (d. M) q # of runs that can be merged together in each pass = min (n. B - 1, n. R) q m Number of passes n. P = (logd. M(n. R)) In our example q d. M = 4 (four-way merging) Ø min (n. B-1, n. R) = min(5 -1, 205) = 4 q Number of passes n. P = (logd. M(n. R)) = (log 4(205)) = 4 Ø First pass: – 205 initial sorted runs would be merged into 52 sorted runs Ø Second pass: – 52 sorted runs would be merged into 13 Ø Third pass: – 13 sorted runs would be merged into 4 Ø Fourth pass: – 4 sorted runs would be merged into 1 15

 External Sort-Merge Blocking factor bfr = 1 record, n. B = 3, b

External Sort-Merge Blocking factor bfr = 1 record, n. B = 3, b = 12, n. R = 4, d. M = min(3 -1, 4) = 2 16

 External Sort-Merge 17

External Sort-Merge 17

 External Sort-Merge m m External Sort-Merge: Cost Analysis Disk accesses for initial run

External Sort-Merge m m External Sort-Merge: Cost Analysis Disk accesses for initial run creation (sort phase) as well as in each merge pass is 2 b q reads every block once and writes it out once m Initial # of runs is n. R = b/n. B and # of runs decreases by a factor of n. B - 1 in each merge pass, then the total # of merge passes is np = logd. M(n. R) m In general, the cost performance of Merge-Sort is q Cost = sort cost + merge cost q Cost = 2 b + 2 b * np q Cost = 2 b + 2 b * logd. M n. R q = 2 b ( logd. M(n. R) + 1) 18

 External Sort-Merge 19

External Sort-Merge 19

 Catalog Information m m File q r: q R: q bfr: # of

Catalog Information m m File q r: q R: q bfr: # of records in the file record size # of blocks in the file blocking factor Index q x: q b. I 1: # of levels of a multilevel index # of first-level index blocks 20

 Catalog Information m Attribute q d: # of distinct values of an attribute

Catalog Information m Attribute q d: # of distinct values of an attribute q sl (selectivity): Øthe ratio of the # of records satisfying the condition to the total # of records in the file. qs (selection cardinality) = sl * r Øaverage # of records that will satisfy an equality condition on the attribute For a key attribute: q d = r, sl = 1/r, s=1 m For a nonkey attribute: q assuming that d distinct values are uniformly distributed among the records q the estimated sl = 1/d, s = r/d m 21

 File Scans m Types of scans q File scan – search algorithms that

File Scans m Types of scans q File scan – search algorithms that locate and retrieve records that fulfill a selection condition. q Index scan – search algorithms that use an index Øselection condition must be on search-key of index. m Cost estimate C = # of disk blocks scanned 22

 Algorithms for SELECT Operations m Implementing the SELECT Operation m Examples: q (OP

Algorithms for SELECT Operations m Implementing the SELECT Operation m Examples: q (OP 1): SSN='123456789' (EMP) q (OP 2): DNUMBER>5(DEPT) q (OP 3): DNO=5(EMP) q (OP 4): DNO=5 AND SALARY>30000 AND SEX=F(EMP) q (OP 5): ESSN=123456789 AND PNO=10(WORKS_ON) 23

 Algorithms for Selection Operation Search Methods for Simple Selection: m S 1 (linear

Algorithms for Selection Operation Search Methods for Simple Selection: m S 1 (linear search) q Retrieve every record in the file, and test whether its attribute values satisfy the selection condition. m q If selection is on a nonkey attribute, C = b q If selection is equality on a key attribute, Øif record found, average cost C = b/2, else C = b 24

 Algorithms for Selection Operation m S 2 (binary search) q Applicable if selection

Algorithms for Selection Operation m S 2 (binary search) q Applicable if selection is an equality comparison on the attribute on which file is ordered. q Assume that the blocks of a relation are stored contiguously q If selection is on a nonkey attribute: ØC = log 2 b: cost of locating the 1 st tuple + Ø s/bfr - 1: # of blocks containing records that satisfy selection condition q If selection is equality on a key attribute: ØC = log 2 b, since s = 1, in this case 25

 Selections Using Indices m S 3 (primary or hash index on a key,

Selections Using Indices m S 3 (primary or hash index on a key, equality) q Retrieve a single record that satisfies the corresponding equality condition Ø If the selection condition involves an equality comparison on a key attribute with a primary index (or a hash key), use the primary index (or the hash key) to retrieve the record. q Primary index: retrieve 1 more block than the # of index levels, Ø C = x + 1: q Hash index: Ø C = 1: for static or linear hashing Ø C = 2: for extendable hashing 26

 Selections Using Indices m S 4 (primary index on a key, range selection)

Selections Using Indices m S 4 (primary index on a key, range selection) q S 4 Using a primary index to retrieve multiple records: Ø If the comparison condition is >, ≥, <, or ≤ on a key field with a primary index, use the index to find the record satisfying the corresponding equality condition, then retrieve all subsequent records in the (ordered) file. q Assuming relation is sorted on A Ø For A v(r) use index to find 1 st tuple = v and retrieve all subsequent records. Ø For A v(r) use index to find 1 st tuple = v and retrieve all preceding records. – OR just scan relation sequentially till 1 st tuple v; do not use index with average cost C = b/2 q Average cost C = x + b/2 27

 Selections Using Indices m S 5 (clustered index on nonkey, equality) q Retrieve

Selections Using Indices m S 5 (clustered index on nonkey, equality) q Retrieve multiple records. q Records will be on consecutive blocks q C = x + s/bfr Ø# of blocks containing records that satisfy selection condition 28

 Selections Using Indices m S 6 -1 (secondary index B+-tree, equality) q Retrieve

Selections Using Indices m S 6 -1 (secondary index B+-tree, equality) q Retrieve a single record if the search-key is a candidate key, ØC = x + 1 q Retrieve multiple records if search-key is not a candidate key, ØC = x + s ØCan be very expensive!. Each record may be on a different block , one block access for each retrieved record 29

 Selections Using Indices m S 6 -2 (secondary index B+-tree, comparison) q For

Selections Using Indices m S 6 -2 (secondary index B+-tree, comparison) q For A v(r) use index to find 1 st index entry = v and scan index sequentially from there, to find pointers to records. q For A v(r) just scan leaf pages of index finding pointers to records, till first entry v q If ½ records are assumed to satisfy the condition, then ½ first-level index blocks are accessed, plus ½ the file records via the index ØC = x + b. I 1/2 + r/2 30

 Complex Selections: 1 2 … n(r) m S 7 (conjunctive selection using one

Complex Selections: 1 2 … n(r) m S 7 (conjunctive selection using one index) q Select i and algorithms S 1 through S 6 that results in the least cost for i(r). q Test other conditions on tuple after fetching it into memory buffer. q Cost of the algorithms chosen. 31

 Complex Selections: 1 2 … n(r) m S 8 (conjunctive selection using composite

Complex Selections: 1 2 … n(r) m S 8 (conjunctive selection using composite index). q If two or more attributes are involved in equality conditions in the conjunctive condition and a composite index (or hash structure) exists on the combined field, we can use the index directly. q Use appropriate composite index if available using one the algorithms S 3 (primary index), S 5, or S 6 (B+-tree, equality). 32

 Complex Selections: 1 2 … n(r) m S 9 (conjunctive selection by intersection

Complex Selections: 1 2 … n(r) m S 9 (conjunctive selection by intersection of record pointers) q Requires indices with record pointers. q Use corresponding index for each condition, and take intersection of all the obtained sets of record pointers, then fetch records from file q If some conditions do not have appropriate indices, apply test in memory. q Cost is the sum of the costs of the individual index scan plus the cost of retrieving records from disk. 33

Complex Selections: 1 2 … n(r) m S 10 (disjunctive selection by union of

Complex Selections: 1 2 … n(r) m S 10 (disjunctive selection by union of identifiers) q Applicable if all conditions have available indices. ØOtherwise use linear scan. q Use corresponding index for each condition, and take union of all the obtained sets of record pointers. Then fetch records from file m READ q “Examples of Using the Cost Functionst” Øpage 569 --570. Øpage 694 -695 34

 Duplicate Elimination m Duplicate elimination can be implemented via hashing or sorting. q

Duplicate Elimination m Duplicate elimination can be implemented via hashing or sorting. q On sorting, duplicates will come adjacent to each other, and all but one set of duplicates can be deleted. q Optimization: duplicates can be deleted during run generation as well as at intermediate merge steps in external sort-merge. q Cost is the same as the cost of sorting q Hashing is similar – duplicates will come into the same bucket. 35

 Algorithms for PROJECT Operation m Algorithm for PROJECT operations (Figure 15. 3 b)

Algorithms for PROJECT Operation m Algorithm for PROJECT operations (Figure 15. 3 b) <attribute list>(R) 1. If <attribute list> has a key of relation R, extract all tuples from R with only the values for the attributes in <attribute list>. 2. If <attribute list> does NOT include a key of relation R, duplicated tuples must be removed from the results. 36

 Algorithms for SET Operations m m m Algorithm for SET operations Set operations:

Algorithms for SET Operations m m m Algorithm for SET operations Set operations: q UNION, INTERSECTION, SET DIFFERENCE and CARTESIAN PRODUCT of relations R and S include all possible combinations of records from R and S. The attribute of the result include all attributes of R and S. Cost analysis of CARTESIAN PRODUCT q If R has n records and j attributes and S has m records and k attributes, the result relation will have n*m records and j+k attributes. CARTESIAN PRODUCT operation is very expensive and should be avoided if possible. 37

 Set Operations m m m R S: (See Figure 18. 3 c) q

Set Operations m m m R S: (See Figure 18. 3 c) q 1. Sort the two relations on the same attributes. q 2. Scan and merge both sorted files concurrently, whenever the same tuple exists in both relations, only one is kept in the merged results. R S: (See Figure 18. 3 d) q 1. Sort the two relations on the same attributes. q 2. Scan and merge both sorted files concurrently, keep in the merged results only those tuples that appear in both relations. R – S: (See Figure 18. 3 e) q keep in the merged results only those tuples that appear in relation R but not in relation S. 38

 b. Project c. Union 39

b. Project c. Union 39

 d. Intersection e. Different 40

d. Intersection e. Different 40

 Aggregate Operations m The aggregate operations MIN, MAX, COUNT, AVERAGE, and SUM can

Aggregate Operations m The aggregate operations MIN, MAX, COUNT, AVERAGE, and SUM can be computed by scanning the whole records (the worst case) m If index exists on the attribute of MAX , MIN operation, then these operations can be done in a much more efficient way: select max/min (salary) from employee q If an (ascending) index on SALARY exists for the employee relation, then the optimizer could decide on traversing the index for the largest/smallest value, which would entail following the right/left most pointer in each index node from the root to a leaf. 41

 Aggregate Operations m m SUM, COUNT and AVG For a dense index (each

Aggregate Operations m m SUM, COUNT and AVG For a dense index (each record has one index entry): q m q Actual number of records associated with each index entry must be accounted for With GROUP BY: the aggregate operator must be applied separately to each group of tuples. q m Apply the associated computation to the values in the index. For a non-dense index: q m Use sorting or hashing on the group attributes to partition the file into the appropriate groups; Computes the aggregate function for the tuples in each group. What if we have Clustering index on the grouping attributes? 42

 Combining Operations using Pipelining m Motivation q A query is mapped into a

Combining Operations using Pipelining m Motivation q A query is mapped into a sequence of operations. q Each execution of an operation produces a temporary result (Materialization). q Generating & saving temporary files on disk is time consuming. m Alternative: q Avoid constructing temporary results as much as possible. q Pipeline the data through multiple operations Øpass the result of a previous operator to the next without waiting to complete the previous operation. 43

 Materialization m Materialized evaluation: q evaluate one operation at a time, starting at

Materialization m Materialized evaluation: q evaluate one operation at a time, starting at the lowest-level. q Use intermediate results materialized into temporary relations to evaluate next-level operations. 44

 Pipelining m Pipelined evaluation q evaluate several operations simultaneously, passing the results of

Pipelining m Pipelined evaluation q evaluate several operations simultaneously, passing the results of one operation on to the next. q E. g. , in previous expression tree, don’t store result of q instead, pass tuples directly to the join. q Similarly, don’t store result of join, pass tuples directly to projection. Much cheaper than materialization: q no need to store a temporary relation to disk. m Pipelining may not always be possible q e. g. , sort, hash-join. m 45

 Using Heuristics in Query Optimization m Process for heuristics optimization 1. The parser

Using Heuristics in Query Optimization m Process for heuristics optimization 1. The parser of a high-level query generates an initial internal representation; 2. Apply heuristics rules to optimize the internal representation. 3. A query execution plan is generated to execute groups of operations based on the access paths available on the files involved in the query. m The main heuristic is to apply first the operations that reduce the size of intermediate results. q E. g. , Apply SELECT and PROJECT operations before applying the JOIN or other binary operations. 46

 Using Heuristics in Query Optimization m m m Query tree: q A tree

Using Heuristics in Query Optimization m m m Query tree: q A tree data structure that corresponds to a relational algebra expression. It represents the input relations of the query as leaf nodes of the tree, and represents the relational algebra operations as internal nodes. An execution of the query tree consists of executing an internal node operation whenever its operands are available and then replacing that internal node by the relation that results from executing the operation. Query graph: q A graph data structure that corresponds to a relational calculus expression. It does not indicate an order on which operations to perform first. There is only a single graph corresponding to each query. 47

 Using Heuristics in Query Optimization m Example: q m For every project located

Using Heuristics in Query Optimization m Example: q m For every project located in ‘Stafford’, retrieve the project number, the controlling department number and the department manager’s last name, address and birthdate. Relation algebra: PNUMBER, DNUM, LNAME, ADDRESS, BDATE ((( PLOCATION=‘STAFFORD’(PROJECT)) DNUM=DNUMBER (EMPLOYEE)) m (DEPARTMENT)) MGRSSN=SSN SQL query: Q 2: SELECT P. NUMBER, P. DNUM, E. LNAME, E. ADDRESS, E. BDATE FROM PROJECT AS P, DEPARTMENT AS D, EMPLOYEE AS E WHERE P. DNUM=D. DNUMBER AND D. MGRSSN=E. SSN AND P. PLOCATION=‘STAFFORD’; 48

 Using Heuristics in Query Optimization 49

Using Heuristics in Query Optimization 49

 Using Heuristics in Query Optimization 50

Using Heuristics in Query Optimization 50

 Using Heuristics in Query Optimization m m Heuristic Optimization of Query Trees: q

Using Heuristics in Query Optimization m m Heuristic Optimization of Query Trees: q The same query could correspond to many different relational algebra expressions — and hence many different query trees. q The task of heuristic optimization of query trees is to find a final query tree that is efficient to execute. Example: SELECT LNAME FROM EMPLOYEE, WORKS_ON, PROJECT WHERE PNAME = ‘AQUARIUS’ AND PNMUBER=PNO AND ESSN=SSN AND BDATE > ‘ 1957 -12 -31’; 51

 Using Heuristics in Query Optimization 52

Using Heuristics in Query Optimization 52

 Using Heuristics in Query Optimization 53

Using Heuristics in Query Optimization 53

 Using Heuristics in Query Optimization 54

Using Heuristics in Query Optimization 54

 Transformation Rules for RA Operations 1. Cascade of : A conjunctive selection condition

Transformation Rules for RA Operations 1. Cascade of : A conjunctive selection condition can be broken up into a cascade (sequence) of individual operations: c 1 AND c 2 AND. . . AND cn(R) = c 1( c 2(. . . ( cn(R)). . . )) 2. Commutativity of : The operation is commutative: c 1( c 2(R)) = c 2( c 1(R)) 3. Cascade of : In a cascade (sequence) of operations, all but the last one can be ignored: List 1( List 2(. . . ( Listn(R)). . . )) = List 1(R) 4. Commuting with : If the selection condition c involves only the attributes A 1, . . . , An in the projection list, the two operations can be commuted: A 1, A 2, . . . , An( c(R)) = c( A 1, A 2, . . . , An(R)) 55

 Transformation Rules for RA Operations 5. Commutativity of ⋈ (or ): The ⋈

Transformation Rules for RA Operations 5. Commutativity of ⋈ (or ): The ⋈ operation is commutative as is the operation: R ⋈c S = S ⋈c R; R S = S R 6. Commuting with ⋈ (or ): If all the attributes in the selection condition c involve only the attributes of one of the relations being joined—say, R—the two operations can be commuted as follows: c(R ⋈ S) = ( c(R)) ⋈ S Alternatively, if the selection condition c can be written as (c 1 and c 2), where condition c 1 involves only the attributes of R and condition c 2 involves only the attributes of S, the operations commute as follows: c(R ⋈ S) = ( c 1(R)) ⋈ ( c 2(S)) 56

 Transformation Rules for RA Operations 7. Commuting with ⋈ (or ): Suppose that

Transformation Rules for RA Operations 7. Commuting with ⋈ (or ): Suppose that the projection list is L = {A 1, . . . , An, B 1, . . . , Bm}, where A 1, . . . , An are attributes of R and B 1, . . . , Bm are attributes of S. If the join condition c involves only attributes in L, the two operations can be commuted as follows: L(R ⋈c S) = ( A 1, . . . , An(R)) ⋈c ( B 1, . . . , Bm(S)) If the join condition c contains additional attributes not in L, these must be added to the projection list, and a final operation is needed. 57

 Transformation Rules for RA Operations 8. Commutativity of set operations: The set operations

Transformation Rules for RA Operations 8. Commutativity of set operations: The set operations and are commutative but – is not. 9. Associativity of ⋈, , , and : These four operations are individually associative; that is, if stands for any one of these 4 operations (throughout the expression), we have (R S) T = R (S T) 10. Commuting with set operations: The operation commutes with , , and –. If stands for any one of these 3 operations, we have c(R S) = ( c(R)) ( c(S)) 58

 Transformation Rules for RA Operations 11. The operation commutes with . L(R υ

Transformation Rules for RA Operations 11. The operation commutes with . L(R υ S) = ( L(R)) υ ( L(S)) 12. Converting a ( , ) sequence into ⋈: If the condition c of a that follows a corresponds to a join condition, convert the ( , ) sequence into a ⋈ as ( c(R S)) = (R ⋈c S) 59

 Heuristic Algebraic Optimization Algorithm 1. 2. 3. Using rule 1, break up any

Heuristic Algebraic Optimization Algorithm 1. 2. 3. Using rule 1, break up any select operations with conjunctive conditions into a cascade of select operations. σc 1 AND c 2 AND … AND cn(R) ≡ σc 1(σc 2(…(σcn(R))…)) Using rules 2, 4, 6, and 10 concerning the commutativity of select with other operations, move each select operation as far down the query tree as is permitted by the attributes involved in the select condition. Using rule 9 concerning associativity of binary operations, rearrange the leaf nodes of the tree: q Position the leaf node relation with the most restrictive σ operations so they are executed first, q Make sure that the ordering of leaf nodes does not cause CARTESIAN PRODUCT operations 60

 Heuristic Algebraic Optimization Algorithm Using Rule 12, combine a cartesian product operation with

Heuristic Algebraic Optimization Algorithm Using Rule 12, combine a cartesian product operation with a subsequent select operation in the tree into a join operation. Combine a with a subsequent σ in the tree into a ⋈ 5. Using rules 3, 4, 7, and 11 concerning the cascading of project and the commuting of project with other operations, break down and move lists of projection attributes down the tree as far as possible by creating new project operations as needed. 4. 6. Identify subtrees that represent groups of operations that can be executed by a single algorithm. 61

Summary of Heuristics for Algebraic Optimization m The main heuristic is to apply first

Summary of Heuristics for Algebraic Optimization m The main heuristic is to apply first the operations that reduce the size of intermediate results. m Perform select operations as early as possible to reduce the number of tuples and perform project operations as early as possible to reduce the number of attributes. (This is done by moving select and project operations as far down the tree as possible. ) m The select and join operations that are most restrictive should be executed before other similar operations. (This is done by reordering the leaf nodes of the tree among themselves and adjusting the rest of the tree appropriately. ) 62

 Query Execution Plans m An execution plan for a relational algebra query consists

Query Execution Plans m An execution plan for a relational algebra query consists of a combination of the relational algebra query tree and information about the access methods to be used for each relation as well as the methods to be used in computing the relational operators stored in the tree. m Materialized evaluation: the result of an operation is stored as a temporary relation. m Pipelined evaluation: as the result of an operator is produced, it is forwarded to the next operator in sequence. 63

Using Selectivity and Cost Estimates in Query Optimization m Cost-based query optimization: q Estimate

Using Selectivity and Cost Estimates in Query Optimization m Cost-based query optimization: q Estimate and compare the costs of executing a query using different execution strategies and choose the strategy with the lowest cost estimate. q (Compare to heuristic query optimization) m Issues q Cost function q Number of execution strategies to be considered 64

Using Selectivity and Cost Estimates in Query Optimization m Cost Components for Query Execution

Using Selectivity and Cost Estimates in Query Optimization m Cost Components for Query Execution 1. Access cost to secondary storage 2. Storage cost 3. Computation cost 4. Memory usage cost 5. Communication cost m Note: Different database systems may focus on different cost components. 65

Using Selectivity and Cost Estimates in Query Optimization m Catalog Information Used in Cost

Using Selectivity and Cost Estimates in Query Optimization m Catalog Information Used in Cost Functions q Information about the size of a file Ø Ø q Information about indexes and indexing attributes of a file Ø Ø Ø number of records (tuples) (r), record size (R), number of blocks (b) blocking factor (bfr) Number of levels (x) of each multilevel index Number of first-level index blocks (b. I 1) Number of distinct values (d) of an attribute Selectivity (sl) of an attribute Selection cardinality (s) of an attribute. (s = sl * r) 66

Using Selectivity and Cost Estimates in Query Optimization m m Examples of Cost Functions

Using Selectivity and Cost Estimates in Query Optimization m m Examples of Cost Functions for SELECT S 1. Linear search (brute force) approach q CS 1 a = b; q For an equality condition on a key, CS 1 a = (b/2) if the record is found; otherwise CS 1 a = b. S 2. Binary search: q CS 2 = log 2 b + (s/bfr) – 1 q For an equality condition on a unique (key) attribute, CS 2 =log 2 b S 3. Using a primary index (S 3 a) or hash key (S 3 b) to retrieve a single record q CS 3 a = x + 1; CS 3 b = 1 for static or linear hashing; q CS 3 b = 1 for extendible hashing; 67

Using Selectivity and Cost Estimates in Query Optimization m m Examples of Cost Functions

Using Selectivity and Cost Estimates in Query Optimization m m Examples of Cost Functions for SELECT (contd. ) S 4. Using an ordering index to retrieve multiple records: q For the comparison condition on a key field with an ordering index, CS 4 = x + (b/2) S 5. Using a clustering index to retrieve multiple records: q CS 5 = x + ┌ (s/bfr) ┐ S 6. Using a secondary (B+-tree) index: q For an equality comparison, CS 6 a = x + s; q For an comparison condition such as >, <, >=, or <=, q CS 6 a = x + (b. I 1/2) + (r/2) 68

Using Selectivity and Cost Estimates in Query Optimization m m m Examples of Cost

Using Selectivity and Cost Estimates in Query Optimization m m m Examples of Cost Functions for SELECT (contd. ) S 7. Conjunctive selection: q Use either S 1 or one of the methods S 2 to S 6 to solve. q For the latter case, use one condition to retrieve the records and then check in the memory buffer whether each retrieved record satisfies the remaining conditions in the conjunction. S 8. Conjunctive selection using a composite index: q Same as S 3 a, S 5 or S 6 a, depending on the type of index. 69

 10. Semantic Query Optimization m Semantic Query Optimization: q m m Uses constraints

10. Semantic Query Optimization m Semantic Query Optimization: q m m Uses constraints specified on the database schema in order to modify one query into another query that is more efficient to execute. Consider the following SQL query, SELECT E. LNAME, M. LNAME FROM EMPLOYEE E M WHERE E. SUPERSSN=M. SSN AND E. SALARY>M. SALARY Explanation: q Suppose that we had a constraint on the database schema that stated that no employee can earn more than his or her direct supervisor. If the semantic query optimizer checks for the existence of this constraint, it need not execute the query at all because it knows that the result of the query will be empty. Techniques known as theorem proving can be used for this purpose. 70

 Example (1) An un-optimized relational algebra expression: Name ( GPA 3. 5 and

Example (1) An un-optimized relational algebra expression: Name ( GPA 3. 5 and Title = 'Ada Programming Language’ and Students. SSN = Enrollment. SSN and Enrollment. Course_no = Courses. Course_no (Students Enrollment Courses)) 71

 Example (2) Initial query tree: Name GPA 3. 5 and Title = 'Ada

Example (2) Initial query tree: Name GPA 3. 5 and Title = 'Ada Programming Language’ and Students. SSN = Enrollment. SSN and Enrollment. Course_no = Courses. Course_no Students Courses Enrollment 72

 Example (3) Perform selections as early as possible. Name Enrollment. Course_no = Courses.

Example (3) Perform selections as early as possible. Name Enrollment. Course_no = Courses. Course_no Students. SSN = Enrollment. SSN Title = 'Ada Programming Language’ GPA 3. 5 Enrollment Courses Students 73

 Example (4) Replace Cartesian products by joins. Name Title = 'Ada Programming Language’

Example (4) Replace Cartesian products by joins. Name Title = 'Ada Programming Language’ GPA 3. 5 Enrollment Students Courses 74

 Example (5) Perform more restrictive joins first. Name GPA 3. 5 Students Enrollment

Example (5) Perform more restrictive joins first. Name GPA 3. 5 Students Enrollment Title = 'Ada Programming Language’ Courses 75

 Example (6) Project out useless attributes early. Name SSN, Name GPA 3. 5

Example (6) Project out useless attributes early. Name SSN, Name GPA 3. 5 Students SSN, Course_no Enrollment 76 Course_no Title = 'Ada Programming Language’ Courses

 Example (7) The optimized algebra expression is: Name (( SSN, Name ( GPA

Example (7) The optimized algebra expression is: Name (( SSN, Name ( GPA 3. 5 (Students))) (( SSN, Course_no(Enrollments)) ( Course_no ( Title = 'Ada Programming Language’ (Courses))))) m Projections and selections on the same relation are usually performed using the same scan of the relation. 77