GIS Data Quality Producing better data quality through

GIS Data Quality Producing better data quality through robust business processes Kim Ollivier Bright. Star TRAINING

Schedule Day 2 Suggested breaks for the following times: Start: 9: 00 Session 1 ( 90 min) Morning tea: 10: 30 to 10: 45 Session 2 ( 105 min) Lunch: 12: 30 to 1: 30 Session 3 ( 90 min) Afternoon tea: 3: 00 to 3: 15 Session 4 ( 105 min) Finish: 5: 00 Each session will have an exercise or interactive discussion

Topics • Metadata • Designing rules • Data warehouse and ETL • Feature maintenance

Metadata n n n Data model Business rules, relations, state Subclasses (lookup tables) GIS Metadata NZGLS and ISO XML Readme. txt or readme. html

Metadata n n n Which standard? ISO 19115, NZGMS Aust asdd. ga. gov. au

Examine Metadata n n n n Geospatial metadata Benefit to users or producer? How do we collect it? Standardisation or not? metadatatopo 250 k_metadata. html metadataDCW_DQ_Project. htm metadatameta. html Morning Tea

Data Quality Rules n n n Attribute domain constraints Relational integrity rules Rules for historical data Rules for state-dependent objects General dependency rules Spatial feature rules

A GIS Data Quality System Assess Data Quality Assessment Data Profiling Improve Data Cleaning Prevent Monitoring Data Integration Interfaces Ensuring Quality of Data Conversion and Consolidation Monitor Recurrent Data Quality Assessment Recognise Building Data Quality Metadata Warehouse

Assessing Quality n n n Project steps Required roles Defining the objectives Designing rules Scorecard and Metadata Frequency of assessment

Building Rules n n n Data profiling Interview users Examine data model Data Gazing Application v data matrix

Attribute Domain Constraints n n n n Lookup tables Numeric ranges Null values Blank values Format constraints Precision Complex domain restraints

Relational Integrity Rules n n Identity rule Reference rules Cardinal rules Inheritance rules

Historical Data n n n Time dependent attribute Value constraints Rates of change Volatility Continuity Granularity

State-dependent Objects start n n n State-transition models States, terminators Actions Active (A) Terminated (T) On Leave (L) Deceased (D) Retired (R)

Event Histories n n An object may have many events Event Overlaps Event Frequencies Event Conditions

Spatial Rules n n n Projection, units Dimensions 2 D, 3 D, M, Z point, line, poly Precision Topology

Valuation Roll n n n Legacy structure, 50 years old Variable maintenance standard Valuer General audit (DQ spec)

Rules Exercise n Split into pairs Examine sample DVR dataset Devise some rules for each category n Verbal discussion with class n n Lunch

Data Warehouse & ETL n n n Why not direct access to online DB? Staging Area Scripting tools Trade-offs KPI for project • better quality than source target

ETL Extract n Extract

ETL Transform n The importance of primary keys

ETL Load n n Batch offline most common Daily status usually enough

Safe Software FME n Examples Afternoon Tea

Data Quality Team IT DQ Team Users

Maintenance of features n n n Time series important Line/polygon features are not atomic Splitting loses inheritance Calculating depreciation Direct editing bypasses business rules

Maintenance of the Quality n n Gardening, not mountain climbing Discussion of course topics

References n n Data Quality, The Accuracy Dimension – Jack E Olson The Data Warehouse ETL Toolkit – Ralph Kimball Please fill in evaluation forms Finish