Geoprocessing Conflation tools Getting Started and Building Workflows

Geoprocessing Conflation tools: Getting Started and Building Workflows Dan Lee (dlee@esri. com) Nobbir Ahmed (nahmed@esri. com) Esri UC 2014 | Technical Workshop |

Agenda What is Conflation? Geoprocessing Conflation Tools Ø Demo 1 – Basic scenarios Conflation Workflows Demo 2 – Unification of overlapping datasets Ø Demo 3 – Edgematching of adjacent datasets Ø Remarks Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Translated by Esri localization What is Conflation? Birleştirme Объединение Assemblage Combinación 合并 Zusammenführung Fusione 補正 ��� Combinação Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Challenges arise when multi-source spatial data are used together Common obstacles in analysis and mapping: Ø Spatial and attribute inconsistency caused by differences in data collection and modeling Ø High cost to fix the problems Ø No existing tools can solve the problem effectively Overlapping datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Adjacent datasets

Conflation reconciles multi-source datasets and optimizes their quality and usability Conflation is the process of： Ø Identifying corresponding features (known as feature matching) Ø Making spatial adjustment and attribute transfer Ø Ultimately, combining matched and unmatched features into one unified dataset with the optimal accuracy, completeness, consistency, and integrity Long-term benefits: Ø No longer living with various imperfect datasets Ø More confidence in reliable analysis and high quality mapping What’s the way to get there? Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Have you used these tools? Geoprocessing Conflation Tools Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows New in Arc. GIS 10. 2. 1

Our initial focuses Develop highly automated tools in Geoprocessing framework: Ø Starting with linear features (roads, parcel lines, etc. ) Ø Aiming at high feature matching accuracy (not promising 100%) Ø Providing information to facilitate post-processing Build practical workflows Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Challenges in feature matching (the foundation of conflation) Complexity Dissimilari ty Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Feature matching (FM) for overlapping datasets Based on proximity, topology, pattern, and similarity analysis, as well as attributes information 1: 1 and 1: m matches m: 1 and m: n matches Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

FM-based tool #1 - Detect Feature Changes (DFC) Finding feature differences Update features vs. base features CHANGE_TYPE in output Ø Spatial (S) change Ø Attribute (A) change Ø Spatial and attribute (SA) change Ø No change (NC) Ø New update feature (N) Ø To-Delete base feature (D) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows DFC

FM-based tool #2 – Transfer Attributes (TA) From source features to target features Ø Transfer fields (e. g. ROAD_NAME) Ø Target features are modified TA Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

FM-based tool #3 – Generate Rubbersheet Links (GRL) Rubbersheeting moves source locations towards target locations based on established links Generate Rubbersheet Links (GRL) Ø From source features to target features GRL Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Followed by Rubbersheet Features (RF) Ø Adjusting input features RF

Edge (feature) matching (EM) and tools for adjacent datasets Based on proximity, topology, and continuity analysis, as well as attributes information Generate Edgematch Links (GEL) Ø From source features to adjacent features GEL Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Followed by Edgematch Features (EF) Ø Connects features guided by the established links EF

Demo 1: Basic scenarios Part 1: Conflate overlapping datasets Part 2: Edgematch adjacent datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Workflow strategy Part 1: conflation of overlapping datasets A popular scenario and requirements: Ø set. A set. B Contains updates Spatially accurate (2) To unify the two datasets into one (3) For all features: Do rubbersheeting adjustment Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows (1) Identify matched and unmatched with combined spatial and attribute information Make a copy (4) For matched: Transfer uncommon attributes (5) (6) set. C Select unmatched Append unmatched Best of both

Part 1: Input streets Together Update features with new streets and attributes Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Base features with spatial accuracy and attributes

This reflects the conflation strategy. With the simple and highly similar demo data, the process produces 100% accurate result. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Part 1: results Attributes transferred Changes detected Rubbersheetin g links generated Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows New features adjusted and added to base

Part 2: edgematching of adjacent datasets Goal - make two adjacent datasets connect Source features Together Adjacent features Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Since the demo data has little ambiguity; the process produces 100% accurate results. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Part 2: results Move endpoint Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Conflation Workflows • Unification of overlapping datasets • Edgematching of adjacent datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Three components in conflation workflows Conflation and evaluation Preprocessing Postprocessing Ø In same projection Ø Conflation tools Ø Queued review Ø Data validation Ø Workflow tools Ø Interactive editing Ø Selection of relevant features Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Workflow strategy Demo 2: Unification of overlapping datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Overview Two road datasets (northeast of Meigs County, OH): Ø Local. NE – 1085 features Ø State. NE – 1013 features Both datasets: Ø Have common and uncommon features and attributes Ø Are well preprocessed Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Breakdown of the workflow (based on Demo 1 – Part 1 workflow) Step 4 QA #4 Step 1 a QA #1 Step 5 Step 1 b Step 3 QA #2 Step 2 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows QA #3

Step 1 a of the workflow with evaluation QA #1 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

DFC result and potential match errors Esri UC 2014 | Technical Workshop Geoprocessing | Conflation tools: Getting started and Building Workflows

QA potential match errors Total 16 CFM_GRP were flagged; 11 had match issues due to data complexity and dissimilarity; 5 were ignorable Match issue due to data complexity Esri UC 2014 | Technical Workshop Geoprocessing | Conflation tools: Getting started and Building Workflows

QA DFC result – CHANGE_TYPE D and N ((CHANGE_TYPE = 'N') OR ( CHANGE_TYPE = 'D' )) AND( (NEAR_DIST > 0) AND (NEAR_DIST < 10)) Wrong N Inspect records with high potential for errors: Ø 35 reviewed Ø 11 wrong Ns or Ds flagged Wrong D Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Feature matching accuracy estimates Matched groups: Overall feature matching accuracy Ø Total: 896 groups (average of matched and unmatched) Ø Correct: 885 groups Ø Incorrect: 11 groups 97. 09% Accuracy = 885 / 896 = 98. 77% Unmatched: Ø Total: 240 (155 Ns + 85 Ds) Ø Correct: 229 (151 Ns + 78 Ds) Ø Incorrect: 11 (4 Ns + 7 Ds) Accuracy = 229 / 240 = 95. 42% (biased by the total count) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Ready to join with inputs to tag Ns and Ds …

Step 1 b of the workflow Extract Matched Features as inputs for GRL Local. NE: 934 non-N out of 1085 State. NE: 935 non-D out of 1013 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Ready for GRL process …

Step 2 of the workflow with evaluation and QA QA #2 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GRL result Generated total 26198 regular links and 10227 identity links Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GRL evaluation results – Intersecting links 54 locations of intersecting links; Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GRL evaluation results – links of different src/tgt vertex types (qa. Notes = 'src_tgt_Vx. Type_diff') AND( (tgt. Vx. Type >=2) OR( src. Vx. Type >=2 )) AND NEAR_DIST = -1 79 of flagged links were more important Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GRL evaluation results – locations of missing links 22 of the 595 source locations of missing links were on nodes; all others are on in-line vertices. 20 ORIG_FID of frequency >5 locations were reviewed and confirmed non-critical. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

QA regular links - summary (qa. Notes = 'src_tgt_Vx. Type_diff') AND( (tgt. Vx. Type >=2) OR( src. Vx. Type >=2 )) Total 241 (0. 92%) of 26198 links were reviewed: Ø 44 were modified Ø 86 were to be removed Ø 111 were ok 42 missing link locations were reviewed: Ø 14 links were added Ø Links at other locations were not critical Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Ready for rubbersheeting …

26126 regular links were selected by (REV_FLAG <> 'Delete') OR( REV_FLAG IS NULL) to participate Step 3 of the workflow with assessments QA #3 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Rubbersheeting result Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GRL result after rubbersheeting Many regular links became identify links Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

How good is the rubbersheeting result? Three indicators showing spatial improvement Improved location alignment Less spatial differences Before RF After RF Regular links 26126 412 Identity links 10227 15456 Link-length distributions before/after RF - spatially closer to target (Not on the same scale due to the big difference in values) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

QA #3 – Check rubbersheeting result Source (original) and target Target features Source adjusted with N features highlighted Ready to do TA … Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Transfer attributes (UC 2014_ID) from adjusted source to target Excluding Ns from adjusted source; excluding Ds from joined target Step 4 of the workflow with evaluation and QA QA #4 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Attribute transfer result Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

QA #4 – Check attribute transfer result NEAR_DIST >=0; no-transfer features found nearby source features for potentially missed matches 32 records were reviewed: Ø 18 were edited with UC 2014_ID values Almost there … Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Select N features from adjusted source and append them to target (CHANGE_TYPE = 'N') AND(( REV_FLAG <> 'wrong. N' ) OR REV_FLAG IS NULL) Final step of the full workflow Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Appended N features in final result Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Unification of overlapping datasets workflow completed! Processing Time Automated processin g Step 1 (a, b) 1 min 3 sec Step 2 1 min 14 sec Step 3 1 min Steps 4, 5 18 sec Total 3 min 35 sec QA #1 Interactive processing (not counting final review) (CFM_GR P and DN) Review Count (locations or feature groups) Edit Count (field values) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows 51 QA #2 QA #3 (links) 283 x QA #4 (attribute transfer) QA Total 32 366 Time (2 -3 review counts per minute) ~ 2 -3 hrs. 46 255 x 18 319

Workflow strategy Demo 3: Edgematching of adjacent datasets Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Breakdown of the workflow (based on Demo 1 – Part 2) Step 7 Step 6 a QA #1 Step 6 b Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Overview Two road datasets (an area in Alabama): Ø Edge. Roads – 7576 features Ø GISRoads – 3634 features Both datasets: Ø Contain roads that are within 1 km to borders Ø Have inconsistent road names Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Step 6 a of the workflow QA #1 Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GEL result Generated 454 links; midpoints of links were created for visualization purpose. Borders were not in the process, but displayed for reference. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Example edgematch links EM_CONF in output Ø 100 (matched with no ambiguity) Ø 50 (spatially matched with unmatched attributes) Ø < 50 (spatially matched with some ambiguity and weak continuity) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

GEL evaluation results 33 locations of intersecting links; 134 links of low EM_CONF values; 62 source dangle locations of potential missing links Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Inspection and editing of edgematch links Reviewed: Ø 33 locations of intersecting links NEAR_DIST >= 0 Ø 98 low EM_CONF links (EM_CONF < 33) AND (REV_FLAG IS NULL) Ø 62 source dangle locations (near links) Summary: Ø 388 (~85%) of total 459 links were good (54 were flagged for recheck) Ø 71 (~15%) of total links were modified, removed, or added Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

What happened to the SRC_FID and TGT_FID of the added or modified links? Step 6 b of the workflow Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Ready for edgematching …

Step 7 of the workflow Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Edgematch result Review flagged locations … Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Edgematching of adjacent datasets workflow completed! Processing Time Automated processin g Step 6 a 6. 52 sec Step 6 b 4. 09 sec Step 6 c 2. 15 sec Total 12. 76 sec QA Links Interactive processing (not counting final review) Review Count (locations or feature groups) Edit Count (field values) Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Time (2 -3 review counts per minute) ~ 193 ~ 1 - 1. 6 hrs. 192

Thanks to: • Department of Public Works Remarks • • Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows (DPW), Los Angeles County, USA. Institut Cartogràfic i Geològic de Catalunya (ICGC), Barcelona, Spain. Ohio State Department of Transportation, USA. National Institute for Water and Atmospheric Research (NIWA) and Land Information New Zealand (LINZ) - Crown Copyright Reserved. All others who supported us along the way.

Conflation can be done more efficiently now in Arc. GIS Desktop Conflation and evaluation Preprocessing Ø Use the best practice Ø Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows Highly accurate results and rich information Postprocessing Ø Small among of review and editing Ø Time worth-spending

Our future work New tools and enhancements Ø Transform Features, Align Features Ø Other feature types Ø Contextual conflation (spatially related features) Integrated review-editing environment Formalization of workflows Ø Common scenario-oriented Ø Guidelines and training Please send us your use cases and requirements Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

We recommend you to consider conflation a higher priority Study the tools and understand the results Ø Use small test areas Begin to build workflows for your organizations Ø Improve data quality and usability Work with broader communities Ø Community mapping Please send us your feedbacks Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Recent papers and other references Lee, D. , Yang, W. , Ahmed, N. (2014). Conflation in Geoprocessing Framework - Case Studies, GEOProcessing, 2014, Barcelona, Spain. http: //goo. gl/i. Oo. SGV Yang, W. , Lee, D. , Ahmed, N. (2014). Pattern Based Feature Matching for Geospatial Data Conflation, GEOProcessing, 2014, Barcelona, Spain. http: //goo. gl/JKGJbo V. Walter and D. Fritsch, “Matching spatial data sets: a statistical approach”, International Journal of Geographical Information and science, vol. 3(5, 1999), pp. 445 -473. M. Zhang and L. Meng, “Delimited stroke oriented algorithm – working principle and implementation for the matching of road networks”, Journal of Geographic Information Sciences, vol. 14(1), June, 2008, pp. 44 -53. L. Li and M. Goodchild, “An optimization model for linear feature matching in geographical data conflation”, International Journal of Image and Data Fusion, vol. 2(4), 2011, pp. 309 -328. Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Reminder Please fill out the session survey: First Offering ID: 1116 Second Offering ID: 1227 Online – www. esri. com/ucsessionsurveys Paper – pick up and put in drop box Thank you for attending! Any questions, comments …? Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows

Esri UC 2014 | Technical Workshop | Geoprocessing Conflation tools: Getting started and Building Workflows