Integration of Multisource Data for Automated Building Extraction

Integration of Multi-source Data for Automated Building Extraction Zhilin Li & Kourosh Khoshelham Dept of Land Surveying & Geo-Informatics Hong Kong Polytechnic University Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 1

Automated building extraction: Problems l l Diversity of buildings: Buildings in a variety of shapes and sizes; Loss of information: Buildings are threedimensional objects but the third dimension is lost in 2 D images. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 2

Automated building extraction: Solutions l To minimise information loss l To compensate for the loss • High percentage of overlap between images • Multi-source images • Multi-viewpoint images • Additional data sources, e. g. 2 -D map data • and height data Knowledge, e. g. building models Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 3

Our methodology l l l Solution 1: Building extraction based single image Solution 2: integration of image data and height data (DSM) Solution 3: integration of 2 D ground plans, image and height data (DSM) Image DSM Model Ground plan Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 4

Process l l l Image + model Image + height + 2 D plan +model Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 5

Approach 1 l l From image to edge From edge to line From line to grouped lines From grouped lines to building (matched with building model selected from library) Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 6

The process l l Locate the building on image (drawing a box); Select a model (of the building type) from model library (parametric models) Model matched with buildings on image (represented by lines which are extracted from image) Parameters for the model is then computed Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 7

From image to edge to line edge detection (e. g. by Canny operator) Recursive split-and-merge line fitting process Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 8

From image lines to grouped lines: Perceptual grouping of image lines l l Perceptual grouping based on two relations that are viewpoint invariant: • • proximity parallelism • If DISTANCE is Close and LENGTH is Short then Proximity. Decision is Probably. Connected. Significance of each relation defined using a number of linguistic fuzzy rules Example of a fuzzy rule for proximity relation: Example of a fuzzy rule for parallelism relation: • If ANGLE is Small and LENGTH is Short then Parallelism. Decision is Probably. Parallel. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 9

Construction of the library of building models Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 10

Model selection and matching l l l Model selection is interactive. A human operator selects a model from the library. Once the model is selected, it will be automatically matched with the grouped image lines. For the matching, corresponding model and image lines must be selected. The model is searched for the model lines that carry the same relations as in the grouped image lines. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 11

Matching procedure l l l The model is parameterized. Initial model parameters are estimated. Model lines are transformed to image space. Distance between the endpoints of every model line and the corresponding image lines are measured. Corrections to model parameters are estimated that minimize the sum of squared distances. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 12

Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 13

fusion of image and height data for automated building extraction l l l By making use of height data these two steps can be automated. In this work a split-and-merge method is developed to fuse image and height data, and automate the model selection. Localization of the building in the image is also carried out automatically by applying a top hat transform to height data. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 14

Integration of image data and height data (DSM) l l Image data used to extract building planes using segmentation Height data used to find building locations Height data used to assist segmentation Also automated model matching Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 15

Building edges from image by segmentation l to partition an image into a finite set of homogenous regions that correspond to surfaces in object space Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 16

Detection of building candidates from height data (DSM) l l l Opening: flattens sharp peak and height blobs Such areas can be retrieved by subtracting the opened DSM from the original DSM The process is called morphological top hat transform Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 17

Problems with image segmentation l Over-segmentation occurs when the segmentation method detects more than one region on a single surface. Under-segmentation occurs when the segmentation method fails to sufficiently separate two or more Workshop on Earth Observation for Urban Planning 18 surfaces. and Management, 20 th November 2006, HK l

A split-and-merge process for refining the initial segmentation by making use of height data l l l Height points are projected from the DSM to image regions. An LMS (least median of squares) estimator is used to fit planar faces to the height points in every image region. Initial segmentation results are refined using the computed planar faces. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 19

Split process l l Original image Over-grown region Two planes found in height data Over-grown region split Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 20

Merge process Every two neighboring regions whose planar faces are coplanar are merged. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 21

An example of the performance of the split -and-merge process Roof planes are identified by checking the height of the region-planes over the DTM. Roof planes are attributed as flat-roof or slanted-roof according to their parameters. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 22

Automated model selection l Number of roof planes are used as an index to select a class of models from the library. l Attributes of the roof planes are used to further limit the number of selected models. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 23

Library of parametric models Model selection is automated. Only a 2 D roof model is matched with the image. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 24

2 D parameterization of the roof models l Since the parametric forms of the roof planes are known, only a 2 D model of the roof is matched with the image lines to compute the boundary of the roof. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 25

Matching a 2 D roof model with image lines Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 26

Approach 3: integration of 2 D plans with image and height data for automated building extraction l l Integration of 2 D ground plans leads to the reconstruction of generic polyhedral models that cover a wider range of buildings. While building detection using top hat transform may include objects other than buildings, using 2 D ground plans and height data leads to a more reliable detection. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 27

Overview of the approach Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 28

Overview of the approach l l Buildings are localized in the image using ground plans and DEM. Assuming that walls are vertical, their parametric forms are derived from the ground plans. The parametric forms of roof planes are derived from the integration of image and height data using the split-and-merge technique. Model faces are intersected and assembled to form a generic polyhedral model using a plane patch reconstruction method Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 29

Experiments and results - Image data consists of an orthoimage in four channels with 0. 5 m ground resolution. - Height data consists of DSM and DTM with 1 m resolution obtained from a laser scanner. - Ground plans are manually digitized from the orthoimage. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 30

Image data and the computed NDVI map Color (RGB) aerial orthoimage Gray-scale image (for segmentation) Color infrared (CIR) orthoimage Computed NDVI map Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 31

Height data and detected building candidates Lidar DSM (first echo) Lidar DSM (last echo) Lidar DTM Detected building candidates Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 32

Results models reconstructed using split-and-merge process and model matching 10 successful reconstruction (parametric models); Front view 3 failure in the split-and-merge process; 1 failure in the model matching process Workshop on Earth Observation for Urban Planning Rear and Management, 20 th November 2006, HK view 33

Result Plane patch reconstruction method Integration of 2 D ground plans manually digitized from orthoimages in the plane patch reconstruction method The use of ground plans for the localization of buildings leads to an improvement in the results of the split-and-merge process B 12 Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 34

Completeness evaluation split-and-merge process and plane patch reconstruction 11 successful reconstruction (generic polyhedral models); 2 failure in the splitand-merge process; Front view 1 failure in the plane patch reconstruction method Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HKRear view 35

Other tests l l Accuracy efficiency Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 36

Conclusions The combination of the proposed approaches in one system will provide the user with a great flexibility to choose the most suitable method based on the available data. Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 37

End of presentation Workshop on Earth Observation for Urban Planning and Management, 20 th November 2006, HK 38