Architectural Style Classification of Building Facades Dr Gayane

Architectural Style Classification of Building Facades Dr. Gayane Shalunts CV Meetup #6 27. 04. 2016.

Overview • Architectural styles: Phases of development that classify architecture in the sense of historic periods, regions and cultural influences • How to know the architectural style of a building facade given only its image? 2 Architectural Style Classification of Building Facades

Google visual search (images. google. com): SUCCESS! 3 Architectural Style Classification of Building Facades

Now try it with a non-landmark building • An ordinary neo-baroque building in Vienna’s 18 th district 4 Architectural Style Classification of Building Facades

Google visual search (images. google. com): FAIL! 5 Architectural Style Classification of Building Facades

Problem Statement • To build a computer vision software tool, which classifies the architectural style of a facade, given its image Architectural style 6 Architectural Style Classification of Building Facades

The Challenges! • Input - unlimited number of 2 D projections (images) of 3 D facades • Dozens of architectural styles • Principle historic phases, e. g. In Gothic - Early Gothic, High Gothic • Local regional signature • Mixture of architectural styles on a single facade • Big intra-class variability • Non-planar historic facades • No image databases labeled by architectural styles 7 Architectural Style Classification of Building Facades

The Methodology

The Algorithm of the Voting of Architectural Elements

The Algorithm of Architectural Style Classification of Building Facades • • • Models the human intelligence for architectural style classification Embeds architectural knowledge General enough to classify any visually distinguishable architectural style Landmark and non-landmark buildings Classification of partly occluded facades Classification of facade parts Planar and non-planar facades Classification of a mixture of architectural styles Software implementation: Pan-European architectural styles • Romanesque (1050 AD-1200 AD) • Gothic (1150 AD -1500 AD) • Baroque (1550 AD -1750 AD) 10 Architectural Style Classification of Building Facades

1. The Segmentation Approaches

The Segmentation of Domes • Dome – a convex roof Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th Iberoamerican Congress on Pattern Recognition (CIARP '12), ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. 12 Architectural Style Classification of Building Facades

The Segmentation of Domes [1] Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th CIARP , ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. [2] Loy, G. , Eklundh, J. : Detecting symmetry and symmetric constellations of features. In: European Conference on Computer Vision. pp. 508 -521 (2006) [3] Felzenszwalb, P. F. , Huttenlocher, D. P. : Ecient graph-based image segmentation. International Journal of Computer Vision 59(2), 167 -181 (2004) 13 Architectural Style Classification of Building Facades

The Segmentation of Domes [1] Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th CIARP , ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. [2] Loy, G. , Eklundh, J. : Detecting symmetry and symmetric constellations of features. In: European Conference on Computer Vision. pp. 508 -521 (2006) [3] Felzenszwalb, P. F. , Huttenlocher, D. P. : Ecient graph-based image segmentation. International Journal of Computer Vision 59(2), 167 -181 (2004) 14 Architectural Style Classification of Building Facades

Validation Checking of The Found Segment • Leap = (Row(k) - Row(1)) > 0, 15 && Leap < min. Leap. Threshold Leap = Col(k) - Col(k - 1) min. Leap. Threshold • 18 pixels, if image resolution < 1 million pixels • 26 pixels otherwise • Roundness = 4 * Area / (π * Major. Axis. Length 2) • if (Roundness > 0, 37 && Dome. Bounding. Box > 1500) Dome is FOUND! else Try 2 nd strongest symmetry axis • Try up to the 5 th strongest symmetry axis (5 found empirically) Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th Iberoamerican Congress on Pattern Recognition (CIARP '12), ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. 15 Architectural Style Classification of Building Facades

The Segmentation of Towers • Tower – part of a building exceptionally high in proportion to its width 16 Architectural Style Classification of Building Facades

The Segmentation of Single Towers [1] Loy, G. , Eklundh, J. : Detecting symmetry and symmetric constellations of featuress. In: European Conference on Computer Vision. pp. 508 -521 (2006) [2] Felzenszwalb, P. F. , Huttenlocher, D. P. : Ecient graph-based image segmentation. International Journal of Computer Vision 59(2), 167 -181 (2004) 17 Architectural Style Classification of Building Facades

The Segmentation of Double Towers 18 Architectural Style Classification of Building Facades

Validation Checking: Domes – roundness, towers - solidity • Solodity = Area / Convex. Hull. Area • if (Solidity > 0, 72 && Tower. Bounding. Box > 5000) Tower is FOUND! else Try 2 nd strongest symmetry axis • Try up to the 2 nd strongest symmetry axis (2 found empirically) 19 Architectural Style Classification of Building Facades

2. The Classification Approaches

The Classification of Windows Romanesque Single arch Double arch Triple arch G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 21 Architectural Style Classification of Building Facades

Gothic Rose window Pointed arch G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 22 Architectural Style Classification of Building Facades

Baroque Triangular pediment Segmental pediment Balustrade G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 23 Architectural Style Classification of Building Facades

The Approach: Bag of visual Words (Bo. W) G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 24 Architectural Style Classification of Building Facades

Finding the best codebook • Exercise k (k-means clustering) and p (SIFT peak threshold) • Maximization(k, p) of the classification rate on the training set G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 25 Architectural Style Classification of Building Facades

Histograms and the classification model a) Romanesque b) Gothic c) Baroque • Codebook size = 90 (k = 30, p = 0. 03) • Nearest Neighbor (NN) classifier • Style -> MAX (Integrated Response (Romanesque, Gothic, Baroque)) G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. 26 Architectural Style Classification of Building Facades

The Classification of Towers Romanesque 27 Gothic Architectural Style Classification of Building Facades Baroque

The Classification of Traceries, Pediments and Balustrades Tracery - Gothic a) Trefoil b) Quatrefoil c) Cinquefoil d) Sexfoil G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Classification of Gothic and Baroque architectural elements, ” in Proc. of the 19 th IWSSIP, ser. LNCS, 2012, Vienna, Austria, pp. 330 -333. 28 Architectural Style Classification of Building Facades

Pediment, Balustrade - Baroque a) Triangular pediment b) Segmental pediment c) Triangular broken on the base d) Segmental broken on the top e) Swan’s neck pediment f) Balustrade G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Classification of Gothic and Baroque architectural elements, ” in Proc. of the 19 th IWSSIP, ser. LNCS, 2012, Vienna, Austria, pp. 330 -333. 29 Architectural Style Classification of Building Facades

The Classification of Domes Hemispherical Dome – Renaissance (Baroque, Neoclassical, Neo-Renaissance, Neo-Baroque) G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 30 Architectural Style Classification of Building Facades

Saucer dome - Islamic Onion dome – Russian Onion dome - Islamic G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 31 Architectural Style Classification of Building Facades

The Classification of Domes 1. Dome width > height -> Islamic 2. Golden color detection in YCb. Cr space -> Russian 3. Bag of visual words G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 32 Architectural Style Classification of Building Facades

2. Golden Color Detection in YCb. Cr Color Space • YCb. Cr advantage – luma-independent • • Y – luminance Cb and Cr – Chrominance • • Discard Y channel Find Cb, Cr ranges for golden color • The challenge! • Highly reflective golden color • Can appear from white to dark brown G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 33 Architectural Style Classification of Building Facades

2. Golden Color Detection in YCb. Cr Color Space • • Find initial Cb, Cr ranges Mean Cb, Cr pixel values of 97 golden patches Sort Cb values in ascending order Correlation! With Cb increase Cr decreases Low Cb, high Cr values – highly saturated golden color Correlation -> instead of 1 Cb, Cr range 4 Cb, Cr subranges Final Cb, Cr range on the training set • Maximize true positives • Minimize false positives Cb (31 -100]; Cr (134 -202] Cb (100 -110]; Cr (131 -142] Cb (110 -115]; Cr (126 -137] Cb (115 -120]; Cr (124 -130] G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 34 Architectural Style Classification of Building Facades

3. The Voting of Architectural Elements

The Voting Scheme • Maximum votes -> Romanesque, Gothic, Baroque • Mixture of Romanesque and Gothic üAll the 3 classes have equal number of votes üRomanesque and Gothic classes have equal votes, which is greater than the votes for Baroque class üRomanesque class has maximum votes; 2 and more Gothic votes Threshold Gothic votes by 2, since 1 vote could be false classification üGothic class has maximum votes; 2 and more Romanesque votes 36 Architectural Style Classification of Building Facades

Performance Evaluation, Results

The Image Datasets • • Challenge! No images datasets labeled by architectural styles Dataset collected from my own and Flickr (www. flickr. com) databases Facades of the corresponding architectural styles Facades featuring the corresponding architectural elements Dataset ground truth annotation by architectural styles Total 2315 images Generalization! Buildings spread over wide geographical area • Europe – Austria, Germany, Sweden, Czech Republic, Hungary, France, Spain, Luxemburg , England, Belgium, Switzerland, Armenia, Russia • North, South America, Asia 38 Architectural Style Classification of Building Facades

The Segmentation of Domes Renaissance, Neo-Renaissance, Baroque, Neo-Baroque, Neoclassical and Islamic styles Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th Iberoamerican Congress on Pattern Recognition (CIARP '12), ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. 39 Architectural Style Classification of Building Facades

The Results of the Dome and Tower Segmentation Approaches Towers Domes • The 1 st approach for segmentation of domes! • The 1 st approach for segmentation of towers! • Number of images 550 • Number of images 325 • Number of domes 550 • Number of towers 544 • Number of buildings 77 • Number of buildings 35 • 1 st strongest symmetry axis 95. 45% • 1 st strongest symmetry axis 93. 23% • Average segmentation rate 96% • Average segmentation rate 94. 77% Ø Robust to high perspective distortions Ø Scale invariant Ø Resolution independent Ø Color-independent – both day and night images Ø Complex scenes Ø Different weather conditions - fog, clouds, etc Gayane Shalunts, Yll Haxhimusa, Robert Sablatnig, “Segmentation of Building Facade Domes, ” in Proc. of the 17 th Iberoamerican Congress on Pattern Recognition (CIARP '12), ser. LNCS, vol. 7441, Buenos Aires, Argentina, 2012, pp. 324 -331. 40 Architectural Style Classification of Building Facades

The Results of the Classification Approaches Windows • Training set – 90 images • Testing set – 310 images • Classification rate on testing set 95. 16% Traceries, pediments, balustrades • Training set – 100 images • Testing set – 420 images • Classification rate on testing set 96. 67% ü Rotation, scale invariant (SIFT) ü Camera viewpoint invariant G. Shalunts, Y. Haxhimusa, R. Sablatnig, “Architectural style classification of building facade windows, ” in Proc. of the 7 th ISVC, ser. LNCS, vol. 6939, Las Vegas, USA, 2011, pp. 280– 289. G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Classification of Gothic and Baroque architectural elements, ” in Proc. of the 19 th IWSSIP, ser. LNCS, 2012, Vienna, Austria, pp. 330 -333. 41 Architectural Style Classification of Building Facades

The Results of the Classification Approaches Towers • Training set – 102 images • Testing set – 442 images • Classification rate on testing set 80. 54% ü Rotation, scale invariant (SIFT) ü Camera viewpoint invariant Domes • Training set – 100 images • Testing set – 420 images • Golden color detection module – 75% • Classification rate without golden color detection – 87. 14% • Final classification rate on testing set 90. 24% G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 41 Architectural Style Classification of Building Facades

The Results of the Dome Classification Approach • Training set - 100 images • Testing set - 420 images • Rotation, scale invariant (SIFT) • Camera viewpoint invariant • Golden color detection module – 75% • Classification rate without golden color detection – 87. 14% • Final classification rate on testing set 90. 24% G. Shalunts, Y. Haxhimusa, and R. Sablatnig, “Architectural style classification of Domes”, in Proc. of the 8 th ISVC, ser. LNCS, Rethymnon, Crete, Greece, 2012, pp 420 -429. 42 Architectural Style Classification of Building Facades

STYLE Demo

What is the architectural style of Votivkirche? 44 Architectural Style Classification of Building Facades

STYLE GUI 45 Architectural Style Classification of Building Facades

The segmented architectural elements – towers, windows, traceries 46 Architectural Style Classification of Building Facades

Votivkirche is Gothic! Well, it is neo-Gothic… 47 Architectural Style Classification of Building Facades

Comparison with the state-of-the-art STYLE Mathias et al. • Algorithm - general • Styles - Romanesque, Gothic, Baroque • Planar, non-planar facades • Full and partly occluded facades, facade parts • Database – photos of individual facades or parts • Algorithm - 3 steps 1. Segmentation of architectural elements 2. Classification of architectural elements 3. Style classification -Voting of elements • Algorithm - specific • Styles - Flemish Renaissance, Haussmannian, Neoclassical • Planar facades • Full facades • Database - photos taken from a mobile mapping van • Algorithm - 4 steps 1. Scene classification No building, Street, Facade, Building part 2. Image rectification (Planar facades) 3. Facade splitting (Facade blocks into individual facades) 4. Style classification M. Mathias, A. Martinovic, J. Weissenberg, S. Haegler, L. Van Gool, “Automatic Architectural Style Recognition”, in Proc. of the 4 th International Workshop on 3 D Virtual Reconstruction and Visualization of Complex Architectures, Trento, Italy, 2011, pp 280 -289. 48 Architectural Style Classification of Building Facades

Conclusions • • • First general algorithm of image-based architectural style classification of facades Three major steps – segmentation, classification, voting of architectural elements Realization on Romanesque, Gothic, Baroque styles The explored elements – dome, tower, window, tracery, pediment, balustrade First approaches of segmentation of domes and towers Segmentation approaches – pipeline of bilateral symmetry detection, graph-based segmentation, image analysis and processing techniques • Classification approaches • Clustering and learning and of local (SIFT) features, Bo. W, NN classifier • Golden color detection in YCb. Cr color space • Voting – maximum votes -> architectural style! • Outlook – possible to extend further with new architectural elements and styles • General algorithm • Software implementation as independent segmentation, classification, voting modules 49 Architectural Style Classification of Building Facades

Thank you!