CUPUM 2007 Foz do Iguau Brazil NEW INFORMATION

CUPUM 2007 – Foz do Iguaçu, Brazil NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT: Intelligent Middleware for Understanding Neighborhood Markets Prof. Joseph Ferreira, Jr. and Dr. Raj Singh MIT, Urban Information Systems Department of Urban Studies & Planning July 12, 2007 CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007

NEW INFORMATION TECHNOLOGIES FOR COMMUNITY DEVELOPMENT l Applaud recent interest in communtiy data centers, mashups, and PPGIS l l Problems l l l But, current practice cannot meet high expectations Data maintenance and currency Limited and inflexible modeling & analysis Solution – more IT ? ? !! l l l Modular service chaining - (geo)web services, SOA Intelligent middleware for modeling and analysis Lessons (even without adding new IT) CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 2

Trends in IT for Community Development l Neighborhood Information Systems l l l Local Data Centers serving NGOs & sharing data PPGIS (public participation GIS - beyond census tables) Public/private/non-profit partnerships Examples: NNIP website: http: //www. urban. org/nnip Mashups and Web 2. 0 l l l Internet GIS: Arc. IMS, Map. Server, Geo. Server… Map ‘mashups’ on Google Earth/Maps, Virtual Earth, Yahoo maps… User-generated content: Flickr photos, social networking… CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 3

GIS Office for State of Massachusetts Mass. GIS: http: //www. mass. gov/mgis/ CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 4

Mass. GIS: Example of very good online mapping and download site for typical US State CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 5

Google Earth Mashup: Nanaimo, Canada l l City website: http: //earth. nanaimo. ca/data. html Nanaimo. Master and Nanaimo. Virtually. Downtown using Google Earth+Map. Guide CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 6

National Neighborhood Indicators Partnership: http: //www. urban. org/nnip Nongovernmental organizations focused on needs of particular communities CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 7

The Providence Plan Providence Rhode Island USA CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 8

Neighborhood Knowledge Los Angeles http: //nkla. sppsr. ucla. edu Land Opportunities Tracking System http: //lots. ucla. edu CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 9

Limitations of Neighborhood Information Systems l Keeping data up to date l l No problem for decennial census data Big problem for land use/ownership (cadastre) and permitting More difficult as more parties try to share data Interpreting & Analyzing ‘official’ data l l l Must clean/re-interpret data for local use Hard to include ‘local knowledge’ Customized analyses are useful but non-trivial CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 10

Intelligent Middleware for Understanding Neighborhood Markets Urban Markets Initiative Pilot Project A collaborative effort among l. Massachusetts Institute of Technology (MIT), l. Boston Metropolitan Area Planning Council (MAPC) l. Boston’s Department of Neighborhood Development l. The Boston Foundation (TBF) l. Open Geospatial Consortium (OGC) with support from the Urban Markets Initiative of The Brookings Institution and participation of local CDCs. CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007

People Working on the ‘Intelligent Middleware’ Project Principal Investigator: Joe Ferreira, Professor, MIT Principal Collaborators: Holly St. Clair, Director, MAPC Data Center Larry Braman, Manager, Mapping and Data Services, Boston DND Charlotte Kahn, Director, Boston Community Indicators, TBF MIT Research Team: Raj Singh, system architect (now at OGC) Jiawen Yang, database services (now Assist. Prof. Georgia Tech) Myounggu Kang, mapping services (now Assist. Prof. Seoul U. ) Mi Diao, Xiongjiu Liao, Weifeng Li (Ph. D candidates) Sponsored by: The Brookings Institution, Urban Markets Initiative CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007

‘Intelligent Middleware’ Project Goals and Concepts Facilitate data sharing within metropolitan areas l Without everyone needing their own ‘data center’ l Because everyone needs to change their copy of the data Allow reinterpretation of official data l Based on local knowledge l Without explicitly changing the original data l Codify local knowledge as rules to interpret official data on the fly Deliver results into familiar desktop environment l Tables into spreadsheet or browser; maps as mashups l Examples: Identify key landowners, targeted land uses, … CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 13

‘Smart’ Middleware Alternative Intelligent Middleware for Understanding Neighborhood Markets Police Dept. Data Sets Assessing Dept. Data Sets Health Dept. Data Sets Housing Dept. Data Sets Read-only, ‘Official’ data repositories Processing Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, … Intelligent data intermediaries Housing portal Service Environmental portal CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 14

One Example: Land use & Ownership Patterns l Want: l l l REPORT & MAP of largest landowners: Top-10 Owners Keep map/report accurate and up to date Allow flexibility in definition of ‘neighborhood’ Allow flexibility in which land uses to consider (residential, open space…) Problem: Parcel map & Assessing database not enough l l Spelling problems: many different spellings of city agencies and other large landowners Updating: old snapshot of parcels is out of date CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 15

Desired Output: Table, Chart, and Map in Familiar Desktop Environment CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 16

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 17

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 18

Middleware Example: Prototype NGO Webpage (for DSNI) CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 19

…but Top-10 Landowner List has Problems: Results BEFORE Standardizing Owner Names For Dudley Street Neighborhood Initiative (DSNI) Neighborhood: OWNER NAME (Boston parcels, FY 2005) parcels acres CITY OF BOSTON 182 37. 85 CITY OF BOSTON BY FCL 123 9. 76 DUDLEY NEIGHBORS INC 121 12. 14 COTTAGE BROOK HOUSING LP 48 2. 38 KEPNES EDWARD N TS 44 4. 02 ALEXANDER MAGNOLIA LP 39 3. 21 BOSTON HOUSING AUTHORITY 36 2. 27 CITY OF BOSTON FCL 27 1. 82 DUDLEY_NEIGHBORS_INC 21 1. 68 RAIMONDI ROBERT A SR TS 19 0. 51 660 75. 6 Total Parcels/Acres owned by largest owners CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 20

Results AFTER Standardizing Some Owner Names For Dudley Street Neighborhood Initiative (DSNI) Neighborhood OWNER NAME (Boston parcels, FY 2005) parcels acres CITY OF BOSTON 339 50. 12 DUDLEY NEIGHBORS INC 145 14. 06 COTTAGE BROOK HOUSING LP 48 2. 38 KEPNES EDWARD N TS 44 4. 02 ALEXANDER MAGNOLIA LP 39 3. 21 BOSTON HOUSING AUTHORITY 36 2. 27 RAIMONDI ROBERT A SR TS 19 0. 51 ORCHARD PK REVITALIZATION 18 0. 81 HOWARD DACIA LLC 16 1. 43 ORCHARD PARK REVITALIZATION 16 2. 92 720 81. 7 Total Parcels/Acres owned by largest owners CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 21

Solving the Spelling Problem l l Problem: many different spellings of city agencies and large landowners in parcel data Solution choices: l l l Top-down: standardize spellings in official datasets Bottom-up: fix spellings in copy of dataset Middle-out: accumulate spelling changes separately and apply on-the-fly as needed …build ‘middleware’ for middle-out strategy CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 22

Example of RULE to Fix Spelling Error • Standardize OWNER names • Create ‘update’ RULE to fix spellings • Apply on-the-fly to Boston’s parcel data (BASETABLE) to produce temporary tables (FACADES) with spelling corrections • Use temporary table for REPORTS and THEMATIC MAPS • Example of one RULE for City of Boston UPDATE fas_parcel_bos 05 SET OWNER_FY 05='CITY OF BOSTON' where OWNER_FY 05 like '%CITY%' and ((OWNER_FY 05 like '%BOSTON%' or OWNER_FY 05 like %BSOTON%') and OWNER_FY 05 not like '%CITYWIDE%' and OWNER_FY 05 not like '%SUITES%' and OWNER_FY 05 not like '%LIGHT%‘ and OWNER_FY 05 NOT LIKE '%PROPERTIES%' and OWNER_FY 05 NOT LIKE '%LLC%' and OWNER_FY 05 NOT LIKE '%L L C%' and OWNER_FY 05 NOT LIKE '%CREDIT UNION%)'; CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 23

‘Intelligent Middleware’ Project Goals and Concepts l Facilitate data sharing within metropolitan areas l Allow reinterpretation of official data based on local knowledge Deliver results into familiar desktop environment l HOW? l Modular design: interoperable parts, distributed access, secure access control l Data intermediary manages ‘virtual’ data center middleware l Middleware uses web services and open standards l Middleware is hidden from end-user applications l User/group logins and SSL provide data sharing control CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 24

‘Smart’ Middleware Alternative Intelligent Middleware for Understanding Neighborhood Markets Police Dept. Data Sets Assessing Dept. Data Sets Health Dept. Data Sets Housing Dept. Data Sets Read-only, ‘Official’ data repositories Processing Normalize, trim, merge, transform, overlay, cookie-cut, extrapolate, qualify, … Intelligent data intermediaries Housing portal Service Environmental portal CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 25

Middleware Components Concept Example Delivery Thematic MAP Map of land owned by largest landowners Google Maps pinmap showing parcels owned REPORT Table listing largest landowners in neighborhood Excel spreadsheet showing acreage of Top-10 owners FACADE (Virtual) parcel records with corrected spellings Manage façade rules and sharing via browser RULES SQL query to standardize spelling of owner names Apply SQL rules that fix parcel data on-the-fly Study Area Service area of Dudley Street Neighborhood Initiative Reports are for userdefined study areas BASETABLE Parcel data from Boston’s Assessing Office Use read-only copy of city data CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 26

Tables are not static but result from ‘web query’: http: //uis. mit. edu/umi/service/report. php? Request=Query. Report& name=top_own_parcels_a_jf 1& tablename=own 05_mixed_after& studyarea=dsni& xsl=http: //uis. mit. edu/umi/xl/report. xsl CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 27

Same URL in Browser: http: //uis. mit. edu/umi/service/report. php? Request=Query. Report& name=top_own_parcels_a_jf 1& tablename=own 05_mixed_after& studyarea=dsni& xsl=http: //uis. mit. edu/umi/xl/report. xsl CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 28

Thematic Maps are also web services: http: //uis. mit. edu/umi/mapservice/umimap. php? request=Get. Map& layers=top_own_parcels_map 1& WIDTH=500&HEIGHT=500& SRS=EPSG: 4326& TRANSPARENT=true Map requests conform to Web Mapping Standard (WMS) of Open Geospatial Consortium (OGC) • Many applications now support WMS • E. g. , Excel, Google, Arc. GIS, … CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 29

A WMS call to the middleware services yields a thematic map top-10 landowners for all of Boston The URL: http: //uis. mit. edu/umi/mapservice /umimap. php? request=Get. Ma p&layers=testjf 0004 The same URL works from inside of Arc. Map (see next slide) CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 30

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 31

Middleware Components l l l Prototype on RHEL 4 Linux PC running open source applications Database server: Postgres with Postgis spatial extensions Mapping server: Map. Server (or Geo. Server) Web server: Apache 2 (with Tomcat) Web services programmed in PHP Additional middleware management tools in AJAX (asymmetric javascript and XML) CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 32

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 33

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 34

CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 35

Middleware Use: More than Spelling Fixes… Understand Land Use and Ownership Patterns: l Identify “top-10” landowners in neighborhood l Trace effective ownership: who controls ABC Corp. l Track (un)desired land use, vacant land suitable for reuse… l Study trends: gentrification impacts of condo conversion, predatory lending practices, … Measure urban performance: l Model affordability, accessibility, … l Estimate retail ‘gaps, ’ development potential, … CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 36

Conclusions: (1) Key Concepts Sharing data within a metropolitan area l l Avoid ‘data center’ limitations as small geographies and large/changing datasets explode data needs Provide mechanism for accumulating, utilizing, and sharing local knowledge about land use, ownership, and market potential (independently of official datasets) Use web services approach, with interoperability standards, to build ‘lego blocks’ for many management and end-user applications. Examine the skill sets and institutional relationships that can take advantage of ‘service oriented architectures’. CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 37

Conclusions: (2) Community Development Implications l l l As volume and detail of ‘relevant’ data grows, traditional sharing of data and tools becomes more problematic l Corrolary: Google Earth is great but can’t meet expectations Web services and ‘middleware’ offer promising avenues for deeper sharing, but l Require different mix of skills and institutional arrangements l Won’t be offered soon by vendors (unless we push!) l Suggests new direction and short-term strategies More complex urban models and performance indicators have limited use unless built as plug-and-play modules that provide tunable web services. CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 38

Thanks… Joe Ferreira, jf@mit. edu Raj Singh, rajsingh@mit. edu ‘Middleware’ project website: http: //uis. mit. edu CUPUM 07: Intelligent Middleware - Joe Ferreira - July 12, 2007 39