Locational Analytics Spatial DecisionMaking and Big Data Research

Locational Analytics, Spatial Decision-Making and Big Data: Research and Teaching Overview of Spatial Big Data and Analytics (8: 40 -9: 15 am) James B. Pick University of Redlands School of Business James_pick@redlands. edu Pre-ICIS Workshop on Locational Analytics, Spatial Decision-Making and Big Data: Research and Teaching Dublin, Ireland, December 11, 2016 Sponsored by SIGGIS Association for Information Systems

The Goal: Solve a Spatial Big Data Problem • Consider if you had the data on all the graduate students studying in the U. S. • 1. 7 million according to U. S. Dept. of Education. • You are analyzing their recording with real-time updating, as the data change from day to day. You have 2 years of the data updated on a daily basis. For each graduate student you have location (lat. -long. ), 25 characteristics, a photo, free-form audio recordings about the student’s background and readiness, and sample video in which the student discusses his/her graduate study goals. • How would you approach organizing the data, so an analyst wishing to study trends in graduate student goals and interests could narrow the data down and do the necessary analytics to gain value? Keep in mind that the data are in varied formats (numbers, addresses (x-y), text, data-base, video, audio). • These types of problems are ones this workshop seeks to introduce the skills to address, and the answers for. 2

Definition of Spatial Big Data • Big Data are “data sets that are so big they cannot be handled efficiently by common database management systems” (Dasgupta, 2013). • Big Data have volume of 100 terabytes to petabytes, have structured and unstructured formats, and have a constant flow of data (Davenport, 2014) • Spatial Big Data represents Big Data in the form of spatial layers and attributes. • There is no standard threshold on minimum size of Big Data or Spatial Big Data, although big data in 2013 was considered one petabyte (1, 000 terabytes) or larger (Dasgupta, 2013). • Big Data are getting unbelievably large • More video is captured daily today than happened in the initial 50 years of television • Amount of data available today. More than 2. 8 zettabytes (2. 8 trillion gigabytes). 3

Big Data – A Brief Review So, we know that “big data” is BIG… But, what does that mean to us? (source: courtesy of Brian Hilton)

New IDC Forecast Sees Worldwide Big Data Technology and Services Market Growing to $48. 6 Billion in 2019, Driven by Wide Adoption Across Industries sss. idc. com 09 Nov 2015 FRAMINGHAM, Mass. , November 9, 2015 – The Big Data market continues to exhibit strong momentum as businesses accelerate their transformation into data-driven companies. This momentum is driving strong growth in big data-related infrastructure, software, and services. A new forecast from International Data Corporation (IDC ) sees the big data technology and services market growing at a compound annual growth rate (CAGR) of 23. 1% over the 2014 -2019 forecast period with annual spending reaching $48. 6 billion in 2019. And a new IDC Special Study examines spending on big data solutions in greater detail across 19 vertical industries and eight big data technologies. "The ever-increasing appetite of businesses to embrace emerging big data-related software and infrastructure technologies while keeping the implementation costs low has led to the creation of a rich ecosystem of new and incumbent suppliers, " said Ashish Nadkarni , Program Director, Enterprise Servers and Storage and co-author of the report with Dan Vesset , Program Vice President, Business Analytics & Big Data. "At the same time, the market opportunity is spurring new investments and M&A activity as incumbent suppliers seek to maintain their relevance by developing comprehensive solutions and new go-to-market paths. " All three major big data submarkets – infrastructure, software, and services – are expected to grow over the next five years. Infrastructure, which consists of computing, networking, storage infrastructure, and other datacenter infrastructure-like security – will grow at a 21. 7% CAGR. Software, which consists of information management, discovery and analytics, and applications software – will grow at a CAGR of 26. 2%. And services, which includes professional and support services for infrastructure and software, will grow at a CAGR of 22. 7%. ……. As big data matures, IDC expects its share of the larger Business Analytics market to increase…. . The availability and skill level of big data IT and analytics talent will also have a direct impact on the market.

(source: courtesy of Brian Hilton)

Sources of Spatial Big Data • Sources of Spatial Big Data include: • GPS, including • GPS-enabled devices • Satellite remote sensing • Aerial surveying • Radar • Lidar • Sensor networks • Digital cameras • Location of readings of RFID • Mobile devices • Internet of things (Partially based on Dasgupta, 2013) 7

Where is this Big Data coming from? It’s from the Mobile Planet and Internet of Everything… We’re About Here (modified from Brian Hilton)

Where is this Big Data coming from? It’s User-Generated Content… (source: courtesy of Brian Hilton)

Where is this Big Data coming from? It’s Sensor Data… (source: courtesy of Brian Hilton)

Where is this Big Data coming from? It’s all these “Smart” “Things”… (source: courtesy of Brian Hilton)

Five V’s of Spatial Big Data • Volume • • Satellite imagery covers the globe so is vast. Sensors are expanding worldwide at a rapid rate. Digital cameras have reached several billion through spatially-reference cell phones. One estimate indicates that 2. 5 quintillion bytes are generated daily worldwide. (www. ibm. com). 2. 5 with 18 zeros. • Variety • The form of data is based on 2 -D or 3 -D points configured as vector or raster imagery. This is entirely different than conventional big data which is alphanumeric or pixel-based (similar to raster but not vector) • Velocity is very fast since imagery travels at speed of light. 12

Five V’s of Spatial Big Data (cont. ) • Veracity Attribute veracity • For attribute (non-spatial) data, do the data meet data quality tests? • Cross checking totals against other sources or historical trends • Examination of outliers • Review and audit of data collection techniques Spatial veracity • For vector data (imagery based on points, lines, and polygons), the quality varies. It depends on whether the points have been GPS determined, or determined by unknown origins or manually. Also, resolution and projection issues can alter veracity. • For geocoded points, there may be errors in the address tables and in the point location algorithms associated with addresses • For raster data (imagery based on pixels), veracity depends on accuracy of recording instruments in satellites or aerial devices, and on timeliness. 13

(source: courtesy of Brian Hilton)

Five V’s of Spatial Big Data (cont. ) • Value • For real-time spatial big data, decisions can be enhance through visualization of dynamic change in such spatial phenomena as climate, traffic, social-media-based attitudes, and massive inventory locations. • Exploration of data trends can include spatial proximities and relationships. • Once spatial big data are structured, formal spatial analytics can be applied, such as spatial autocorrelation, overlays, buffering, spatial cluster techniques, and location quotients. 15

How does Big Data differ from traditional datasets used for over 15 years? You can see that the traditional datasets could be quite large, but they were traditionally formatted in spreadsheets or data-bases, tended to be static, and were designed to prove hypotheses. (Modified from Davenport, 2014) By contrast, Big Data has the 5 Vs and can use machine learning, which pushes out solutions by seeing what works in big datasets. The statistical term is exploratory. 16

Spatial Big Data – Example of Locations and Movement of Central New York City Taxicabs, based on space, time, and attributes A user-friendly interface Taxi. Vis allows users to view and analyze the patterns and movements of over 173 million taxi trips daily in central NYC. The data from NY Taxi and Limousine Commission gives pickup and drop off locations, time, and attributes. Commercial map rendering is done using Google Maps, Bing Maps and Open. Street Map. Simple or complex queries can be done. Balance between simplicity and expressiveness. The example shows taxi trips from lower Manhattan area to La. Guardia airport area (upper part of image) and Kennedy airport area (lower part). The volume of trips are given in the lower hourly graphs for Sundays in May 2011 (left) and Monday (right), with blue for La. Guardia and red for Kennedy. (Source: Ferreira et al. , 2013) 17

New York City Taxi example – further capabilities • Side-by-side “sensor” maps over time • Visual queries for pick-up AND dropoff • Constraints of attributes of taxi id, distance traveled, fare, and tip amount • Enables economic analysis • Complex queries. • Use set-theoretic functions on simple queries • Level-of-detail reduced the number of points shown on the map. • Done by hierarchical sampling of point cloud • Density heat maps • Different visualizations (Source: Ferreira et al. , 2013) 18

Spatial Big Data and Analytics NYC Taxi Data - includes driver details, pickup and drop-off locations, time of day, trip locations (longitude-latitude), cab fare and tip amounts. An analysis of the data, for instance, shows that: • Almost 50% of the trips did not result in a tip, • The median tip on Friday and Saturday nights was typically the highest, and • The largest tips came from taxis going from Manhattan to Queens. Was a tip paid for the trip? (Binary Classification) What was the tip amount range? (Multiclass Classification) What was the tip amount? (Regression) How agglomerated are the origin points of the taxi rides? (Spatial Autocorrelation, Moran’s I)

Spatial Autocorrelation Patterns Measured by Moran’s I Source: Longley, P. et al. (2011). Geographic Information Systems & Science, Wiley, p. 103. 20

Big Data Analytic Traditional Techniques What is enabling them? • • • Classification Clustering Regression Simulation Anomaly Detection Numerical Forecasting Optimization Geographic Mapping … Limitations. For Big Data, they often cannot handle well the 3 V’s of volume, velocity, and variety They tend to work best with “Small Data” (modified from Brian Hilton)

“Non-traditional” Big Data Analytic Techniques • Ensemble methods • Combine multiple models, e. g. linear regression, decision tree, neural network, spatial autocorrelation work together to yield one answer. • Commodity models • Apply complex models to address only the high-value data. • For most of the data, use simple, less resource -intensive model(s) • Modern Data Visualization • Multiple graphs and charts linked to the same underlying Big Data, and displayed in Dashboards, including maps • Space-Time slider visualiizations, showing locational changes in a movie-like sequence. • 3 -D Displays. 3 -D Mapping. (Partial source: Franks, 2012) • Text Analysis (Content Analysis) • Appropriate for unstructured text. Opens up social media, call center conversations, etc. for powerful analytics. Parse the text and use the components to extract meaning, valence, and feelings. • Spatial Analysis • Spatial sampling, auto-correlation, continuous contours (ocean, air), etc. • Analytic Point Solutions • Software to solve very specific Big Data, Analytics problems. (e. g. Esri’s Arc. Logistics. • Virtual Reality • Google VR • Can include fictional or actual geographic mapping • Machine Learning • AI-based programs that can learn without having been specifically pre-programmed them for the application. • “Intelligent” Robotics is one type • Neural networks verges on ML, but they are often restricted to learning in specialized ways

Example of Spatial Space-Time Big Data and Analytics NYC Taxi Data – 48 hour period – 30 and 31 December 2013 Emerging Hot Spot Analysis Space-Time Cube Analysis

Spatial Big Data and Analytics Sporadic Hot Spots Oscillating Hot Spots New Hot Spots (source: courtesy of Brian Hilton)

Spatial Big Data and Analytics Oscillating Hot Spots Sporadic Hot Spots New Hot Spots (source: courtesy of Brian Hilton)

Big Data Analytic Platforms What is enabling them? • Lower Cost • Greater Storage (HD and RAM) • Faster Input / Output Operations • Faster Processing • Increased Bandwidth Since 1990, the average price per MB of memory has dropped from $59 to 0. 49 cents – a 99. 2% price reduction. At the same time, the capacity of a memory module has increased from 8 MB to a 8 GB. (source: Microsoft, courtesy of Brian Hilton)

Spatial Big Data Platforms CEP = complex event processing, SOLAP = spatial online analytical processing. ETL = extract, transform and load, UI/UX = user interface/user experience design. Interactive Analytics System—adopted from Lee and Kang (2015) 27

Big Data Analytic Platforms What is enabling them? • Cloud / Distributed Computing • New Data Management Tools (Hadoop, etc. ) • New Technologies (Spark, etc. ) • Ease-of-Use (Browser-based, etc. ) (source: courtesy of Brian Hilton)

Big Data Analytic Software - Tableau Example of the Benefits of Big Data and Analytics Analysis of Building Permits over five years in Seattle, Washington, using Tableau is a good teaching software product for spatial big data. It allows import of very large data-sets from Excel (a million+ records are fine), as well as data-bases. Tableau has limited analytics and simple mapping. However, it has strength in its intuitiveness, user friendliness, and ease in composing Dashboards, such as the on the right.

Example’s “Big Data” Set (50552 rows) What’s missing for this example of Big Data? Sufficient Volume? Variety Velocity

Big Data Analytic Platforms How do we use them for Analysis? (source: courtesy of Brian Hilton)

Dr. John Snow Dr. Snow is frequently referred to as the 'father of public health. ' In 1854 a cholera epidemic raged across Europe. The onset of the disease is sudden and death can result in as little as a week. In London, one devastating outbreak claimed the lives of more than 500 people in just ten days. The search for the cure and the cause was furious and fruitless. Dr. Snow had observed cholera first-hand in 1831 as an apprentice surgeon, but it was only 17 years later, in 1848 -1849, that he developed a new theory for the mechanism of cholera transmission. Contrary to the prevailing belief, Snow argued that cholera was a disease of the gut and that the causal agent must enter through the mouth and then multiply within the gut of the sufferer, subsequently spreading to others. Dr. Snow reasoned that broad transmission of cholera had to be due to contaminated drinking water. In September 1854, when Dr. Snow was called on to examine the causes of the cholera epidemic, he turned immediately to the water supply. His previous research suggested that the localized nature of the outbreak would mean that the cause had to be a contaminated pump or well, rather than a problem with the general water supply. He discovered that while there were five water pumps in the neighborhood, most of the deaths took place near the pump on Broad Street. Upon further investigation he discovered that among the deaths of people situated farther from the Broad Street pump, half of the deceased preferred the water from the Broad Street pump to their nearer pump, and another third attended school near the ill-fated pump. Upon presentation of his findings to community leaders, the handle of the Broad Street pump was removed, and the epidemic quickly abated. Further investigation of the well discovered that a sewer pipe underground was leaking raw sewage into the drinking water of the Broad Street pump. Dr. Snow realized that a spot map illustrating the location of the deaths in the Broad Street cholera outbreak would be a useful addition to his report. Snow's famous map was first exhibited at a meeting of the Epidemiological Society of London in December 1854. (source of this slide and next 7 maps: courtesy of Brian Hilton)

John Snow Map, 1854 Soho, London, England Cholera deaths are in black Pic ca dil ly Cir cu s Regent Street

John Snow Map, 1854 Soho, London, England

John Snow Map, 1854 Soho, London, England Pump locations are circled

Wa ter Pum p John Snow Map, 1854 Soho, London, England s a Loc n tio a N nd u er b m o ath e f. D

John Snow Map, 1854 Soho, London, England

160+ Years Later Soho, London, England

2015 map / 1854 map Soho, London, England Locations of water pumps and deaths

2015 map / 1854 map Soho, London, England Density of location of deaths

2015 map / 1854 map Soho, London, England Br St d a o mp u P et re Statistically significant “hot spots” of deaths

Applications of Spatial Big Data and Analytics • • • Politics Transportation Supply Chain Management Public Safety Urban Traffic Emergency Management Healthcare Energy and Environment Climate Science Marketing/Advertising 43

Energy management at Bathworks using Big Data, with mapping • American Bathworks Inc. is a manufacturer and supplier of bathroom plumbing features for buildings in U. S. Spatial big data is important. • Delivery fleet. For any vehicle, the facilities manager knows in real time the locations, distance traveled for one day or total, average, peak speeds, acceleration/braking patterns (Spatial). If the patterns are wasteful of energy or risky for the driver, reminder e-mails and text messages are sent. • If this approach seems invasive to some employees, they can elect a non-company car. • Energy management group monitors and controls energy consumption of Bathworks’s heating air conditioning, and ventilation ((HVAC) systems. • More than 23, 000 building spaces are monitored by “temperature, humidity, light levels, and human presence. ” (Spatial analytics of big data – could be done using GIS software, analytics software, or spatial analytics software) • Active building control of temperature, windows, shades. Know about occupancy of parts of building, airflow maintenance. (Source: Davenport, 2013) 44

Electric Utilities, a laggard in Big Data, but catching up • Utilities need to provide more informed support for “enterprise decisions around where to invest in new generation sources, transmission lines, and operational questions about real-time energy management decisions, and how consumers utilized energy. “ • Since all these factors have spatial components, GIS should be a major part of the much expanded gas usage facilities and consumer uses of energy. • All these factors depend on their spatial location, so GIS permeates what can be done with spatially-referenced GIS data-sets. Mobile GIS is also highly relevant in collecting field information as well as conducting repairs and maintenance in the field. • The rapidly growing renewable energy sources of solar, wind, and geothermal are all geographically based, and add to utilties spatial data. (Modified from Davenport, 2013) 45

Spatial Big Data and Analytics How do we / will we use them for spatial-temporal: analysis? data mining? machine learning? knowledge discovery? visualization? …

Spatial Big Data and Analytics What are / will be the workflows? How will data move through these platforms? data > non-spatial analysis > spatial analysis data > spatial analysis > non-spatial analysis > spatial analysis

Questions still unanswered with Big Data • How will Spatial Big Data affect organizational processes. • One possible trend is towards centralization of data in the Cloud, after decades of decentralization. • Concern about privacy invasion and targeting from Big Data. • The appeal to unsuspecting users can come from it being “clothed” in social media (Foursquare) or retail discounting. • A backlash against this intrusion is likely • How will Big Data and Analytics change decision-making. • To what extent will human managers and decision-makers override the results of Big Data. 48

Summary on Big Data, Spatial Big Data, and Analytics • Big Data refers to huge data-sets that overflow ordinary data management systems. • The 5 V’s define big data including Volume, Variety, Velocity, Veracity, and Value. • Spatial Big Data is Big Data that is spatially referenced, so in addition to common analytics techniques, mapping and spatial analytics can be applied. • Ordinary, small-data approaches will not work, because most of the traditional techniques cannot perform exploration of massive data sets. • Big Data methods allow multidimensional screening and “data mining” to locate parts of the mass that are showing interesting relationships, trends, or comparisons. • Those interesting parts of a Big Data Set can be sorted into small data-sets that can have the more powerful traditional analysis methods applied to them. • The management issues of Big Data are not yet figured out. • Success need to be studied from a management and organizational standpoint to understand what works managerially and results in profits and other benefits. 49

Questions? ? Discussion