Big Data and Official Statistics The UN Global

Big Data and Official Statistics The UN Global Working Group http: //unstats. un. org/unsd/trade/bigdata/ Ronald Jansen Chief, Trade Statistics Branch United Nations Statistics Division jansen 1@un. org and Big. Data@un. org

Overview What is Big Data? What is Big Data for Official Statistics? What are the Challenges? UN Global Working Group on Big Data for Official Statistics

What is Big Data? Big Data characteristics q Volume q Velocity q Variety Big Data benefits v Auto-generated v Timely v Granular Big Data sources: o Mobile Devices o Digital Transactions o Sensors o Social Networking

Big Data for Development UN GLOBAL PULSE o Research & Development o Big Data Partnerships o Pulse Lab Network § New York § Jakarta § Kampala

Big Data for Development UN Global Pulse o Early warning o Real-time awareness o Real-time feedback

What is Big Data for Official Statistics? Big Data sources / Statistics • Mobile Phone Positioning Data / Tourism statistics • Supermarket scanner data / Price statistics • Vehicle tracking device / Transport statistics • Twitter/ Consumer confidence statistics • Satellite imagery / Agriculture statistics

What are the Challenges? Big Data Challenges • Methodology • Privacy • IT infrastructure • Human skills • Partnerships

Methodological Challenges Methodology • Representativeness ? • Volatility (no time series? ) • Standardization ? • Modelling

Partnership Challenges Big Data providers Data Sources National Statistical Offices - Standards and methodology Big Data partnerships opportunities Research institutes and technology providers - Upstream processing - IT infrastructure, Academia and scientific communities - Analytical capacity and modelling

Example of use of Mobile Phone data for Official Statistics

Mobile Phone Positioning Data Call Detail Record (CDR) Data Detail Record (DDR) (Internet Protocol Detail Record) Location Updates (LA) Radio coverage updates (Abis data)

Example 1: Estonia Data provider/ source • Mobile phone operator • Provides raw data to research company Research company Shared responsibility for data processing and analysis • Provides IT infrastructure • Preparation of data • Ensuring privacy and confidentiality • Providing modeling and Academia analytics National Statistical Office • Receives preprocessed and aggregated data • Prepares and disseminates statistics

Example 2: Netherlands Responsibility for data Data source/ provider • Vodafone • Makes data available to research institute Research company • Intermediate commercial partner • Provides IT platform for data filtering and privacy National Statistical Office • Receives preprocessed Big Data • Analyze, prepare and disseminate statistics

UN Global Working Group on Big Data for Official Statistics

Milestones Ø Feb 2013 Friday Seminar on Big Data Ø Mar 2014 Report to Statistical Commission Decision 45/110 – GWG on Big Data Ø Oct 2014 Conference on Big Data – Beijing Ø Mar 2015 Report of GWG to Stat Commission Ø Oct 2015 Conference on Big Data – Abu Dhabi

UN Global Working Group on Big Data Mandate (Decision 45/110 – 2014) 1. Provide strategic vision, direction and coordination of a global programme on Big Data for official statistics; 2. Promote practical use of sources of Big Data for official statistics, while building on the existing precedents in Big Data and finding solutions for i. Methodological issues, ii. Legal issues of access to data sources; iii. Privacy issues iv. Data security issues; v. Cost benefit analysis

UN Global Working Group on Big Data Mandate (continue) 3. Promote capacity building, training and sharing of experience; 4. Foster communication and advocacy of use of Big Data for policy applications; 5. Build public trust in the use of private sector Big Data for official statistics including issues of confidentiality and privacy.

Composition of the GWG Countries Australia Denmark Italy Mexico Netherlands USA Bangladesh China Colombia Morocco Philippines Tanzania Egypt UAE Oman International Organizations ITU OECD UN Global Pulse UNECE UNESCAP UNSD World Bank

Task Teams 1. 2. 3. 4. 5. 6. 7. 8. Advocacy and Communication Big Data and SDG indicators Access and Partnerships Skills, Training, Capacity Building Cross-cutting issues (Quality framework) Mobile Phone Data Satellite Imagery Social Media Data

Strong participation o o o o Orange, Telenor, World Economic Forum, NASA, Google, Data Pop, World Pop, Flowminder, UNU-EHS, ODI (UK), Positium, UPenn, Harvard

Advocacy and Communication 1. Advocacy/information materials on Big Data for official statistics, such as brochures, papers, presentations, videos, etc. ; 2. Good practices of using big data in official statistics, such as use cases; and 3. Joint materials with other task teams to promote and communicate the outcomes of their work

Big Data and SDG indicators 1. Assessment of the potential use of Big Data for monitoring progress on the 169 SDG targets by conducting a survey, taking account of the draft proposals of indicators, 2. An inventory of research work on Big Data that could be used to estimate one or more SDGs, and 3. Good practices and lessons learned from a pilot research in 1 -2 countries on estimating 2 -3 SDG indicators using Big Data.

Access and Partnerships 1. 2. 3. 4. Assessment of existing good practices of data access A set of principles as conditions for access to Big Data sources Templates for MOUs or services agreements Good practices and examples of partnerships

1. Training, Skills and Capacity Building Assessment tool for the needs of Big Data skills and for the institutional ‘readiness’ for Big Data skills; 2. Corresponding training curriculum; 3. Technical seminar on Big Data in 2015; and 4. MOU for a global network of training institutions on Big Data

Cross-cutting issues 1. 2. 3. Inventory of methodologies, data analytics /visualisation tools as well as quality assurance frameworks and refined classification of Big Data; Examples and good practices for the repository of Big Data projects; Assessment of Big data sources for the production of official statistics, determine their impact on the production process and identify best practices for managing the necessary changes within an NSO

Big Data sources n Lessons learned from one or more pilot projects using mobile phone data / satellite imagery / social media data; n Guidelines on the use of Big Data sources for official statistics, including template of a project plan

Big Data and Classification s

Classification of Type of Big Data 1. Social Networks (human-sourced information) 2. Traditional Business systems (process-mediated data) 3. Internet of Things (machine-generated data)

Classification of Type of Big Data Social Networks (human-sourced information) 1100. Social Networks: Facebook, Twitter, Tumblr etc. 1200. Blogs and comments 1300. Personal documents 1400. Pictures: Instagram, Flickr, Picasa etc. 1500. Videos: Youtube etc. 1600. Internet searches 1700. Mobile data content: text messages 1800. User-generated maps 1900. E-Mail

Classification of Type of Big Data Traditional Business systems (processmediated data) 21. Data produced by Public Agencies 2110. Medical records 22. Data produced by businesses 2210. Commercial transactions 2220. Banking/stock records 2230. E-commerce 2240. Credit cards

Classification of Type of Big Data Internet of Things (machine-generated data) 31. Data from sensors 311. Fixed sensors 3111. Home automation 3112. Weather/pollution sensors 3113. Traffic sensors/webcam 3114. Scientific sensors 3115. Security/surveillance videos/images 312. Mobile sensors (tracking) 3121. Mobile phone location 3122. Cars 3123. Satellite images 32. Data from computer systems 3210. Logs 3220. Web logs

Global survey (June-Aug 2015) Big data are data sources with a high volume, velocity and variety of data, which require new tools and methods to capture, curate, manage, and process them in an efficient way

Part I: Office Management of Big Data strategy – Advocacy and Communication Big Data strategy – linking to the SDG indicators Access, privacy and confidentiality Part II: Current status of Big Data projects Data sources Technologies Quality assessment Part III: Guidelines on Big Data for Official Statistics

Questions ? Ronald Jansen United Nations Statistics Division jansen 1@un. org