Session 6 Open Data Deidentification data management www

Session 6 Open Data De-identification & data management www. bitss. org @ucbitss

RT 2 Roadmap Motivating Issues To achieve Researchers degrees of freedom Scientific misconduct Publication bias Failure to replicate Open materials, data, code, & access Transparent reporting & disclosure Reproducible & replicable results Cumulative meta-analyses Organized Workflow and File Management (OSF, Github) Design Conduct Dissemination Archiving Pre-Registration Data Management Transparent Reporting & Disclosure Data Repositories Pre-Analysis Plans Version Control Preprints Dynamic Documents Power Planning Open Notebooks (Jupyter/Docker) Open Access www. bitss. org @ucbitss

Danae Roumis, Social Impact: I work primarily in impact evaluation, with a focus on public health programs (infectious diseases, water & sanitation). www. bitss. org @ucbitss

• Monitoring, evaluation, strategic planning, and capacity building services to advance development effectiveness • Impact evaluation, surveys & large-scale data collection • 120+ countries in all regions & sectors • Agriculture & food security, democracy & governance, education, energy, environmental protection, health, humanitarian assistance, water & sanitation, youth development • USG & non-USG • USAID, Millennium Challenge Corporation, State Dept. , USDA, + • Foundations, NGOs, DFID, World Bank, + socialimpact. com

Overview: Open Data for Research Transparency and Reproducibility • What? Why? Where? When? • Issues • Ownership • Informed consent • De-identification • Sharing identifiable data

What is open data? Public The degree to which government data are treated with a presumption that favors openness to the extent permitted by law and subject to privacy, confidentiality, security or other valid restrictions Accessible The degree to which Open Data are made available in convenient, modifiable and open formats that can be retrieved, downloaded, indexed and searched Described How fully Open Data are described so that consumers of the data have sufficient information to understand their strengths, weaknesses, analytical limitations and security requirements and how to process them Reusable Whether Open Data are made available under an open license that places no restrictions on their use Complete Whether Open Data are published in primary forms (i. e. , as collected at the source), with the finest possible level of granularity that is practicable and permitted by law and other requirements Timely How soon Open Data are made available so as to preserve their value Whether a point of contact is designated to assist with data use and Managed Post-Release respond to complaints about adherence to these Open Data requirements Source (& lots more info): World Bank Open Data Toolkit

Why open data? Changing norms for RESEARCHERS Key to reproducing results Required by JOURNALS Required by FUNDERS

Why open data? • Funders moving in this direction • USG (NSF, NIH, HHS, USAID, others) = $140 billion/annually • World Bank • Foundations (Gates, Hewlett, others) • Journals moving in this direction • 3, 200 journals sign on to Transparency and Openness Promotion (TOP) Guidelines • American Economic Association (AEA) Data Editor

Where do we post open data? • Growing number of data repositories/services • May depend on requirements of funder, journal • Examples include: • World Bank Microdata Catalog • Dataverse • MCC Evaluation Catalog • Best practice: Data + code is posted with other study materials (design report, sampling strategy, analysis reports, etc. )

Issues to consider: risks • Benefits: transparency, accountability, research • Risks: exposing personal, sensitive information creates potential risk for respondent(s) #beneficen ce #nonmalfe asance Researcher’s task: Maximize benefits of opening data while minimizing risks to individuals HHS Belmont Report

Issues to consider: risk • MA Group Insurance Commission (GIC) re-identification

Issues to consider: risk • AOL search terms, 2006 • > 20 m. search queries for > 650 k users, over 3 mo. Period • NYT was able to re-identify individuals • Search terms not scrubbed, contained names, ages, places, etc. • US Voter data breach, 2017 • 99% of 200 m. voter records exposed to firm working for RNC • British Pregnancy Advisory Service data breach, 2012 • Details of 10 k individuals exposed to hacker

Issues to consider: risk Source: The Guardian, Secret Aid Worker

Issues to consider: varies by data What kind of data do you use? Publicly accessible? Administrative: Utility, Personnel Primary: Facility, School, Community Who owns it? What was promised at informed consent? Can it be de-identified? Primary: Household, Individual

Issues to consider: ownership • Planning early for transparency • Consider who owns data • If not researcher-owned, inquire about future data sharing early • Build this into requests for access to data • Build this into informed consent • If proprietary data can’t be shared, researcher should document how obtained the data and all other steps (pre-analysis plan, code, etc. ) • Useful reference: • Administrative data and reproducibility

Issues to consider: informed consent • Informed consent • Respondents have right to know access and uses of data (including quantitative, qualitative, photos, media), especially with regard to personally identifiable information (PII) • Who will have access to data provided (de-identified, identifiable) • How these data will be used • Who can be contacted with concerns • Don’t promise confidentiality if not necessary (but best to be specific about intentions for sharing/posting data) • Early planning is key

Issues to consider: informed consent • Sample language for informed consent "All personally identifying information collected about you will be destroyed once it is no longer needed for the study. ” “Any personal information that could identify you will be removed or changed before files are shared with other researchers or results are made public. " "If you decide to be in this study, the study researchers will get information that identifies you and your personal health information. This may include information that might directly identify you, such as your name and address. This information will be kept for the length of the study and a fixed period afterwards (x- years). After that time it will be destroyed or de-identified, meaning we will replace your identifying information with a code that does not directly identify you. ” • Funders may also have suggested language Source: Utrecht University Data Protocols

Also plan for… • Data collection, management, analysis, data publication • Data access, when, and how (IRB) • Collaboration with other institutions, partners, investigators (data sharing agreements, etc. ) • Replicability of analysis with de-identified data • Logistics: • • Assessment of re-identification risk De-identification Documentation Data publication

Issues to consider: de-identification • “Removes identifying information from a dataset so that individual data cannot be linked with specific individuals. ” • – NIST. IR. 8053, here • “De-identified data refers to data through which a link to a particular individual cannot be established. This often involves “scrubbing” the identifiable elements of personal data, making it “safe” in privacy terms while attempting to retain its commercial and scientific value. ” • Future of Privacy forum, here • Anonymisation is the process of turning data into a form which does not identify individuals and where identification is not likely to take place. This allows for a much wider use of the information. The Data Protection Act controls how organisations use ‘personal data’ – that is, information which allows individuals to be identified. ” • “Anonymisation is of particular relevance now, given the increased amount of information being made publicly available through Open Data initiatives and through individuals posting their own personal data online. ” • UK ICO, here

Source: Future of Privacy Forum, link here

De-identified vs. anonymous Source: Future of Privacy Forum, link here

Issues to consider: de-identification • Promises • Did you promise confidentiality in informed consent? • If yes, need to care a lot about de-identification • What kinds of risks? • Financial, legal, medical, reputational, etc. • Methods & Approaches? • Varies, we’ll review some • Some fields more prescriptive • e. g. HIPAA Expert Determination vs. Safe Harbor • k-anonymity

Issues to consider: de-identification • Using dataset on its own • Forward & backward re-identification • Indirect identifiers, small populations, inferential disclosure • RCTs: public lottery • By linking with other datasets • Linkage, “Jigsaw” • Not limited to (free or) publicly available data • Risks to participants if re-identified • Consider potential for group harm • Consider sensitivity of info in context • Consider “small numbers” • Intruders • Consider motivations, existing info, and means to re-identify

De-identification Did you collect personal and identifiable information? What is the risk that respondents can be reidentified? What are risks to respondents if the data are re-identified? Who might want access? Why? What means do they have to do so? What measures can be taken to reduce risk of reidentification? How has that affected usefulness of the data to others?

De-identification 1. Identify risks for re-identification 2. Apply de-identification measures 3. Examine data again / test (intruder testing) May also include external review 4. Publish data + META-DATA + transparency statement

How to de-identify • Goal: reduce risk of disclosure • + Reduce information loss • Suppression • Generalization • Perturbation “There is no particular method that is universally the best option for every covered entity and health information set. ” – Guidance on HIPAA privacy rule

How to de-identify 1. Direct Identifiers Suppress or generalize As early in data cleaning & analysis as possible; Save securely, with key to link back, accessible only by IRBapproved personnel • • Individual names Phone numbers Addresses GPS coordinates 2. Indirect Identifiers Generalize In some cases, may consider suppressing • • • Area names Cluster codes (linkable) Account numbers (linkable) Other data (inferential disclosure; outliers) Check text fields in survey Sensitive Data Highly context-dependent; judgment of researcher • • Income/expenditures, durable assets Political views & activity Trauma, abuse, medical information Witness or participation in illegal activity þ TIP Consider naming variables to indicate identifiers e. g. “q 102_di” or “q 504 b_ii” making it easy to account for identifiers later þ TIP In Stata: -lookfor- searches for words within variable names & labels (e. g. “name”, “contact”, “phone”, “gps”) In Stata: -egen…strpos()searches for specific words within strings (if you know what to look for)

How to de-identify: HIPAA safe harbor (2)(i) The following identifiers of the individual or of relatives, employers, or household members of the individual, are removed: (A) Names (B) All geographic subdivisions smaller than a state, including street address, city, county, precinct, ZIP code, and their equivalent geocodes, except for the initial three digits of the ZIP code if, according to the current publicly available data from the Bureau of the Census: (1) The geographic unit formed by combining all ZIP codes with the same three initial digits contains more than 20, 000 people; and (2) The initial three digits of a ZIP code for all such geographic units containing 20, 000 or fewer people is changed to 000 (C) All elements of dates (except year) for dates that are directly related to an individual, including birth date, admission date, discharge date, death date, and all ages over 89 and all elements of dates (including year) indicative of such age, except that such ages and elements may be aggregated into a single category of age 90 or older (D) Telephone numbers (L) Vehicle identifiers and serial numbers, including license plate numbers (E) Fax numbers (M) Device identifiers and serial numbers (F) Email addresses (N) Web Universal Resource Locators (URLs) (G) Social security numbers (O) Internet Protocol (IP) addresses (H) Medical record numbers (P) Biometric identifiers, including finger and voice prints (I) Health plan beneficiary numbers (Q) Full-face photographs and any comparable images (J) Account numbers (R) Any other unique identifying number, characteristic, or code, except as permitted by paragraph (c) of this section [Paragraph (c) is presented below in the section “Re-identification”]; and (K) Certificate/license numbers (ii) The covered entity does not have actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information

How to de-identify • Goal is to maintain integrity of data while achieving goal of sufficiently protecting respondents from re-identification • Order matters: Removing direct IDs and names/IDs of small geog. areas can limit amount of other data manipulation • Drawback - loss of transparency on study locations • Consider restricted-access dataset • For indirect identifiers & sensitive data, consider: • Observability of the attribute • Stability over time • Keep the link for re-identification by authorized parties

How to de-identify: indirect & sensitive data • Options: Outliers Continuous Categorical String Recoding, top-coding, winsorizing Grouping, Rounding Coarsening, collapsing categories Redacting or pseudonymizing • Consider effect on analysis & reproducibility • Commensurate with assessed risk of re-identification • Keep in mind: sensitive data = highly context specific

How to de-identify • Outliers – recoding must be tied to legitimate deidentification risk, considering other de-identification methods undertaken • Goal: retain integrity of the data • Top-coding at 99% or 95% or 92% could all be appropriate, dependent on the data and each unique survey

De-identifying other types of data • Principles are the same! Measures taken should be commensurate with assessed risk to respondents. • Qualitative þ TIP • Redacting personal, sensitive information Flag potentially identifying information • Pseudonymization for people, places while you are coding or reading qualitative • Geospatial transcripts. • Aggregating, masking, jittering & Keep a de • Media (audio, photo, video) identification log. • Blurring, bleeping Youth Employment Evaluation • Case studies of youth over time • Raw notes of analytical value • Youth given pseudonyms Public Financial Management Evaluation • Enterprise surveys and interviews, sensitive topics • Raw transcripts withheld

De-identifying qualitative data Log Source: UK Data archive Source - UK ICO Anonymization code of practice

De-identifying geospatial data • Aggregating • Is the data meant to represent a specific area? • Does this provide adequate protection? • Masking • Perturbation, displacement, scale/rotation • “jittering”, “dithering” Despite the development of several different masking techniques, there is no general consensus on which technique is most suited for a particular task. Zandbergen 2014 • Analytical value vs. privacy risks • How important is preserving the spatial pattern? (e. g. cluster/hot spot) See: Zandbergen 2014

De-identifying geospatial data - aggregation Tile maps solve issues of area distortion (think US electoral college map)

De-identifying geospatial data – heat maps Source - UK ICO: Anonymization code of practice

De-identifying geospatial data – masking

Media • Faces, identifying aspects of background in photos, identifying aspects of photo (license plates) • Doors (tracking panel or cohort) • Audio – voices considered PII, identifying sounds in background

Assessing re-identification risk “What is an acceptable level of identification risk for an expert determination? There is no explicit numerical level of identification risk that is deemed to universally meet the “very small” level indicated by the method. The ability of a recipient of information to identify an individual (i. e. , subject of the information) is dependent on many factors, which an expert will need to take into account while assessing the risk from a data set. This is because the risk of identification that has been determined for one particular data set in the context of a specific environment may not be appropriate for the same data set in a different environment or a different data set in the same environment. As a result, an expert will define an acceptable “very small” risk based on the ability of an anticipated recipient to identify an individual. ” Source: HHS HIPAA de-identification guidelines, link in notes

Assessing re-identification risk • MA Group Insurance Commission (GIC) re-identification • Researcher showed 87% of US population could be reidentified using ZIP code, birthdate, and sex • Influenced development of HIPPA Privacy rule

Assessing re-identification risk Consider: • Who might want to identify, why • Link with other published datasets • Spontaneous recognition • Variable combinations (rare / unique combinations) • Nosy neighbor Linkage…HIPAA guidelines list 3 conditions: 1. 2. 3. de-identified data are unique or “distinguishing” data source with which it could be linked mechanism to relate de-identified & identified data sources • lack of a readily available naming data source does not imply that data are sufficiently protected, but that it is harder to re-identify given data sources at hand • “Intruder tests” - Internal & external testing • Useful references (UK ICO): • • Privacy impact assessment code of practice (here) Data sharing code of practice (here)

Intruders • UK Office for National Statistics - Intruder scenarios: • attempt to match a number of individuals or other statistical units in the data with other sources; these will share a characteristic of interest such as ethnic group, household composition or occupation • attempt to find a specific individual, household or business as they have knowledge of information unique to this record (including, perhaps, response knowledge) • attempt to find an arbitrary record in order to demonstrate that the published data are not secure • attempt to find a record in the survey and use this information to find out more about this record externally; this is the reverse of the previous types of attack Source: UK ONS Policy for social survey microdata

De-identification 1. Identify risks for re-identification 2. Apply de-identification measures 3. Examine data again / test (intruder testing) May also include external review 4. Publish data + META-DATA + transparency statement 5. Be kind to replicators?

Issues to consider: sharing identifiable data • Who might you want to share with? • Collaborators, co-investigators, research assistants may need to work with identifiable datasets • New research team members • Replicators • Restricted access (requests) • Data use agreements, MOUs • Cold room, data centers

Restricted access • “Limited access is particularly appropriate for the handling of anonymised data derived from sensitive source material or where there is a significant risk of reidentification. ” – UK ICO • Some safeguards • • • Purpose limitations Encryption Physical environment Background checks Penalties Source - UK ICO: Anonymization code of practice Data sharing code of practice

Example: Treatment variable risk • Evaluation of Tanzania Energy project (MCC) • Conducted a public lottery for 30 villages to receive a subsidized connection rate in 2012 • When prepared baseline data for public use, realized that deidentification needed to take place at REGION level, a high geographic unit with populations of 1 -2 million • If retained an identifier for the lottery treatment variable, could re-identify the village, which significantly increased reidentification risk for households • Solution: Treatment variable not included in public use dataset, included in a restricted access dataset

Posting de-identified data • Meta-data • Transparency statement or researcher disclosure • De-identified data used for analysis? • How does dataset posted differ from reported analysis • Why data was withheld or de-identified • Codes (well-commented) • Other required documentation (funders, journals) • How to request access to restricted data, if applicable

When to open data? • Plan early (notice a theme? ) • Determine researcher embargo period (if applicable) and clearly communicate it • Timely data is useful data, but…. . sometimes can’t know all future risks – balance with demand/usability and managing risk • If collecting multiple rounds of data, decision between individual data releases and grouped data release – pros and cons to each • Some general rules? It depends… • 6 -12 months following publication of report • With journal publication

Ongoing debates • Journal requirements • • • Commitments to Open Data – TOP Journal editors’ transparency statement (JETS) + Researcher "Protest" More controversy Journals committing resources: • AEA Data Editor • Role of IRBs • Some other issues: • • • Agencies develop their own guidelines Funder guidelines/policies/requirements and local laws FOIA, open government licenses Lack of enforcement Reduced ability to trace co-located research

De-identification and reproducibility De-identification for protection of human subjects Open Data for transparency and reproducibility

Take-aways • De-identification based on assessment of risk • Considers risk & usefulness of data • Risk is based on context – no one size fits all approach to how to balance these two opposing forces • Plan early • • • Informed consent Analysis on de-identified data Regulations/agreements about sharing Share everything else if data can’t be shared Set up of restricted access