Improving Bayesian Prediction of Daily Response Propensity in

  • Slides: 16
Download presentation
Improving Bayesian Prediction of Daily Response Propensity in Responsive Design with Data-Driven Priors Brady

Improving Bayesian Prediction of Daily Response Propensity in Responsive Design with Data-Driven Priors Brady T. West 1, 2 Michael R. Elliott 1, 2, 3 James Wagner 1, 2 Stephanie Coffey 2, 3 Xinyu Zhang 1 1 Survey 2 Research Center, Institute for Social Research, Univ. of MI-Ann Arbor Joint Program in Survey Methodology, Univ. of MD-College Park 3 U. S. Census Bureau Big. Surv 20: November 13, 2020 1

Acknowledgements / Disclaimers • This work was supported by a grant from the National

Acknowledgements / Disclaimers • This work was supported by a grant from the National Institutes for Health (#1 R 01 AG 058599 -01; PI: James Wagner) • The National Survey of Family Growth (NSFG) is conducted by the Centers for Disease Control and Prevention's (CDC’s) National Center for Health Statistics (NCHS), under contract # 200 -2010 -33976 with University of Michigan’s Institute for Social Research with funding from several agencies of the U. S. Department of Health and Human Services, including CDC/NCHS, the National Institute of Child Health and Human Development (NICHD), the Office of Population Affairs (OPA), and others listed on the NSFG webpage (see http: //www. cdc. gov/nchs/nsfg/). The views expressed here do not represent those of NCHS nor the other funding agencies. Big. Surv 20: November 13, 2020 2

Research Problem • Responsive Survey Design (RSD) has been developed by survey methodologists as

Research Problem • Responsive Survey Design (RSD) has been developed by survey methodologists as a principled framework for reducing survey errors and costs via real-time monitoring of survey paradata • Despite numerous documented successes of the effectiveness of this technique, there also a number of documented cases where it has not worked well (Tourangeau et al. 2016) • A possible explanation for this lack of consistent success may be the absence of a unifying analytic framework for RSD • In particular, estimates of response propensity (RP) may be noisy (and misleading) early in a data collection, when paradata are sparse Big. Surv 20: November 13, 2020 3

A Possible Solution • Bayesian methods would seem to be a natural fit for

A Possible Solution • Bayesian methods would seem to be a natural fit for RSD: • Survey managers wish to update prior beliefs about key design parameters with new paradata from the field, and make decisions accordingly • Research Question: Can Bayesian methods improve the accuracy of daily predictions of RP early in a data collection, when available information from the field is sparse? • Answering this question requires an approach for gathering prior information about the coefficients in an RP model • Here we elicit prior information using 1) data collected from previous waves of similar surveys, and 2) estimates from related literature; see Coffey et al. (2020) for an approach using expert elicitation Big. Surv 20: November 13, 2020 4

Prior Formulation: Four Possible Approaches 1. Estimate the RP models using standard frequentist approaches,

Prior Formulation: Four Possible Approaches 1. Estimate the RP models using standard frequentist approaches, with no priors and only the cumulative data available on a given day (STANDARD) 2. Analyze multiple prior data collection periods, and develop precisionweighted normal priors for the coefficients (PWP) • Example (8 periods): mean = , variance = 3. Analyze only the most recent data collection period, and define a normal prior based on the final estimate of a given coefficient using all available data (the mean) and its variance (LAST) 4. Literature Review: Identify published studies containing information about the same RP model coefficients, and aggregate that information to define normal prior distributions for the coefficients (LIT) Big. Surv 20: November 13, 2020 • Use non-informative / flat priors when no information is available in the literature 5

Data: National Survey of Family Growth (NSFG) • We analyze data from 13 quarters

Data: National Survey of Family Growth (NSFG) • We analyze data from 13 quarters of the NSFG (6/2013 – 9/2016) • The NSFG applies RSD, using a two-phase sample design and intervening with particular subgroups when certain outcomes seem sub-optimal (Wagner et al. 2012); heavy reliance on daily RP • Our focus is on the daily probability of responding to a screening interview, attempted with sampled households to identify eligible persons between the ages of 15 and 49 for the main interview Big. Surv 20: November 13, 2020 6

Response Propensity Modeling • We fit a discrete-time hazard model to data on 119,

Response Propensity Modeling • We fit a discrete-time hazard model to data on 119, 981 contact attempts from the eight most recent quarters, where the binary outcome was {1 = completed screener, 0 = other}; good fit overall • Backward selection used to identify a robust set of predictors across these eight quarters (sampling frame / commercial data, paradata) • We fit this reduced model to the “final” contact attempt data from each of five recent quarters (Q 16 – Q 20), and saved “final” predicted probabilities of daily response for each case (benchmarks) • For our priors based on historical data, we then fit this same model to data from the eight prior quarters (PWP) or the last quarter (LAST) Big. Surv 20: November 13, 2020 7

Literature Review Results (LIT) • Once the robust set of predictors was identified, we

Literature Review Results (LIT) • Once the robust set of predictors was identified, we scoured the literature for articles / reports presenting estimated coefficients for similar / related predictors in studies of response propensity • Article details / coefficients available upon request • We identified eight articles / reports, presenting information for 33 of the 75 coefficients in the final RP model (44%) • We examined mean coefficients and standard errors across the relevant articles, and used normal priors with mean 0 and variance 10 for the 42 coefficients with no prior information in the literature Big. Surv 20: November 13, 2020 8

Analytic Approach • For days 7 -84 in one of the five most recent

Analytic Approach • For days 7 -84 in one of the five most recent quarters, we used PROC MCMC in SAS to fit the same RP model to all available data on that day, specifying the priors according to one of our approaches • Using 5, 000 posterior draws of each coefficient in the RP model on that day, we computed 5, 000 corresponding draws of the predicted probabilities of response for each case on that day, and averaged them • For each case, we compared the Bayesian RP estimate to the benchmark estimate of RP based on all data from that quarter • i. e. , early on, can we obtain the estimate based on all data from the quarter? • On each day of the quarter, we examined the mean difference for each method, in addition to the standard error of the mean Big. Surv 20: November 13, 2020 9

Results: Trends in Mean Differences Zoom in to focus on days 7 -16, early

Results: Trends in Mean Differences Zoom in to focus on days 7 -16, early in the quarter: evidence of advantages! Big. Surv 20: November 13, 2020 10

Results: Comparisons of Bias and RMSE Big. Surv 20: November 13, 2020 11

Results: Comparisons of Bias and RMSE Big. Surv 20: November 13, 2020 11

Summary of Results • Most noticeable (and consistent) advantages are in the middle periods

Summary of Results • Most noticeable (and consistent) advantages are in the middle periods of each quarter, when interventions are often considered • While there may be advantages earlier in the quarter as well, we did not find consistent evidence of this across all five quarters • Mean differences approach zero faster when using the Bayesian methods, and in particular when using the historical data methods • Moderate evidence in favor of the historical data methods, but the literature method is still competitive if no historical data are available! Big. Surv 20: November 13, 2020 12

Discussion Points • Clearly these approaches need replication in other contexts: • Fewer predictors

Discussion Points • Clearly these approaches need replication in other contexts: • Fewer predictors of RP available; stronger predictors of RP available • Less historical data • We also considered “dynamic” priors that were re-formulated after each day, and didn’t find any substantial benefit to that approach • One prior elicitation method that we didn’t consider here: surveying data collection managers and methodologists about their expectations regarding these coefficients for contact-attempt-level RP • See Coffey et al. (2020) for more details about this approach • Need experimental evaluation of benefits in actual RSD! Ongoing… Big. Surv 20: November 13, 2020 13

Thank You! • Please direct any questions to bwest@umich. edu • References: • Coffey,

Thank You! • Please direct any questions to [email protected] edu • References: • Coffey, S. , West, B. T. , Wagner, J. , Elliott, M. R. (2020). What Do You Think? Using Expert Opinion to Improve Predictions of Response Propensity Under a Bayesian Framework. methods, data, analyses, 14(2). Available at https: //mda. gesis. org/index. php/mda/article/view/2020. 05. • Tourangeau, R. , Brick, J. M. , Lohr, S. and Li, J. (2016). Adaptive and responsive survey designs: a review and assessment. Journal of the Royal Statistical Society - Series A, 180(1), 203 -223. • Wagner, J. , West, B. T. , Kirgis, N. , Lepkowski, J. M. , Axinn, W. G. , and Kruger-Ndiaye, S. (2012). Use of Paradata in a Responsive Design Framework to Manage a Field Data Collection. Journal of Official Statistics, 28(4), 477 -499. • West, B. T. , Wagner, J. , Coffey, S. *, and Elliott, M. R. (Under Review, JSSAM). Deriving Priors for Bayesian Prediction of Daily Response Propensity in Responsive Survey Design: Historical Data Analysis vs. Literature Review. Working paper: https: //arxiv. org/abs/1907. 06560. Big. Surv 20: November 13, 2020 14

Appendix: Early Results Big. Surv 20: November 13, 2020 15

Appendix: Early Results Big. Surv 20: November 13, 2020 15

Appendix: Late Results Big. Surv 20: November 13, 2020 16

Appendix: Late Results Big. Surv 20: November 13, 2020 16