Oops I DID It Again Advanced Differencein Differences

Overview Brief Review of Difference-in-Differences Power and Sample Size Calculations for D-I-D and Binary

D-I-D Basics: A Brief Review § Outcome measured at two time points § D-I-D

Dataset for Examples Value-Based Insurance Designs and Medication Non. Adherence § Kaiser Permanente members

D-I-D Sample Size and Power Calculations Basic Steps 1. Specify your model 2. Estimate

D-I-D Sample Size and Power Calculations Assumptions 1. Mixed model w/random intercept 2. Unexposed:

PROC GLMPOWER Code Example Size Title "VBID D-I-D Required Sample Size: PROC GLMPOWER"; proc

PROC GLMPOWER Output 1: Output from PROC GLMPOWER for D-I-D Required Sample Size at

D-I-D and Binary Outcomes Analysis Methods § Only two outcome measures, so D-I-D models

D-I-D and PROC GENMOD Examples Absolute Risk Differences title "Population-Level Marginal D-I-D Model: PROC

D-I-D and PROC GENMOD Examples Relative Risks and Relative Risk Ratios title "Population-Level Marginal

D-I-D and PROC GENMOD Results Marginal Binomial Distribution Populati Binomial Distribution Poisson Distribution Log

Propensity Scores and D-I-D Balancing Case-Mix Differences § Recent method used to balance case-mix

Propensity Scores and D-I-D Use the conditional probability of being exposed given a set

Propensity Scores and D-I-D Stage 1 Tackled Using PROC PSMATCH § Run logistic regression

Propensity Scores and D-I-D Stage 1 Code Using PROC PSMATCH title "VBID D-I-D: PROC

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Visual inspection of the

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Characterist ic Female Age,

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Distributional balance of continuous

Propensity Scores and D-I-D Stage 2 ATE Weighted D-I-D title "D-I-D model for change

Difference in Difference-in-Differences (DDD) Assessing Heterogeneity of Treatment Effects § Formal comparison of 2

Difference in Difference-in-Differences (DDD) SAS Code for DDD model title "D-I-D model for change

Difference in Difference-in-Differences (DDD) DDD Output Contrast Estimate Results Label low_ses=1, pre, VBID Mean

Difference in Difference-in-Differences (DDD) DDD Output Analysis Of GEE Parameter Estimates Empirical Standard Error

Concluding Remarks § D-I-D has several methodological benefits and results have a causal interpretation

Acknowledgements § Mary E. Reed, Ph. D § Andrew J. Karter, Ph. D §

Questions? 27 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For

Contact Information § E. Margaret Warton, MPH – Kaiser Permanente Division of Research; Oakland,

Slides: 28

Download presentation

Oops, I D-I-D It Again! Advanced Difference-in. Differences Models in SAS® E Margaret Warton and Melissa M Parker September 6, 2018 Kaiser Permanente Research

Overview Brief Review of Difference-in-Differences Power and Sample Size Calculations for D-I-D and Binary Outcomes Propensity Scores and D-I-D Difference-in-Differences 2 October 30, 2020

D-I-D Basics: A Brief Review § Outcome measured at two time points § D-I-D is the difference in slopes between two groups § Interaction term between exposure and measurement period (time) § How much more (or less) did the exposed group’s outcome change between time 1 and time 2 compared to the unexposed group’s outcome? 3 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Dataset for Examples Value-Based Insurance Designs and Medication Non. Adherence § Kaiser Permanente members in 2013 § All on non-deductible insurance plan § All changed to deductible insurance plan on 1/1/2014 – Exposed: VBID benefit provides free medications for chronic conditions – Unexposed: no VBID pharmacy benefit § Outcome of interest: non-adherence to medications for chronic conditions § D-I-D answers the question: Did the VBID plan mitigate the effect of the increased deductible cost on medication non-adherence? 4 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D Sample Size and Power Calculations Basic Steps 1. Specify your model 2. Estimate outcomes and group ratios 3. Estimate standard deviations and correlation 4. Create an "exemplary dataset" 5. Use PROC GLMPOWER to calculate sample size or power 5 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D Sample Size and Power Calculations Assumptions 1. Mixed model w/random intercept 2. Unexposed: 40% to 45% Exposed: 40% to 42% proc mixed data=vbid method=reml; class study_id exposed; model non_adh = post exposed post*exposed/solution; random int/subject=study_id type=vc ; run; Unexposed: Exposed = 3: 2 data 3. Std Dev: 0. 03 (or 3%) Correlation: ρ = 0. 80 4. Exemplary dataset code adh; Input exposed $ adh_pre adh_post wt; datalines; unexposed. 40. 45 3 exposed. 40. 42 2 ; 6 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

PROC GLMPOWER Code Example Size Title "VBID D-I-D Required Sample Size: PROC GLMPOWER"; proc glmpower data = adh; class exposed; weight wt; model adh_pre adh_post = exposed; repeated time; power effects=(exposed) mtest = hlt alpha = 0. 05 power = 0. 9 ntotal =. stddev = 0. 03 matrix("adhcorr") = lear (0. 8, 1, 2, 1 2) corrmat = "adhcorr"; run; 7 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

PROC GLMPOWER Output 1: Output from PROC GLMPOWER for D-I-D Required Sample Size at 90% Power § Total sample size of 1, 755 is adequate to detect a 3% difference in change in non-adherence § 3: 2 ratio requires 1053 unexposed and 702 exposed § Study cohort has 1458 unexposed and 1024 8 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D and Binary Outcomes Analysis Methods § Only two outcome measures, so D-I-D models are population-level marginal models § Different modeling solutions available in SAS – PROC GENMOD: ANOVA, repeated measures – PROC MIXED, PROC GLIMMIX : Mixed models § PROC GENMOD: Choice of link and distribution – Rarity of outcome – Preference for absolute risk differences vs relative risks 9 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D and PROC GENMOD Examples Absolute Risk Differences title "Population-Level Marginal D-I-D Model: PROC GENMOD with dist=Poisson and link=identity"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='non-vbid') / param=ref; model non_adh=post|exposed /dist=poisson link=identity type 3; repeated subject=study_id /type=un; format exposed. ; run; 10 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D and PROC GENMOD Examples Relative Risks and Relative Risk Ratios title "Population-Level Marginal Model(RRR): PROC GENMOD with dist=binomial and link=log"; proc genmod data=vbid descending; class study_id exposed (order=formatted ref='Non-VBID') / param=ref; model non_adh= post|exposed /dist=binomial link=log type 3; estimate 'RR: EXPOSED' exposed 1 /exp; estimate 'RR: POST' post 1 /exp; estimate 'RRR: POST*EXPOSED' post*exposed 1 /exp; repeated subject=study_id /type=un; format exposed. ; run; 11 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

D-I-D and PROC GENMOD Results Marginal Binomial Distribution Populati Binomial Distribution Poisson Distribution Log Link on Relative Risk Ratios Identity Link Model Absolute Risks (Exponentiated) Non. Adheren ce Estimate 95% CI (0. 382, Intercept 0. 406 (0. 381, 0. 431) 0. 406 0. 432) (-0. 033, Post -0. 012 0. 009) 0. 970 0. 920, 1. 021 (-0. 026, Exposed 0. 013 0. 052) 1. 038 (0. 938, 1. 135) § D-I-D is -1. 7% prevalence of non-adherence declined more in VBID Exposed group (NS) * (-0. 050, (0. 885, § Poisson and binomial identity link are identical (binomial is more efficient) Post 0. 016) 1. 040) -0. 017 0. 959 § Binomial log exponentiated=relative risk – No quick intuition on prevalence of non-adherence – 0. 96 Post/Pre relative risk of poor adherence was lower in VBID than in non-VIBD 12 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Balancing Case-Mix Differences § Recent method used to balance case-mix differences so that exposed and reference groups are as similar as possible § Plan carefully in the design stage to eliminate imbalances § In observational research residual imbalances are likely Reed ME et al. , Health Aff, 2017. 13 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Use the conditional probability of being exposed given a set of observed covariates to balance case mix differences in the analysis. § Stage 1: Development of the propensity score (PS) model ─ Check for acceptable level of balance ─ Determine how PS will be used in the analysis ─ Weights, matching, stratified sample, covariate ─ Long, iterative process ─ Ends when satisfactory covariate balance has been achieved § Stage 2: Run D-I-D in PS weighted, matched or stratified sample 14 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 1 Tackled Using PROC PSMATCH § Run logistic regression model or import precalculated PS § Assess covariate balance – Variable information (mean, STD, prevalence) – Standardized difference § difference in means (prevalence) between the exposed and reference groups divided by the pooled standard deviation § 0 means no difference; -0. 10 ≤ d ≤ 0. 10 indicates a negligible difference – Ratio of variances ideally equal to 1 – Graphical exploration with boxplots, CDF plots, quantile plots § Create and output matched sample or PS weights 15 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 1 Code Using PROC PSMATCH title "VBID D-I-D: PROC PSMATCH for Propensity Scores"; proc psmatch data=VBID; class exposed; psmodel exposed (treated=last)= age_at_index female low_ses hispanic black asian other race_miss english comorbid 1 -comorbid 3; assess ps lps var=(female white age_at_index low_ses hispanic black asian other english comorb 0 -comorb 3) / plots(nodetails)=(cdfplot boxplot(display=(ps age_at_index)) stddiff(ref=0. 10) ) weight=atewgt(stabilize=no) ; output out (obs=all)=outx lps=lps ps=ps atewgt(stabilize=no)=iptw_u ; run; 16 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Visual inspection of the plot shows how covariate imbalances are removed after PS weighting. 17 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Characterist ic Female Age, mean (SD) Race Black White Asian Other All VBID N=1024 No VBID N=1458 42% 50. 4 (8. 3) 48% 52. 0 (8. 0) 14% 64% 14% 5% 12% 57% 23% 7% ATE Weighted Variance Std Diff Ratio VBID N=1024 No VBID N=1458 -0. 19 1. 076 45% 51. 4 (7. 9) 46% 51. 3 (8. 4) 0. 08 0. 14 -0. 23 -0. 04 13% 61% 19% 6% 13% 60% 20% 6% -0. 12 0. 978 1. 193 0. 943 0. 690 0. 849 Std Diff Varianc e Ratio -0. 01 0. 998 0. 01 0. 001 0. 02 -0. 02 0. 007 0. 883 1. 002 0. 993 0. 972 0. 974 Hispanic ethnicity 18% 15% 0. 09 1. 179 17% 16% 0. 01 1. 027 § Assessment of covariate balance after weighting English Spkr 99% 94% 0. 32 0. 100 97% 96% 0. 05 0. 778 – All standardized differences between -0. 10 and 0. 10 – Variance ratios closer to 1, with the exception of age 18 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 1 OUTPUT using PROC PSMATCH Distributional balance of continuous covariates is similar and aligned to the overall sample after weighting. 19 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Propensity Scores and D-I-D Stage 2 ATE Weighted D-I-D title "D-I-D model for change in medication adherence, Main effects model in ATE weighted cohort" ; proc genmod data=xwt descending; class study_id exposed (order=formatted ref='Non-VBID') / param=ref; model non_adh = post|exposed /dist=binomial link=identity type 3; repeated subject=study_id /type=un; weight iptw_u ; format exposed. ; run; Non-Adherence Intercept Post Exposed (VBID) Exposed* Post (DID) Unadjusted Estimate 95% CI 0. 406 (0. 381, 0. 431) -0. 012 (-0. 033, 0. 009) 0. 013 (-0. 026, 0. 052) -0. 017 (-0. 050, 0. 016) ATE Weighted Estimate 95% CI 0. 408 (0. 382, 0. 434) -0. 014 (-0. 035, 0. 008) 0. 002 (-0. 044, 0. 044) -0. 021 (-0. 057, 0. 015) § Poor adherence decreased by 2. 1% more in VBID compared to non. VBID (NS) ─ Exposed: 41. 0% (pre) to 37. 5% (post) ─ Reference: 40. 8% (pre) to 39. 4% (post) 20 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Difference in Difference-in-Differences (DDD) Assessing Heterogeneity of Treatment Effects § Formal comparison of 2 or more D-I-D estimates § Null hypothesis § The D-I-D among patients living in low SES neighborhoods is equal to the D-I-D among patients who do not live in low SES neighborhoods. § DDD estimator is the parameter estimate of the low_ses*exposure*post interaction term 21 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Difference in Difference-in-Differences (DDD) SAS Code for DDD model title "D-I-D model for change in medication adherence, Testing for EM by SES in ATE weighted cohort"; proc genmod data=xwt descending ; class study_id; model non_adh = post exposed post*exposed low_ses*post*exposed /dist=binomial link=identity type 3; repeated subject =study_id /type=un; estimate 'low_ses=1, pre, VBID' intercept 1 exposed 1 low_ses 1 ; estimate 'low_ses=1, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1 low_ses*post*exposed 1 ; estimate 'low_ses=1, pre, NON-VBID' intercept 1 low_ses 1 ; estimate 'low_ses=1, post, NON-VBID' intercept 1 post 1 low_ses 1 ; estimate 'low_ses=0, pre, VBID' intercept 1 exposed 1 ; estimate 'low_ses=0, post, VBID' intercept 1 post 1 exposed 1 post*exposed 1 ; estimate 'low_ses=0, pre, NON-VBID' intercept 1 ; estimate 'low_ses=0, post, NON-VBID' intercept 1 post 1 ; estimate 'low_ses=1, DID' post*exposed 1 low_ses*post*exposed 1 ; estimate 'low_ses=0, DID' post*exposed 1 ; weight iptw_u ; run; 22 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Difference in Difference-in-Differences (DDD) DDD Output Contrast Estimate Results Label low_ses=1, pre, VBID Mean L'Beta Standar L'Beta Mean Confidence Estimat d Alph Confidence Chi- Pr > Estimate e Error a Square Chi. Sq Limits 0. 4703 0. 413 0. 527 0. 4703 0. 0291 0. 05 0. 413 0. 5274 261. 43<. 0001 3 4 3 low_ses=1, post, VBID 0. 4091 0. 329 0. 488 0. 4091 0. 0405 0. 329 0. 4885 101. 99<. 0001 7 5 7 low_ses=1, pre, NON 0. 4680 0. 418 0. 517 0. 4680 0. 0252 0. 05 0. 418 0. 5174 344. 65<. 0001 VBID 6 4 6 low_ses=1, post, NON 0. 4544 0. 405 0. 503 0. 4544 0. 0252 0. 05 0. 405 0. 5038 325. 22<. 0001 VBID 0 8 0 low_ses=0, pre, VBID 0. 3956 0. 362 0. 428 0. 3956 0. 0168 0. 05 0. 362 0. 4285 554. 63<. 0001 7 5 7 § Among patients living in low SES neighborhoods, the prevalence of low_ses=0, post, VBID 0. 3672 0. 333 0. 400 0. 3672 0. 0171 0. 05 0. 333 0. 4006 463. 67<. 0001 poor adherence decreased by 4. 76% (NS) more in VBID compared to 8 6 8 reference group. 0. 3933 0. 365 0. 421 0. 3933 0. 0142 0. 05 0. 365 0. 4211 769. 23<. 0001 low_ses=0, pre, NONVBID 5 1 5 § Among patients not living in low SES neighborhoods, the prevalence of low_ses=0, post, NON 0. 3797 0. 352 0. 407 0. 3797 0. 0141 0. 05 0. 352 0. 4073 724. 76<. 0001 poor adherence decreased by 1. 48% (NS) more in VBID compared to 23 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only. VBID 0 3 0 reference group.

Difference in Difference-in-Differences (DDD) DDD Output Analysis Of GEE Parameter Estimates Empirical Standard Error Estimates 95% Standard Confidence Parameter Estimate Error Limits Intercept 0. 3933 0. 0142 0. 3655 0. 4211 Post -0. 0136 0. 0108 -0. 0348 0. 0076 Exposed 0. 0023 0. 0212 -0. 0392 0. 0438 post*exposed -0. 0148 0. 0178 -0. 0497 0. 0201 LOW_SES 0. 0747 0. 0270 0. 0219 0. 1276 § The DDD= -3. 28%. -0. 0328 0. 0465 -0. 1240 0. 0584 post*exposed*LOW_SES Z 27. 74 -1. 26 0. 11 -0. 83 2. 77 -0. 71 Pr > |Z| <. 0001 0. 2085 0. 9125 0. 4064 0. 0056 0. 4804 § This means the DID (post – pre) among patients living in low SES neighborhoods was 3. 28% more extreme than the DID (post - pre) among patients not living in low SES neighborhoods. § The difference did not reach statistical significance. 24 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Concluding Remarks § D-I-D has several methodological benefits and results have a causal interpretation as long as time ordering is preserved. § Sample size and power calculations for D-I-D models are easy to do in PROC GLMPOWER. § Use PROC GENMOD for population level marginal effect estimates of pre-post change in a binary outcome. – Log link for relative risk, Identity link for absolute risk § PROC PSMATCH is your one-stop shop for developing a propensity score model, assessing covariate balance and creating a matched cohort or propensity score weights. § Use DDD to assess heterogeneity of treatment effects. 25 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.

Contact Information § E. Margaret Warton, MPH – Kaiser Permanente Division of Research; Oakland, CA – margaret. m. warton@kp. org § Melissa M. Parker, MS – Kaiser Permanente Division of Research; Oakland, CA – melissa. m. parker@kp. org 28 October 30, 2020 | © 2011 Kaiser Foundation Health Plan, Inc. For internal use only.