Analytic Methodology Work Group Meeting National Health Security

  • Slides: 11
Download presentation
Analytic Methodology Work Group Meeting, National Health Security Preparedness Index February 2, 2015 April

Analytic Methodology Work Group Meeting, National Health Security Preparedness Index February 2, 2015 April 8, 2015 NHSPI Program Management Office University of Kentucky

Agenda Scaling Measures Old = actual/max ; trimming at +2 S. D New =

Agenda Scaling Measures Old = actual/max ; trimming at +2 S. D New = (actual – min)/(max-min) Weighting Approach Old = equal weighting of items within subdomains gives rise to large differences in implicit weighting in the overall Index New = weights based on expert panel Delphi ratings An Equal Opportunity University �

Validation studies with 2014 Index % Impact on Final Index Score 131 74 130

Validation studies with 2014 Index % Impact on Final Index Score 131 74 130 156 60 152 109 162 33 158 129 107 112 120 132 151 77 148 111 122 116 114 101 50 67 95 61 47 128 147 125 123 134 145 105 54 51 48 43 41 143 146 164 103 98 96 100 29 62 159 70 30 20 15 32 19 12 21 17 78 75 72 68 63 85 83 94 92 90 87 166 157 84 6 11 4 1 7 39 38 149 81 177 174 172 57 55 171 168 Index Measures These differential outcomes are a function of model structure (implicit weighting) as well as the scaling approach. 0. 00% 0. 50% 1. 00% 1. 50% 2. 00% 2. 50%

Scaling and Weighting Don’t want the measure scaling to have a differential impact on

Scaling and Weighting Don’t want the measure scaling to have a differential impact on Index scores Dichotomous v. continuous Do want the weighting to be explicit and reflect underlying science

Sensitivity Analyses Index Measures % Impact on Final Index Score M 307 M 229

Sensitivity Analyses Index Measures % Impact on Final Index Score M 307 M 229 M 309 M 67 M 174 M 62 M 151 M 24 M 287 M 69 M 308 M 148 M 160 M 298 M 310 M 61 M 72 M 292 M 153 M 300 M 294 M 168 M 140 M 262 M 71 M 81 M 175 M 164 M 306 M 291 M 303 M 312 M 348 M 331 M 264 M 50 M 170 M 52 M 46 M 319 M 347 M 33 M 254 M 330 M 82 M 139 M 6 M 188 M 161 M 106 M 7 M 11 M 15 M 286 M 9 M 12 M 208 M 335 M 150 M 222 M 84 M 189 M 332 M 118 M 345 M 343 M 341 M 338 M 35 M 68 M 228 M 217 M 290 M 20 M 17 M 220 M 171 M 91 M 293 M 115 M 273 M 197 M 202 M 346 M 36 M 277 M 275 0. 0% Current Structure & Scaling Equal weight, New Scaling 0. 5% 1. 0% 1. 5% 2. 0% 2. 5%

The Difference of a Scaling Change Index Measures % Impact on Final Index Score

The Difference of a Scaling Change Index Measures % Impact on Final Index Score M 273 M 196 M 257 M 276 M 258 M 34 M 32 M 271 M 161 M 68 M 66 M 63 M 61 M 293 M 291 M 348 M 319 M 317 M 315 M 54 M 313 M 311 M 307 M 308 M 305 M 301 M 303 M 299 M 297 M 295 M 169 M 167 M 153 M 151 M 148 M 331 M 254 M 140 M 138 M 329 M 81 M 344 M 342 M 340 M 338 M 332 M 118 M 115 M 336 M 72 M 150 M 227 M 107 M 333 M 71 M 10 M 189 M 175 M 173 M 346 M 36 M 263 M 50 M 261 M 164 M 260 M 52 M 46 M 171 M 90 M 47 M 287 M 16 M 3 M 259 M 216 M 12 M 9 M 208 M 15 M 13 M 290 M 23 M 220 M 22 M 19 M 17 0. 0% Equal Weight, Original Scaling Equal weight, New Scaling 0. 5% 1. 0% 1. 5% 2. 0% 2. 5%

Sensitivity Analyses Index Measures % Impact on Final Index Score M 307 M 229

Sensitivity Analyses Index Measures % Impact on Final Index Score M 307 M 229 M 309 M 67 M 174 M 62 M 151 M 24 M 287 M 69 M 308 M 148 M 160 M 298 M 310 M 61 M 72 M 292 M 153 M 300 M 294 M 168 M 140 M 262 M 71 M 81 M 175 M 164 M 306 M 291 M 303 M 312 M 348 M 331 M 264 M 50 M 170 M 52 M 46 M 319 M 347 M 33 M 254 M 330 M 82 M 139 M 6 M 188 M 161 M 106 M 7 M 11 M 15 M 286 M 9 M 12 M 208 M 335 M 150 M 222 M 84 M 189 M 332 M 118 M 345 M 343 M 341 M 338 M 35 M 68 M 228 M 217 M 290 M 20 M 17 M 220 M 171 M 91 M 293 M 115 M 273 M 197 M 202 M 346 M 36 M 277 M 275 0. 0% Current Structure & Scaling Current Structure, New Scaling 0. 5% 1. 0% 1. 5% 2. 0% 2. 5% 3. 0%

Scaling Measures New scaling [(actual – min)/(max-min)] does not exercise a differential impact on

Scaling Measures New scaling [(actual – min)/(max-min)] does not exercise a differential impact on Index scores Dichotomous & continuous measures scaled similarly

Weighting • Expert panel, online Delphi approach – What gets weighted? • measures, subdomains,

Weighting • Expert panel, online Delphi approach – What gets weighted? • measures, subdomains, and/or domains? – Weighted with respect to what? • state-specific risks, all hazards, uniform planning scenarios? – Who are the experts?

Upcoming Meetings April 21, Model Design (Virtual)

Upcoming Meetings April 21, Model Design (Virtual)

For More Information National Program Office Supported by The Robert Wood Johnson Foundation Glen

For More Information National Program Office Supported by The Robert Wood Johnson Foundation Glen P. Mays, Ph. D. , M. P. H. glen. [email protected] edu Anna Goodman Hoover, Ph. D. Anna. [email protected] edu Michael Childress, M. A. Michael. [email protected] edu Email: [email protected] edu Web: www. nhspi. org / www. publichealthsystems. org Journal: www. Frontiersin. PHSSR. org Archive: works. bepress. com/glen_mays Blog: publichealtheconomics. org National Coordinating Center