UNDERSTANDING NAVIGATING THE CMF CLEARINGHOUSE RATING SYSTEM CHANGES

UNDERSTANDING & NAVIGATING THE CMF CLEARINGHOUSE RATING SYSTEM CHANGES FOR PRACTITIONERS Thursday, February 18, 2021 2: 00 – 3: 00 PM EST

Webinar "Housekeeping" • • • Rejoining the Program All Lines are Muted Control Panel Questions/assistance Webinar Recording

Poll • Please identify your employer type. (select one answer) – Academic – Private – State/Federal – Regional/Local – Other

Today’s Presenters Karen Scurry Office of Safety Jerry Roche Office of Safety John Mc. Fadden Resource Center David Petrucci Resource Center

Agenda • Overview of CMF Clearinghouse Changes – Jerry Roche, FHWA Office of Safety • CMF Rating Transition – John Mc. Fadden, FHWA Resource Center • Questions & Answers – Dave Petrucci, FHWA Resource Center • Wrap-up – Karen Scurry

Poll • How often do you use the CMF Clearinghouse? (select one answer) – Daily – Weekly – Monthly – Once or Twice Per Year – Never

Overview of CMF Clearinghouse Changes Jerry Roche, FHWA Office of Safety

CMF Clearinghouse Website http: //www. cmfclearinghouse. org

Updated CMF Star Rating Page

Comparison Spreadsheet

Star Quality Rating Page

CMF Details Page

Score Details

Summary • Become familiar with new criteria • Determine how this might impact you and your job

Poll • On a scale of 1 (not important) to 5 (very important), please rate the relative importance of the CMF star quality ratings to you and your work? (select one answer) – – – 5 (very important) 4 (somewhat important) 3 (indifferent) 2 (less important) 1 (unimportant)

CMF Clearinghouse Rating Transition John Mc. Fadden, FHWA Resource Center

Background • The current star rating system has been in use since the inception of CMF Clearinghouse in 2009 • NCHRP 17 -72 recently developed an updated rating system for CMFs with the purpose of updating the rating/inclusion process for the 2 nd Edition of the Highway Safety Manual • We have been working on updating the CMF Clearinghouse rating system to be consistent with the NCHRP 17 -72 rating scheme

Current CMF Star Rating System • Single rating scheme for Before/After, Cross-Sectional, Meta-Analysis, and Meta-Regression studies • CMFs are rated based on five criteria: – – – Study Design (weight of 2) Sample Size (weight of 2) Standard Error (weight of 1) Potential Bias (weight of 1) Data source (diversity in states) (weight of 1) • Points are assigned to each criteria based on the level of rigor (excellent = 2 points, fair = 1 point, or poor = 0 points)

Current CMF Star Rating System Score Star Rating 14 (max possible) 5 Stars 11 - 13 4 Stars 7 - 10 3 Stars 3 -6 2 Stars 1 -2 1 Star 0 0 Stars • Overall Score = (2 * study design) + (2 * sample size) + standard error + potential bias + data source • Max Possible Points = 14 – Very few 5 Star CMFs unless the study included multiple States

NCHRP 17 -72 CMF Rating System

NCHRP 17 -72 Update Crash Modification Factors for the Highway Safety Manual • The objectives of this research are to: – Assess the current criteria and existing process for evaluating and identifying the quality of CMFs for appropriate use with the HSM. – Develop proposed revisions to the criteria and process, including how existing and new CMFs may be incorporated in the HSM. Provide guidance for practitioner application of the revised process – Apply the evaluation criteria to identify and assess CMFs and develop a list of appropriate CMFs for the HSM.

NCHRP 17 -72 Rating Procedure • The rating system developed by NCHRP 17 -72 is much more detailed and provides scores for different factors including sample size, study design, methodology, and statistical significance • A separate rating criteria has been developed for Before/After, Cross-Sectional, Meta-Analysis, and Meta. Regression studies (detailed scoring criteria is available at http: //cmfclearinghouse. org/changes. cfm) • Overall Procedure – Points are assigned based on multiple factors • Levels within factors and points for each level – Total score calculated by adding the points; maximum possible score is 150

NCHRP 17 -72 Rating Criteria Before/After and Cross-Sectional Studies Factors Subcategories Total Points Data Number of miles/sites and number of crashes for reference/comparison and treatment groups 55 Study Design and Statistical Methodology Potential biases, accounting for trends, and appropriateness of statistical procedures 75 Statistical Significance Statistical significance of CMF 20

NCHRP 17 -72 Rating Criteria Meta-Analysis Studies Factors Subcategories Total Points Methodology and Data Consistent methodology and crash type and severity definitions across included studies 55 Individual CMF Quality of studies used and standard errors of the individual CMFs. 35 Appropriateness of Combining Use of appropriate methods to estimate the combined CMF 40 Statistical Significance Overall statistical significance of the combined CMF 20

NCHRP 17 -72 Rating Criteria Meta-Regression Studies Factors Methodology and Data Individual CMF Quality Appropriateness of Statistical Analysis Subcategories Consistent methodology and crash type and severity definitions across included studies Quality of studies used and standard errors of the individual CMFs. Use of appropriate statistical methods to estimate crash modification function including use of appropriate model and functional forms Total Points 55 35 60

Converting NCHRP 17 -72 Ratings to Star Rating NCHRP 17 -72 Rating Score Star Rating in CMF Clearinghouse 135 -150 5 star 110 -134 4 star 75 -109 3 star 35 -74 2 star 0 -34 1 star

Changes to Legacy CMF Star Ratings NCHRP 17 -72 Rating Score Legacy Rating Score 5 Star 4 Star 3 Star 2 Star 1 Star (0 -34) 3 21 116 341 215 2 Star (34 -74) 1 89 639 666 59 3 Star (75 -109) 67 485 1903 264 25 4 Star (110 -134) 91 954 988 11 4 5 Star (135 -150) 137 210 56 0 0

Changes to Current CMF Star Ratings • The conversion scale leads to many CMFs being rated differently from their current ratings in the CMF Clearinghouse – 52. 12% of the CMFs would maintain their current star ratings – 16. 16% of CMFs currently rated 3 Stars or better would be rated 2 Stars or lower

Example Ratings under the Current CMF Clearinghouse & NCHRP 17 -72 Rating Systems

CMF ID 8441: 5 Star Rating (both Current and 17 -72 Rating Schemes) Current CMF Clearinghouse Rating Scheme Study Design Sample Size Standard Error Potential Bias Data Source OVERALL SCORE STAR RATING 2 2 2 14 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 10 20 25 5 10 10 10 150 5 Star

CMF ID 8441: 5 Star Rating (both Current and 17 -72 Rating Schemes) Current CMF Clearinghouse Rating Scheme Study Design Sample Size Standard Error Potential Bias Data Source OVERALL SCORE STAR RATING 2 2 2 14 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 10 20 25 5 10 10 10 150 5 Star

CMF ID 8441: 5 Star Rating (both Current and 17 -72 Rating Schemes) Current CMF Clearinghouse Rating Scheme Study Design Sample Size Standard Error Potential Bias Data Source OVERALL SCORE STAR RATING 2 2 2 14 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 10 20 25 5 10 10 10 150 5 Star

CMF ID 460: 5 Star Rating (Current) and 1 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes for all sites combined Rating for at least two years with actual/estimated traffic volume counts in the study period Rating for similarity of sites with and without the treatment (selection bias) Rating for appropriate model form Rating for appropriate functional form Rating for appropriate consideration of omitted variable bias Rating for appropriate consideration of correlation between independent variables Rating for appropriate consideration of spatial and temporal correlation Rating for CMF significance level OVERALL RATING STAR RATING 0 0 0 10 0 0 20 30 1 Star

CMF ID 460: 5 Star Rating (Current) and 1 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes for all sites combined Rating for at least two years with actual/estimated traffic volume counts in the study period Rating for similarity of sites with and without the treatment (selection bias) Rating for appropriate model form Rating for appropriate functional form Rating for appropriate consideration of omitted variable bias Rating for appropriate consideration of correlation between independent variables Rating for appropriate consideration of spatial and temporal correlation Rating for CMF significance level OVERALL RATING STAR RATING 0 0 0 10 0 0 20 30 1 Star

CMF ID 460: 5 Star Rating (Current) and 1 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 5 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes for all sites combined Rating for at least two years with actual/estimated traffic volume counts in the study period Rating for similarity of sites with and without the treatment (selection bias) Rating for appropriate model form Rating for appropriate functional form Rating for appropriate consideration of omitted variable bias Rating for appropriate consideration of correlation between independent variables Rating for appropriate consideration of spatial and temporal correlation Rating for CMF significance level OVERALL RATING STAR RATING 0 0 0 10 0 0 20 30 1 Star

CMF ID 318: 1 Star Rating (Current) and 3 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 1 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 0 0 5 10 10 10 0 25 5 10 105 3 Star

CMF ID 318: 1 Star Rating (Current) and 3 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 1 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 0 0 5 10 10 10 0 25 5 10 105 3 Star

CMF ID 318: 1 Star Rating (Current) and 3 Star Rating (17 -72 Rating Scheme) Current CMF Clearinghouse Rating Scheme Rating Based on Adjusted Standard Error STAR RATING 1 Star NCHRP 17 -72 Rating Scheme Rating for number of miles/sites of reference/comparison group Rating for number of crashes in reference/comparison group Rating for number of miles/sites of treatment group Rating for number of before plus expected after crashes Rating for providing at least one traffic volume count in before and after periods Rating for reference/comparison group and treatment group having similar AADT Rating for reference/comparison group and treatment group having similar roadway characteristics Rating for CMF significance level Rating for addressing RTM or system wide implementation Rating for accounting for spillover/crash migration Rating for accounting for changes in traffic volume Rating for accounting for time trends and other changes Rating for appropriate SPF functional form OVERALL RATING STAR RATING 10 0 0 5 10 10 10 0 25 5 10 105 3 Star

Poll • Do you expect the new ratings to impact you and/or your customers? (select one answer) – – – Yes, definitely Yes, likely but not in a significant way No, it’s unlikely No, absolutely not I’m not sure

Question & Answers Dave Petrucci, FHWA Resource Center

What should I do next, if anything? • It depends, but if you, your organization, or your projects are using CMF's from the Clearinghouse, and their star quality ratings are of interest, we encourage you to review the changes discussed today, and continue to select the most appropriate CMF's as you would define them.

Where can I find CMF score details? • While visiting the Clearinghouse and viewing a CMF/CRF ID Report, to the right of the star quality rating, select VIEW SCORE DETAILS, and the list of score details will be provided, showing rating inputs, outputs, and updated score according to the applicable study type.

Has FHWA changed CMF's? • No, the CMF/CRF values themselves remain the same. The only changes are the updated score and resultant star quality rating using the rating system discussed during this webinar.

Will users still have access to the previous star quality ratings? • Yes, the Clearinghouse includes a comparison spreadsheet that contains both legacy and updated star quality ratings. All CMF's added to the Clearinghouse after February 15, 2021 will be rated using the updated rating system. Visit this URL below: http: //www. cmfclearinghouse. org/collateral/Ratings_Comparison. xlsx

Is the new rating system better? • The new rating system is more rigorous and objective than the legacy rating system. While the updated score and star quality rating provide an indication of the quality of the study that produced the CMF, users should ensure that any selected CMF is applicable to their project based on road and other conditions.

QUESTIONS?

Wrap-up Karen Scurry, FHWA Office of Safety

Education & Outreach

NHI CMF Training • Application of CMFs – FHWA-NHI-380093 • Science of CMFs – FHWA-NHI-380094 • Developing Quality CMFs – FHWA-NHI-380119 https: //nhi. fhwa. dot. gov

Poll • What additional information would be helpful to you related to the new rating system (select all that apply)? – – – More in-depth training on the new system Question & Answer Session Technical Assistance Presentation Other (enter in the chat or Q/A)

For additional information • Visit: • http: //www. cmfclearinghouse. org/ • Sign up for the CMF CH Newsletter • Contact: – Karen. Scurry@dot. gov – Jerry. Roche@dot. gov – John. Mc. Fadden@dot. gov – Dave. Petrucci@dot. gov