Travel Model Improvement Program Model Validation Improvement Efforts

Travel Model Improvement Program Model Validation Improvement Efforts presented to Transportation Planning Applications Committee (ADB 50) presented by Sarah Sun Federal Highway Administration David Kurth Cambridge Systematics, Inc. January 13, 2009 Transportation leadership you can trust.

Travel Model Improvement Program (TMIP) Goals Conduct outreach and education to advance the state of the practice Promote research on modeling Improve model quality 1

Travel Model Improvement Program Outreach Training Disseminating ideas Engaging stakeholders Facilitating research implementation 2

TRB Special Report 288 http: //onlinepubs. trb. org/onlinepubs/sr/sr 288. pdf 3

TRB Special Report 288 – Model Practice Shortcomings Too frequently found • Inadequate data • Optimism bias • Quality control • Validation errors 4

TRB Special Report 288 – Summary Recommendations Develop and implement new modeling approaches Ensure better practice Federal, state, regional collaboration 5

Travel Model Validation Practices Peer Exchange – May 9, 2008 In response to the recommendation of TRB Special Report 288 • take steps to ensure better practice Sponsored by FHWA through TMIP • Project manager – Sarah Sun • Project staff – CS and TTI Exchange participating experts • 9 from public agencies • 2 from academia • 4 from consulting • 4 from Federal agencies (FHWA & FTA) 6

Peer Exchange Objective Use the experience of each panel member to provide practical advice and “action items” useful for moving the validation of travel models closer to “What it should be. ” “…What should it be? ” 7

Summary of Current MPO Practices Model Component Validation • Independent data sparse • Estimation and validation using household survey data • CTPP used for home-based work validation Overall Model Validation • Traffic counts • Transit boardings • Focus on statistics: R 2, % RMSE, % Error, VMT Error… (Documented) Sensitivity Tests Lacking • Few temporal validations 8

Summary of Current MPO Practices Guidelines and “Standards” Used: • Calibration and Adjustment of System Planning Models (1990) • TMIP Model Validation and Reasonableness Checking Manual (1997) • NCHRP Report 365 (1998) • NCHRP Report 255 (1992) • Various state guidelines 9

Peer Exchange Summary Assessment of Current Practices Unwarranted focus on attaining standards • Valid = attain measure of accuracy ⇒ “You get the $$$” • Double-edged sword • Perpetuates myths regarding predictive power of models Model validation and techniques are not context specific • Consider “business process” • Consider intended use of model & forecasts Planning for and documentation of validation is lacking 10

Peer Exchange Summary Improved Validation Process Steps to verify the ability of the model system to make reasonable predictions over a range of development patterns, transportation operations, and external factors. Frank Koppelman 11

Peer Exchange Summary Improved Validation Process Model validation must be context specific • Type of analysis considered • Scope of analysis considered • Agency business process being supported Model validation must consider the entire model Consider the validity of the validation data Document validation expectations and results 12

Peer Exchange Summary “ 7 Questions” How important is it to match base-year observations? • Important, but not sufficient to establish validity How close is “close enough? ” • It all depends… What steps are needed to ensure accurate validation data? • A conscious, planned effort • Initiate when model estimations / updates are planned • Budget necessary resources 13

Peer Exchange Summary “ 7 Questions” How should model sensitivity be tested? • Forecasting & backcasting • Test range of alternatives to determine validity Other tests to assess ability to forecast future travel? • Peer review panels • Analysis of components of change (à la FTA New Starts) Should risk analysis of future forecasts be performed? • Depends on decisions being made… 14

Peer Exchange Summary “ 7 Questions” What is the role of validation documentation in raising model credibility? • Necessary for understanding models, resulting in… − confidence in models − knowledge of the limitations of models 15

Model Validation Practices Peer Exchange White Paper See… • http: //tmip. fhwa. dot. gov/res ources/clearinghouse/brow se/list/6/1285 • http: //t/ 16

Travel Model Validation and Reasonability Checking Manual Draft In Preparation • Incorporates White Paper suggestions All elements of current manual • Expanded focus beyond trip-based • Focus on temporal validation and sensitivity testing Model validation plan specification • Identification of data requirements & budgeting Validation documentation recommendations 17

Travel Model Validation and Reasonability Checking Manual Chapters Introduction Model Validation Plan Specification Validating Model Inputs Socioeconomic Models Amount of Travel / Activity 18

Travel Model Validation and Reasonability Checking Manual Chapters Trip Distribution / Destination Choice / Location Choice Mode Choice / Auto Occupancy Time-of-Day Choice / Time-of-Day Split / Directional Split Factors Assignment Procedures Temporal Validation and Sensitivity Testing Validation Documentation 19

Update of the Travel Model Validation and Reasonability Checking Manual Published manual • Web document • Searchable & keyword hyperlinks • Possible wiki format Publicize updated manual • Conferences • TMIP e-mail list • Virtual Mentoring & Technical Support Center (VMTSC) • Webinars 20

Information The Travel Model Improvement Program (TMIP) • http: //tmip. fhwa. dot. gov/ • Suggestions, comments or good practices to share − Contact Sarah Sun at sarah. sun@dot. gov or (202) 493 -0071 21