New Calibration and Validation Standards for Travel Demand

New Calibration and Validation Standards for Travel Demand Modeling presented to 12 th TRB Transportation Planning Applications Conference, May 2009, Houston, TX presented by Robert G. Schiffer, AICP Thomas F. Rossi Cambridge Systematics, Inc. Study prepared for Florida Department of Transportation January 12, 2009 Transportation leadership you can trust.

Presentation Overview Background Literature Review Recommended Calibration and Validation Guidelines and Standards • LRTP Models with Transit • Other Model Applications Calibration and Validation Best Practices Guidelines for Model Application Next Steps 1

Background Follow-up to Phase I Study on model parameters • Phase I Final report is available for downloading at − http: //www. fsutmsonline. net/images/uploads/mtf-files/FSUTMSCube_Parameters. pdf Phase II Study on calibration standards included four subtasks • Literature Review • Model Calibration/Validation Guidelines and Standards • Best Practices for Model Calibration/Validation • Documentation – 1) Calibration and Validation Standards; 2) Best Practices for Model Validation; 3) Guidelines for Model Application 2

Background (continued) Calibration versus validation − Calibration – process where models are adjusted to simulate or match observed travel behavior in the study area − Validation – procedure used to adjust models to simulate base-year traffic counts and transit ridership figures Standards versus guidelines/benchmarks − Standards – desirable accuracy levels for comparing estimated versus observed metrics − Benchmarks – documented statistical ranges from literature review, model outputs, NHTS, etc. Purposes of validation process − Level of comfort to planners, agency staff, and elected officials − Evidence that model is accurate enough for specific application − Accounts for errors in observed comparative data 3

Literature Review Checklist of Available Validation Standards from Literature – Trip Generation 60+ documents reviewed • Specific models − Technical reports − Model outputs • Reference reports − Federal agencies/TMIP − State DOT guidelines and standards Prepared ranges of acceptability Statistic Standard Population/Employment Ratio Benchmark 40 -60% Person Trips/Person 3. 64 – 3. 87 Document(s) Cited Iowa DOT Peer Review (39) Validation and Reasonableness (14) Person Trips/Person (Urban) 2. 54 University of Wisconsin (16): Kentucky Statewide Model/NPTS Person Trips/Person (Rural) 2. 57 University of Wisconsin (16): Kentucky Statewide Model/NPTS Person Trips/HH 8. 5 – 10. 5 University of Tennessee (59) Person Trips/HH 6. 8 – 12. 4 Validation and Reasonableness (14), NCHRP 365 (15) Person Trips/DU 14. 1/14. 5/11. 8/7. 6 Calibration and Adjustment 6) – population sizes: 50 -100/100250/250 -750/750 k+ Person Trips/DU 9. 2/9. 0/8. 6/8. 5 NCHRP 365 (15) – population sizes: 50 k-200 k/200 k-500 k/500 k 1 M/1 M+ Vehicle Trips/DU 9. 15 VTRC (29) 78. 5%/21. 5% VTRC (29) Resident/Commercial Neighborhood Trips Person Trips/Employee 1. 29 – 1. 40 TAZs/Population 1 TAZ/1 k Population Iowa DOT Peer Review (39) Person Trips/TAZ 25 k or less Iowa DOT Peer Review (39) Percent Trips by Purpose – HBW* 18% – 27% University of Tennessee (59) Percent Trips by Purpose – HBNW 47% – 54% University of Tennessee (59) Percent Trips by Purpose – NHB 22% – 31% University of Tennessee (59) Percent Trips by Purpose – HBW* 17% – 23% Validation and Reasonableness (14), NCHRP 365 (15) Percent Trips by Purpose – HBNW 52% – 60% Validation and Reasonableness (14), NCHRP 365 (15) 23% – 25% Validation and Reasonableness (14), NCHRP 365 (15) 0. 90 -1. 10 Percent Trips by Purpose – NHB Unbalanced Attractions/Productions 4 Validation and Reasonableness (14)

Recommended Guidelines and Standards Checking Input Data Socioeconomic data • Visual comparisons • Statistical comparisons • Regionwide comparisons (below) County − Persons per DU (or HH) − Employment/ population ratio − Autos/DU (or HH) Census Data 2000 2003 Duval 779, 618 817, 480 Clay 141, 671 Percent Difference 2005 5% 762, 674 810, 493 6% 157, 502 11% 139, 036 167, 020 20% 57, 903 61, 625 6% 56, 897 64, 695 14% 124, 458 142, 869 15% 120, 738 150, 084 24% 1, 103, 650 1, 179, 476 7% 1, 079, 345 1, 192, 292 10% St Johns Benchmarks/Settings Low High Regionwide Persons/DU (or HH) 2. 0 2. 7 Regionwide Employment/Population Ratio 0. 45 0. 75 Regionwide Autos/DU (or HH) 1. 75 2. 10 Approximate Population/TAZ N/A 5 3, 000 Statistic Percent Difference 2000 Nassau Total NERPM Data

Recommended Guidelines and Standards Checking Input Data (continued) Highway network data Transit network data Highway and transit speed data • Logical hierarchy • Balance highway and transit Terminal times • Logical hierarchy • Phase I Report 6

Recommended Guidelines and Standards Trip Generation Benchmarksa Aggregate trip rates • Person trips/TAZ • Person trips/person • Person trips/DU (or HH) a Statistic Low High Person Trips/TAZ N/A 15, 000 Person Trips/Person 3. 3 4. 0 Person Trips/DU (or HH) 8. 0 10. 0 HBW Person Trips/Employee 1. 20 1. 55 Generally excludes nonmotorized trips; including motorized trips could increase person trips per DU up to 11. 5. • HBW person trips/employee Total unbalanced attractions versus productions by purpose • Preferred +/-10%; acceptable in some instances +/-50% Percent external-external trips by zone/station • Great variation expected (4 -21 percent range documented) 7

Recommended Guidelines and Standards Trip Generation (continued) Percent trips by purpose Benchmarks Statistic Low (Percent) High (Percent) Percent Trips by Purpose – HBW 12 24 Percent Trips by Purpose – HBSH 10 20 Percent Trips by Purpose – HBSR 9 12 Percent Trips by Purpose – HBSC 5 8 Percent Trips by Purpose – HBOa 14 28 Percent Trips by Purpose – HBNWb 45 60 Percent Trips by Purpose – NHBc 20 33 a HBO includes a variety of special trip purposes depending on the model (e. g. , airport, college, and shop). b HBNW accounts for all home-based trip purposes except HBW. c NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate. 8

Recommended Guidelines and Standards Trip Distribution Benchmarks Average trip length by purpose Trip length frequency distributions by purpose Coincidence ratios by purpose – measures the percent of area that coincides for two trip length frequencies Coincidence Ratio = 0. 82 12 35 Average Trip Length – HBSH (minutes) 9 19 Average Trip Length – HBSR (minutes) 11 19 Average Trip Length – HBSC (minutes) 7 16 Average Trip Length – HBOa (minutes) 8 20 Average Trip Length – NHBb (minutes) 6 19 Average Trip Length – IE (minutes) 26 58 6% Observed (ATL = 18. 9 Min) 4% 2% 20 30 Travel Time (in Minutes) 40 50 9 60 Standards Mean Trip Length, Observed Total Trips +/-3% Trip Length Frequency Distribution versus Observed +/-5% 70% a HBO includes a variety of special trip purposes, depending on the model (e. g. , airport, college, and school). b NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate. c Some lower coincidence ratios have been deemed acceptable for trip purposes that had relatively few trips and therefore higher error rates. Estimated (ATL = 18. 2 Min) 10 Average Trip Length – HBW (minutes) Coincidence Ratios by Purposec 8% 0 High Statistic Percent of Total Trips 0% Low Statistic

Recommended Guidelines and Standards Trip Distribution (continued) Percent intrazonal trips by purpose Map-based (“node-point”) charts • Zone-based • Number of trips • Trip productions/attractions by purpose Benchmarks Statistic Low High Percent Intrazonal – HBW 1% 4% Percent Intrazonal – HBSH 3% 9% Percent Intrazonal – HBSR 4% 10% Percent Intrazonal – HBSC 10% 12% Percent Intrazonal – HBOa 3% 7% Percent Intrazonal – NHBb 5% 9% Percent Intrazonal – Total Trips 3% 5% a b Standards Statistic Percent Intrazonal, Observed Total Trips Acceptable Preferable +/-3% +/-5% 10 HBO includes a variety of special trip purposes, depending on the model (e. g. , airport, college, and school). NHB includes combined purposes for NHB Work and NHB Nonwork, where appropriate.

Recommended Guidelines and Standards Mode Choice Mode Zero-Vehicle Households One-Vehicle Households Two-Vehicle Households Three-Vehicle Households Walk 5, 000 6, 000 4, 000 3, 000 Bike 2, 000 1, 000 500 200 - 130, 000 350, 000 200, 000 Shared Ride 2 Persons 6, 000 15, 000 20, 000 10, 000 Shared Ride 3 Persons 1, 000 2, 000 4, 000 2, 000 • Mode Local Bus, Walk 6, 000 7, 000 4, 000 1, 000 Local Bus, PNR - 500 • Auto ownership level Local Bus, KNR - 200 2, 000 500 Express Bus, Walk 1, 000 500 • Geographic subarea Express Bus, PNR - 2, 000 4, 000 2, 000 Express Bus, KNR - 200 500 1, 000 400 - 300 500 Mode split targets (ideal) Drive Alone • Trip purpose LRT, Walk LRT, PNR LRT, KNR Polk 2000 Census 2000 1990 Trips Percent of Trips Number Percent of Trips Drive Alone 244, 414 79. 69% 188, 259 80. 47% Drive Alone 84. 14% One Passenger 51, 465 16. 78% 38, 212 16. 33% Carpool 15. 07% Two+ Passenger 9, 637 3. 14% 7, 023 3. 00% Total Transit 1, 206 0. 39% 465 0. 20% Transit 0. 78% Trip Allocation By Mode HBW 11

Recommended Guidelines and Standards Mode Choice (continued) Mode splits by observed calibration targets Total area transit trips, estimated versus observed Transit trips between districts • Tabular comparisons (CTPP) • Desire lines Mean trip length, estimated transit trips versus observed Standards Statistic Total Area Transit Trips versus Observed Transit Trips between Districts Low High +/-1% +/- 2% Compare model trip table against CTPP or HH survey Mean Trip Length Transit Trips versus Observed +/-5% +/-15% Mode Splits by Observed Calibration Targets +/- 2% -0. 6 -0. 1 Elasticity of Demand with Respect to LOS Variables 12

Recommended Guidelines and Standards Trip Assignment Standards Acceptable Preferable Freeway Volume-over-Count +/- 7% +/- 6% Arterial Volume-over-Count +/- 15% +/- 10% Collector Volume-over-Count +/- 25% +/- 20% Frontage Road Volume-over-Count +/- 25% Statistic Volume-over-count ratios +/-1 lane percent error (calculated based on FDOT LOS Handbook) Freeway Peak Volume-over-Count 75% of links @ +/-20%; 50% of links @ +/-10% Major Arterial Peak Volume-over-Count 75% of links @ +/-30%; 50% of links @ +/-15% Assigned VMT-over-Count Areawide +/-5% +/-2% Assigned VHT-over-Count Areawide +/-5% +/-2% Assigned VMT-over-Count by FT/AT/NL +/- 25% +/- 15% Assigned VHT-over-Count by FT/AT/NL +/- 25% +/- 15% Aggregate VMT Standards Acceptable Preferable Percent Error – LT 10, 000 volume (2 L road) 50% 25% Percent Error – 10, 000 -30, 000 (4 L road) 30% 20% Percent Error – 30, 000 -50, 000 (6 L road) 25% 15% Percent Error – 50, 000 -65, 000 (4 -6 L freeway) 20% 10% Percent Error – 65, 000 -75, 000 (6 L freeway) 15% 5% Percent Error – GT 75, 000 (8+L freeway) 10% 5% Statistic • VMT/HH (60 -75) • VMT/person (24 -32) • VMT/commercial vehicle (3 -25%) 13

Recommended Guidelines and Standards Trip Assignment (continued) Standards Acceptable Preferable RMSE – LT 5, 000 AADT RMSE LT 5, 000 AADT 150% 45% RMSE – 5, 000 -9, 999 AADT RMSE 5, 000 -9, 999 AADT 45% 35% RMSE – 10, 000 -14, 999 AADT RMSE 10, 000 -14, 999 AADT 35% 27% RMSE – 15, 000 -19, 999 AADT RMSE 15, 000 -19, 999 AADT 35% 25% RMSE by volume group RMSE – 20, 000 -29, 999 AADT RMSE 20, 000 -29, 999 AADT 27% 15% RMSE – 30, 000 -49, 999 AADT RMSE 30, 000 -49, 999 AADT 25% 15% Transit assignment validation RMSE – 50, 000 -59, 999 AADT RMSE 50, 000 -59, 999 AADT 20% 10% RMSE – 60, 000+ AADT RMSE 60, 000+ AADT 19% 10% RMSE Areawide 45% 35% Statistic Screenline volume-over-count • By volume and location Benchmarks Statistic Estimated-over-Observed Transit Trips Low High +/- 9% +/- 3% Standards Statistic Acceptable Preferable Acceptable Error – Transit Screenlines +/-20% +/-10% Transit Ridership – <1, 000 Passengers/Day +/-150% +/- 100% Transit Ridership – 1 k-2 k Passengers/Day +/- 100% +/- 65% Transit Ridership – 2 k-5 k Passengers/Day +/- 65% +/- 35% Transit Ridership – 5 k-10 k Passengers/Day +/- 35% +/- 25% Transit Ridership – 10 k-20 k Passengers/Day +/- 25% +/- 20% Transit Ridership – >20, 000 Passengers/Day +/- 20% +/- 15% 14

Calibration and Validation Best Practices Steps in Model Validation and Calibration Process Iterative process Initiate Inventory Process must acknowledge • Availability of behavioral data (or lack thereof) Institutional Framework Secondary Data Collection Primary Data Collection Data Checking Model Estimation Model Implementation • Variability in data accuracy Iterate Model Calibration • Regional issues to be evaluated with the model Model Validation Model Application • Need for future year sensitivity testing NO Satisfactory Results? YES Must validate each step (i. e. , not just assignment) Continual Model Maintenance, Application 15

Calibration and Validation Best Practices Guidance on Validation and Adjustment Matching base year statistics is not sufficient to say model is validated Ideal combination of skills Validation and Reasonableness Checks Estimation Calibration Validation Application • Local area knowledge Iterate • Familiarity with sources for transferable parameters • Understanding of what constitutes acceptable results • Experience with cause and effect of model adjustments • Quality and availability of data 16 Iterate

Calibration and Validation Best Practices Special Validation Considerations Requirements by study type • FTA New Starts projects • Subarea and corridor validation Statewide Model Urban Model Area Study Area Within One Model Area Statewide Model Urban Model Area 1 Urban Study Model Area 2 Study Areas Within Two Model Areas Statewide Model Study Area Urban Model Area Study Area Outside Urban Model Areas Example of Coding Penalties for HOV Lanes • Site impact studies General Purpose Lanes HOV Lanes Other validation practices General Purpose Lanes • Adjustment of congested speeds Example of Coding Prohibitors for Proper Access Would not cross traffic to take loop ramp • Use of special generators • Transferable parameters • Impact of new paradigms Cannot cross median to access driveway (centroid connector) Red lines represent penalized or prohibited movements. 17

Guidelines for Model Applications Stability of Model Parameters Static versus dynamic parameter settings • Generally “locked down” during validation • Limited trend data to back up values for future years History has shown changes in… • Nonhome-based trip rates • Trip length frequencies • Auto occupancy rates • BPR curves 18

Guidelines for Model Applications Typical Model Applications and Guidelines Developed list of 14 different model application types and relevant model guidance unique to each (e. g. , forecasting external trips for MPO LRTP Updates, etc. ) Acknowledge uncertainty in forecasting process • • • MPO LRTP Updates Comprehensive Plans SIS/FIHS Planning Campus Master Plans Concurrency Applications Development of Regional Impacts (DRIs) Congestion Management Systems Air Quality and Climate Change Corridor Studies § Corridor Feasibility Studies; § FTA New Starts/Small Starts Applications; § Project Development and Environment (PD&E) Studies; § Interstate Master Plans; § Interchange Justification/Modification Reports (IJR/IMR); and § Toll Feasibility Studies. 19

Guidelines for Model Applications Model Application Checks ü Review logic of demographic forecasts at region and subarea ü Generate color-coded plots of highway network characteristics ü Compare base and future year trip productions and attractions by purpose at the regional and subarea level ü Compare base and future year trip distribution patterns ü Review logic of changes in mode splits resulting from scenario testing that would seemingly benefit one mode over another ü Compare traffic estimates on specific corridors and screenlines between base and future years and build and no-build conditions 20

Next Steps FDOT is conducting peer review of Final Report Validation checklist and new standards being tested to • Review/comment on recent validation studies • Ratchet up expectations for ongoing validation studies Model Step Model Statistic to Evaluate Acceptable Range of Values Low High Accuracy Standard Cube, GIS Visual and Statistical Comparisons/Checks Recommended Comparisons and Calculation Methods/Comments Input Data Socioeconomic Data Persons/DU (or HH) 2. 00 2. 70 N/A NHTS > 2. 46 FL – 2. 59 U. S. Employment/Population Ratio 0. 35 0. 75 N/A Autos/DU (or HH) 1. 75 2. 10 N/A Approximate Population/TAZ N/A 3, 000 N/A Recommendations from TAZ White Paper Highway Network Data Cube, GIS visual and statistical comparisons/checks Check hwy network, prohibitors, tolls, paths Highway Speed Data ensure logical hierarchy by AT/FT/NL; survey chk Will provide acceptable ranges in sep. table Transit Network Data chk access links; chk routing against GIS data Checks for transit network, access, paths 21 Document checks for households and employment