Agency Capabilities and Performance in Applying BCA Another

Agency Capabilities and Performance in Applying BCA Another Modest Proposal for Reform Richard B. Belzer Regulatory Checkbook Mt. Vernon, VA rbbelzer@post. harvard. edu

History of Federal Practice Formalized since 1981 n n Agency responsibility to perform BCA OMB responsibility to perform peer review Controversies and conflicts are hard-wired n n n With Congress and among agencies Staffing, organization, and reprogramming Methodological issues and research priorities Society for Benefit Cost Analysis

‘CSI: Executive Branch’ Bureaucratic biology n BCA: A bacterial infection Agencies have developed antibodies n n n OMB: Dissociative Identity Disorder (ICD-9 -CM Dx 300. 14) Performs external, independent peer review of RIAs… n n Program control over / co-option of agency economists Methodological corruption Benefits can be found if you look hard enough Costs can be ignored if you are determined enough …but advance the President’s agenda… …without being held publicly accountable for the difference where these functions conflict Administrative Procedure Act: Placebo therapy Society for Benefit Cost Analysis

Evidence of ICD-9 -CM Dx 300. 14 Caveats Agency (not OMB) estimates From a draft report Invalid Assumptions All ‘major rules’ are included Agencies adhered to OMB Circular A-4 Estimates of costs, risks, and benefits are unbiased Uncertainty is unimportant Society for Benefit Cost Analysis

Agency Use of Science in BCA: April 1998 -- March 2000 73 rules, 36 about risk Embedded bias n n Uncertainty ignored n n n 7 (‘judgment’) 2 (assumptions) 6 (upper bounds) 1 (sensitivity analysis) 6 (point estimates) 2 (ranges) 1 (distribution) Motive for performing BCA n n Statutory reqmt: 100% EO 12866: 14% Society for Benefit Cost Analysis

Examples: Major Rules from 1988 -1998 ‘Hazardous Waste Corrective Action’ n ‘Partially-Cooked Beef Patties’ n C: < $10 million; B: not estimated ‘Seafood HACCP’: impossible benefits n n C: ~$1 trillion; B: not estimated Reduced risks from seafood (‘consumers underestimate true risk’) Increased consumer confidence in seafood safety (‘consumers overestimate true risk’) ‘Meat & Poultry HACCP’: imaginary benefits n Assume regulation reduces risk 90% Society for Benefit Cost Analysis

Examples: Major Rules from 1988 -1998 ‘Hazardous Waste Corrective Action’ n ‘Partially-Cooked Beef Patties’ n C: < $10 million; B: not estimated; R: ~ 30 deaths/yr ‘Seafood HACCP’: impossible benefits n n C: ~$1 trillion; B: not estimated; R: ~0 Reduced risks from seafood (‘consumers underestimate true risk’) Increased consumer confidence in seafood safety (‘consumers overestimate true risk’) ‘Meat & Poultry HACCP’: imaginary benefits n Assume regulation reduces risk 90% Society for Benefit Cost Analysis

Examples: Major Rules from 2007 -08 5 Major Rules n n n PTO ‘Limits on Claims’ PTO ‘Limits on Continuations’ PTO ‘IDS’ PTO ‘Appeals’ DHS ‘No Match’ OMB made aware of costs but deemed none of them ‘major’ Society for Benefit Cost Analysis

Examples: Major Rules from 2007 -08 5 Major Rules n n n PTO ‘Limits on Claims’ PTO ‘Limits on Continuations’ PTO ‘IDS’ PTO ‘Appeals’ DHS ‘No Match’ OMB made aware of costs but deemed none of them ‘major’ Society for Benefit Cost Analysis $1 billion!

BCAs Can Be Validated, But They Aren’t Often easier than ex ante analysis Very limited federal agency interest Limited external WTP for ‘academic’ work HACCP Implementation Period Society for Benefit Cost Analysis

Dx 2001: Centralized Review Doesn’t Work Limited and progressively smaller review staff Transaction-driven, and hence reactive Review occurs after decisions have been made Asymmetrical rules of engagement Limited enforcement tools Enforcement has limited utility Presidential initiatives are exempt Congressional sensitivity is unbounded Statutory/judicial deadlines trump OMB review OMB has limited political and public support Society for Benefit Cost Analysis

Rx 2001: Failed Remedies Persuasion Co-option RIA Guidance Return to sender Suspension of review Society for Benefit Cost Analysis

Rx 2001: First Modest Proposal Failed Remedies Persuasion Co-option RIA Guidance Return to sender Suspension of review Society for Benefit Cost Analysis Possible Ideas Regulatory budgeting ‘Prompt’ letters Publish peer reviews BCA blueprints Integrate BCA & PRA

Rx 2001: Implemented 2001 -09 Failed Remedies Persuasion Co-option RIA Guidance/Cir. A-4 Return to sender Suspension of review Society for Benefit Cost Analysis Possible Ideas Regulatory budgeting ‘Prompt’ letters Publish peer reviews BCA blueprints Integrate BCA & PRA Information quality Peer review policy

How Have These Remedies Fared? Guidance, ‘prompts’, and returns Society for Benefit Cost Analysis

OMB Review: Lion or Pussycat? Lion n Reagan I Bush 41 Bush 43 (2001 only) Pussycat n n n Reagan II Clinton I & II Bush 43 (2002 -07) Society for Benefit Cost Analysis

How Have These Remedies Fared? Guidance, ‘prompts’, and returns Peer review policy (2005) Society for Benefit Cost Analysis

How Have These Remedies Fared? Guidance, ‘prompts’, and returns Peer review policy (2005) n n n Covers ‘influential’ scientific, technical, & statistical information used in BCA’ Good idea, badly implemented Too much agency discretion Agencies control charge and reviewers Reviewers cannot veto Agency BCAs reviewed by OMB are exempt! Society for Benefit Cost Analysis

How Have These Remedies Fared? Guidance, ‘prompts’, and returns Peer review policy (2005) Information quality Society for Benefit Cost Analysis

Information Quality Law and Guidelines ‘Information Quality Act’ (2001) 44 U. S. C. 3516 note OMB government-wide guidance (2002) n n Reduce error: Pre-dissemination review Correct error: Administrative procedures Agency responses n n All complied except DHS Almost all were not serious Society for Benefit Cost Analysis

Information Quality Terms and Definitions Utility Integrity Objectivity n Substance n Presentation n “accurate, reliable, and unbiased” “accurate, clear, complete, and unbiased manner” “within a proper context” Procedure (transparency, reproducibility) Society for Benefit Cost Analysis

Types of Information Quality Error Substance Transcription Units Excess precision Misapplication Misuse Bias (not policy-neutral) Society for Benefit Cost Analysis Presentation Caveats omitted Sources omitted Context omitted Data/results censored ‘Best’ models only Bias (not policy-neutral) Procedural Non-transparency Irreproducibility

Types of Bias Incidental n Unavoidable n Not material to decision-making No model is perfect, data are limited and expensive Purposeful n n n ‘ 2 + 2 = 5 for large values of 2’ Best justification given: ‘avoids false negatives’ Consequence: many false positives Conflates analytic and decision-making functions Lets analysts make policy decisions Fundamentally incompatible with information quality and BCA Society for Benefit Cost Analysis

An Example of Freely Advertised Purposeful Bias “EPA’s policy is that risk assessments should not knowingly underestimate or grossly overestimate risks. This policy position prompts risk assessments to take a more ‘protective’ stance given the underlying uncertainty with the risk estimates generated. ” U. S. Environmental Protection Agency Office of the Science Advisor. 2004. An Examination of EPA Risk Assessment Principles and Practices. Available: http: //www. epa. gov/osainter/pdfs/ratffinal. pdf Society for Benefit Cost Analysis

Why Biased Risk Assessment Is So Toxic to BCA Even if every other methodological problem in BCA is solved… n n BCA results are invalid and highly misleading Baseline risks are overstated Benefits of regulation are overstated CE-analysis yields unreliable rankings Rank order would be correct if bias was constant Bias is not constant Tolerating bias incentivizes bias Some interests are made better off if risk assessment and BCA are discredited Society for Benefit Cost Analysis

Flexible Principles: Activists & BCA Complaint Regulation Sophisticated Sabotage (2004) Undercounted Benefits Overcounted Costs Too hard/unethical to quantify Benefits Double-counting Costs Discounting Rate too high Intergenerational benefits Overwhelming Existence value Massive Biases in data & analysis Must be ‘protective’ Society for Benefit Cost Analysis

Flexible Principles: Activists & BCA Complaint Regulation Sophisticated Sabotage (2004) Roads, Bridges, Dams, etc. Damming the West (1973) Undercounted Benefits Costs Overcounted Costs Benefits Too hard/unethical to quantify Benefits Costs Double-counting Costs Benefits Discounting Rate too high Rate too low Intergenerational benefits Overwhelming Overwrought Existence value Massive Negligible Biases in data & analysis Must be ‘protective’ Inexcusable always Society for Benefit Cost Analysis

Regulatory Checkbook Study of 5 -Year Agency Performance Procedural compliance n n Timely? Informative? Interpretively correct? Forgiving of limited sophistication? Substantive compliance n n n Accurate: Is response true? Honest: Are legitimate complaints admitted? Responsive: Is the remedy reasonable? Society for Benefit Cost Analysis

Agency Performance on the Least Demanding Procedural Index Requests for Correction of Information Quality Errors, 2002 -2007: Selected Agency Response Times (Days) ‘Timel % Avg SD Min Max y’ ‘Timel Agenc N y Cases (days) (days) y’ HHS 33 171 128 35 60 12% EPA 41 175 128 17 689 90 15% Society for Benefit Cost Analysis

Why Is Agency Performance So Bad? Institutional n n n Externally imposed Some errors are useful Acknowledging error increases litigation and compromises enforcement Legal n n WH decision to issue guidance, not promulgate a rule Limited enforcement of guidance by OMB No enforcement of guidance by courts No congressional oversight Society for Benefit Cost Analysis

What Can the New Society for Benefit Cost Analysis Do? Publish a journal n n Publish an external, authoritative manual n n Some theory Lots of applications OMB guidance documents (1990, 2003) need improvements that OMB staff cannot make An external authority is better Peer review federal agency BCAs n n n Use information quality levers (insist on objectivity) Convene genuinely independent panels M*A*S*H approach (‘meatball economics’) Society for Benefit Cost Analysis