Technology Program Evaluation Methodologies from the Advanced Technology

Technology Program Evaluation: Methodologies from the Advanced Technology Program Survey Development, Uses, and Findings Stephen Campbell Technology Innovation Program National Institute of Standards and Technology Collaborative Expedition Workshop #71 National Science Foundation March 18, 2008 National Institute of Standards and Technology • U. S. Department of Commerce

Framework Ø Concepts to measure § Mission § Broader research interests Ø Operationalizing the concepts § Quantitative and/or qualitative § Absolute and/or relative Ø Uses and findings from the data § Summary stats § Fuller analysis (regression) National Institute of Standards and Technology • U. S. Department of Commerce

Different Surveys for Different Questions Ø Did ATP select the “right” types of projects? § “Process” evaluation § Driven by mission and selection criteria Ø What are the determinants of success in ATP projects? § “Project” evaluation § Define measures of success § Are the selection criteria correlated with success? What are the other determinants? National Institute of Standards and Technology • U. S. Department of Commerce

Evaluating the Selection Process Ø Selection criteria § Innovative § High-risk § Encourage longer research time horizons § Foster collaboration § Broad-based benefits (diffusion) § Appropriate need for ATP funding (counterfactual) Ø Used the Survey of ATP Applicants § Three surveys fielded for award competitions in 2000, 2002, 2004 § Surveyed both awardee and non-awardee companies § Combined with administrative data National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Selection Criteria (Example for Risk) Ø Quantitative measures for technical risk of ATP project and for a “typical” project at the company Ø Did ATP fund higher risk projects and/or projects with a greater “stretch” in the risk profile? Ø Survey questions § “From 0 to 100%, what would you say is the approximate probability that your proposed ATP project could fully achieve its technical goals? (Try to answer based on how you thought about your project when you proposed it to ATP)” § “What is the approximate probability that a typical R&D project at your company could fully achieve its technical goals? ” National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Risk (cont’d) National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Selection Criteria (Example for Time Horizon) Ø Quantitative measures for time to impact on revenues of ATP project and for a “typical” project at the company Ø Did ATP fund longer time horizon projects and/or projects with a greater “stretch” in the time horizon? Ø Survey questions § “Approximately how many years after the start of your ATP proposed project could you expect results to first have an impact on company revenues? (Try to answer based on how you thought about your project when you proposed it to ATP)” § “Approximately how many years after the start of a typical R&D project could you expect results to first have an impact on company revenues? ” National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Time Horizon (cont’d) National Institute of Standards and Technology • U. S. Department of Commerce

Regression Analysis of the Selection Process Ø Projects with greater technical risk were more likely to receive an award Ø Projects with a longer time horizon were more likely to receive an award Ø Companies receiving an ATP award are able to leverage additional funding to the line of research around the ATP project in the post-application period Ø Evidence of a “value-added” for the ATP selection panels National Institute of Standards and Technology • U. S. Department of Commerce

Evaluating Project Success Ø Content areas § Key Personnel § Subcontracting § Company characteristics § ATP project characteristics § Research effort § Project management § Research outputs § Technology commercialization Ø Uses the Business Reporting System § Awarded companies surveyed annually from project start to project end § Awarded companies surveyed 2, 4, and 6 years after project completion in a “Post Project Survey” National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Technical Diffusion Ø Categorical measures for research results and know-how becoming known to others outside of the company § Within 2 years of project end § 2 to 5 years after project end § 5 to 10 years after project end § 10 or more years after project end § Never Ø Survey questions § “How quickly do you expect critical research results from this project to become known to other outside your company? ” § “How quickly do you expect critical research ‘know-how’ from this project to become known to other outside your company? ” National Institute of Standards and Technology • U. S. Department of Commerce

Technical Diffusion – cont’d Ø Research results § Within 2 years of project end § 2 to 5 years after project end § 5 to 10 years after project end § 10 or more years after project end § Never 72% 24% 3% 0% 1% Ø Research know-how § Within 2 years of project end § 2 to 5 years after project end § 5 to 10 years after project end § 10 or more years after project end § Never 39% 42% 12% 5% 2% National Institute of Standards and Technology • U. S. Department of Commerce

Operationalizing Continued Technology Development Ø Categorical measures for the significance of technical and non- technical challenges to overcome before achieving widespread benefits of the ATP technology § Very significant § Moderately significant § Somewhat significant § Not significant Ø Survey questions § “How significant are any additional technical (research, development) challenges that still need to be addressed in order to achieve widespread commercialization? ” § “How significant are any additional non-technical (regulatory, business) challenges that still need to be addressed in order to achieve widespread commercialization? ” National Institute of Standards and Technology • U. S. Department of Commerce

Technology Development – cont’d Ø Technical challenges § Very significant § Moderately significant § Somewhat significant § Not significant 22% 42% 29% 7% Ø Non-technical challenges § Very significant § Moderately significant § Somewhat significant § Not significant 22% 41% 27% 11% National Institute of Standards and Technology • U. S. Department of Commerce

Regression Analysis of the Determinants of Project Success Ø Small companies are more likely than medium or large size Ø Ø companies to experience early revenues from products incorporating the ATP-funded technology Joint venture (non-lead) partners are less like than single companies or joint venture leads to realize both research and commercial outputs University participation in a project has a positive effect on knowledge creation and dissemination, as measured by journal publications Projects with university participation are less like to stop early for adverse reasons, which may reflect essential differences in motivation or characteristics of these projects Projects with greater innovativeness and risk (closer to the research frontier) are more likely to realize technical and commercial outputs National Institute of Standards and Technology • U. S. Department of Commerce

Concluding Thoughts Ø ATP was “blessed” to be such a controversial and scrutinized program Ø Early dedication to evaluation must be accompanied by resources (don’t forget about underlying database construction, maintenance, and integration issues) Ø Will always face a budget constraint and must prioritize § Basic reporting § Larger, broader research portfolio interests § “Boundary of the firm” issues National Institute of Standards and Technology • U. S. Department of Commerce