National Science Foundation Division of Science Resources Statistics

National Science Foundation Division of Science Resources Statistics Record Keeping Studies – Love ‘Em or Leave ‘Em International Conference on Establishment Surveys Montreal, Quebec, CA June 19, 2007 Jeri Metzger Mulrow, National Science Foundation Stanley Freedman & Robert Rutchik, Energy Information Administration National Science Foundation Division of Science Resources Statistics www. nsf. gov/statistics/

National Science Foundation Division of Science Resources Statistics Outline of Today’s Talk • Background and impetus for study • What we hoped to learn • The record keeping study design • Results and some conclusions • Lessons learned

National Science Foundation Division of Science Resources Statistics NSF Survey of Industrial R&D Background • Collects information on research and development (R&D) • Long running, annual survey (1953) • Company level (not establishment level) • Intended to cover all for-profit, public or private, non-farm companies with 5 or more employees operating in the U. S. • Self administered paper form • Response rates ~ 80%

National Science Foundation Division of Science Resources Statistics Industry Expert Panel Literature Review Methodological Work – Response Studies, Cognitive Interviews, Respondent Debriefings Record Keeping Study Redesign Efforts Data User Needs Workshops Process & Methods Evaluation s

National Science Foundation Division of Science Resources Statistics OMB Guidance – Pre-Survey Design Visits for Establishment Surveys • Is the information available? • How closely aligned are the records with the requested data? • Are the record keeping systems compatible with the collection instruments? • How difficult is it to provide the data?

National Science Foundation Division of Science Resources Statistics NSF Questions of Interest • Where are the records kept? • Who has access to them? • How many record keeping systems exist in the organization? • Which ones contain the data of interest? • How knowledgeable is the person about the data? • How often are the records updated? • What specific data items are kept?

National Science Foundation Division of Science Resources Statistics Overview of Recordkeeping Study • Preparations (October 2004 - June 2005) • Background interviews and project design • Question and protocol development • Phase 1: (July-September 2005) • Broad understanding of firms’ R&D definitions, organization and availability of records • Phase 2: (June-August 2006) • Detailed review of financial recordkeeping practices • Phase 3: (TBD) • Recordkeeping practices on specific priority areas

National Science Foundation Division of Science Resources Statistics Preparations • October 2004 - June 2005 • Eight (8) pre-interviews with R&D or innovation experts – – – • • Business schools Accounting firms Government organizations Developed protocol Developed recruitment strategy Tested with five (5) companies Revised strategy and protocol

National Science Foundation Division of Science Resources Statistics Phase 1 • July - September 2005 • 20 company visits • Sample of companies – – Used public lists Unknown if respondents to R&D survey Covered a variety of industries Included 10 small companies (Small Business Innovation Research) • Did not mention the R&D survey • Broad understanding of how the company performed R&D • How they defined R&D • General availability of records

National Science Foundation Division of Science Resources Statistics Phase 2 • June – August 2006 • Evaluated the results from Phase 1 • Revised the approach – – – Focused more on large companies Focused on specific information and records Requested to speak to people familiar with the financial records of the company rather than the R&D • 9 company visits • Did not mention the R&D survey

National Science Foundation Division of Science Resources Statistics Industry by Company Size (to date)

National Science Foundation Division of Science Resources Statistics Job Function by Company Size (to date)

National Science Foundation Division of Science Resources Statistics What happened? • Pre-interviews provided good insights • Difficult to gain access to companies • Companies did not provide access to records However -- • Extremely valuable to talk with companies • Gained useful knowledge

National Science Foundation Division of Science Resources Statistics Did we answer our questions? • Where are the records kept? – Different records kept in different places • Who has access to them? – Different people have access to different types of information • How many record keeping systems exist in the organization? – Financial/accounting often separate from HR separate from licenses and patents • Which one contains the data of interest? – All of them

National Science Foundation Division of Science Resources Statistics Did we answer our questions? (2) • How knowledgeable is the person about the data? – We did not speak to the people who created the data – People in one part of the organization did not know about how things were done in other parts of the organization • How often are the records updated? – ? • What specific data items are kept? – Varies across companies

National Science Foundation Division of Science Resources Statistics What did we learn about R&D? • Most companies do not have a formal definition of R&D • Informal definitions of R&D are not consistent within or across companies • The line between research (R) and development (D) is not clear or consistent • R&D is done in a variety of ways – In a central lab – At each business line – Anywhere in between

National Science Foundation Division of Science Resources Statistics What else did we learn? • R&D details are tracked and managed at the project level • Companies can provide a total cost for R&D BUT the components vary • Companies know who is paying for the R&D – themselves or others • Companies can separate domestic from international spending • Many collaborate with universities to do R&D

National Science Foundation Division of Science Resources Statistics Our conclusions • Provide clear definitions of R&D on the survey • Tell them what to include or exclude in the R&D totals • Involve more than one person to answer the survey • High level R&D data are available but many of the details aren’t kept in the record keeping systems

National Science Foundation Division of Science Resources Statistics Lessons Learned • Company visits are extremely valuable • Flexibility is good, but still need a framework • Talk to more than one person in the organization, if possible • The more you know, the more you learn • Record keeping is just one piece of the puzzle

National Science Foundation Division of Science Resources Statistics Love ‘Em or Leave ‘Em?