Common Guidelines for Education Research and Development The

Common Guidelines for Education Research and Development: The National Science Foundation and the Institute for Education Sciences at the U. S. Department of Education Edith Gummer MSP Conference 12/16/2013

Joint Committee’s Assumptions � Knowledge development is not linear, but multi- directional � Scientific contributions by multiple researchers, research teams and practitioners are needed � An efficient research and development program may require fewer studies as one moves from foundational to impact evaluation studies � Implementation research is important for design and development projects and for impact evaluations � Outcomes may be education and/or learning in multiple settings 2

Goals of the Common Guidelines Project �Improve the quality and pace of findings from education research and development proposals �Develop an education infrastructure that supports more rapid and efficient knowledge development �Aid NSF and ED in making informed choices about where to invest scarce research and development dollars �Provide clarity for the field (and within the two agencies) 3

What’s Included in Education “Research and Development”? 1. Proposals that seek to test, develop, or refine theories of education and learning that advance knowledge acquisition and the factors that facilitate or impede learning 2. Proposals that design and test new approaches to teaching and learning 3. Proposals that investigate some application of practice (e. g. , a new curriculum, a new technology) 4. Proposals that study the impact of an education practice or policy (e. g. , what effects are College and Career ready standards in mathematics having on instruction? ) 4

Types of Studies � � � Foundational research and development ◦ Fundamental knowledge that may contribute to teaching and/or learning Early stage/exploratory ◦ Examines relationships among constructs to establish logical connections Design and development ◦ Design and iteratively develop particular interventions (programs, policies, practices or technologies); can also pilot test fully developed intervention to see if it achieves its intended outcomes 5

Types of Studies IMPACT STUDIES � Efficacy Studies ◦ Estimate the impacts of strategies under optimal conditions of implementation � Effectiveness Studies ◦ Examine implementation and estimate impacts similar to routine practice but still on a limited scale � Scale-up Studies ◦ Explore implementation and estimates impacts under conditions that prevail under wide-scale adoption 6

Document Organization � Introductory material � NSF and ED missions- distinct but complementary � Guidelines for six types of research and development proposals ◦ ◦ Foundational Early Stage and Exploratory Design and Development Impact Studies �Efficacy �Effectiveness �Scale-up � Conclusions and recommendations for use 7

General Structure of the Document: All six study types have these three components that call for a description of: (1) the purpose of the study; (2) the reason it’s important (entry guidelines); and (3) the likely outcomes (exit guidelines) Section Heading Content & Example Subheadings Purpose • Gives an overview of the Study Type. • No subheadings Justification. Entry • Significance for policy and/or practice • Basis in prior theory & evidence Evidence. Exit • Likely project outcomes • Plans for the research & development • Plan for external feedback 8

A Joint Committee of ED and NSF Membership Directed the Development of these Guidelines Ex Officio: Joan Ferrini-Mundy Assistant Director, EHR and John Easton, Director, Institute of Education Sciences Co-Chairs: Janice Earle, EHR and Rebecca Maynard, National Center for Evaluation and Regional Assistance, ED Members: � � ED: Elizabeth Albro, IES; Karen Cator, Office of Technology; Michael Lach, Office of the Secretary; Loy Lesnick, Center for Evaluation and Regional Assistance; Ruth Curran Neild, National Center for Evaluation and Regional Assistance; Lynn Okagaki, IES; Jefferson Pestronk, Office of Innovation and Improvement; Anne Ricciuti, IES; Allen Ruby, IES; Deborah Speece, National Center for Special Education. NSF: Jinfa Cai, EHR-DRL; Edith Gummer, EHR-DRL; Jim Hamos, EHR-DUE; Gavin Fulmer, EHR-DRL; Janet Kolodner CISE and EHR-DRL; Susan Winter. SBE 9

Do these Guidelines apply to all education projects at NSF? � These Guidelines are most relevant for NSF programs and projects that undertake education research and development activities. Some solicitations will explicitly reference the Guidelines. � Not intended for outreach or scholarship activities. 10

How do the guidelines affect NSF's Merit Review criteria? � The Guidelines are not intended to supplant, but are consistent with the Merit Review criteria. ◦ One element of the intellectual merit criterion for proposals is whether the project can advance knowledge and understanding. ◦ In addition, the intellectual merit criterion calls for a well-reasoned, well organized plan based on a sound rationale along with a mechanism to assure success. 11

Do the Guidelines preclude or privilege any research methodologies? � No. The Guidelines do not preclude or favor any research methods, but they do underscore the importance of ensuring that the methods are well described, justified, and appropriate to the research questions that are posed. � Qualitative and quantitative approaches may be used in all of the six research genres that are described in the Guidelines. 12

How will reviewers be informed about the Guidelines? � Reviewers will be informed of the Guidelines through multiple approaches. The Guidelines will be: ◦ posted on the NSF website, ◦ referred to in program solicitations, ◦ discussed in reviewer webinars and orientations, and ◦ presented at PI and other professional meetings. 13

What are the implications of the Guidelines for external evaluation of projects? � The Guidelines include recommendations for all types of studies that call for external feedback on the work being proposed. ◦ External feedback can include a number of approaches including third party evaluation, program officer evaluation, and/or regular feedback from advisory groups. ◦ It is up to the proposer to explain (a) why the external feedback planned is appropriate, and (b) how it is aligned with program requirements. 14

Will these Guidelines preclude innovative projects? � No. The Guidelines are intended to help PIs in proposal preparation. The key point of the Guidelines is to ensure that projects are explicit about their research questions, methods and analytic approaches in their proposals. These criteria should be relevant for all types of education R&D efforts. 15

Additional Information on the Common Guidelines available on NSF website: NSF-127: http: //www. nsf. gov/pubs/2013/nsf 13127. pdf � FAQs Point of contact for NSF: Janice Earle jearle@nsf. gov 16