NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Quality in Education Research Norman L. Fortenberry, Sc. D. Director, CASEE http: //www. nae. edu/CASEE nfortenb@nae. edu 202 -334 -1926 University of Idaho May 1, 2003

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Schedule Summary of NRC Report on “Scientific Research in Education” edited by Shavelson and Towns (30 min) Break (10 min) Five (sets of) questions on the implications for Engineering education research (50 min)

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Context of NRC Report Sparked by l Elementary and Secondary Education Act reauthorization and its legislated definition of “acceptable” research as well as l Growing enthusiasm for “evidencebased” research.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Nature of Science (1/2) l Continual process of rigorous reasoning supported by a dynamic interplay among methods, theories, and findings. l Builds understanding in the forms of models or theories that can be tested, advances are achieved by the self-regulated norms of the scientific community over time. l The path of progress is circuitous, indirect, and takes time; it depends on professional skepticism and criticism.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Nature of Science (2/2) l Rarely does a single study produce unequivocal and durable results and a synthesis across studies is often necessary. l The enterprise depends upon a health community of researchers. l There are guiding principles that serve as norms, not rigid standards for judging individual studies.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (1/6) Pose significant questions that can be investigated empirically, • The testability and refutability of scientific claims • or hypotheses is an important feature of scientific investigations that is not typical in other forms of inquiry. The questions, and the designs developed to address them, must reflect solid understanding of the relevant theoretical, methodological, and emperical work that has come before.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (2/6) Link research to relevant theory, • Every scientific inquiry is linked either implicitly or explicitly to some overarching theory or conceptual framework that guides the entire investigation. • Science generates cumulative knowledge by building on, refining, and occasionally replacing, theoretical understanding.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (3/6) Use methods that permit direct investigation of the question, • Methods can only be judged in terms of their • appropriateness and effectiveness in addressing a particular research question. Moreover scientific claims are significantly strengthened when they are subject to testing by multiple methods. Particular research designs and methods are suited for specific kinds of investigations and questions, but can rarely illuminate all the questions and issues in a line of inquiry. Therefore, very different methodological approaches must often be used in various parts of a series of related studies.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (4/6) Provide a coherent and explicit chain of reasoning, • At the core of science is inferential reasoning. Making • • • scientific interferences is not accomplished merely by applying an algorithm for using accepted techniques in correct ways. Rather, it requires the development of a logical chain of reasoning from evidence to theory and back again that is coherenet, shareable, and persuasive to the reader. The validity of inferences made through this process is strengthened by identifying limitations and biases, estimating uncertainty and error, and crucially, systemically ruling out plausible counterexplanations in a rational, compelling way. Detailed descriptions of procedures and analyses are critical to permit others to critique, analyze, and replicate.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (5/6) Replicate and generalize across studies, and • Since all studies rely on a limited set of • observations, a key question is how individual findings generalize to broader populations and settings. Ultimately, scientific knowledge advances when findings are reproduced in a range of times and places and when findings are integrated and synthesized.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Guiding Principles (6/6) Disclose research to encourage professional scrutiny and critique. • Scientific studies do not contribute to a larger body of knowledge until they are widely disseminated and subjected to professional scrutiny by peers. • The objectively of science derives from publicly enforced norms of the professional community of scientists, rather than the character traits of any individual person or design features of any study.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Contexts of Research in Education (1/3) l While all sciences share common principles, every field of study develops a specialization as the principles are applied. l Education occurs within an interaction among stakeholders, is highly value laden, and involves a diverse array of people and political forces. l These features require attention to the physcial, social, cultural, economic, and historical environment in the research process since these contexts often influence research results.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Contexts of Research in Education (2/3) l Multiple disciplinary features bear on the study of education (e. g. , content and process). l Conducting education research that involves studying humans is governed by the need to ensure ethical treatment of these participants. l Education research depends on its relationships with practice.

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Contexts of Research in Education (3/3) l The features of education, in combination with the guiding principles of science, set the boundaries for the design of scientific education research. l The design of the study does not make the study scientific. A wide variety of legitimate scientific designs are available for education research (from randomized experiments to ethnographic case studies).

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Implications for Engineering Education Research Will the Engineering community accept education research as “research” and not simply another form of “teaching” – implications for faculty workload and rewards? Can this issue be separated from the validity of the Boyer model? l ? ? ?

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Implications for Engineering Education Research Even if accepted as a valid form of research, does education research have a place in Engineering departments (i. e. , why not leave it to the social scientists and educators) l ? ?

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Implications for STEM Education Research The physics community has built theoretical underpinnings to support research efforts, can/should engineering disciplines do likewise? l ? ? ?

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Implications for STEM Education Research How might the Engineering community build the capacity for the conduct, review, and communication of rigorous education research? Until such capacity is built, how do we manage the transition; that is, who judges the early efforts? l ? ? ?

NATIONAL ACADEMY OF ENGINEERING OF THE NATIONAL ACADEMIES Center for the Advancement of Scholarship on Engineering Education Implications for STEM Education Research How is a community of researchers best built and maintained? Is such research limited to post-tenure faculty? Can this be done effectively if done only on nights and weekends? Is there room for a single specialist in a large department? l ? ? ?