Common Metrics Strategies for Teacher Preparation Survey Use

Common Metrics: Strategies for Teacher Preparation Survey Use in Accreditation and Program Improvement American Association of Colleges for Teacher Education February 25, 2016 Daria Taylor Paul, Minnesota State University, Mankato Mark Baron, University of South Dakota Kerry Hogan, Winona State University Sally Baas, Concordia University, St. Paul Alan Olson, Valley City State University Robin White, FHI 360

Agenda • Overview of NEx. T initiative and common metrics instruments • NEx. T common metrics in context of multiple measures; common metrics framework • Technical aspects and transition to IHE management of survey administration • Use of common metrics data for program improvement • Use of common metrics data for CAEP accreditation • Statewide use of common metrics instruments in ND • Plans/process for sharing instruments more broadly • Questions and discussion

NEx. T Overview • 14 colleges and universities working together to transform how teachers are recruited, prepared, placed, and supported, using data to drive continuous improvement. • NEx. T institutions have program specific goals in each of these areas. • NEx. T sites have formal relationships with P-12 partners and work with them to meet the established goals.

The NEx. T Common Metrics Framework • Development of a valid and reliable set of common survey instruments • Decision-making by consensus across 14 IHEs • Alignment of items across four surveys administered at different points in time • Use of psychometric analyses to guide survey revisions

Surveys Entry Survey • Who are the teacher candidates? • What encouraged them to become teachers? Exit Survey • How do candidates feel about their preparation? • How will we contact them after graduation? Transition to Teaching • What are graduates’ perceptions about their preparation and effectiveness after the first year of teaching? Supervisor Survey • What are the supervisors’ perceptions of the graduates’ effectiveness as firstyear teachers?

Data Governance Recommendations • Designed to guide responsible use and sharing of common metrics data with various internal and external audiences • Created by subcommittee of IHE representatives and approved by entire common metrics work group

Developmental Continuum

Key Elements 1. Performance should be presented with multiple measures and with responses to multiple questions. 2. IHEs should practice “responsible transparency” with all audiences to ensure an accurate representation of their programs. 3. IHEs should understand acknowledge the limitations of perceptual data and low response rates.

Key Elements 4. The focus of data sharing should be on program improvement and teacher effectiveness. 5. When sharing data, the audience should be considered.

Distribution of Higher Education Partners Valley Partnership • Minnesota State University, Moorhead • North Dakota State University • Valley City State University Minnesota State University, Mankato University of South Dakota, Vermillion NDSU USD Saint Cloud State University of Minnesota, Twin Cities Winona State University Twin Cities Private College Consortium (TC 2) • Augsburg College • Bethel University • Concordia University–St. Paul • Hamline University • St. Catherine University • University of St. Thomas

Lead Institution Responsibilities: NDSU and USD The Work: • • • Data use agreements Data transfer protocol Data cleaning Aggregate report Institutional reports Continuous psychometrics Rationale: • Enhanced efficiency • Better continuity • More equitable workload • More effective oversight • Improved accuracy

Technical Aspects – Survey Tools • • Qualtrics Survey Monkey Google Forms Paper & Pencil – Methods • Online • Classroom – Target Populations • Names • Cohorts • Demographic Information

Technical Aspects • Qualtrics – Relatively easy to use – Customizable (logos, etc. ) – Allows multiple platforms • Multiple Operating Systems • Desktops/Laptops/ Handhelds • Social media /Text messaging – Produces clean data – IHEs retain autonomy/ confidentiality – Panels – Scheduled mailings/Targeted reminders

Survey Reliability Analyses

Transition Point Chart

Data Driven Decision-Making Data Retreats Data Carousel The DR was a systematic, structured, and accessible approach to harnessing the power of the data: 1. 2. • • • Six retreats/year Faculty engagement Build assessment literacy Investigate unit & program-level data Data carousel worksheets 3. 4. 5. What do these data tell us? What do these data not tell us? What else would we need to know? What good news is here for us to celebrate? What needs for school improvement might these data indicate?

Faculty Use of CM Data Establish program goals and priorities Enhance a program of study or curriculum Improve/modify course content Incorporate new pedagogies or technologies in the classroom • Alter field experience hours and requirements • Use data in program evaluations • •

Institutional Example of Data Use Winona State University • Local analysis • Data visualization • Change from previous administration • Summary points • Long-term trends over time • Consistent design

Concordia Univ. St. Paul and CAEP-CIP • Background: Education Department is few in numbers and large in aspirations and plans for the future based on data from multiple measures. • Continuous improvement with reliance on Ne. XT surveys for baseline data combined with other measures • Using the data for program improvement – Earlier interviews for program – Procedures for working with ELLs who are pursuing teaching credentials – Data days for faculty and with stakeholders – Curriculum map – Use of integrated platform (Live Text) for all teacher education procedures

From the CAEP Report • The goals for improvement appear to be related to the overall rationale and cross over all programs. • The goals are related to improvement of the data collection system as a whole so that assessment data can be evidence of recruiting, supporting, and completion of students from a broad range of backgrounds and diverse populations. • CM are being used for program quality and improvement. The potential for the CIP to have a positive impact on the EPP and candidates is good.

Data Integr ity Relationsh ips Learning is like standing on the edge of a chasm! Don’t try to get over it one step at a time, but just take a giant leap!

Statewide use of NEx. T Common Metrics Surveys North Dakota Association of Colleges for Teacher Education (NDACTE) NEx. T Common Metrics surveys were shared with NDACTE representatives and 12 institutions in North Dakota. • Dickinson State University • Fort Berthold Community College • Mayville State University • Minot State University • North Dakota State University • Sitting Bull College • • • Turtle Mountain Community College United Tribes Technical College University of Jamestown University of Mary University of North Dakota Valley City State University

Actions involved in the planning stages for the ND pilot use of the Common Metrics Surveys • Statewide Participation o Benefits and challenges of common assessments • Survey Access o Permission from NEx. T Common Metrics group and Bush Foundation TEI to use the instruments in their entirety • Vision for Data Collection, Distribution, and Sharing o ABCs to IRBs

Implementation of Common Metrics in ND Communication and training to ensure: • Validity • Reliability • Common time lines Surveys administered in 2014 -2015 and aggregate report prepared at North Dakota State University by Dr. Stacy Duffield.

Data Analysis and Sharing (A sample of a several aggregate items are shown in the table below) • Common Metrics surveys gather quality data from multiple perspectives • Data useful to inform decisions for the improvement of teacher preparation Each institution has: • its own data to share with its stakeholders for program improvement. • data with common language and assessments that invite collaboration with other institutions. • aggregate data that allows institutions to identify similar strengths and areas for improvement in teacher preparation across the state.

Reflection and Future Plans The NEx. T Common Metrics group improvements: – Survey administration and data cleaning – Response rates and tracking of graduates – Improvements of aggregate data reports – Analysis of assessment instruments – Data governance – Data sharing with stakeholders – Program improvement – Training

Summary of Significant Benefits 1. Survey data that provide multiple perspectives to inform program decisions for improvement. 2. Common assessments and language to improve teacher preparation across the state’s teacher education institutions. 3. Reliable and valid assessment instruments aligned with In. TASC standards to assist in state and CAEP accreditation.

National Scale-up • NEx. T organizational governance and common metrics oversight/design teams • CAEP Endorsement • AACTE communications • 3 Tier Vision for instrument sharing

TIER ONE • Surveys shared with individual IHEs, organizations, collaboratives • Agreements re: Sharing Protocols (data templates, data governance, privacy protections) • Technical Assistance and Professional Development provided (fee structure TBD) • Individual sites responsible for data collection, analysis, reporting

TIER TWO • Surveys shared at state organization level • Agreements re: Sharing Protocols (data templates, data governance, privacy protections) • State organization responsible for data collection, analysis and reporting • Optional Data Sharing Agreements for interstate analysis and reporting

TIER THREE • Surveys shared with individual IHEs, organizations, collaboratives, states • Agreements re: Sharing Protocols (data templates, data-sharing, data governance, privacy protections) • Participation in NEx. T aggregate N • NEx. T assistance with data collection, analysis and reporting • Technical Assistance and Professional Development provided (fee structure TBD) • Reports include aggregate N analyses • Participation fees (TBD)

Questions and Discussion Representatives interested using the NEx. T Common Metrics surveys should contact Dr. Daria Taylor Paul, Minnesota State University, Mankato daria. paul@mnsu. edu