To See or Not to See Practical Issues

To See or Not to See: Practical Issues for Classroom Observation in Evaluation AEA 2013 Conference October 18, 2013 Martha A. Henry, Ed. D, CEP Keith S. Murray, CEP

Classroom Observations in Education Evaluation: Background • Professional Development = $$$ • PD Implementation Change • Implementation is the closed box

How do we know what’s inside? By opening the box and looking inside………

What can you tell about these classrooms just by looking?

What can you tell about these classrooms just by looking?

Classroom Observations: The Challenges

Classroom Observations: The Challenges

Alternatives to Evaluator Observation: The Downside • Teacher Self-Reports (Unreliable, potential for bias)

Alternatives to Evaluator Observation: The Downside • Lesson Plans/Other Data from Teacher Records (Documents do not show teaching practice in action) • Proxy observation by project team, mentors, coaches (Non-evaluative perspective, bias, training need)

Other Things to Consider Instrumentation Subject Selection The Observer Protocols and Practicalities Data Analysis and Use

Observation Instrumentation • Don’t assume validity of widely used instruments. • Alignment between project philosophy and instrument.

Observation Instrumentation

Observation Instrumentation Examples: • For STEM, Inside the Classroom Observation and Analytic Protocol (Horizon Research, 2002), or • Reformed Teaching Observation Protocol (ACEPT, 2000) (www. mahenryconsulting. com/supportmaterials)

The Observer The Ideal observer doesn’t exist How can we account for less than ideal conditions?

Subject Selection Focus Size Representative Attrition

Protocols and Practicalities How Many? When?

Protocols and Practicalities Protection of Human Subjects Confidentiality and anonymity Contact with students

Protocols and Practicalities Doubling up – administrator interviews, community review, document review

Data Analysis and Use • Compiling data/ensuring data quality • Types of analyses. • www. mahenryconsulting. com/supportmaterials • Determining what results are important. • Reporting and disseminating results

Example of Reporting Project Impact on Teachers: Practice First Observation Second Observation Capsule Ratings of Classroom Observations Third Observation (N=41 observations among 14 teachers) Ratings: 1=ineffective instruction, 2=elements of effective instruction, 3=beginning stages of effective instruction, 4=accomplished, effective instruction, 5=exemplary instruction.

What Is Gained by Good Classroom Observations? e tic c pra Co m an nne o o r cti d s s ou on la c tco o f o on i t m f PD e a t n p e es y m e T pl m i f o e c n Use of technology Evide Class a t room a d e cultu tiv a m re r o f r e g n o Str

Contact Marty Henry mahenry@mahenryconsulting. com Keith Murray keithsmurray@mahenryconsulting. com www. mahenryconsulting. com