Critical Issues in Evaluating Educational Technology Mark Hawkes

Critical Issues in Evaluating Educational Technology Mark Hawkes Dakota State University Presentation at the American Evaluation Association Annual Conference, Nov. 2, 2000

School Expenditure on Technology 1997 -98 1998 -99 1999 -2000 $5. 2 Billion $6. 0 Billion $7. 2 Billion

The Secretary’s Conference on Educational Technology Evaluating the Effectiveness of Technology The effectiveness of technology is embedded in the effectiveness of other school improvement efforts. Standardized test scores offer limited formative information with which to drive the development of a school’s technology program.

Critical Issues… Current evaluation practices need broadening. Schools must document and report their evaluation findings in ways that satisfy diverse stakeholders’ need to know. Everyone must agree on a common language and standards of practice for measuring how schools achieve that end.

Critical Issues. . . The role of teachers is crucial in evaluating the effectiveness of technology in schools. Some existing policies need to be “transformed” to match the new needs of schools using technology.

Scope of Evaluation: status then Technology focus: Drill-and-practice software, integrated learning systems, videotaped lessons, computer-animated picture books, low -level problem-solving and simulation computer software Professional Development: Participation, topics, accessibility, quality Hardware capability: Processing speed, memory, application functionality Student products Rate of implementation

Scope of Evaluation: status now Technology: information-processing and productivity tool Professional Development: relationship to student performance with technology Breadth of technology vision n Level and breadth of ownership Community involvement Propensity for new practice

Metrics: status then Standardized measures (IGAP, ITBS, CTBS) Student-to-computer ratio Survey: n n Self-report (competency, nature of use) Affect oriented Behavioral indicators n n n Attendance/Graduation rates Principal referrals Post-secondary options Teacher discourse: Technology-centered

Metrics: status now Critical incident analysis n Incidents of learning activities that reflect jobrelated skills and performances w Token Economies w Media Production w Comparative science Nature of discourse, signs of: n n Teacher-centered—nature of pedagogy Student centered—constructivist pedagogy Classroom observation of innovative uses

Role of Teacher: status then Hands on time with the technology n n n Trips to the lab Types of activities Tools used Report affective measures n Student attitudes toward technology Document categories of outcomes (motivational, behavioral)

Role of Teacher: status now Teacher activity logs Journaling on how teachers implemented technology facilitated instructional strategies with corresponding outcomes Content area professionals in examining how technology is linked to state learning standards Examining relationship between instructional practices and teaching and learning roles

Technology Policy: status then Lab topology; limited access detached form the learning experience Access to technology resources (equity) Focus with use policy and security n n Acceptable Internet use Email accounts

Technology Policy: status now • Aligning school support systems with technology outcomes • Libraries as information centers • Continuous financial support to sustain system components • Exploring technology use opportunities that extends learning beyond the school day • Extended lab hours • Lap top check-out

Inputs Infrastructure Wireless features Restructured leadership New roles for support services Individual Staff development plans Competitive collaboration grants Outputs Student motivation Accessibility Improved PD Improved use of services Renewed teacher skill building Improved efficiency in technology use

Over the two years. . . Continued reliance on standardized measures Policy at all levels remain prohibitive in many cases Improved support for technology purchases and professional development Limited examination of systemic indicators: - Vision of technology use - Support for technology implementation - Community roles and involvement - Administrative effectiveness

Over the two years. . . Parsing the effects of technology from the numerous school improvement ventures in the school proves virtually impossible Growth needed in collaborative “gap analysis” and deployment of technology to address those gaps Efforts to evaluate look quasiexperimental, have more attributes of research than evaluation

Slides available at: www. homepages. dsu. edu/hawkesm/ Presentations. htm Paper available at: www. ed. gov/Technology/Tech. Conf/1999/conf sum. html