Evaluating Technologys Impact on Teaching and Learning Educational

Evaluating Technology’s Impact on Teaching and Learning Educational Technology Integration

Contact Information zjsun@sun-associates. com y 978 -453 -3070 zwww. sun-associates. com/eval/sample y. This presentation is linked to that page

Where Do We Stand? z. Hot topics y. Your District y. State y. National z. Your expectations for today

Workshop Goals z. To overview a scheme for creating a formative evaluation of technology’s impact z. To review related issues and developments visa vis our experience in a particular school district (Fayette County, KY) z. To create an action plan for conducting a similar assessment in your own district

A Framework for Review

Why Evaluate? z. To realize your investment in technology y. What sort of “difference” has all of this technology made? z. To continue your school or district technology planning effort y. Evaluation is a key component of ongoing technology planning!

Evaluation Starts with Technology Goals z. Evaluation should be rooted in a strong educational technology plan which. . . y. Is more than an infrastructure plan y. Focuses on technology’s impact on teachers and students y. Has clear goals and objectives for what you want to see happen

Skeletal Plan Format

Your Goals? z. Using the Plan Mapping Worksheet, map your plan in terms of vision, goals, and actions.

Goals Lead to Questions z. What do you want to see happen? y. These are your goals z. Achieving these goals requires a process y. Are you performing the process steps? z. All of this can be measured through a formative evaluation

The Evaluation Process z. Create meaningful evaluation questions z. Design relevant indicators z. Organize those indicators in a performance rubric which can be used to measure progress and achievement z. Collect data to score the rubrics z. Create a report which applies the evaluation in a formative manner

Take a break. . .

FCPS Example Questions z. How have students been impacted by technology integration? z. Are teachers using technology in ways that match district goals for technology and use the potential for instructional technology? z. Have we adequately allocated district technology resources?

Processes of Technology Evaluation z. What are you looking for? y. Ways to make the implementation of technology better y. Teaching and learning impacts — this is not about counting machines!

Developing Indicators z. What is it that you want to measure? y. Impact on teaching and learning y. Change y. Progress towards desired outcomes z. You need indicators for progress to be measured

z. Indicators should reflect your school’s unique goals and aspirations y. Arise from your plan y. Rooted in your vision and goals y. Indicators must be indicative of your unique environment. . . what constitutes success for you might not for someone else.

Indicators of Engaged Learning z. Engaged learners are y. Responsible for their own learning y. Strategic problem-solvers y. Energized by learning y. Collaborative learners z. These are all observable characteristics z. These indicators can help inform your student impact rubric

Levels of Teacher Appropriation

z. Point is. . . There are things we can look for in terms of student and teacher behavior. z. The goal in creating indicators is to link aspects of behavior to levels of achievement

What’s Realistic? z. When developing questions and indicators, what makes sense visa vis the research on technology’s potential impacts? z www. sun-associates. com/eval/resources. html

What the Research Says z. CBI generates measurable increases in mastery of core content y. Kulik’s meta-analysis y. Fletcher’s evaluation of military training z. Pace, repetition, and standardization promote memorization of “facts” z. Kulik’s work shows that CBI did virtually nothing in terms of changing student “attitude” toward subject matter

z. Productivity tools -- particularly tools for writing -- encourage students to produce more and pay more attention to the quality of their work. y. This more free-form use of technology cuts directly to the heart of the connection between attitude toward learning and student achievement y. Students will do more of what they like to do

z. Technology improves the learning environment y. Motivates students y. Makes learning more exciting y. Makes learning more relevant to real-world tasks z. None of these benefits come solely through technology!

z. It gets more gray when considering “cognitive uses” y. Cooperative and collaborative tasks y. Communication y. Problem solving y. Using technology as a key element of a rich, supportive, educational environment

Student Performance in Rich Educational Environments z. Higher rates of attendance and graduation (ETS report) z. Higher qualification rates for honors programs z. Greater development of critical thinking and problem-solving skills z. Generally higher scores on traditional assessments z. Lower numbers of disciplinary incidents

z. These benefits have been observed/documented specifically in those classrooms where technology has been a key intervention. . . but never the only intervention!

What Other Interventions? z. More teacher training y. Usually in relation to using the technology tools as part of an overall reformed instructional environment z. More individualized, student-centered, learning y. Usually as a result of problem-based, cooperative learning environments where technology is the catalyst for change

So, What Changes? z. The student outcomes from participation in a technology-enriched environment are indeed positive z. Measures are usually in terms of new things that can be done versus increases on traditional assessments z. Nevertheless, traditional assessments do show improvement. . . it just occurs over time and is part of a broader, system-wide, picture of change.

Try a Sample Indicator z. Using the Developing Indicators worksheet, let’s develop a few sample indicators y. Work in several groups y. Select someone to report out y. Try to develop some consensus on indicators which would work across districts

FCPS’s Indicators z. Use of technology positively impacts and fosters the student’s motivation to engage in learning practices that lead to new ways of thinking, understanding, constructing knowledge, communicating results, and acquiring basic skills

z. The faculty and staff are proficient, knowledgeable, and current with available technology and translate that knowledge into relevant learning opportunities for students z. Teachers create learning opportunities and physical environments that allow students to assume more independent roles in their own learning through their use of technology.

z. All teachers and learners throughout the district have sufficient access to technology-based productivity tools, online services, media-based instructional materials, primary sources of data, and adequate support for using these resources so as to enrich and extend their learning goals.

Rubrics www. fayette. k 12. ky. us/central/edtech/suneval. asp

Rubric Tips z. Even number of levels z. Create from ideal to lesser z. Yes, Yes But, No z. Be as descriptive as possible z. Do not get excessively quantitative

Take a break. . .

FCPS Rubric Scores

Student Impact, 2 z. Technology is being used daily as a basic tool within a range of traditional learning activities z. Some evidence that technology is supporting new ways of thinking and learning z. Most student technology use is very teacher -directed and occurs in settings outside of the classroom

Teacher Fluency z. Teacher Fluency = 2. 5 y. Teachers are reasonably fluent, but need to do more to connect technology to the core curriculum z. Use of Technology with Students = 2. 25 y. Teachers direct students to use technology, but need to use technology to empower students as independent, engaged, learners

z. Teacher Productivity = 4 y. Teachers are very proficient in using technology as a tool for personal and professional productivity z. Administrator Support = 2 y. Administrators use technology, but need to advocate and support more integrated uses among their teaching staff

Infrastructure Allocation, 2. 75 z The basic infrastructure -- hardware, network, software -- is complete and adequate z Individuals in Technology Support functions (schools and district) are doing an exemplary job and should receive much of the credit for the advances we have seen thus far z Schools need even more technology resource teacher support z Efforts must continue to map technology on to the core curriculum

Sum It All Up? z. Fayette County is doing very well! y. Based on our research nationwide, FCPS is clearly above average in most areas of its technology implementation z. Teachers are using technology in ways that are consistent with their overall instructional approach y. Directed instruction y. Technology as a “skill” versus an integrated element of the learning process

z. Technology can do more, and teachers can do more y. Professional development and increased levels of resource teacher support will make this happen y. The full report contains a number of recommendations for professional development and policy designed to positively impact performance on the indicators

To Summarize. . . z. Start with your plan z. From your goals, develop indicators and a performance rubric z. Accumulate data and/or develop your data collection tools y. How will you collect the data necessary to document your stage of fulfillment?

Data Collection z. Review Existing Data y. Current technology plan y. Curriculum y. District/school improvement plans zwww. sun-associates. com/eval/sample z. Create a checklist for data collection

z. Collect New Data y. Focus Groups/Interviews x. Teachers x. Parents x. Students x. Administrators x. Other stakeholders y. Classroom Observations

y. Surveys x. Creating good surveys — what do you want to know • • • length differentiation (teachers, staff, parents, community, etc. . ) quantitative data attitudinal data timing/response rates (getting returns!) xwww. sun-associates. com/eval/samplesurv. html

z. Solicitation of teacher/parent/student stories y. This is a way to gather truly qualitative data y. What does the community say about the use and impact of technology?

z. Compile the report z. Determine how to share the report y. School committee presentation y. Press releases y. Community meetings

Conclusion z. Build evaluation into your technology planning effort z. Remember, not all evaluation is quantitative z. You cannot evaluate what you are not looking for, so it’s important to — z. Develop expectations of what constitutes good technology integration

Technology and Traditional Assessment Methods z. Does technology use result in higher test scores? z. Probably, but there is not a direct link z. Technology alone is not helpful in improving student performance z. A high degree of technology integration is more than likely to be an indicator itself of high performing schools and students

Is There a Connection? z. High scores on technology integration rubrics generally equate to successful teaching and learning practices -regardless of whether these practices have any specific relation to technology ye. g. , Constructivism, Engaged Learning, and Student-Centered Classrooms

z. Technology supports these practices and these practices are shown to improve student learning z. Schools where these practices are in place and technology is used appropriately are high performing schools z. It’s not the technology, it’s the teaching!

z. We measure how well the technology is used to support successful teaching practices. z. Our formative evaluations highlight how to improve teaching, technology integration, and ultimately student success

Next Steps? z. Awareness building y. Establish the need for evaluation y. Establish a process z. Division leadership y. As with technology implementation in general, this is very important y. The evaluation will only be as good as it is taken seriously

z. Evaluation committee membership y. Stakeholders! z. Are tech plans in shape? y. Ideally, the evaluations will be based in sound visions and goals and will take into account existing data on completed actions y. Do you divisions have this capacity?

z. Evaluation work can be used to rejuvenate the local revision process, but this will take time and organization z. Training y. It takes about a day to train people in this approach/process y. Regional workshops? y. District teams?

z. Standardizing and cross-division review y. People should share and review their questions, indicators, and rubrics before setting out to conduct an evaluation z. Think about data collection y. Integrating a meaningful technology impact survey into existing survey efforts y. Integrating observation data into existing teacher/class observations

More Information zjsun@sun-associates. com y 978 -453 -3070 zwww. sun-associates. com/eval/sample y. This presentation is linked to that page