Change is a Process Not an Outcome Implications

Change is a Process, Not an Outcome Implications for Evolving Federal Evaluation Policy Dianna L. Newman, Ph. D. University at Albany, SUNY Anna F. Lobosco, Ph. D. NYS Developmental Disabilities Planning Council

The Problem Increasing federal policy focus on programs intended to promote systems change Systems change versus service delivery Systems change is a complicated and intricate process with much “trial & error” as inputs shift and are converted to outputs & outcomes Federal stewardship includes a heavy reliance on logic models and outcomes based evaluation models —but what happens when: Economy shifts and resources are realigned to preserve services, but fidelity to the outcomes side of the logic model must be maintained The logic model doesn’t work and it impedes efforts rather than encouraging more feasible results

Federal Policy on Performance Management Achieve ambitious performance goals Constructive process ◦ ◦ performance review Trends Causal factors Action to outcomes alignment Links to detailed analysis Share information with the public (Metzenbaum, 2010)

Programs Managers Must… Set specific outcome-focused goals Measure progress Track completion of key milestones Compare programs across peers Look for factors government can influence Adopt evidence-based strategies Confirm achievement of intended outcomes Quickly adjust ineffective strategies Report to the public in useful & accessible ways

Logic Models in Federal Policy “Logic models illustrate the causal relationships among program elements and define program success” (OMB guidance to program managers, 2006) Program Logic Models “warmly embraced” by OMB (Jules & Rog, 2007) Clear focus on outcomes & impact to inform policy development/refinement— though more recent acceptance of program improvement & sustainability

Logic Model Worksheet INPUTS What we invest OUTPUTS OUTCOMES—IMPACT Activities Participation Short Medium Longterm What we do Who is reached? What do we think participants will know, feel, or be able to do after participating in the program? How do we think the participant s will behave or act differently after participatin g in the program? What kind of impact can result if the participants behave or act differently after participating in the program?

Logic Model INPUTS XXXXX Staff Time and Effort Existing Resources/Ser vice System Parent and Self -Advocate Advisors Technical Assistance/ Consultants Curricula and Training Information Technology (IT) Resources Steering Committee and Subcommittee Input from Community Leaders and Organizations Input from Other State Agencies Provider/Provid er Association Advisors XXXProgram Infrastructure Logistical Support/ Transportation for Meetings GOAL STRATEGIES OUTPUTS OUTCOMES Increased choice and control: developme nt / enhancem ent of selfdirected service delivery system. Increase number, capacity and efficiency of new and existing organizations, including XXXXX, that deliver individualized services across the life span, based on defined criteria. Conduct both internal and external public outreach about personcentered design and individualized service options, with a special focus on transition periods for youth and the aging. Streamline documentation through the use of electronic tools and technology. Address issues of recruiting and retaining a high-quality workforce. Strengthen and promote individual and family activities within communities through volunteerism, the use of natural supports, and relationship building. Strengthen quality assurance and quality improvement protocols to insure emphasis on individualized outcomes and quality of life indicators. Make understandable information about state and voluntary providers and the quality of services and supports available to all interested parties in accessible formats. Summary of national research • Pilot projects of evidence-based, exemplary practices with assessments of success • Definitions of: “individualized services”, “quality of life”, “quality services”, “community involvement” and “natural supports” • Outreach curriculum/program • Enhanced marketing materials • Universal electronic Medicaid Services Coordination (MSC) forms/infrastructure support • XXXXX/provider interconnectivity • Electronic signature process • Training on electronic records • Recommendations from direct support professional advisory process • Plan to address future workforce needs • Media products to promote acceptance of individuals with disabilities/public service announcement (PSA) campaign • Quality Assurance protocols that reflect input from individuals and families on quality of life • Enhanced customer satisfaction survey activities • Publicly accessible information on provider performance and service quality A culturally competent infrastructure to assist people in accessing individualized services and supports Service plans will reflect culturally diverse person centered values, promote efficient communication and enhance staff service as a result of reduced paperwork A quality, stable workforce to support individualized, person centered services and supports Community participation, relationship building and the use of natural supports in services Public access to information on key indicators of state and voluntary provider quality and performance data, as well as individual satisfaction with supports and services

Safe Schools/Healthy Students Final Logic Model Worksheet Needs and Gaps Needs 1. During the SCSD 2005/2006 school year, there were 1825 violent incidents. Gaps PBIS is not offered districtwide. Only 9 of 15 SCS have implemented PBIS. 2 nd Step Violence Prevention is only implemented at the 6 th grade level. Goals 1. To create and maintain a safe and violence free environment in all SCSD schools. SS/HS Final Logic Model – Putting It All Together SS/HS Element(s) Objectives Activities Partners Environments and Violence Prevention Activities 1. 1. To reduce violent incidents by a minimum of 2. 5% in every SCSD school building for each program year 1. Expand the PBIS model district wide. (1. 1) 2. Expand Second Step Violence Prevention to include grades 7 th & 8 th (1. 1) 3. Implement Olweus Anti. Bullying Program (grades 3 -8). (1. 1) 4. Deploy School Resource Police Officers in secondary schools. (1. 1) 5. Coordinate violence prevention strategies with the SCSD Security Dir. and School Safety Office. (1. 1) SCSD Implement PBIS, Second Step and Olweus with Violence Prevention Training Specialist to be hired with SS/HS funds; Police Dept. Add 3 SRO’s for middle schools with SS/HS funds; provide 2 SRO’s for SHS w/ local funds. Process Measures Indicators and GPRA 1. Number of teachers trained in PBIS at the high school level. 2. Number of turnkey trainers trained in 2 nd Step at the middle school level. 3. Number of Olweus turnkey trainers trained at the elementary school level. 4. Number of resource officers hired at the middle school level. 5. Number of prevention trainings at all levels. 1. 1. To reduce violent incidents by a minimum of 2. 5% in every SCSD school building for each program year as measured by discipline referral data reported on the NYS Violence And Discipline Incident Report (VADIR).

Strategic Plan Graphic

Systems Change Defined Systemic change can be described as a sustainable change in some aspect of the service or support availability; design or delivery that leads to positive or meaningful outcomes for people that also has the potential for replication (Promise of Opportunity, 2000) Systems change has been defined as: the moving of knowledge into practice by showing that it is accepted, incorporated, and maintained (Newman, 2001)

Systems Change Frameworks Organization #1 (Promoting Systems Change, 2008) Organization #2 (Impact & Outcomes, 2000) A conceptualization that includes: A process that includes: Improve the knowledge base Selecting social strategies Engaging Stakeholders Support policy entrepreneurs Using unexpected events (Tipping Points) Analysis and objective setting o Information gathering & goal development Introducing change & involving the system o o Engage the system Engaging partner(s)/collaborators Demonstrations & other strategies Staff development Infusing change into the system o Sustainability, i. e. continuation, replication Evaluation, renewal & extension o Evaluation o Information dissemination o Systems advocacy

The Three I Model for Evaluating Systems Change Initiation Implementation Impact • Visioning • Objectives • Strategies • • Activities Outcomes Formative evaluation Modification • Change • Summative evaluation • Re-Visioning

Three I Model for Evaluating Systems Change Initiation ◦ Visioning ◦ Goals & Objectives ◦ Strategies Implementation ◦ ◦ Activities Intermediate Outcomes Formative Evaluation Modification Impact ◦ Change ◦ Summative Evaluation ◦ Re-visioning

Three I Model for Evaluating Systems Change Phase Change Cycle Visioning Initiation Goals & Objectives Strategies Activities Implementation Intermediate Outcomes Formative Evaluation Modification Change Impact Summative Evaluation Revisioning Action Steps Learning Use Outcome

Table 1 Implementation Phase of Systemic Change in School-wide Policies and Procedures* Activity L L L L Activities Prioritizing resources and advocating for technology growth Executing fixed technology-based operational planning meetings Supervising educators in integration roles Increasing uses of technology in managerial tasks Promoting specific grade level or subject area applications Formative Evaluation Acquiring feedback on policies and procedures from educators and technology planners Defining and addressing continuing needs Regularly reviewing technology goals, policies, and procedures Using conflicts or issues that arise as impetuses for evaluation Modification Discontinuing ineffective and continue support of successful technology applications Evaluating educators’ skills and tailoring in-house efforts to specific needs Shifting from dependence on other sources of professional development and support to sustainable in-house efforts Shifting from lab-based resources to mobile or classroom-based technologies Learning L L H H L L L H H Use Increased educator understanding of priorities, decision making processes, and feasible resources Reflection on costs and benefits of growth Knowledge of best practices and valuable skills Solicitation of further training based on school and individual needs L Understanding of staff concerns and effective practices Identification of areas in need of strengthening, technical support, or change Evaluation of technology-based goal achievement levels Understanding of applications and skills that should be deepened and new applications to broaden L School-wide understanding of boundaries and capacities of technology infrastructure and staff abilities Understanding of process and product outcomes of increased technology integration that are likely to sustain over time Simplification of the processes of technology learning and use for educators L L H H Outcome Solidification of effective top-down policies and procedures Definition of observable indicators for successful technologyintegrated teaching Investigation of creative approaches to address scarcities of resources Delineation of a bottom-up planning approach L Engagement of a problem solving process to address concerns Prioritization of in-house, just-intime training and support for educators Address levels of success in meeting technology goals with staff and committee Investigation of new resources for strengthening technology uses L Integration of existing barriers and facilitators into school-wide planning Recurrent promotion of effective technology professional development and resources Evaluation of new roles that external service providers can play within the school Increase in teacher selfdetermination in technology acquisition and use L L L H H H Effective technology uses are reinforced Acceptable uses and expectations are widely disseminated Deeper understanding of feasible and impractical applications New solutions to technologybased problems are communicated and piloted Staff confidence in technology administration Continuation of training in applications proven effective Increasing in-house professional development and support offered by trainees and staff Development of new operational goals for addressing needs Initiation of new resource acquisition Shifting of staff roles and infrastructure to manage technology integration needs Continual distribution of technology learning and use across the school Comprehensive growth in baseline skills and confidence with technology Increase in technology uses to address Standards, authentic assessment, and state testing Preparedness for regular modifications

Systems Change as a Platform

“Legs” of Sustainable Systems Change POLICIES & PROCEDURES DESIGN & DELIVERY OF SERVICES INFRASTRUCTURE EXPECTED OUTCOMES/ EXPERIENCES

“Braces” of Systems Change CLIMATE & CULTURE CAPACITY BUILDING � LEADERSHIP � SUPPORTS FOR SUSTAINABILITY

The Case of the missing INFRASTRUCTURE

Relationship of Federal Policy—Program Evaluation—Systems Change Federal Policy on Performance Management Logic Model Program Evaluation 3 I Model ? ? ? Systems Change

An Example Initiation Inputs Processes/ Activities Implementation Outputs Impact Intermediate Outcomes Information Base Design and Delivery of Services Strategies Policies and Procedures Stakeholders Infrastructure Policy Entrepreneurs Expectations and Outcomes of Consumers Long-term Outcomes Sustainability Unexpected Events Intervening Variables Supports for Sustainabilit y & • Leadership Advancement • Capacity Building • Climate/Culture Elements of Long-term Change • Table-top not Braces

An Evaluator’s View Initiation Implementation Impact Outcomes Short-Term Intermediate Long-Term Consensus on vision Informal knowledge of activities Systemic change (P&P, infra. , D&D, Expectations) Delineation of strategies Intermediate program outcomes Summative evaluation – does the change meet expectations? Identification of objectives Formal knowledge from formative evaluation Reflection & re-visioning plans (sustaining, maintaining Measurable outcomes Modifications & success of modifications Resource identification – present & needed Identification of key stakeholders Short term program outcomes

An Example of Findings Growth in baseline skills & effective practice Inspecting Stand-alone Increased infrastructure & observing use technology curriculum Careful selection of equipment & criteria for evaluating impact PD Comparison of initial goals & activities to outcomes new objectives, shifting resources & acquisitions practice Stronger infrastructure decisions confidence in technology administration Goal achievement at desired levels Deeper understanding of applications & skills New need-based operational goals established Increased use of technology in meeting learning standards use of data in decision making Assessing capacity to meet emerging need Planning for sustainability on specific skills Integration of best Empowered teachers & students to construct own knowledge Informed Identify rifts between demand capacity Re-visioning on review & evaluation based

Relationship of Federal Policy—Program Evaluation—Systems Change Federal Policy on Performance Management Logic Model Program Evaluation 3 I Model ? ? ? Systems Change

Relationship of Federal Policy—Program Evaluation—Systems Change Federal Policy on Performance Management Logic Model ? ? ? 3 I Model Systems Change Program Evaluation

Guided Discussion What’s missing? ◦ And how do we get those factors in place? What are the challenges to get those missing factors in place? How does the missing factor effectively overlap with: ◦ Logic models ◦ 3 I Model

Logic Models and Evaluation Policy Related to Systems Change Possibilities for the Future: Fidelity vs. Evolution Adaptations of logic model methodology Realistic balance between attaining outcomes and documenting the systems change process in changing conditions Strengthen connection between evaluation and laws passed by Congress

References Jules & Rog (2007) Metzenbaum (2010) Newman, D. (2001) (OMB guidance to program managers, 2006) Promise of Opportunity (2000) Promoting Systems Change (2008)

Contact Information Dianna L. Newman, Ph. D. dnewman@uamail. albany. edu University at Albany/SUNY 1400 Washington Avenue School of Education, 2 nd floor Albany, NY 12222 518 -442 -5018 Anna F. Lobosco, Ph. D. Anna. lobosco@ddpc. ny. gov NYS Developmental Disabilities Planning Council 155 Washington Ave. , 2 nd floor Albany, NY 12210 518 -486 -7505

Appendices More examples follow:

Table 2 Impact Phase of Systemic Change in Infrastructure* Activity Change LShifting from uninformed acquisition of resources to careful selection of equipment that is projected to have positive impacts HShifting from concrete, long-term infrastructure to a flexible system based on changing needs Summative Evaluation LInspecting infrastructure and observing usage as means of assessing utility HDiscussing infrastructure strengths and weaknesses in formal planning HSoliciting feedback from stakeholders via surveys Revision HAssessing infrastructure’s capacity to meet emerging needs identified by stakeholders and through formal technology plans Learning L H Establishment of criteria for evaluating impacts of infrastructure acquisition and utilization Conceptualizing the value of regular data in making decisions and the potential for flexibility in infrastructure change to increase interest and appropriate use Use L H H Improved technology usage and monitoring of technology usage Increased empowerment of stakeholders to impact infrastructure Stimulating administrative willingness and ability to be flexible in infrastructure change Outcome L H Increased independence and utility of technology infrastructure within the school Increasing potential for infrastructure to meet the high expectations and increasing demands of stakeholders LIdentification of frequently and properly used technologies and infrequently or improperly used technologies HUnderstanding of barriers and facilitators associated with infrastructure change LInformed decision making in acquiring, updating, or phasing out specific infrastructure components HImprovements in the development of new solutions to infrastructure problems LIncreased stakeholder adaptation to infrastructure change possibilities and constraints HPrevention of debilitating infrastructure failures and more seamless transitions to new infrastructure HComparison of initial goals and activities to outcomes HIdentification of new objectives related to resource shifting and acquisitions HEstablish new objectives based on changing needs, uses, and available resources HConvergence of infrastructure and administrative revisioning HExpansion and sustainability of infrastructure planning H 2) Increased diversity, competence, and satisfaction in technology usage *L represents characteristics of schools lacking evidence of systemic change or demonstrating only basic change. H represents additional characteristics seen only in schools demonstrating advanced and comprehensive evidence of systemic change

Table 3 Initiation Phase of Systemic Change in Curriculum Development and Instruction* Activity Learning Use Outcome Vision LIdentifying technology resources that can be used to enhance instruction LDiscussing best practices in technology integration with other teachers HDistinguishing grade level and subject area appropriateness HPrioritizing professional development and integration activities that can most effectively enhance student learning Knowledge of the existence of limited technologies and professional development opportunities Recognition of ways in which technology can be used to enhance curriculum development and instruction H Basic understanding of age appropriateness and applicability of certain tools to the given curriculum Decision to pursue additional information and opportunities to increase expertise in using computers within educational contexts H Development of more extensive criteria for assessing value of technologies and training opportunities Development of entry-level familiarity with technology and its potential roles Perceived need to adapt to changes in technology and use them as means of varying instruction H Capacity to set clear, detailed goals and objectives for improving technology integration efforts Objectives LDefining roles that technology should play in curriculum design, instruction, and student learning LIdentifying familiarity, confidence, and skill development as desired training outcomes Focusing on using technology applications in different contexts and teaching students to evaluate their effectiveness Prioritizing seamless integration of key technologies over broad knowledge Knowledge of realistic potential roles of technology within the classroom and school contexts Understanding of unique challenges posed by technology as an information source and means of productivity H Analysis of student understanding and perceptions concerning technology H Identification of barriers to overcome in fostering seamless integration L Setting practical and observable objectives for technology integration in lesson planning and execution Establishing benchmarks for themselves and their students within content and contextual frameworks. L Strategies LSeeking training and regular technical support both local and external experts LIncreasing exposure to available technologies Using technology applications regularly to maintain effective practices L L L Selection of tools for integration into curriculum development and instruction Individualized plans for inclusion and integration of technology that is reflective of the building wide plan but that is adaptive to unique classroom needs. L H Acquisition of skills Discrimination of most worthwhile tools and applications Increased skill in acquiring and sharing new knowledge and confidence in creating individualized changes. H Communication with technology staff and other educators for direction toward professional development and appropriate resources Prioritization of student-centered learning in the process of learning to integrate technology Professional technology-based goals and objectives match those delineated in school-wide technology planning Planned inclusion of technology tools within overall curriculum. L Planned inclusion of technology in specified instructional practices. Teacher and classroom specific plans for implementation. *L represents characteristics of schools lacking evidence of systemic change or demonstrating only basic change. H represents additional characteristics seen only in schools demonstrating advanced and comprehensive evidence of systemic change

Table 4 Implementation Phase of Systemic Change in Curriculum Development and Instruction* Activity Adding technology applications to previously used teaching approaches Modeling appropriate technology uses for students and other educators Implementing new teaching methods based on technology learning Soliciting support follow-up professional development Intermediate Outcomes LSpecific curriculum and instruction practices are changed to reflect technology infusion LTeachers are more comfortable with and confident in their use of specific tools. Administrators begin to include use of technology as part of their review and hiring process. Faculty begin to include technology as part of student outcomes. Formative Evaluation H Identifying skills that constrain technology integration HIdentify infrastructure and policies that constrain technology integration. Learning Use Outcome How to use and comfort with technology for planning and aiding instruction. Motivational impact of technology and its role in meeting standards -based curriculum. Greater familiarity with alternative teaching methods. The importance of student-centered learning and the role that technology can play in supporting the process. The ability to turnkey train and support other teachers L The value of the use of technology integration as it impacts professional evaluations and status among peers is increased The importance of technology integration as part of the building-wide effort H Increased awareness of possible uses of technology and the cycle of outcomes Motivation factor in increased involvement for both teachers and students Delineation of further skills need for change. Knowledge of outcomes assists teachers and administrators in implementing more studentcentered assessment to correlate with student-centered learning. Use of technology is now included as part of the assessment rubric L L Revision of instructional approaches to conform to technology constraints H Reallocation of resources to overcome barriers to technology integration. Need for additional resources to bolster skills HNeed for additional support for all program components HBarriers to implementation H H H The inclusion of technology in autonomous classroom activities or in limited support of student-based outcomes. Technology integrated with different teaching strategies Increased learning of skills and integration methods Increased ability to assist others via role modeling. L L L H H Requests and reallocation of resources to meet skills constraints. HPrioritization of barriers. HRequests and reallocation of resources to meet prioritized barriers. L H H Greater teacher familiarity and comfort with technology Increased role modeling of basic skills to students Increased expectation that students will utilize selected basic skills Use of more student centered approaches that integrated technology Selective involvement in mentoring programs Plans for additional implementation in future curriculum design efforts. More inclusion of skills in daily expectation of teachers and students. Increased value placed on the use of technology in all phases of teaching. Increased value placed on gaining integration skills that further building-wide goals.

Table 4 Implementation Phase of Systemic Change in Curriculum Development and Instruction* Activity Modification L Original vision reexamined with emphasis on need to lower immediate expectations HOriginal strategies and activities reexamined with emphasis on identifying barriers across and within systems. Learning L Areas in which staff need additional skills or training were identified with emphasis on making a new vision. HBarriers and facilitators to integration that were faculty related were identified. Use Outcome Resources were allocated for additional training of staff and revision of instructional HInformal needs assessments were conducted to determine and prioritize reallocation of resources, policies, and procedures. L Modified vision to reflect staff’s ability based on training. HModified strategies, activities, and resources to support original vision. L *L represents characteristics of schools lacking evidence of systemic change or demonstrating only basic change. H represents additional characteristics seen only in schools demonstrating advanced and comprehensive evidence of systemic change