CHAPTER 5 Accountability Assessing Needs Determining Outcomes and

CHAPTER 5: Accountability: Assessing Needs, Determining Outcomes, and Evaluating Programs Transforming The School Counseling Profession Fourth Edition Bradley T. Erford

Accountability involves responsibility for professional actions. In the context of a school counseling program, this may involve: 1) Identifying and collaborating with stakeholder groups 2) Collecting data and assessing needs of students, staff, and community 3) Setting goals and establishing objectives based on data 4) Implementing effective interventions 5) Measuring the outcomes of interventions 6) Using results for program improvement 7) Sharing results with major stakeholders Numerous contemporary factors have called for greater accountability from professional school counselors and an increased focus on assessment. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -2

5 Questions to Address Accountability in the School Counseling Program 1. Is a comprehensive, standards-based program in place? (program 2. What are the needs of the school’s student population when compared to these standards? (needs assessment) 3. What services were implemented to address the identified needs and standards? (service assessment) 4. What was the result of the implemented services? (results or 5. How well is the professional school counselor performing? evaluation or audit) outcomes studies) (performance evaluation/appraisal) Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -3

Advantages & Disadvantages of Accountability Studies ADVANTAGES DISADVANTAGES Data is better than perception when guiding decision-making. Outcome measures take some training and skill to develop. Accountability helps demonstrate necessity, efficiency, and effectiveness of program. Quality outcome research and evaluation takes time and resources. Studies can help identify professional development needs. Many do not understand the nature and purpose of accountability because of misperceptions or previous “bad” experiences. Professional school counselors can network to share program results. Data is sometimes “over interpreted” or given undue meaning. Conducting accountability studies is a professional responsibility and demonstrates commitment to improvement. Comprehensive evaluations are seldom conducted, leaving the “big picture” incomplete. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -4

Accountability: Evaluating Programs and Assessing Outcomes It is a professional and ethical responsibility for professional school counselors to evaluate their program to ensure that the services provided to students, school personnel, and parents are of high quality and effective in meeting intended needs. Without accountability data to back up service provision, school counseling services could be among the first “nonessential services” to go during budget cutbacks. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -5

Needs Assessment Purposes of needs assessment: • Helps professional school counselors understand the needs of various subpopulations of a school community. • Helps establish the priorities that guide the construction of a comprehensive developmental school counseling program. • Assesses not only what currently is, but what should be. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -6

Data Driven Needs Assessment Data-driven decision-making deals with real needs and impact, and begins with an analysis of school-based performance data. Aggregated data: results are lumped together to show total grade level or school-wide (average) results. • Helpful in understanding how the average students perform in a given class, grade, or school. • Tells very little about the diversity of learner performance or needs. • Tells nothing about how various subgroups or subpopulations performed. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -7

Data Driven Needs Assessment (cont. ) Disaggregated data: results are broken down by subpopulations so that performance differences between and among groups can be analyzed. • Analysis usually involves intergroup differences (male vs. female, race, ethnicity) • Helps to provide hard evidence of gaps in student performance, rather than relying on perceptions. • Provides direction for the types of strategies and interventions needed to close these achievement gaps. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -8

Interpreting Norm-referenced Scores: The Basics Percentile rank: • Most easily understood by visualizing a line up of 100 individuals all with certain characteristics in common, with the 1 st student in the line as the lowest performing student and the 100 th student in the line the highest performing. • A student’s place indicates their relative standing compared to other students the same age across the country (national percentile rank). Quartile: • Interpretive statistic which divides the percentile rank distribution into 4 segments. – Q 1 = 0 -25, Q 2 = 26 -50, Q 3 = 51 -75, Q 4 = 76 -100 Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -9

Interpreting Norm-referenced Scores: The Basics (cont. ) Stanine: • Short for standard nine • Divides a normal distribution into nine segments, the 1 st represents the lowest level of performance and the 9 th represents the highest level of performance • Unlike quartiles, stanines represent onehalf standard deviation units and may be comprised of varying percentages of the population Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -10

Perceptions-based Needs Assessments Professional school counselors are often interested in what teachers, parents, and students perceive as primary needs to be addressed in a developmental way. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -11

Frequency of Conducting a Needs Assessment It is best to follow a continuous cycle of assessing programmatic needs. Breaking down needs into areas of academic, career, and personalsocial development can guide the rotation of needs assessments. Assessing needs is part of a much bigger endeavor: implementing curricular changes to continuously improve the comprehensive developmental counseling program. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -12

Populations to be Assessed It is most practical and efficient to seek out those stakeholders who are informed and likely to respond, such as teachers, administrators, students, and parents. Although more difficult to obtain large responses, valuable information can be garnered from community organizations, local businesses, and the general public as well. When considering the return rate, a higher return rate leads to a lower sampling error, which leads to greater confidence in the accuracy of the results. The highest priority needs should be those agreed to by all or most populations assessed. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -13

Design Issues in an Efficient Needs Assessment Stone and Bradley (1994) recommended seven methods for determining needs: 1) Questionnaires and inventories 2) Analysis of records 3) Personal interviews 4) Counseling statistics 5) Classroom visits 6) Use of outside consultants 7) Systematic evaluation of the guidance program What is most important is that the needs assessment uses objective methods for data gathering and analysis. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -14

Steps to Form an Efficient Needs Assessment 1. Decide what you need to know. 2. Decide on the best approach to derive what you need to know. 3. Develop the needs assessment instrument or method. 4. Enlist the support of colleagues and a few individuals from the target groups to review and try out items for understanding. 5. Implement the final version on the target groups. 6. Tabulate, analyze, and interpret the results. 7. Translate the results into programmatic goals and objectives. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -15

Design Issues in an Efficient Needs Assessment (cont. ) It is important to take the population into consideration when designing a scale: • For teachers—should ask for grade level, number of students in the class, etc. • For parents—should ask for the names of the parent’s children in case responses require contact by the counselor. • For students—should ask for student’s grade and homeroom teacher’s name. Questions or response stems should be short, to the point, and easily understood. Reading level of the items should be appropriate for the target audience. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -16

Response Formats If the purpose is to determine the importance or frequency of a potential problem, it is generally best to use a multipoint scale with 3 -5 choices such as: 1) Rarely 2) Sometimes 3) Frequently 4) Most of the time 5) Almost always It is important to provide a clear description of each choice. Be wary of response choices such as: • “Never” and “always, ” as they are rarely ever the case. • “Yes” and “no”, as they force an all or nothing response. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -17

Design Issues in an Efficient Needs Assessment (cont. ) It is important to include a frequency count in a needs assessment in order for professional school counselors to not only assess the importance of an issue, but also determine how many, and which students are in need of a specific service. Computing the results of a needs assessment is probably best accomplished by assigning a number value to each response category and averaging all responses for a given item. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -18

Converting Needs to Program Goals and Objectives If the needs assessment was designed correctly, translating the results into goals and learning objectives is relatively easy. Step 1: Prioritize the needs in order of importance and their relationship to existing components of the program. Step 2: Match up or translate the needs into goals included in the national standards. Step 3: Operationalize the goals through development of learning objectives. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -19

Program (Process) Assessment Program Assessment: a systematic examination of a program’s content. Program evaluation asks 2 questions: • Is there written program documentation? • Is the written program being appropriately implemented? Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -20

Program Assessment (cont. ) A program audit should be conducted at the end of the year and reports should address: Program strengths • Areas in need of improvement • Long and short-term goals • ASCA suggested that program criteria be evaluated on the following response choices: • • • None (not in place) In Progress (begun, but not completed) Completed (but perhaps not implemented) Implemented (fully implemented) Not Applicable (when the criteria do not apply) Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -21

Service Assessment Service assessments are sometimes requested by guidance supervisors and demanded by superintendents and school boards to document how counselors are spending their time. Two types are commonly used: • Event-topic counts – Involve documenting each time an individual is contacted or provided with a counseling service and the nature of the topic addressed. • Time logs – Involve documenting and categorizing activities for every minute of the workday. Unfortunately, service assessments give no information about the quality or effectiveness of counselor interventions. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -22

Results or Outcomes Evaluation Answers the question, “How are students different as a result of the program? ” (Johnson & Johnson, 2003) Assessment is not a discrete component, but rather an integrated part of a continuous process for program improvement. The evaluation cycle: • Questions Evidence Interpretation Conclusions Change Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -23

Results or Outcomes Evaluation (cont. ) Assessment of school counseling programs IS: • • a way to answer important program-related questions. the responsibility of an accountable professional school counselor. a cooperative endeavor with other school counseling program advisory committee (SCPAC) members and stakeholders. Assessment of school counseling programs IS NOT: • the evaluation of an individual professional school counselor. • an assault on the professional school counselor’s freedom. • a mandate for standardized tests or curricula. • all figured out. ongoing and evolving, a means to a better end - better education for all students. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -24

Important Assessment Terms Evaluation: the measurement of worth; indicates that a judgment will be made regarding the effectiveness of a program. Evidence: any data that will help make judgments or decisions; can be quantitative or qualitative. Formative evaluation: evaluative feedback that occurs during the implementation of a program; allows corrective action to occur if a program is shown to be off course. Summative evaluation: feedback collected at a specific endpoint in an evaluation process. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -25

Important Assessment Terms (cont. ) Stakeholder: anyone involved in or potentially benefiting from the school counseling program. Baseline: any data gathered to establish a starting point. Inputs: any resources that go into a program. Pretest: a measure administered before a program is implemented. Posttest: a measure administered after the program or intervention has been completed. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -26

Sources of Evidence People Products Students Portfolios Teachers Performances Staff Administration Use of rating from external examiners Observations Local tests Parents Purchased tests Employers Student self-assessments Graduates Community resource agents Surveys Interviews Focus groups Student work Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -27

Practical Program Evaluation Considerations To be of practical value, assessment must be connected to real program concerns as well as the core values of the school or program. Suggestions: • Avoid overwhelming the data collectors. • Focus on only one or several important questions at a time. • Always select measures that will yield reliable and valid scores for the purposes under study. • Involve the relevant stakeholders. • Use a variety of approaches. • Do not reinvent the wheel—use what you are already doing to generate useful data. • Don’t be afraid to call upon outside experts to consult on the development and evaluation of a program. • Start small and build upon what is found to work. • Remember: different can still be effective. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -28

Aggregated Outcomes Aggregation can be a valuable tool in evaluating how well school counseling programs meet higher level standards, such as the ASCA National Standards. Objective 1 Outcome 1 Objective 2 Objective 3 Outcome 2 Standard 1 Curriculum begins with a statement of standards which are further described through a series of outcomes, which are broken down into measurable objectives. Objective 4 Objective 5 Outcome 3 Objective 6 Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -29

Designing Outcome Studies In most instances, professional school counselors should measure outcomes or results by designing a research-type study. Answering the following questions can help the professional school counselor determine which research design to use: • Has the treatment already been implemented? • Can I randomly assign participants to treatment conditions? • Can I conduct (one or several) pretests, posttests, or both? True experimental designs involve randomization of participants and allow causative conclusions to be reached. Professional school counselors need to select or design an outcome measure with sufficient sensitivity and direct applicability to the issue being studied in order to receive significant results. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -30

Common Designs Used for Outcomes Research Pretest-posttest single group design Case study Static-group comparison Two-sample pretest-posttest design Nonequivalent control group design Time series design Randomized pretest-posttest control group design Randomized post-test only control group design Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -31

Principles to Consider When Developing an Outcome Measure from Scratch Use simple language. Ask only about things that the respondent can be expected to know. Make the question specific. Define terms that are in any way unclear. Avoid yes-no questions. Avoid double negatives. Don’t ask double-barreled questions. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved Use wording that has been adopted in the field. Include enough information to jog people’s memories. Look for second-hand opinions only when first hand information is unavailable. Be sensitive to cultural differences. Learn how to deal with difficult respondent groups. 1 -32

Common Data Collection Methods Interviews Observations Written questionnaires, surveys and rating scales Program records and schedules Standardized and educator-made tests Academic performance indicators Products and portfolios Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -33

Action Research Action research allows professional school counselors to focus on changing social, ecological, or client conditions in particular situations by creating a study and intervention to explore and solve a particular problem, usually in the client’s environment. Involves the analysis of data to improve practice and solve practical problems. Advantages to action research: It requires minimal training, helps develop effective, practice-based solutions for practical problems, and creates a collaborative atmosphere where professionals work together to address and improve conditions affecting students. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -34

Action Research (cont. ) Steps to developing an action research plan: • Identify and clarify the research problem • Gather data (should be gathered using a specific or particular group of individuals) • Analyze and interpret data • Create an action plan (should support and validate gathered data) • Evaluate and reflect (critically and periodically evaluate the plan’s effectiveness) Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -35

Reporting the Results It is important that professional school counselors involve the school counseling program advisory committee (SCPAC) in every step of the report writing process. A comprehensive report may be helpful for analysis by the SCPAC, however a 1 -2 page executive summary should also be prepared for administrators, system administrators, and the school community. After results have been reviewed by the SCPAC, professional school counselors, and administration, it is important to release the results of the program outcomes to relevant stakeholder groups at regular intervals. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -36

Reporting the Results (cont. ) Possible ways to disseminate the school counseling program results: • Written report • Verbal presentation • Multimedia presentation • Journal articles • Web page • Videotape • Posters • E-mail • Text messaging • Newspaper article Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -37

Performance Appraisal The effectiveness of a school counseling program often relies on the competence and efficiency of its implementers. The services provided by, and responsibilities of, the professional school counselor are the most complex of any school employee. It is best practice to have counselor supervisors coordinate in the appraisal of counselor performance. Criteria for any evaluation always should be available well in advance. Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -38

Performance Appraisal (cont. ) Most professional school counselor performance appraisals are comprised of a rating system that aggregates or averages responses across a variety of categories or work-related responsibilities. The primary focus of the appraisal system should be to enhance the quality and competence of all those evaluated. It is important that professional school counselors be held to performance standards such as those provided by the ASCA National Model (2005). Bradley T. Erford, Transforming The School Counseling Profession, Fourth Edition. © 2015, 2011, 2007, 2003 by Pearson Education, Inc. All Rights Reserved 1 -39