Response to Intervention Monitoring Student Progress at the

Response to Intervention Monitoring Student Progress at the Secondary Level Jim Wright www. interventioncentral. org

Response to Intervention “Few agree on an appropriate curriculum for secondary students…; thus it is difficult to determine in what areas student [academic] progress should be measured. ” -- Espin & Tindal (1998) Source: Espin, C. A. , & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed. ) Advanced applications of curriculum-based measurement. New York: Guilford Press. www. interventioncentral. org 2

Response to Intervention RTI Literacy: Assessment & Progress-Monitoring • • • To measure student ‘response to instruction/intervention’ effectively, the RTI model measures students’ academic performance and progress on schedules matched to each student’s risk profile and intervention Tier membership. Benchmarking/Universal Screening. All children in a grade level are assessed at least 3 times per year on a common collection of academic assessments. Strategic Monitoring. Students placed in Tier 2 (supplemental) reading groups are assessed 1 -2 times per month to gauge their progress with this intervention. Intensive Monitoring. Students who participate in an intensive, individualized Tier 3 intervention are assessed at least once per week. Source: Burns, M. K. , & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. www. interventioncentral. org 3

Response to Intervention Measuring General vs. Specific Academic Outcomes • General Outcome Measures: Track the student’s increasing proficiency on general curriculum goals such as reading fluency. An example is CBM-Oral Reading Fluency (Hintz et al. , 2006). • Specific Sub-Skill Mastery Measures: Track short-term student academic progress with clear criteria for mastery. An example is CBAMath Computation Fluency (Burns & Gibbons, 2008). Sources: Burns, M. K. , & Gibbons, K. A. (2008). Implementing response-to-intervention in elementary and secondary schools: Procedures to assure scientific-based practices. New York: Routledge. Hintz, J. M. , Christ, T. J. , & Methe, S. A. (2006). Curriculum-based assessment. Psychology in the Schools, 43, 45 -56. www. interventioncentral. org 4

Response to Intervention Local Norms: Screening All Students (Stewart & Silberglit, 2008) Local norm data in basic academic skills are collected at least 3 times per year (fall, winter, spring). • Schools should consider using ‘curriculum-linked’ measures such as Curriculum-Based Measurement that will show generalized student growth in response to learning. • If possible, schools should consider avoiding ‘curriculum-locked’ measures that are tied to a single commercial instructional program. Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 225 -242). Bethesda, MD: National Association of School Psychologists. www. interventioncentral. org 5

Response to Intervention Local Norms: Using a Wide Variety of Data (Stewart & Silberglit, 2008) Local norms can be compiled using: • Fluency measures such as Curriculum-Based Measurement. • Existing data, such as office disciplinary referrals. • Computer-delivered assessments, e. g. , Measures of Academic Progress (MAP) from www. nwea. org Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 225 -242). Bethesda, MD: National Association of School Psychologists. www. interventioncentral. org 6

Response to Intervention Measures of Academic Progress (MAP) www. nwea. org www. interventioncentral. org 7

Response to Intervention Applications of Local Norm Data (Stewart & Silberglit, 2008) Local norm data can be used to: • Evaluate and improve the current core instructional program. • Allocate resources to classrooms, grades, and buildings where student academic needs are greatest. • Guide the creation of targeted Tier 2 (supplemental intervention) groups • Set academic goals for improvement for students on Tier 2 and Tier 3 interventions. • Move students across levels of intervention, based on performance relative to that of Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. peers (local norms). Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 225 -242). Bethesda, MD: National Association of School Psychologists. www. interventioncentral. org 8

Response to Intervention Apply the ‘ 80 -15 -5’ Rule to Determine if the Focus of the Intervention Should Be the Core Curriculum, Subgroups of Underperforming Learners, or Individual Struggling Students (T. Christ, 2008) – – – If less than 80% of students are successfully meeting academic or behavioral goals, the intervention focus is on the core curriculum and general student population. If no more than 15% of students are not successful in meeting academic or behavioral goals, the intervention focus is on small-group ‘treatments’ or interventions. If no more than 5% of students are not successful in meeting academic or behavioral goals, the intervention focus is on the individual student. Source: Christ, T. (2008). Best practices in problem analysis. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 159 -176). www. interventioncentral. org 9

Response to Intervention Local Norms: Supplement With Additional Academic Testing as Needed (Stewart & Silberglit, 2008) “At the individual student level, local norm data are just the first step toward determining why a student may be experiencing academic difficulty. Because local norms are collected on brief indicators of core academic skills, other sources of information and additional testing using the local norm measures or other tests are needed to validate the problem and determine why the student is having difficulty. … Percentage correct and rate information provide clues regarding automaticity and accuracy of skills. Error types, error patterns, and qualitative data provide clues about how a student approached the task. Patterns of strengths and weaknesses on subtests of an assessment can provide information about the concepts in which a student or group of students may need greater instructional support, provided these Source: subtests Stewart, L. H. & Silberglit, B. (2008). and Best practices in developing academic local norms. In A. are equated reliable for these Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 225 -242). Bethesda, MD: p. Psychologists. 237 www. interventioncentral. org Nationalpurposes. ” Association of School 10

Response to Intervention How Does a Secondary School Determine a Student’s Math Competencies? “Tests [to assess secondary students’ math knowledge] should be used or if necessary developed that measure students’ procedural fluency as well as their conceptual understanding. Items should range in difficulty from simple applications of the algorithm to more complex. A variety of problem types can be used across assessments to tap students’ conceptual knowledge. ” p. 469 Source: Ketterlin-Geller, L. R. , Baker, S. K. , & Chard, D. J. (2008). Best practices in mathematics instruction and assessment in secondary settings. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 465 -475). www. interventioncentral. org 11

Response to Intervention Making Use of Existing (‘Extant’) Data www. interventioncentral. org

Response to Intervention Extant (Existing) Data (Chafouleas et al. , 2007) • Definition: Information that is collected by schools as a matter of course. • Extant data comes in two forms: – Performance summaries (e. g. , class grades, teacher summary comments on report cards, state test scores). – Student work products (e. g. , research papers, math homework, Power. Point presentation). Source: Chafouleas, S. , Riley-Tillman, T. C. , & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www. interventioncentral. org 13

Response to Intervention Advantages of Using Extant Data (Chafouleas et al. , 2007) • Information is already existing and easy to access. • Students are less likely to show ‘reactive’ effects when data is collected, as the information collected is part of the normal routine of schools. • Extant data is ‘relevant’ to school data consumers (such as classroom teachers, administrators, and members of problemsolving teams). Source: Chafouleas, S. , Riley-Tillman, T. C. , & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www. interventioncentral. org 16

Response to Intervention Drawbacks of Using Extant Data (Chafouleas et al. , 2007) • Time is required to collate and summarize the data (e. g. , summarizing a week’s worth of disciplinary office referrals). • The data may be limited and not reveal the full dimension of the student’s presenting problem(s). • There is no guarantee that school staff are consistent and accurate in how they collect the data (e. g. , grading policies can vary across classrooms; instructors may have differing expectations regarding what types of assignments are given a formal grade; standards may fluctuate across teachers for filling out disciplinary referrals). • Little research has been done on the ‘psychometric Source: Chafouleas, S. , Riley-Tillman, T. C. , & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and of instruction. Newdata York: Guilford Press. adequacy’ extant sources. www. interventioncentral. org 17

Response to Intervention Universal Screening at Secondary Schools: Using Existing Data Proactively to Flag ‘Signs of Disengagement’ “Across interventions…, a key component to promoting school completion is the systematic monitoring of all students for signs of disengagement, such as attendance and behavior problems, failing courses, off track in terms of credits earned toward graduation, problematic or few close relationships with peers and/or teachers, and then following up with those who are at risk. ” Source: Jimerson, S. R. , Reschly, A. L. , & Hess, R. S. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 1085 -1097). Bethesda, MD: National Association of School Psychologists. p. 1090 www. interventioncentral. org 18

Response to Intervention Mining Archival Data: What Are the ‘Early Warning Flags’ of Student Drop-Out? • • A sample of 13, 000 students in Philadelphia were tracked for 8 years. These early warning indicators were found to predict student drop-out in the sixthgrade year: Failure in English Failure in math Missing at least 20% of school days Receiving an ‘unsatisfactory’ behavior rating from at least one teacher Source: Balfanz, R. , Herzog, L. , Mac. Iver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist, 42, 223– 235. . www. interventioncentral. org 19

Response to Intervention What is the Predictive Power of These Early Warning Flags? Number of ‘Early Warning Flags’ Probability That Student Would in Student Record Graduate None 56% 1 36% 2 21% 3 13% 4 7% Source: Balfanz, R. , Herzog, L. , Mac. Iver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle grades schools: Early identification and effective interventions. Educational Psychologist, 42, 223– 235. . www. interventioncentral. org 20

Response to Intervention ‘Elbow Group’ Activity: What Extant/Archival Data Should Your RTI Team Review Regularly? • Discuss the essential extant/archival data that your RTI Team should review as ‘early warning indicators’ of students who are struggling. • What process should your school adopt to ensure that these data are reviewed regularly (e. g. , every five weeks) to guarantee timely identification of students who need www. interventioncentral. org 21

Response to Intervention Grades as a Classroom-Based ‘Pulse’ Measure of Academic Performance www. interventioncentral. org

Response to Intervention Grades & Other Teacher Performance Summary Data (Chafouleas et al. , 2007) • Teacher test and quiz grades can be useful as a supplemental method for monitoring the impact of student behavioral interventions. • Other data about student academic performance (e. g. , homework completion, homework grades, etc. ) can also be tracked and graphed to judge intervention effectiveness. Source: Chafouleas, S. , Riley-Tillman, T. C. , & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www. interventioncentral. org 23

Response to Intervention Marc Ripley 2 -Wk 9/23/07 4 -Wk 10/07/07 6 -Wk 10/21/07 (From Chafouleas et al. , 2007) 8 -Wk 11/03/07 10 -Wk 11/20/07 12 -Wk 12/05/07 Source: Chafouleas, S. , Riley-Tillman, T. C. , & Sugai, G. (2007). School-based behavioral assessment: Informing intervention and instruction. New York: Guilford Press. www. interventioncentral. org 24

Response to Intervention Online Grading Systems www. interventioncentral. org 25

Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement www. interventioncentral. org

Response to Intervention Curriculum-Based Measurement: Advantages as a Set of Tools to Monitor RTI/Academic Cases • Aligns with curriculum-goals and materials • Is reliable and valid (has ‘technical adequacy’) • Is criterion-referenced: sets specific performance levels for specific tasks • Uses standard procedures to prepare materials, administer, and score • Samples student performance to give objective, observable ‘lowinference’ information about student performance • Has decision rules to help educators to interpret student data and make appropriate instructional decisions • Is efficient to implement in schools (e. g. , training can be done quickly; the measures are brief and feasible for classrooms, etc. ) • Provides data that can be converted into visual displays for ease of communication Source: Hosp, M. K. , Hosp, J. L. , & Howell, K. W. (2007). The ABCs of CBM. New York: Guilford. www. interventioncentral. org 27

Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Reading: These 3 measures all proved ‘adequate predictors’ of student performance on reading content tasks: – Reading aloud (Oral Reading Fluency): Passages from content-area tests: 1 minute. – Maze task (every 7 th item replaced with multiple choice/answer plus 2 distracters): Passages from content-area texts: 2 minutes. Vocabulary matching: 10 vocabulary items Source: Espin, C. – A. , & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed. ) Advanced applications of curriculum-based measurement. New York: Guilford Press. and 12 www. interventioncentral. org definitions (including 2 distracters): 28

Response to Intervention Oral Reading Fluency Probe Sample http: //www. rti 2. org/rti 2/oral. Readings www. interventioncentral. org 29

Response to Intervention Maze Probe Sample http: //www. rti 2. org/rti 2/mazes www. interventioncentral. org 30

Response to Intervention Curriculum-Based Evaluation: Math Vocabulary Format Option 1 • 20 vocabulary terms appear alphabetically in the right column. Items are drawn randomly from a ‘vocabulary pool’ • Randomly arranged definitions appear in the left column. • The student writes the letter of the correct term next to each matching definition. • The student receives 1 point for each correct response. • Each probe lasts 5 minutes. • 2 -3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 397 -418). www. interventioncentral. org 31

Response to Intervention Curriculum-Based Evaluation: Math Vocabulary Format Option 2 • • • 20 randomly arranged vocabulary definitions appear in the right column. Items are drawn randomly from a ‘vocabulary pool’ The student writes the name of the correct term next to each matching definition. The student is given 0. 5 point for each correct term and another 0. 5 point if the term is spelled correctly. Each probe lasts 5 minutes. 2 -3 probes are given in a session. Source: Howell, K. W. (2008). Best practices in curriculum-based evaluation and advanced reading. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 397 -418). www. interventioncentral. org 32

Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Mathematics: Single-skill basic arithmetic combinations an ‘adequate measure of performance’ for low-achieving middle school students. Websites to create CBM math computation • www. interventioncentral. org probes: • www. superkids. com Source: Espin, C. A. , & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn (Ed. ) Advanced applications of curriculum-based measurement. New York: Guilford Press. www. interventioncentral. org 33

Response to Intervention Assessing Basic Academic Skills: Curriculum-Based Measurement Writing: CBM/ Word Sequence is a ‘valid indicator of general writing proficiency’. It evaluates units of writing and their relation to one another. Successive pairs of ‘writing units’ make up each word sequence. The mechanics and conventions of each word sequence must be correct for the student to receive credit for that sequence. CBM/ Word Sequence is Source: Espin, C. A. , & Tindal, G. (1998). Curriculum-based measurement for secondary students. In M. R. Shinn Advanced applications of curriculum-based measurement. New York: Guilford Press. the(Ed. ) most comprehensive CBM writing www. interventioncentral. org 34

Response to Intervention Breaking Down Complex Academic Goals into Simpler Sub-Tasks: Discrete Categorization www. interventioncentral. org

Response to Intervention Identifying and Measuring Complex Academic Problems at the Middle and High School Level • Students at the secondary level can present with a range of concerns that interfere with academic success. • One frequent challenge for these students is the need to reduce complex global academic goals into discrete sub-skills that can be individually measured and tracked over time. www. interventioncentral. org 36

Response to Intervention Discrete Categorization: A Strategy for Assessing Complex, Multi-Step Student Academic Tasks Definition of Discrete Categorization: ‘Listing a number of behaviors and checking off whether they were performed. ’ (Kazdin, 1989, p. 59). • Approach allows educators to define a larger ‘behavioral’ goal for a student and to break that goal down into sub-tasks. (Each sub-task should be defined in such a way that it can be scored as ‘successfully accomplished’ or ‘not accomplished’. ) • The constituent behaviors that make up the larger behavioral goal need not be directly related to each other. For example, ‘completed homework’ may include as sub-tasks ‘wrote down homework assignment correctly’ and ‘created a work plan before starting homework’ Source: Kazdin, A. E. (1989). Behavior modification in applied settings (4 ed. ). Pacific Gove, CA: th Brooks/Cole. . www. interventioncentral. org 37

Response to Intervention Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills Tina was struggling in her mathematics course because of poor study skills. The RTI Team and math teacher analyzed Tina’s math study skills and decided that, to study effectively, she needed to: q q Check her math notes daily for completeness. Review her math notes daily. Start her math homework in a structured school setting. Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. q Spend sufficient ‘seat time’ at home each day completing homework. q Regularly ask math questions of her teacher. www. interventioncentral. org 38

Response to Intervention Discrete Categorization Example: Math Study Skills General Academic Goal: Improve Tina’s Math Study Skills The RTI Team—with teacher and student input—created the following intervention plan. The student Tina will: q Approach the teacher at the end of class for a copy of class note. q Check her daily math notes for completeness against a set of teacher notes in 5 th period study hall. q Review her math notes in 5 th period study hall. q Start her math homework in 5 th period study hall. q Use a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. q Enter into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. q Stop by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that 39 www. interventioncentral. org

Response to Intervention Discrete Categorization Example: Math Study Skills Academic Goal: Improve Tina’s Math Study Skills General measures of the success of this intervention include (1) rate of homework completion and (2) quiz & test grades. To measure treatment fidelity (Tina’s follow-through with subtasks of the checklist), the following strategies are used : q Approached the teacher for copy of class notes. Teacher observation. q Checked her daily math notes for completeness; reviewed math notes, started math homework in 5 th period study hall. Student work products; random spot check by study hall supervisor. q Used a highlighter and ‘margin notes’ to mark questions or areas of confusion in her notes or on the daily assignment. Review of notes by teacher during T/Th drop-in period. q Entered into her ‘homework log’ the amount of time spent that evening doing homework and noted any questions or areas of confusion. Log reviewed by teacher during T/Th drop-in period. q Stopped by the math teacher’s classroom during help periods (T & Th only) to ask highlighted questions (or to verify that Tina understood that week’s instructional content). Teacher observation; student sign-in. www. interventioncentral. org 40

Response to Intervention RTI Teams: Recommendations for Data Collection www. interventioncentral. org

Response to Intervention RTI Teams: Recommendations for Data Collection • Collect a standard set of background information on each student referred to the RTI Teams should develop a standard package of background (archival) information to be collected prior to the initial problemsolving meeting. For each referred student, a Team might elect to gather attendance data, office disciplinary referrals for the current year, and the most recent state assessment results. www. interventioncentral. org 42

Response to Intervention RTI Teams: Recommendations for Data Collection • For each area of concern, select at least two progressmonitoring measures. RTI Teams can place greater confidence in their progressmonitoring data when they select at least two measures to track any area of student concern (Gresham, 1983)-ideally from at least two different sources (e. g. , Campbell & Fiske, 1959). With a minimum of two methods in place to monitor a student concern, each measure serves as a check on the other. If the results are in agreement, the Team has greater assurance that it can trust the data. If the measures do not agree with one another, however, the Team can investigate further to determine the reason(s) for the apparent discrepancy. www. interventioncentral. org 43

Response to Intervention RTI Teams: Recommendations for Data Collection • Monitor student progress frequently. Progress-monitoring data should reveal in weeks--not months-whether an intervention is working because no teacher wants to waste time implementing an intervention that is not successful. When progress monitoring is done frequently (e. g. , weekly), the data can be charted to reveal more quickly whether the student’s current intervention plan is effective. Curriculum-based measurement, Daily Behavior Report Cards, and classroom observations of student behavior are several assessment methods that can be carried out frequently. www. interventioncentral. org 44

Response to Intervention Monitoring Student Academic Behaviors: Daily Behavior Report Cards www. interventioncentral. org

Response to Intervention Daily Behavior Report Cards (DBRCs) Are… brief forms containing student behaviorrating items. The teacher typically rates the student daily (or even more frequently) on the DBRC. The results can be graphed to document student response to an intervention. www. interventioncentral. org 46

Response to Intervention Daily Behavior Report Cards Can Monitor… • • • Hyperactivity On-Task Behavior (Attention) Work Completion Organization Skills Compliance With Adult Requests Ability to Interact Appropriately With Peers www. interventioncentral. org 47

Response to Intervention Daily Behavior Report Card: Daily Version Jim Blalock Mrs. Williams www. interventioncentral. org May 5 Rm 108

Response to Intervention Daily Behavior Report Card: Weekly Version Jim Blalock Mrs. Williams Rm 108 05 05 07 05 06 07 40 0 www. interventioncentral. org 05 07 07 05 08 07 05 09 07 60 60 50

Response to Intervention Daily Behavior Report Card: Chart www. interventioncentral. org

Response to Intervention Student Case Scenario: Jim • • Jim is a 10 th-grade student who is failing his math course and in danger of failing English and science courses. Jim has been identified with ADHD. His instructional team meets with the RTI Team and list the following academic and behavioral concerns for Jim. Does not bring work materials to class Fails to write down homework assignments Sometimes does not turn in homework, even when completed Can be non-compliant with teacher requests at times. www. interventioncentral. org 51

Response to Intervention www. interventioncentral. org 52

Response to Intervention Closing Thoughts About Local Norms & RTI Assessment at the Secondary Level… www. interventioncentral. org

Response to Intervention Local Norms: Set a Realistic Timeline for ‘Phase-In’ (Stewart & Silberglit, 2008) “If local norms are not already being collected, it may be helpful to develop a 3 -5 year planned rollout of local norm data collection, reporting, and use in line with other professional development and assessment goals for the school. This phased-in process of developing local norms could start with certain grade levels and expand to others. ” p. 229 Source: Stewart, L. H. & Silberglit, B. (2008). Best practices in developing academic local norms. In A. Thomas & J. Grimes (Eds. ), Best practices in school psychology V (pp. 225 -242). Bethesda, MD: National Association of School Psychologists. www. interventioncentral. org 54

Response to Intervention Activity: How ‘RTI Ready’ is Your Middle or High School in Measuring Academic Skills? In your ‘elbow groups’: • Discuss the range of assessments that your school can access to measure the current performance and progress of students in basic academic skill areas. • How ‘RTI ready’ is your school in measuring these academic skills? www. interventioncentral. org 55

Response to Intervention End www. interventioncentral. org 56

Response to Intervention RTI Team Initial Meeting Form: Secondary Student Progress. Monitoring Page 15 www. interventioncentral. org 57

Response to Intervention RTI Team Initial Meeting Form: Secondary Student Progress. Monitoring Page 16 www. interventioncentral. org 58

Response to Intervention RTI Team Initial Meeting Form: Secondary Student Progress. Monitoring Page 17 www. interventioncentral. org 59

Response to Intervention www. interventioncentral. org 60

Response to Intervention www. interventioncentral. org 61