Experiences and Lessons Learned Using Publishers Online Assessment
Experiences and Lessons Learned Using Publishers’ Online Assessment Systems for OSEP Reporting Presenters: Barbara Jackson – Munroe-Meyer Institute, UNMC Jan. Thelen, Nebraska Dept. of Education Ruth Littlefield, New Hampshire Dept. of Education August 27 th, 2007
State Overview 2
What is Results Matter in Nebraska? n A statewide child and family outcomes system n Designed to improve programs and supports n For all young children birth to five, including Part C and Part B 619 n Served by school districts, the Early Development Network, and their partners n Results Matter applies to all school-based early childhood programs in Nebraska 3 Results Matter in NE
Results Matter in Nebraska Purpose… n improve experiences, learning, development and lives of young children (birth to age five) and their families n inform program practices n demonstrate program effectiveness n guide the development of local and state policies and procedures n provide data to demonstrate results 4
Results Matter in Nebraska calls for curriculum and child outcomes assessment that. . . n is based on ongoing observation of children engaged in real activities, with people they know, in natural settings n reflects evidence-based practices n engages families & primary care providers as active participants n integrates information gathered across settings n is individualized to address each child’s unique ways of learning n informs decisions about day-to-day learning opportunities for children n reflects that development and learning are rooted in culture and supported by the family 5 Results Matter in NE
Implementation Process for Results Matter in Nebraska n NECTAC state planning framework with state management team in January 2005 State & Local Infrastructure n Personnel Development n Implementation/Practices n Child/Family/Program Outcomes n n Engage State Level Child Measurement Task Force on implementation process n Implemented six pilot sites in 2006 n Statewide implementation in January 2007 6 Results Matter in NE
Results Matter in Nebraska Three child measurement tools…. n The High/Scope Child Observation Record (COR) n The Creative Curriculum Developmental Continuum n Assessment, Evaluation and Programming System (AEPS) Use of publishers’ online systems for supporting documentation of required assessments 7 Results Matter in NE
NH Preschool Outcome Measurement System n A statewide system to measure outcomes for preschoolers with disabilities n Designed to improve programs & supports n For all preschool children with disabilities with an IEP (effective November 1, 2006) n May be used more broadly, at local discretion 8
NH Preschool Outcome Measurement System n GSEG Cornerstone grant for Part C & 619 Engage stakeholders n Align Child Outcomes (Birth through 5) with Early Learning Guidelines n Consider how to address family outcomes n Federal reporting AND Improving outcomes n Focus on progress, not comparison n 9
NH Preschool Outcome Measurement System n Two Lead agencies n Part C: 10 Area Agencies with 14 programs n Part B 619: 177 School districts with multiple programs n Shared goal: improve results for children with disabilities and their families n Part C: COSF n 619: Publishers web-based data systems n Behind the scenes conversion to OSEP reporting requirements 10
NH Preschool Outcome Measurement System Three child measurement tools… n n Assessment, Evaluation, and Programming (AEPS®): http: //www. brookespublishing. com/store/books/brickeraeps/osep. htm Brigance® Inventory of Early Development II (IED-II): http: //www. curriculumassociates. com The Creative Curriculum® Developmental Continuum Assessment for Ages 3 -5: http: //www. creativecurriculum. net/ Allowed for flexibility Assessor preference n Best fit for the child n 11
State Administration 12
NDE Administrative Structure n 254 School Districts & 19 Regional Service Units n State License Subscriptions n School Districts n Head Start Programs n NDE memos n Authorization forms completed as part of enrollment process n NSSRS: NDE student identification number 13 Results Matter in NE
NDE Online Structure 14 Results Matter in NE
NDE Contracts and Costs n Jan 2006 NDE emails surveys to districts to collect: n Information on # of children in school district early childhood programs n Selection of tool(s) n Fall 2006 developed contracts with three Publishers n Brookes Publishing – AEPSi. com $11. 00 n Pearson Learning – Highscope. net $11. 00 n Teaching Strategies – Creativecurriulum. net $10. 95 n Fall 2007 Head Start Agencies contract with NDE to be part of the license 15 Results Matter in NE
NH Administrative Structure n 177 School districts in 80+ School Administrative Units n NHDOE memos n Statewide Preschool Technical Assistance Network (PTAN) Professional development, TA and networking n Immediate communication vehicle n 16
NH Administrative Structure n NHSEIS n (NH Special Education Information System) n Student Identification numbers n Limited authority to view personal data n Limited resources to fund the system n High level of commitment to quality and simplicity for districts 17
NH Contracts n Local Districts sign on with publishers n No State-Level Umbrella agreements n Districts negotiate directly with publishers n NHDOE: online access to reports for OSEP n district and state level (# & %) n run 3 reports and compile as needed n Districts access to all assessment data n Inform instruction n Demonstrate effective practices n Promote discussion with families and other early childhood partners 18
NH Cost n Survey Monkey: estimate number of children and users for each tool (for pricing & training) n State provided estimated children and or users to publishers n Paid for by local districts n NHDOE worked with publishers to establish best rate (not as good as umbrella agreement) 19
NH Cost n AEPS n One-time state fee n Establish data reporting system & Create state account: $1750 n Special rate to Beta Testers: $9/child n Rate for other NH users: $11/child n Complete AEPS®, Second Edition (Administration Guide, Test for Birth to Three Years and Three to Six Years, Curriculum for Birth to Three Years, Curriculum for Three to Six Years) $239. 00 with 10% discount when subscribe to AEPSi 20
NH Cost n Brigance IED II n One Inventory per user $159. 00 each n One Developmental Record Book per student $35. 00/10 -pack or $329. 00/100 -pack n Web-based IED-II Management System annual subscription $124. 95 each user first year ($99 for subsequent years) n State account charged at same rate as local user ($124. 95) 21
NH Cost n Creative Curriculum n $11. 75 per child n No state fee n No additional cost for tools/materials (unless want hard copies) 22
NDE Customizing Needs n Funding Source-allows for differential children Special Education Funding Early Intervention Part C (Birth-3) Early Childhood Sp Ed Part B (3 -5) Not Applicable State Funded EC Funding Source EC Grant Program Even Start Migrant Program Title I Not Applicable Other Funding Source Head Start Early Head Start Not Applicable n Child Newly Verified (IFSP/IEP) after 8. 1. 06 n Added 10 Character ID # Yes /No 23 Results Matter in NE
NH Customizing Needs n NH Head Start Collaboration office is working with NHDOE to better coordinate outcome measurement across systems n Need to consider Work Sampling (used by some NH Head Start) n Work Sampling: user enters assessment information n the COSF data point is generated n the user can adjust up or down n 24
NH Customizing Needs n Each Publisher offers different benefits and challenges n Each has a representative dedicated to NH n Areas for negotiation: Tracking students who move within the state n Online and phone TA for users n Ease of federal reporting n Always ask the next question! 25
Components for Implementation 26
NH Training/Professional Development n Assessment Fair n May 2006 n Publishers paid us: they provided demonstrations and resource n PTAN n local networking n phone and online TA n Coordinated series of publisher trainings n Preschool Leadership Conference (Sept 07) 27
NH Supports and Resources n Statewide Preschool TA Network (PTAN) n http: //ptan. seresc. net/ n Preschool Outcome Measurement Toolkit n NHDOE Special Education Memos n Links to ECO n Developing new FAQ 28
NDE Training/Professional Development n Statewide Training Series on Assessment – Rolled out January 2006 n Publishers provided first training (January) n State trainers provided at three regional sites (Spring) n Results Matter State Conference on Measurement – Spring 2007 n Regional Training Series on Early Childhood Curriculum – Planned for Fall 2008 n Developed collaboratively and supported through the Early Childhood Training Center 29 Results Matter in NE
NDE Supports and Resources n http: //www. nde. state. ne. us/ECH. html Click on Results Matter n NDE Administrative Guidelines n NDE Assessment Fidelity Process 30 Results Matter in NE
Why implement a fidelity process? n Maximize the credibility of observational data collected for Results Matter n Reliable and valid data maximizes quality of information n Consistency among providers in scoring observations (intra- and inter-reliability) n Documentation to support scoring (validity) n Quality information forms the basis for program planning 31 Results Matter in NE
Who does the plan affect? n All providers responsible for administering and scoring observational assessments for Results Matter n Early Childhood Practitioners n Early Childhood Special Education Practitioners n Related Services Providers 32 Results Matter in NE
Levels of Expertise Fidelity process is based on providers’ experience n Tier 1: Providers new to the assessment n Tier 2: Providers experienced with assessment n Tier 3: Experienced providers who can mentor others 33 Results Matter in NE
Five Components of the Fidelity Process 1. 2. 3. 4. 5. All providers must meet reliability standards by April 1, 2008 New provider orientation Continued group monitoring Individual monitoring Ongoing mentoring of providers 34 Results Matter in NE
Sample Plan & District Level Strategies Tier 1: Process for Providers New to the Assessment Tool Check all that are part of your plan and additional activities as needed. Approximate # of providers at this level: ____ Score state reliability tapes and achieve reliability standard Shadow experienced provider, documenting observations and ratings, checking for reliability Supervisor reviews samples of supporting documentation 35 Results Matter in NE
Reports Available and How They Are Used 36
AEPSi n Near entry only reports now available n OSEP near entry – near exit report available as of Sept 2007 n State and district level reports n Districts access online information about their children and programs 37
AEPSi – Near Entry Report 38
Creative Curriculum OSEP Reports n Child Progress Reports n Development of 9 point scale to assess at-risk children n Creative. Curriculum. net + Special Needs Accountability Pack (SNAPTM) and OSEPreports. net n District and State level reports n Local school districts: all information on their children/programs n 39
Creative Curriculum 40
High/Scope COR n Child Outcome Reports n OSEP report – Data Downloaded to High/Scope Foundation for analysis n Report is in progress – available Sept 2007 41 Results Matter in NE
NH Brigance Inventory of Early Development 42
Publisher/State Challenges n Different stages of online development n Different levels of experience n Creating interface with state data systems and processes n Addressing children with significant delays n Responding to OSEP timelines n Emerging OSEP guidance 43
Discussion 44
- Slides: 44