CostEffectiveness in EducationMysteries and Revelations Henry M Levin

Cost-Effectiveness in Education--Mysteries and Revelations Henry M. Levin AERA Meetings, New York April 15, 2018

Eclectic Career— Theme of Education and Democracy � Before 1968, Brookings Institution, Government Finance. � 1968 -75—Educational productivity and Equity. � 1975 -85—Workplace Democracy and Schools and Educational Finance. � 1985 -2000 Accelerated Schools. � 1968 -Pres. Cost-Effectiveness and Benefit. Cost Analyses & School Choice.

Educational Productivity and Equality 1968 -75. � Schools and Inequality, with others (MIT Press, 1971). � Community 1970). � Costs Control of Schools (Brookings to the Nation of Inadequate Education. A Report for the Select Committee on Equal Educational Opportunity, U. S. Senate. (Washington, DC, 1972).

Workplace Democracy and Education 1975 -85 � Worker Cooperatives in America, with R. Jackall (Univ. of California Press, 1984). � Schooling and Work in the Democratic State, with M. Carnoy (Stanford Univ. Press, 1985).

Accelerated Schools, 1985 -2000 � 1040 Schools in 41 states dedicated to providing gifted and talented education to all students, academic enrichment through research, arts, community and personal projects, building on strengths. � Resource Guide for Accelerated Schools, with W. Hopfenberg et al. (Jossey Bass, 1993). � Displaced by NCLB demands and “testprep”.

Cost-Effectiveness and Benefit. Cost Analysis in Education, 1968� “Cost-Effectiveness in Evaluation Research. ” M. Guttentag and E. Struening, (eds. ), Handbook of Evaluation Research, Vol. 2. (Sage Publications, 1975), Chapter 5. � Cost-Effectiveness: A Primer (Sage Publications 1983). � Cost-Effectiveness Analysis: Methods and Applications, � 2 nd Ed. with P. Mc. Ewan. (Sage Publications, 2001). Economic Evaluation in Education Cost-Effectiveness and Benefit-Cost Analysis, 3 rd Ed. , With Mc. Ewan, P. , Belfield, C. , Bowden, B. , & Shand, R. (Sage Publications, 2018).

What is Cost-Effectiveness? � Most evaluations of educational interventions focus on effectiveness? � The users, educational decision-makers, must also take costs into account. � Among alternatives with similar levels of effectiveness, wide differences in costs. � Cost-effectiveness analysis compares the costs of alternatives for achieving a given result.

Why Is It Important? � There might be large differences in costs among alternatives with similar effectiveness. � In studies of the What Works Clearinghouse effectiveness, differences varied by six-fold and more in costs for given results in dropout prevention and early literacy. � By choosing more cost-effective programs, resources are saved and can be used for new initiatives or to enhance other interventions.

Rising Resource Constraints � In 2015 more than half of states were spending less on K-12 education than in 2007. � Education as % of state-local budgets dropped from 26% in 1977 to 22% in 2015. � � In last 3 decades, state-local spending on prisons has risen at 3 times rate of Pre-K-12. � New federal tax law restricts state-local tax deductions to $10, 000. Pressure on taxpayer. � States facing serious underfunding of pensions and rising demands for aging demography.

CE is Rarely Done � Usually effectiveness results are reported with no attempt to estimate costs. � Even when done, it is usually done erroneously. � Borrowing from Lee Shulman on ATI Studies: “Effectiveness (aptitude) is measured with calipers but costs (treatments) are measured with a witching rod”.

Why Costs are not Measured. � Evaluators are focused on effectiveness. � Assumes costs are easily known or accessible from traditional school accounting systems (They are not. ) � Schools focus more on overall budgetary constraints than comparing costs of specific alternatives.

Cost-Effectiveness not understood � There are no demands for cost-effectiveness comparisons by school boards or state authorities. � The cost-effectiveness tool is not taught in evaluation courses or evaluation textbooks. � Cost-effectiveness is used in education as a rhetorical term used to “sell” a specific reform without providing evidence.

Clune Study (2002) � Clune evaluated the ERIC database and found more than 9, 000 documents under the identifier “costeffectiveness”. � � He selected 541 papers at random with this identifier. � � He found that more than 80 percent simply used the term to market or praise a favored intervention. � Less than 2 percent of papers attempted any costeffectiveness evaluation, even a flawed one.

Cost-Effectiveness Requirement � Requires ◦ ◦ ◦ ◦ similar metrics of effectiveness. Test Scores in same subject Graduation Rates Student Satisfaction Student Behavior Post-School Employment Civic Behavior Bullying Attendance

Specific Studies on CBCSE website (www. cbcse. org) � Computer-Assisted Instruction. � Peer tutoring, class size, length school day, . � Dropout prevention. � Early childhood literacy. � Socio emotional learning. � Lecture vs. small group instruction. � Educational technology. � Vouchers � Adolescent literacy.

Proper Cost Measurement � Ingredients Method. � Must recognize all ingredients required to deliver effectiveness results. � Each ingredient valued at market prices or “shadow price” as estimated by economic methods. � Uses economic concept of opportunity cost, the loss of value when assigned to best alternative use.

Ingredients Method: � 1 - Start with goal of intervention, theory of action, mechanism, and process. � � 2 - Specify ingredients required to produce effectiveness result that was found? � 3 - Set out details in categories: personnel, facilities, equipment and materials, other inputs, client requirements. � Attention to qualities, e. g. skills of personnel.

Qualitative Researchers � Role for a qualitative researcher to document the intervention process and implementation. � At present, in many cases must be done retrospectively by following up past effectiveness studies. � Retrospective studies use reports, periodic observations of extant programs when possible, and interviews with knowledgeable staff.

No Ingredient is Free � All ingredients have costs, no matter who provides them. � Even Resources from other government agencies, philanthropy, and volunteers. � Objective is to determine total value of ingredients that produced effectiveness results. � Transferred resources from other school programs are not free since other outcomes must be sacrificed to obtain them.

Cost Spread Sheet

Cost. Out � Refers to software prepared to assist in estimating costs using ingredients approach. � Licensed free to users. Presently more than 1, 500 licensees. � Details can be found at www. cbcse. org.

Effectiveness Studies �We seek experimental and quasiexperimental studies of effectiveness. �We seek validation of effectiveness through multiple trials or methods. �What �In Works Clearinghouse is a source. some cases, we have pursued our own effectiveness studies.

Earliest Study in Literature � 1970 -Journal of Human Resources Cost-Effectiveness of Teacher Selection. � Used Coleman data to compare a strategy of selecting teachers with higher verbal scores versus more experience. � Effectiveness was from production function on 6 th grade student achievement. � Cost was from earnings function to find the market returns to verbal score vs. experience.

CE Results Teacher Selection

Evaluating Nation at Risk Reforms � Nation at Risk, 1983, listed at least 30 different reforms that were “needed”. � Raising graduation requirements with more math, science, foreign language; longer school days, longer school years, more use of technology. � States responded.

Search for Evidence � Very few of the Nation at Risk reforms had evidence for common achievement measures. � We found some evidence on: �Class size—Gene Glass Meta Analysis. �Increase in length of school day—Time in Learning. �Computer-Assisted Instruction—Los Angeles Experiment with ETS. �Adult and Cross-Age Tutoring—Boise Experiment.

Annual Cost Per Student Per Subject and Effect Size Mathematics Computer Assisted Instruction ES � Increase Instructional Time � Reduce Class Size � ◦ ◦ � 35 -30 30 -25 25 -20 35 -29 Tutoring ◦ Adult Component ◦ Peer Component $ 119 (. 12) 61 (. 03) 45 63 94 201 (. 06) (. 07) (. 09) (. 22) 827 212 (. 67) (. 97)

Cost-Effectiveness of Four Interventions Cost-Effectiveness Ratio Average, of Mathematics and Reading Effect Sizes for Each $100 Cost Per Student Per Subject in 1980 prices Cross-age Tutoring: 0. 34 Peer Component 0. 22 0. 07 0. 15 . 11. 09. 08. 09 Combined Peer and Adult Program Adult Component Computer Assisted Instruction Reducing Class Size from: 35 to 30 30 to 25 25 to 20 35 to 20 Increasing Instructional Time (one hour per day)

Comments � Reducing Class Size and Lengthening School Day have low cost-effectiveness (about same as Tennessee Class size for large reduction of CS). � Adult Tutoring—High ES, but costly. � CAI—Modest Cost; Modest ES. � Peer Tutoring—High Cost because needs training and adult supervision, but high ES. � Cost-effectiveness of peer tutoring is double that of CAI.

Additional HS Grads Per 100; Program Cost Per Student; Cost Per Additional Graduate First Things First (FTF) Comprehensive school reform of: small learning communities with dedicated teachers; family advocates; and instructional improvement efforts. Class size reduction (CSR) 4 years of schooling (grades K-3) with class size reduced from 25 to 15. 16 $5, 500 $59, 100 11 $13, 10 0 $143, 600

Benefit-Cost Analysis � Measure of Return on Investment. � Difference between Benefits and Costs (rate of return, benefit-cost ratio, net present value). � Social Investment and Return. � Taxpayer Benefits: Additional Tax Revenues and Reduced Costs of Services such as Public Health, Public Assistance, and Criminal Justice.

Good Educational Investments � Result may be persuasive to funding authorities when benefits are large relative to costs. � Examples of Productive Investments that have influenced further funding: ◦ Head Start and Early Childhood. ◦ High School Graduation. ◦ College Graduation

ASAP—Associate Degrees � City University of New York increase Associate Degree Graduates. � 60 % higher costs, but doubled graduates from about 25% to 55% and reduced cost by $6, 500 graduate. Is cost-effective. � � Taxpayer benefits are more than 3 times the overall costs. � Results persuaded funding appropriations from 1, 000 students to 20, 000 students

Details of BC Study for ASAP and… � www. cbcse. org � H. Levin & E. Garcia (2018), “Accelerating Community College Graduation Rates: A Benefit–Cost Analysis” Journal of Higher Education, 89(1), pp. 1 -27. � Studies show strong benefits relative to cost for social and emotional learning and wraparound school services.

Cost-Effectiveness and Benefit-Cost Promote Educational Quality and Equity (1) Improved education with better resource allocation. (2) More resources for program expansion. (3) Results promote inclusion for those with least family resources. (4) Promote reductions in costs and student loan debt. (5) CB results favor increased participation and completion rates at all levels.

Thanks for Attending. I also thank my colleagues, Clive Belfield, Brooks Bowden, Rob Shand, and Patrick Mc. Ewan for their valued contributions.