October 8 2019 Kimpton Hotel Monaco Baltimore Issues

October 8, 2019 Kimpton Hotel Monaco Baltimore

Issues around Using Academic Return on Investment for Informing and Improving Decisions Bo Yan, Jefferson County Public Schools Louisville, KY

A-ROI: What Is It? Return Investment 3 Program Effect Program Cost

A-ROI: What Is It For? • Reduce dimensionality • Compare costeffectiveness Source: Fiona M. Hollands, Michael J. Kieffer, Robert Shand, Yilin Pan, Henan Cheng & Henry M. Levin (2016). Cost. Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research, Journal of Research on Educational Effectiveness, 9: 1, 30 -53, DOI: 10. 1080/19345747. 2015. 1055639 4

A-ROI: What Is It For? • Reduce dimensionality • Compare costeffectiveness Source: Government Finance Officers Association (2016). Model A-ROI Presentation: An Example from Smarter. School. Spending. org 5

A-ROI: What Is It For? Return • Reduce dimensionality High Return Low Cost High Return High Cost Low Return Low Cost Low Return High Cost • Compare costeffectiveness Cost 6

Five Issues Assume the effectiveness data is obtained through rigorous methods More on this later. . . 7

Issue 1: Validity Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part I - Validity 8

Issue 2: Uncertainty • Uncertainty around individual A-ROI estimates • Uncertainty around comparing A-ROI estimates Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part II - Uncertainty 9

Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 10

Issue 3: Commensurability 11

Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation • Timing Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 12

Issue 3: Commensurability 13

Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation • Timing • Context • Similar A-ROI results Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 14

Issue 3: Commensurability Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 15

Issue 3: Commensurability Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 16

Issue 4: Cost • Difference between budget, actual spending, and true cost • Opportunity cost • Ingredients method • Total cost vs. cost per pupil • Decision-relevant cost 17

Issue 5: Unit of Decision Making • Cost-effective at the program level vs. at the building level • Differentiated decisions • Implications 18

Group Discussion 19 Decision Makers Researchers • Does all this make sense to you? • Any important issues missing? • What to present for decision making? • How should these issues be addressed? • How should these issues be dealt with? • How to share A-ROI results with decision makers?

Sharing

IES Research Project Develop A-ROI Method Compare it with more rigorous methods 21