October 8 2019 Kimpton Hotel Monaco Baltimore Issues
October 8, 2019 Kimpton Hotel Monaco Baltimore
Issues around Using Academic Return on Investment for Informing and Improving Decisions Bo Yan, Jefferson County Public Schools Louisville, KY
A-ROI: What Is It? Return Investment 3 Program Effect Program Cost
A-ROI: What Is It For? • Reduce dimensionality • Compare costeffectiveness Source: Fiona M. Hollands, Michael J. Kieffer, Robert Shand, Yilin Pan, Henan Cheng & Henry M. Levin (2016). Cost. Effectiveness Analysis of Early Reading Programs: A Demonstration With Recommendations for Future Research, Journal of Research on Educational Effectiveness, 9: 1, 30 -53, DOI: 10. 1080/19345747. 2015. 1055639 4
A-ROI: What Is It For? • Reduce dimensionality • Compare costeffectiveness Source: Government Finance Officers Association (2016). Model A-ROI Presentation: An Example from Smarter. School. Spending. org 5
A-ROI: What Is It For? Return • Reduce dimensionality High Return Low Cost High Return High Cost Low Return Low Cost Low Return High Cost • Compare costeffectiveness Cost 6
Five Issues Assume the effectiveness data is obtained through rigorous methods More on this later. . . 7
Issue 1: Validity Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part I - Validity 8
Issue 2: Uncertainty • Uncertainty around individual A-ROI estimates • Uncertainty around comparing A-ROI estimates Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part II - Uncertainty 9
Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 10
Issue 3: Commensurability 11
Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation • Timing Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 12
Issue 3: Commensurability 13
Issue 3: Commensurability • Different outcomes • Multiple outcomes • Linear extrapolation • Timing • Context • Similar A-ROI results Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 14
Issue 3: Commensurability Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 15
Issue 3: Commensurability Five Issues around Using Academic Return on Investment (A-ROI) to Inform and Improve Decisions: Part III - Commensurability 16
Issue 4: Cost • Difference between budget, actual spending, and true cost • Opportunity cost • Ingredients method • Total cost vs. cost per pupil • Decision-relevant cost 17
Issue 5: Unit of Decision Making • Cost-effective at the program level vs. at the building level • Differentiated decisions • Implications 18
Group Discussion 19 Decision Makers Researchers • Does all this make sense to you? • Any important issues missing? • What to present for decision making? • How should these issues be addressed? • How should these issues be dealt with? • How to share A-ROI results with decision makers?
Sharing
IES Research Project Develop A-ROI Method Compare it with more rigorous methods 21
- Slides: 21