Edtech ROI: If Not Test Scores, What Should You Measure and How?

"Did scores go up after you brought in the AI tool?" is a natural question, but it comes with a trap. A single semester's change in scores has a teacher change, exam difficulty, and even seasonal factors mixed into it, so the effect of the tool alone is hard to pull out. The moment you try to convert ROI into one line of test scores, the measurement loses its credibility. Impact has to be looked at from several angles. Measure it badly and you will miss the tools that work while clinging to the ones that do not.

The value you miss by looking only at scores

The return on an investment sometimes shows up more strongly outside the scores. Measure only what is visible and you lose the invisible value wholesale.

• Teacher time: How many hours a week grading and materials prep have gone down. That time gets reinvested into feedback and conferences. Two hours saved a week is more than 30 hours over a semester.
• Engagement: Behavioral indicators such as assignment submission rates, how often students speak up, and absence rates. They are leading signals that move before scores do.
• Equity: How fast lower-achieving students are catching up. Even if the average holds still, a narrowed gap is a major result.
• The affective domain: A short survey on confidence and interest in learning. A recovery of interest shows up late, in next year's scores.

Four steps for designing your measurement

1. Secure a baseline: Record the indicators above for a month before adoption. A claim of impact without a baseline is only an impression.
2. Set a comparison group: Where possible, use a comparable class in the same grade as a comparison group. Only a comparison group separates "thanks to the tool" from "it was always like that."
3. Mix your measures: Gather quantitative indicators like scores together with qualitative material like student and teacher interviews. The numbers tell you what changed; the interviews tell you why.
4. Weigh it against cost: Divide the money and time you put in by the time saved and the indicators that improved. Convert the hours saved into an hourly rate and the figure comes out surprisingly large.

Numbers not going up and there being no effect are two different things. Start by suspecting what you failed to measure.

Not misreading your results

Data is honest, but interpretation often deceives us.

• Watch for the novelty effect: High engagement right after adoption may just be newness. You have to look at the numbers three months later to know what is real.
• The trap of averages: Behind a rising average, the lower group can be slipping further behind, so look at the distribution.
• Overtrusting a single indicator: One indicator improving does not mean the whole picture improved. Judge in bundles.

Carrying results through to decisions

The purpose of measurement is a decision, not a report. Even well-gathered data ends up as paper in a drawer if it does not lead to a decision.

• Review each term: Once a semester, pull the indicators together and decide whether to expand, hold, or scale back. Measurement without a decision leaves nothing but fatigue.
• Feed it back to the classroom: Return the results to teachers so everyone learns together what worked. That feedback is what creates next semester's motivation.
• Update the benchmark: Make one year's result the new baseline for the next, so improvement accumulates.

Key takeaways

Edtech ROI has to be viewed not as the single indicator of test scores but as a bundle spanning time, engagement, equity, and feelings about learning. Put a baseline and a comparison group in place first and mix the quantitative with the qualitative, and you will be left holding a persuasive case at the next budget review.