Defining Learning KPIs: Telling Vanity Metrics from Real Ones

"LMS logins this term are up 40 percent over last year." This sentence, a regular in school reports, actually tells you nothing. More logins say nothing about whether learning happened. Numbers like this are called vanity metrics. They look impressive and report well, but they never change a decision. The first job in defining learning KPIs is weeding out the plausible-looking vanity metrics.

Three questions that expose a vanity metric

When a number comes up as a KPI candidate, run it through these three questions.

1. If this number gets worse, what do we change? If no answer comes to mind, it is decoration for a report, not a KPI. There is no reason to track a metric that never connects to an action.
2. Does the number alone tell you good from bad? A high login count can be bad news. It might mean students couldn't find their way and bounced in and out of the same page thirty times. Only a metric with a clear direction calls for action.
3. Is it causally linked to the learning goal? Long video watch time does not mean deeper understanding. All the time the video played to an empty chair is in that total too.

Keep measuring only what is easy to measure, and you forget how to measure what actually matters.

What real KPIs look like

Metrics like these, by contrast, call for a next action the moment they move.

• Concept mastery rate: The share of students who passed each core standard. Track it at the end of every unit and you can see which concept the class is stuck on.
• Success rate after a retry: The share of problems a student got wrong first and then solved correctly. It is the most honest indicator that learning actually happened. First-attempt accuracy reveals prior knowledge; retry success reveals the effect of the teaching.
• Time from request for help to resolution: How long it takes from a student posting a question to getting an answer. Anything over 24 hours means there is a hole in your support system.
• Share of self-directed learning: The proportion of learning activity students started on their own, with no instruction from a teacher. A rise in this share is the best signal you have that a learning culture is taking root.

With metrics like these, "what do we change" follows almost automatically when the number goes bad. If the success rate after a retry sits at 40 percent, for instance, it means students are guessing again without knowing why they were wrong, so the conclusion - rework the explanations and the way hints are given - arrives on its own.

One more thing: don't set many KPIs at the start. It is better to keep the core metrics the whole school tracks together to three or fewer. Past ten metrics, everyone stops looking at the dashboard, and all you have left are numbers nobody owns. Choose few and look deeply: that is the first principle of running KPIs.

Key takeaways

A good KPI is not a flashy number but a number that tells you what to do when it gets worse. Clear away vanity metrics like logins, page views, and watch time, and concentrate on a small set causally tied to your learning goals, such as mastery rate and retry success rate. Looking hard at three metrics always beats glancing at thirty.