Edtech Pilots: How to Start Small With One Class for One Semester

When a new learning tool turns up, "school-wide adoption this year" gets said easily in a principals' meeting. But spread a tool that has not been adequately tested across 1,000 people at once and the cost of failure is multiplied by 1,000 too. A pilot is a device for penning the risk inside one class and one semester. Experimenting small and learning big is the whole point. Schools that let their enthusiasm run ahead, started with the entire building, and then quietly set the tool down in the second year are not a rare sight in the field.

Three criteria for narrowing a pilot's scope

If what you are testing is vague, the pilot ends as nothing more than "we tried it." Settle the following three things clearly first.

• Who: Limit it to one or two volunteer teachers and one class of 25 to 30 students. Motivated teachers produce the first data. Data from a teacher conscripted against their will measures resistance, not the tool.
• How long: One semester (about 16 weeks) is about right. Too short and the learning effects become the problem; too long and sunk cost does. Four weeks measures only novelty, and a year makes you miss the moment to stop.
• The question: Write it as one testable sentence, along the lines of "does this tool raise the unit test average by 5 points?" or "does it cut lesson prep by two hours a week?" A vague question like "does the teaching get better?" can be argued as correct on any result at all, which makes it impossible to test.

Fix the measures and the exit conditions in advance

A pilot's real value lies in fixing the stopping criteria in advance. Fail to set them before you start and by the time it is wrapping up, the effort already spent makes an objective judgment hard.

1. Baseline measurement: For two weeks before the start, record scores, time, and satisfaction under the existing approach. A claim of impact with no baseline is no more than the impression that "it seems better."
2. A midpoint check (week 8): Look at attendance, assignment submission rates, and teacher fatigue. Catching a problem at this point is what saves the second half of the term.
3. Exit conditions: Build the door out first, as in "stop if teachers' weekly workload has actually gone up by three hours at week 8." Pulling out is not failure; it is part of the design.
4. A closing report: Decide expand, hold, or discard with a one-page result sheet. This report becomes the most valuable asset the next person in charge of adoption will have.

A good pilot tells you not "it worked" but "what to change next time." Even a pilot that exposed shortcomings has more than done its job if it spares you the trial and error of a school-wide rollout.

Common traps that shake a pilot

Even having started small, fall into the following traps and the data gets contaminated.

• The favoritism effect: A class the principal visits often is bound to perform better than usual. You have to allow for the observer effect and read the results conservatively.
• The star teacher trap: Test only with the school's most seasoned veteran and the result will not reproduce for an ordinary teacher.
• Selection bias: A class that volunteered may have had a good atmosphere to begin with, which makes it easy to overestimate the effect.

Key takeaways

A pilot is not an occasion for showing off a tool but an experiment in service of a decision. Simply holding to narrowing the who, the how long, and the question, and fixing the baseline and the exit conditions in advance cuts the trial and error of a school-wide rollout dramatically. Next semester, start with one class and one teacher. That single semester's data will protect the following three years of budget.