One Prompt Can Shake Your Grades: Principles for Designing a Grading Prompt

If you have ever graded the same student response twice and gotten two different scores, the problem is most likely not the response but an ambiguous prompt. Three points the first time and two on a rerun means the criteria are shifting. A grading prompt is the constitution of your assessment. Write it vaguely and the results come out vague too. Before you switch to a pricier model, the right order is to sharpen the prompt first.

A shaky prompt versus a steady one

"Grade this response" is the worst possible instruction. You are leaving the machine to assign a score without knowing what full marks even means. A steady prompt spells out the following.

• What earns full marks: Enumerate the elements that have to be present for a full score.
• Partial credit rules: Set arithmetically how many points each element is worth.
• What not to deduct for: Make clear what is not being assessed, such as spelling, handwriting, or style of expression.
• Output format: Receive the score, the reasoning, and the location of the quoted evidence in a fixed structure.

Making the partial credit rules arithmetic is especially central to consistency. Phrasing like "partial credit if it is somewhat correct" gets read differently every time. Write it so it can be counted, as in "2 of the 3 core concepts present earns 2/3 credit."

A procedure for checking stability

1. Grade the same 5 responses three times each.
2. If a score on the same response shifts, make the criterion wording more specific. Whatever shifts is precisely what is ambiguous.
3. Pin borderline cases, the genuinely ambiguous responses, right into the prompt as examples.

A good grading prompt is finished by a few examples of "a response like this gets this score." Examples are stronger than rules.

Traps people fall into

• An overly long prompt gets its later instructions ignored. Put the core rules up front. The more important something is, the earlier you write it.
• Adjectives like "strictly" and "generously" are not criteria. Every person reads them differently. Convert them into numbers.
• Providing a model answer helps, but include a line like "credit equally valid answers too" so it does not harden into the only correct answer.
• Once you revise the prompt, rerun some previously graded responses and confirm the scores stay consistent.

The skeleton of a good grading prompt

Do not try to write a perfect prompt from scratch; filling in the following skeleton is faster. Build the frame once and you can reuse it by swapping in a new item.

1. Role and context: Start by writing what subject, what grade, and what the item is assessing.
2. Conditions for full marks: List the elements that all have to be present for a full score.
3. Partial credit table: Lay out arithmetically how many points a given number of elements earns.
4. Exclusions: Write out plainly what you will not deduct for.
5. Borderline examples: Pin in two or three confusing responses along with the correct score for each.
6. Output template: Specify the format so you receive score, reasoning, and location of evidence in that order.

Fill in these six boxes and your scores hold steady in a way a vague one-line instruction cannot touch. Item 5 in particular drives the results, so keep growing the prompt by adding every ambiguous response you meet while grading as another example.

Key takeaways

Consistency in grading comes not from model performance but from precision in the prompt. Break full marks into elements, make partial credit arithmetic, and pin in borderline examples, and the scores stop shifting. A prompt you stabilize once gets reused as is next semester, so the time you spend refining it early becomes your longest-lasting asset.