Every measurement course teaches the same sequence: draft the items, try them out, analyze the results, revise, then administer. Almost nobody does it. The reason isn’t laziness — it’s that a classical try-out requires a second group of comparable students you don’t have, and by the time you’d have them, the exam is next week.
This guide covers the real procedure, what it actually buys you, and the shortcuts that catch most broken items without a full try-out.
The big idea: item analysis after the exam tells you which items were broken after they’ve already cost your students marks. A pilot test moves that same diagnosis earlier — before anyone’s grade depends on it.
What a pilot test is actually for
A try-out answers four questions before the exam counts:
- Is any item miskeyed? By far the most common defect, and the most unfair.
- Is the exam the right length for the time allotted? Students running out of time turns a knowledge test into a speed test.
- Are the distractors working? An option nobody picks is a wasted option; an option that outpulls the key is either a misconception or an ambiguity.
- Is the difficulty spread reasonable? An exam where everything is easy ranks nobody; one where everything is hard ranks noise.
Notice what’s not on that list: content validity. That comes from your Table of Specifications, not from a try-out. A pilot test checks whether items function, not whether they cover the right material.
The classical procedure
1. Build the instrument to your TOS. Include roughly 20–25% more items than the final exam needs — the try-out will kill some, and you want survivors to spare.
2. Recruit a comparable group. Students who have already completed the same course content but whose grades won’t be affected — a previous section, a parallel section under another instructor, or students from the term before.
3. Administer under real conditions. Same time limit, same format, same instructions. A try-out run casually tells you nothing about timing.
4. Score and run item analysis. Difficulty index (p) and a discrimination statistic for every item, plus a distractor tally for multiple choice.
5. Apply decision rules. The conventional ones:
| Result | Decision |
|---|---|
| p 0.30–0.80 and discrimination ≥ 0.30 | Keep |
| Discrimination 0.20–0.29 | Revise — usually the stem or one distractor |
| p below 0.20 or above 0.90 | Revise — too hard or too easy to inform anything |
| Negative discrimination | Check the key first — this is a miskey until proven otherwise |
| Discrimination below 0.20 after revision | Reject |
6. Rebuild the final form from surviving items, keeping the TOS distribution intact. If killing three items leaves a competency underrepresented, replace them rather than shortening that row.
Sample size — the honest numbers
Item statistics are estimates, and small groups produce unstable ones.
| Try-out group | What you can trust |
|---|---|
| Under 20 | Miskeys and obvious ambiguity only. Discrimination values are noise. |
| 25–40 | Difficulty is usable; discrimination is directional, not precise. |
| 100+ | Both statistics are reasonably stable. |
| 200+ | Publication-grade. Rarely available for a course exam. |
For classroom purposes, 30 is enough to catch the defects that matter. You are not validating a standardized instrument; you are trying not to give students a broken exam.
Why almost nobody does this
Be honest about the constraints, because they’re real:
- A comparable group that hasn’t seen the material is hard to find and harder to schedule.
- Anyone who takes the try-out has now seen your items — exposure is a genuine cost.
- Your final exam typically doesn’t exist until days before it’s given.
- The whole cycle costs a week you don’t have.
These are why the textbook procedure stays in the textbook. The useful question isn’t “how do I run a proper try-out” — it’s “what fraction of the benefit can I get for an hour of work?”
Four shortcuts that catch most of the damage
Answer your own exam cold, from the key. Print it, wait a day so you’re not reading from memory, and take it. This alone catches most miskeys and every item with two defensible answers. Time yourself — then multiply by three for a realistic student estimate.
Have one colleague answer it. A second content expert who disagrees with your key has found an ambiguous item. This is the single highest-yield ten minutes available.
Reuse analyzed items. Items that already have clean statistics from a previous administration don’t need re-testing. Over a few semesters you accumulate a bank of known-good items, and only the new ones need scrutiny — which is the practical version of what the classical procedure is trying to achieve. Our guide to parallel forms covers how to rotate them.
Simulate the responses. Instead of recruiting students, model how a realistic class would answer and run item analysis on the modelled responses. It costs minutes rather than a week and exposes nobody to your items.
What simulated pilot testing can and can’t tell you
This is worth being precise about, because it’s easy to oversell.
It does catch: structurally broken items — stems that don’t point at a single answer, distractors that are indefensible or too obviously wrong, items whose difficulty is wildly out of line with the rest of the exam, and answer keys that contradict the item.
It does not replace real data. A modelled difficulty index is an estimate of how a typical class would perform, not a measurement of how your class did. Use it to decide what to fix before the exam; use real item analysis afterward to decide what goes in the bank.
Treat it the way you’d treat a colleague’s review: a competent second opinion that catches obvious defects, not an oracle.
Frequently asked questions
Can I pilot test on my own class?
Not on the exam they’re about to take — that’s just giving them the exam twice. You can try out items on a different section, or on your current class for a future semester’s exam.
How many items should I over-draft?
About 20–25%. For a 50-item final, draft 60–65. Expect to lose a handful and to revise several more.
Is a pilot test the same as validation?
No. Validation is about whether the instrument measures what it claims to measure — that’s a TOS-and-expert-review question. A pilot test is about whether the items function statistically. You need both, and they’re different activities.
What if the try-out shows my whole exam is too hard?
Check the TOS first. An exam that’s uniformly too hard is usually mis-levelled against the cognitive distribution you planned, not badly written. Rebalancing Bloom’s levels fixes more of these than rewriting items does.
Do I need to pilot test essay or performance items?
The classical indices don’t apply. What you pilot instead is the rubric — have two raters score the same three sample responses independently. If they disagree by more than one level on any criterion, the descriptors need rewriting, not the task.
Run the try-out without recruiting anyone
Skoolari’s Pilot Simulation (Pro plan, exam-type assessments) models a class of student responses against your draft exam and returns a full item report before you administer it — per-item difficulty, discrimination, distractor pull, and a Keep / Revise / Reject verdict with the reasoning shown for each item. You fix the broken items while fixing them is still free.