Learning Calibration: Compare Confidence With Checked Results

You expect to answer most questions on a topic correctly. A checked practice attempt can tell you whether that expectation fits the task. Comparing predictions with results is a way to examine calibration.

The aim is to make better-informed study decisions. Both high and low confidence can be inaccurate, and neither tells you someone’s ability on its own.

Two kinds of judgment accuracy

Absolute calibration concerns the match between a predicted level of performance and the observed level. If you predict a score of 80% on a defined question set, compare that prediction with the score using the same marking rule.

Relative accuracy concerns whether your judgments distinguish stronger from weaker items or topics. You could rank your topics correctly while predicting scores that are too high for all of them. You could also predict the overall score well while being wrong about which topics need work.

Prinz, Golke and Wittwer (2020) analyzed relative metacomprehension accuracy: how well learners’ predictions distinguished their comprehension of different texts. They found a modest average relationship, with variation across study conditions. The paper appeared in Educational Research Review. Its finding is not a measure of everyone’s overconfidence or a guarantee about an individual learner.

Use a check that matches your goal

Soderstrom and Bjork (2015) explain why performance during acquisition may give an incomplete picture of long-term learning. Looking at an answer can make a task feel easy without establishing that you can produce it later.

Choose a check that resembles the knowledge you need. For an exam, that might be answering a short question or solving a problem. For a work task, it might be explaining a procedure or applying it to a new example. Keep reference materials available when the real task permits them; define the conditions before judging the result.

A delayed check can provide evidence about later performance. The delayed-judgment finding from Nelson and Dunlosky (1991) concerned paired-associate recall under specific conditions. It should not be turned into a promise that every delay improves comprehension or predicts a final exam score.

Keep a simple prediction record

1. Choose a question set and marking rule. Write down when you will attempt it and whether you will use references.
2. Predict your score before the attempt. If you rate individual questions, record those ratings before checking the answers.
3. Complete the attempt under the conditions you chose. Mark against a reliable source and keep partial-credit decisions consistent.
4. Record the predicted and observed scores beside the date and topic. A prediction of 80% followed by 60% shows a 20 percentage-point overestimate on that attempt.
5. Use several comparable attempts to look for a pattern. Revisit missed concepts and check them again; adjust the task if it does not represent your goal.

One result is not a diagnosis. Question difficulty, guessing, small samples and unclear answer keys can affect the comparison. A high-confidence answer that is wrong is useful feedback about that answer, rather than proof that your whole judgment system is broken.

Use the record to choose your next task

Look for topics where predicted confidence stays high but checked performance remains low. Review the source, work through an explanation or practise another problem. Also notice topics where performance is stronger than expected.

Generated practice can help supply questions, but its answer key needs review. Piply does not become a validated calibration assessment simply because you use it for practice.

Start with one manageable question set and keep the first prediction alongside the checked result. Build the record when it helps you decide what to practise next.