What “accuracy” means on a fact-check product

Accuracy is a property of a task, a source set, and a decision threshold.

Accuracy has a denominator

“How accurate is it?” sounds like one question. It is several. Accurate on which claims? Against which reference? At what point in time? Under what threshold? A single percentage can conceal all of these choices.

For a fact-check product, define the task before measuring it. A system that identifies whether a source exists is doing something different from a system that judges whether a statement is supported. A system tested on settled claims may behave differently on fast-moving news.

The reference itself needs care. Some claims have a clear answer in a public record. Others require expert interpretation or remain contested. A benchmark should mark those differences. Reviewers should not be forced into a binary label when the evidence supports a qualified answer. The question asked of reviewers should match the question the product presents to users.

Labels are decisions

Every verdict threshold creates trade-offs. A cautious system may leave more claims unresolved. A permissive one may produce more decisive labels, with more false confidence. Neither is automatically better. The right choice depends on the cost of missing a useful lead and the cost of publishing a mistaken conclusion.

Machine-learning evaluation commonly separates precision and recall. Those measures are useful, but they do not replace a task definition. Calibration also matters: when a system expresses confidence, that confidence should correspond to observed outcomes over a defined set. The calibration literature explains why a confident score is not automatically a reliable probability.

For example, a team may prefer high precision when publishing an automatic correction. It may prefer higher recall when collecting leads for a human researcher. The same model can be appropriate for one job and unsafe for the other. Metrics should therefore be attached to a workflow and a decision, not promoted as a general score for the whole product.

Measure the edges

Average performance is not enough. Look at source quality, language, topic, claim type, and time delay. Check how often the system abstains. Review cases with missing context. Review cases where a citation is related but does not support the conclusion.

Human review should be part of the measurement plan. Reviewers need clear instructions and a way to record disagreement. Reference sets should be documented and refreshed. If a product changes its retrieval sources or model, measure again. Results from one configuration should not be presented as a permanent property.

Watch for leakage and selection effects. If test examples resemble the material used to build the system, a high result can overstate general performance. If only easy, well-documented claims are measured, the result says little about ambiguous posts or developing events. Report the sample, exclusions, time window, and uncertainty around the estimate. A small difference may not be meaningful.

Evaluation should include abstentions and corrections. A system that declines difficult cases may be more useful than one that forces a label. Record why reviewers overruled a result. Those reasons can reveal missing source types, ambiguous instructions, or a threshold that does not fit the intended workflow.

Use accuracy with humility

For a reader, the practical question is not “Can I trust the score?” It is “What can I inspect?” A useful product shows the claim, the evidence, the date, and the remaining uncertainty. It makes correction possible.

Accuracy can also change after launch. Sources disappear. Definitions shift. New facts arrive. A monitoring process should sample live results and send difficult cases back for review. Corrections should be logged, not treated as embarrassing exceptions. This gives users a better picture of where the system works and where it needs help.

Prove is designed as an aid to that process. Open a result, follow its citations, and decide whether they support the wording in front of you. If a result is unclear, that is information too. It can signal that a human should investigate further.