Can a Fast Decision Model Check Legal Citations?

Introduction

Generative models now draft research memos, submissions and advice that cite case law. The characteristic failure is not a fabricated case; it is a real case cited for something it does not say. A proposition may keep the source’s words while reversing its meaning (“must” becomes “need not”) or quietly dropping the condition on which the court’s statement depended. Checking every citation by hand is slow; checking with a second large model is expensive and adds latency to every draft.

A new class of model offers a third option. Jev (TypeSafe AI, released September 2026) is marketed as a “System One Model”: it does not generate text, but answers typed questions (a choice among labels, a score, or the probability that a statement is true) with calibrated probabilities, in well under a second, at a fraction of a cent per decision . The name invokes the familiar distinction between fast, intuitive judgement and slow, deliberate reasoning . We use the label descriptively, as a hypothesis about which errors a fast model handles well, not as a claim about how the model works: if it is apt, such a model should be good at citation errors an expert sees at a glance and weaker at errors that require careful reading. That is testable, and it maps directly onto a practical design question: which checks can be delegated to a fast model, and which need a slower one?

This paper reports a pre-registered evaluation of that question on Australian case law. Our contributions are:

1. A citation-support benchmark built from 150 real pinpoint citations in New South Wales (NSW) judgments, with off-topic near-misses, meaning-altering distortions split in advance into glance and careful-reading tiers, and a smaller set of naturally occurring AI drafting errors.

2. A comparison of nine checker configurations spanning a typed decision model, four language models from two vendors with four ways of eliciting confidence, a natural language inference (NLI) cross-encoder and a lexical baseline, measured for ranking quality, calibration, selective accuracy, latency and run-to-run stability.

3. Evidence on where a fast decision model is and is not sufficient for checking citation support, and a set of methodological lessons, several of which cut against our own earlier plans.

Background and related work

Legal AI reliability.

Studies of legal research tools have documented hallucinated and mis-cited authority even in retrieval-augmented products . Benchmarks such as LegalBench measure legal reasoning broadly; CLERC targets case retrieval and citation-grounded generation. Our task is narrower and closer to deployment: given a proposition and the paragraph cited for it, decide whether the paragraph supports it.

Support checking as entailment.

Deciding whether a source supports a claim is a form of textual entailment . NLI models have been used to check factual consistency of generated text, often by splitting a long source into windows and taking the maximum entailment score . We include such a model as the standard non-LLM baseline.

Confidence and selective prediction.

A checker is useful in practice only if its confidence separates answers that can be accepted from answers that need a human. This is selective prediction . Neural models are often miscalibrated ; language models can express uncertainty through token probabilities, verbal statements or agreement across samples, with differing reliability . We report calibration (Brier score , expected calibration error ) and accuracy on the most confident fraction of answers.

Jev in legal settings.

Public evidence is thin. TypeSafe reports large gains from Jev reranking on CLERC . A legal AI vendor reported gains from Jev in reranking and compliance checks but a failure on groundedness scoring, where answers with fabricated citations scored higher than honest ones . An independent reranking benchmark on Finnish legal texts found Jev roughly tied with a commercial cross-encoder . We are not aware of a published evaluation of Jev’s confidence for legal verification.

Benchmark construction

Source corpus

We used a local corpus of 202,276 Australian decisions (NSW courts and tribunals, the Federal Court and the High Court), derived from the Open Australian Legal Corpus and supplemented by direct collection, with about 1.06 million extracted citation links. Each link records the citing context and a treatment label (for example applied, discussed, distinguished).

Propositions

We sampled citations meeting all of the following (fixed seed): the cited decision is on NSW Caselaw, so its numbered paragraphs can be fetched; the treatment is applied; the citing text gives exactly one pinpoint, of at most three paragraphs; no signal word (cf, contra, but see, compare) precedes the citation; the matter is not criminal; and no subject heading contributes more than 12 propositions. This yielded 150 propositions across 65 subject headings, citing decisions of the NSW Supreme Court (50), Court of Appeal (44), Civil and Administrative Tribunal (44) and Land and Environment Court (10), among others. A language model (Claude Sonnet) read only the citing passage, never the cited paragraph, and stated in one sentence the proposition for which the case was cited, naming no case. The proposition is therefore written without sight of the evidence it is later tested against. It paraphrases the citing judge’s own statement of what the case stands for, but its phrasing is the model’s, so any stylistic habits of that model are shared by every item (see Limitations). The selection filters give a deliberately clean subset of citation practice: one pinpoint, an applying treatment and no contrary signal.

Paragraphs

Cited paragraphs were fetched from NSW Caselaw and split on the site’s numbered-paragraph markup. A first build dropped quoted blocks and indented sub-lists that follow a numbered paragraph, so 12 of 100 supported test items ended “…his Honour said:” with the support missing. This was found in a dry run of the label review (§3.5); the parser was fixed, all paragraphs rebuilt with propositions and item identities unchanged, and every model rerun. The median paragraph is 799 characters; paragraphs were capped at 4,000 characters (6 truncated). The 150 propositions were split 50 for calibration (used only to fit mappings for baselines that need one) and 100 for testing.

Three item sets

Test 1: off-topic near-misses.

Each test proposition appears with its cited paragraph (supported) and with one near-miss (not supported): either another paragraph of the same judgment at least five paragraphs from the pinpoint, or the cited paragraph of a different proposition under the same subject heading. The test set has 100 supported, 64 same-case and 36 same-area items.

Test 2: meaning-altering distortions.

Each test proposition appears as written (supported) and in two distorted versions (not supported), all against its own cited paragraph. A language model wrote the distortions under fixed rules that required most of the wording to be kept (median 97% of content words retained; items below 60% were dropped):

• Tier A, single-word change (“glance”): negation (53), modal flip such as may/must (36), quantifier (6), changed number or threshold (3), role swap (2).

• Tier B, scope or condition change (“careful reading”): overstated scope (47), dropped condition or exception (42), a matter the court assumed stated as decided (6), reversed onus (2).

The tiers are defined by the kind of edit, fixed before any checker ran; “glance” and “careful reading” are shorthand for the expectation that the first kind is easier to spot, not a measured property of each item. The two kinds overlap in practice: a blind model review (§3.5) judged most tier B items, as well as tier A items, obvious on one read. The results below therefore compare edit types, and the System-1 framing is an interpretation of them rather than something they prove.

Natural AI drafting errors (exploratory).

To complement deliberate distortions, a small model (GPT-4.1-nano) was given only each cited paragraph and asked to state the proposition it establishes “as a junior lawyer would in a research memo”. Two models from different vendors that are not among the checkers (Claude Opus 5.5 and GPT-5.5) independently judged each draft; their exact agreement was 81%. Under the pre-registered rule only 2 drafts were clearly unsupported, too few for inference; the natural set is therefore reported as exploratory, with a post-hoc label treating drafts that both judges rate “partly” or “not” supported as errors (23 of 81 test items). These errors are typically overgeneralisations.

Label quality

The labels in tests 1 and 2 are structural rather than matters of legal judgement. A paragraph counts as supporting a proposition because a later court cited it, at that pinpoint, for that proposition; a near-miss counts as unsupported because it is a different paragraph; a distortion counts as unsupported because it was edited under fixed rules to change the meaning. The study concerns these mechanics of citation support, not contested questions of law. Two sources of noise remain: pinpoint citations are sometimes loose, and an edit may fail to change the meaning. To estimate them, a separate model (Claude Opus 5.5) completed a blind review of a stratified sample of 70 items (40 from test 1, 30 from test 2) under the pre-registered stopping rules. It agreed with the automatic labels on 82% of test 1 items (98% if “partly supported” counts as supported) and judged 1 of 20 distortions still supported, so neither stopping rule would trigger. It judged tier A distortions “obvious at a glance” in 9 of 10 cases and tier B in 6 of 10. The natural set uses model judgements as labels and is treated as exploratory throughout. The lawyer review of the same sample that was planned in the pre-registration has not been carried out.

Checkers

Every checker saw the identical proposition and paragraph and produced 𝑝 =𝑃⁡(supported); the decision is 𝑝 ≥0.5 and its confidence is max⁡(𝑝,1 −𝑝).

• Jev (jev-1.13.0): one call with the proposition and paragraph as state and a single probability-of-truth question, “Does the paragraph support the proposition?”.

• Claude Haiku 4.5, stated: an answer plus a stated probability (0 to 100).

• Claude Haiku 4.5, agreement: the one-word answer sampled five times at temperature 1; 𝑝 is the share answering “supported” .

• Claude Sonnet 5, stated: as Haiku stated.

• Claude Sonnet 5, thinking: as above with adaptive extended thinking at high effort.

• Claude Opus 5.5, stated: as Sonnet stated, with the model’s default adaptive thinking (tests 1 and 2 and the GPT-written distortions only; Opus was a label judge for the natural set).

• GPT-4.1-mini, token probability: the one-word answer at temperature 0; 𝑝 is the normalised probability of the “supported” token among the top five first tokens.

• NLI cross-encoder: deberta-v3-base-zeroshot-v2.0 run locally; maximum entailment probability over 400-token windows, mapped to 𝑝 by logistic regression fitted on the calibration split.

• Word overlap: the share of the proposition’s content words present in the paragraph, mapped by logistic regression fitted on the calibration split.

Why these models.

The checkers were chosen to cover the three main ways of getting confidence out of a language model (a stated probability, agreement across repeated samples, and the probability the model assigns to its answer token) and two levels of effort (a direct answer and extended thinking), from two vendors. Token probabilities are the most direct measure of a language model’s own confidence, but OpenAI’s current GPT-5 series are reasoning models that did not return them at the time of testing; GPT-4.1-mini is the most recent OpenAI model that does, so it represents that method rather than OpenAI’s strongest model. Claude Sonnet 5 (with and without thinking) and Claude Opus 5.5 represent frontier models. Claude Haiku 4.5 is the fast, low-cost tier closest to Jev’s intended role. GPT-5.5 and Opus 5.5 also wrote the second distortion set and judged the natural set respectively, and were not used as checkers on sets they helped build. OpenAI’s newest reasoning model (GPT-5.6) was planned as a further frontier checker but could not be run because the account’s API credits ran out.

A locally run open-weights typed-decision model (Laya) was included in the first plan; it scored 61.5% on test 1 and was dropped after running out of memory on the rebuilt, longer paragraphs.

Measures and pre-registration

Four plans were frozen before the runs they govern, and every deviation from them is logged. The main measures are AUROC (threshold-free ranking of supported above unsupported items), accuracy on each checker’s most confident 25%, 50% and 75% of answers (selective accuracy), Brier score, expected calibration error (ECE, 10 bins), the number of distinct confidence values produced, median latency and input tokens. Latency is the wall-clock time of each call over the vendors’ public APIs, measured from Sydney in September 2026 with several requests in flight at once; it reflects network distance, load and serving infrastructure as much as the models, and is reported as an observed deployment characteristic rather than a property of the models themselves. Differences are tested by a paired bootstrap over propositions (10,000 resamples, 95% intervals). Pre-registered hypotheses for test 2 were: H1 Jev beats word overlap on glance errors; H2 Jev’s glance AUROC is within 0.05 of the slow model’s and its latency is under a tenth of the slow model’s; H3 Jev versus Haiku on glance errors (no direction); H4 Jev drops more than the slow model from glance to careful errors. A third plan added the thinking model, calibration for test 2, the GPT and NLI checkers, three repeated runs, and the natural set. A fourth added the robustness checks in §6.6 (two-stage checking, a direct test of the threshold explanation, and distortions written by a second vendor) and the Opus checker.

Results

Test 1: off-topic near-misses

When wrong paragraphs are simply off topic, every competent checker is near ceiling and a free word-overlap rule is within a point of the best (Table 1). Judges tend to paraphrase the paragraphs they cite, so shared wording carries most of the signal. Haiku’s stated confidence was poor here for a reason we did not anticipate: in 72 of 300 replies it answered “not supported” with a confidence of 72 to 99, reporting confidence in its own answer rather than the probability of support the prompt asked for. Read that way (a post-hoc correction), it scores 98.0% on its confident half; the prompt was reworded for test 2.

Test 2: meaning-altering distortions

Pre-registered hypotheses.

H1 supported: Jev beat word overlap on glance errors by 0.396 AUROC (95% CI 0.352 to 0.435); keeping the wording defeats the lexical rule entirely. H2 not supported, on the latency clause only: Jev’s glance AUROC was higher than Sonnet’s (+0.017 against Sonnet with thinking, CI −0.001 to +0.037; +0.015 against Sonnet stated), comfortably inside the 0.05 margin, but it was 4.9 times faster, not the 10 times required. Sonnet’s adaptive thinking engaged on only 27% of items (median 29 output tokens), so the slow model was not very slow on this task. H3: Jev beat both Haiku variants on glance errors (+0.064, CI +0.034 to +0.098 against stated; +0.057, CI +0.023 to +0.092 against agreement). H4 as predicted: Jev’s AUROC fell more from glance to careful errors than Sonnet’s (−0.130 vs −0.087).

Careful-reading errors.

Jev caught fewer careful-reading errors at its default threshold (55% against Sonnet’s 74 to 75%). Its overall ranking of those items was not significantly different from Sonnet with thinking (AUROC difference −0.017, CI −0.048 to +0.013), which suggested the gap might only be a matter of where the threshold sits; §6.6.0.2 tests that directly and finds it is not. Jev was weakest on dropped conditions (60% caught vs 76%) and on the six “assumed as decided” items. The NLI cross-encoder caught the most distortions at its fitted threshold (87% and 79%), but only by rejecting 44% of genuine propositions; its ranking (AUROC 0.839 and 0.769) sits just below Jev and the Sonnet configurations, a strong result for a free local model.

Calibration and selective accuracy

On glance items (Table 3), Jev was better calibrated than every Sonnet, Haiku and GPT configuration (Brier 0.127, ECE 0.087) and had the highest accuracy on its confident half among the pre-registered checkers (93.0%); the pre-registered comparison with the next best, Sonnet with thinking, was a tie (+2.0 points, CI −2.0 to +6.0). Opus 5.5, added later, was better still (95.0% on its confident half, Brier 0.122, ECE 0.069). Checkers that produce few distinct confidence values cannot support fine thresholds: Haiku agreement produced 5 values on these items and a single value on the natural set.

Stability

Across three independent runs (Table 4), Jev and GPT-4.1-mini with token probabilities were the most stable checkers. Jev’s glance AUROC varied by 0.004 and 1.0% of its test 2 decisions changed between runs, against 1.3% for Haiku agreement, 5.7% for Haiku stated and 6.1% for Sonnet stated.

Natural AI drafting errors (exploratory)

On the 81 natural test items (23 post-hoc errors), Sonnet stated ranked best (AUROC 0.816), followed by Sonnet with thinking (0.794), Jev (0.771) and Haiku stated (0.754); GPT-4.1-mini reached 0.653, the NLI cross-encoder 0.579, Haiku agreement collapsed to a single confidence value (0.500), and word overlap fell below chance (0.391), because overgeneralised drafts reuse the paragraph’s vocabulary. With silver labels and 23 errors this is indicative only, but it points the same way as tier B: the errors language models make unprompted are mostly overstatements, the kind of error on which the fast checker is weakest.

Robustness checks

A fourth plan, frozen before these analyses were computed, tested three things the preceding results rely on. The first two re-analyse existing outputs and are post hoc relative to plans 1 to 3; the third is a new test.

Two-stage checking.

We simulated the design the results point to: Jev decides the items it is most confident about and the rest are sent to Sonnet with thinking (Table 5). On test 2, sending Jev’s least confident 60% of items to Sonnet matched Sonnet-alone accuracy (79.1%) at 80% of its expected latency; sending 30% came within 1.0 point at half the latency. On test 1, where errors are easier, sending 30% matched Sonnet exactly at half the latency. The combination never exceeded Sonnet alone: Jev’s value in a pipeline is to reduce how much work reaches the slower model, not to catch errors the slower model misses. This is a simulation replayed over outputs already collected on the same items, not a prospective test of a deployed router. A production system would fix its escalation share on separate data, and its savings would depend on its own mix of errors and on the latencies of its deployment.

Is Jev’s careful-reading shortfall a threshold effect?

No. Setting Jev’s threshold so that it accepts the same share of genuine propositions as Sonnet with thinking (76%), Jev caught 55% of careful-reading distortions against Sonnet’s 74% (difference −19.6 points, CI −28.6 to −10.6). A stricter threshold fitted on the calibration split (0.71) raised Jev’s catch rate to 70%, but only by accepting fewer genuine propositions (70%). Part of the gap is where the threshold sits; most of it is that Jev reads subtle distortions less well than a frontier model. The similar AUROC values reflect differences elsewhere on the curve, not at the operating point a checker would use.

Distortions written by another vendor.

Because Claude Sonnet wrote the test 2 distortions, we had GPT-5.5 (OpenAI) write a second set from the same propositions with the identical prompt and filter (99 glance and 98 careful-reading test items) and reran the checkers (Table 6). Jev’s glance result held: its AUROC on the GPT-written set was 0.913, level with Sonnet with thinking (+0.005, CI −0.020 to +0.031), and Jev, Sonnet stated and Sonnet with thinking were the top three pre-registered checkers under either writer. There was no sign of a home advantage: the Anthropic checkers scored slightly higher, not lower, on GPT-written distortions. GPT-5.5’s careful-reading distortions were harder for every checker (AUROC down by 0.09 to 0.15), and Jev remained slightly behind Sonnet on them (−0.026, CI −0.053 to +0.002). GPT-4.1-mini could not be rerun on this set because the OpenAI account’s credits were exhausted.

The strongest frontier model.

Claude Opus 5.5, added under the fourth plan, was the best checker on every distortion measure (Tables 2 and 6). On Sonnet-written glance errors its lead over Jev was not significant (AUROC 0.920 vs 0.906; difference −0.014, CI −0.037 to +0.006), but on GPT-written glance errors it was (0.954 vs 0.913; −0.041, CI −0.069 to −0.017), and on careful-reading errors it was clearly ahead under either writer (0.845 vs 0.776, CI of the difference −0.097 to −0.042; and 0.709 vs 0.655, CI −0.083 to −0.025). It also had the best calibration on glance items. It was the slowest checker, at a median 3.3 seconds per check, 12 times Jev’s latency, and it produced a median 141 output tokens per answer against Jev’s none. On test 1 it tied Jev (98.0% on the confident half). The precise reading is narrow: Jev matches Sonnet’s ranking of glance errors. It does not match Sonnet on careful-reading errors or on the natural drafting errors, and it does not match Opus on any distortion measure.

Cost and latency

Jev used a median 564 input tokens per check (output is not billed), about US$0.02 per thousand checks at its published price , with a median latency of 275 ms. The language-model checkers used 290 to 530 input tokens per call (Haiku agreement 1,634 over five calls) and 22 to 70 output tokens.

Discussion

What a fast checker buys.

On single-word changes to the operative claim, a model that does no visible reasoning ranked items as well as Claude Sonnet with thinking enabled, was better calibrated than it, was among the most stable across runs, and answered in about a fifth of the time, and this held when a different vendor’s model wrote the errors. That is the extent of the claim. On changes to scope or conditions it was clearly weaker than Sonnet (about 20 points fewer caught at the same acceptance rate), it trailed Sonnet on the natural drafting errors, and the strongest model tested, Opus 5.5, was better than it throughout, at twelve times its latency. This pattern is consistent with the System-1 label, read descriptively: the fast model does well where the error sits in a single word and less well where it sits in what the proposition leaves out. It supports a two-stage design for legal drafting tools, in which a fast model screens every citation and the uncertain share goes to a slower model or a person. In our simulation, sending 30% onward came within a point of the slower model’s accuracy at half its latency; matching it fully needed 60%. Propositions that state rules with conditions or scope are the natural candidates for the slower path.

What the synthetic errors measure.

The distortion sets are controlled stress tests: each changes one thing about a real proposition and keeps the rest. They measure how sensitive a checker is to specific kinds of meaning change, not how often those changes occur in real drafts, so the catch rates should not be read as production error rates. The small natural set is the closest we come to real drafting behaviour, and it suggests that unprompted model errors are mostly overstatements, closer to tier B than tier A. That makes the careful-reading results the more important ones for deployment, and they are where the fast checker is weakest.

Most of the difficulty is in the benchmark, not the model.

Two of our findings are about test design. First, off-topic negatives make citation checking look easy: a word-overlap rule tied every model on test 1 and was useless on test 2. Benchmarks for citation verification should include meaning-altering negatives that preserve wording. Second, confidence elicitation is fragile. The same model moved from last to joint first on test 1 depending on whether its stated number was read as confidence in support or confidence in its answer, and sampling-based agreement produced too few distinct values to threshold. Reports of “calibration” for language models should state how confidence was obtained and how many distinct values it takes.

Context from earlier experiments.

Two exploratory studies informed this design and are summarised in the supporting materials. A routing benchmark (which court’s case law to search) turned out to be solvable by a lookup on subject headings (200/200), so it could not separate models; the lesson was to run trivial baselines first. A case-retrieval study found that fewer than 14% of the cases judges actually relied on reached any ranker, so reranking with Jev or Haiku could not fix what search had missed; Jev and Haiku ranked candidates about equally well.

Limitations.

• Scope. One jurisdiction (NSW), mostly post-2010 decisions, single cited paragraphs, and 100 test propositions. Nothing here tests statutes, contracts, other jurisdictions, reasoning that spans several paragraphs, long authorities, or the groundedness of a whole document.

• A simplified subset of citation practice. Only citations treated as applied, with one pinpoint and no signal word, from non-criminal matters, were used. Real drafts also cite distinguished cases, multiple pinpoints and propositions supported by wider context, which are harder to check.

• Synthetic errors. Both distortion sets are model-written stress tests, not samples of real drafting mistakes (see Discussion); the natural set that approximates real behaviour is small and exploratory.

• Model-phrased propositions. The propositions were phrased by an Anthropic model from the citing text. A second distortion set written by GPT-5.5 gave the same conclusions, but the propositions themselves were not rephrased, so stylistic habits of the phrasing model could make support easier or harder to judge for particular checkers.

• The tier split is by edit type. Glance and careful-reading tiers overlap in how hard they are to spot, so the results compare kinds of edit and do not establish a System-1/System-2 distinction.

• Latency and the pipeline are deployment-specific. Latencies depend on network, region and load, and the slower model thought on only a minority of items, so it was not a deliberately slow checker. The two-stage result is a simulation on the same items, not a deployed routing policy.

• Model coverage. OpenAI’s newest reasoning models were not evaluated as checkers (the planned GPT-5.6 run could not proceed), and GPT-4.1-mini is missing from the second distortion set for the same reason. Jev is a hosted service in early access, and its prices and versions may change; we pinned jev-1.13.0.

Conclusion

On single-word changes to what a proposition claims, a fast typed-decision model ranked supported and unsupported citations as well as Claude Sonnet with extended thinking, with better calibration and stability, at lower observed latency and negligible cost, and the result held when a different vendor’s model wrote the errors. It is not a careful reader: even at a matched acceptance rate it let through about 20 points more of the overstated and conditionally qualified propositions, which are also the errors language models make on their own, and the strongest frontier model tested was more accurate throughout, at far higher latency. As a first-pass screen that sends its uncertain share to a slower model it fits the task well; in our simulation that cut the slower model’s workload by 70% and the expected latency by half, for about a one-point cost in accuracy. Used alone, it does not.

Data, code and disclosures

The pre-registration documents, item sets, model outputs and scoring code are available from the author on request. The author has no commercial relationship with TypeSafe AI, Anthropic or OpenAI. The experiments were designed and run with the assistance of an AI coding agent (Claude Code); the author is responsible for all content.