Name-Swapping and Redaction for Offshore Legal AI: A Preliminary Evaluation on Invented Australian Matters

Foundry Labs

Drafted with AI assistance (Claude, Anthropic). The experiments were designed and run in the same working session, with the open-source Mike application (MikeOSS) as the test harness.

1. Introduction

A firm that wants to use a low-cost model hosted outside Australia on client work must keep client identities out of what that model sees. One approach is to replace names with placeholders or invented names before sending a document, and to convert the answer back afterwards. The practical questions are whether this costs too much accuracy to be worthwhile and whether it is reliable enough to trust. Which model to use for which task is a separate question that we do not address here.

This report documents a first, deliberately small attempt to answer them with measurements. It makes four contributions. First, a controlled generator of invented matters in which the number of name forms per person (full name, titled surname, first name, nickname, initials, and so on) is set by code, so that name-linking difficulty is a dial and not an accident of the data (Section 3). Second, an automatic scorer for fact questions over those matters, with a self-test of 937 checks (Section 3.3). Third, results for the effect of name swapping against redaction on two models (Section 5). Fourth, a reliability fault in the pipeline's name detector, found by this evaluation and fixed (Section 6).

The programme grew from an earlier set of experiments on name swapping using a handful of hand-built matters, which found that swapping to placeholders cost little on clean matters and 1 to 4 points on a 10-point scale on a deliberately messy one, that the style of replacement (numbered, invented, or short codes) made no measurable difference, and that a hand-built, form-by-form realistic swap of the messy bundle scored the same as the original (5.8 against 5.6, five runs each). Those results motivated the present design, which isolates linking difficulty from replacement style.

2. Related Work

Pseudonymisation and redaction have been compared for downstream utility before. Deußer et al. [1] evaluate five models on eleven benchmarks and report that reversible, entity-preserving pseudonymisation outperforms redaction, that more capable models lose more, and that telling the model about the anonymisation does not help; their replacements are mostly numbered placeholders with a category tag, they use an LLM-based entity detector, and they do not discuss nicknames, partial names or legal text. Lothritz et al. [2] compare six anonymisation strategies on nine tasks with two pre-trained encoder models and find a modest loss, smallest for pseudonymisation with random names. Replace-and-restore designs for cloud models appear in [3] and [4]; [3] reports fewer semantic utility errors than unnamed baselines on chat data. Legal pseudonymisation tools that combine name detection with coreference to give one pseudonym per person include PSILENCE [5] for arbitration documents, evaluated on detection accuracy rather than on a language model's task performance; CON-QA [6] studies contract question answering with a local model handling sensitive entities. We did not find a study that varies the number of name forms per entity and measures task accuracy against it, which is the design here. This is a negative search result from web searches and abstract-level reading, not a systematic review.

3. Materials and Methods

3.1 Invented matters and the alias dial

Each of eight matters has eight documents (an agreement, an email thread, a chat log, a sworn statement, two letters, a record, and an internal note), about 1,000 words in total, and 16 questions with short answers. People and businesses are generated by code from seeded name pools; a model (Claude Sonnet 5.5) then writes the documents and questions using mention tokens such as {{P3:nick}}, and code validates and renders them. Because every name is a token, the exact position and form of every mention is known. A matter's alias level limits which forms may appear: A0 full names only; A1 adds titled surnames; A2 adds first names, nicknames and dotted initials; A3 adds bare surnames, plain initials, email addresses and chat handles, and includes two people who share a surname and two who share a first name. Two matters were generated at each level (Table 1). A validator rejects any name written outside a token; an earlier version of the generator lacked that check, and matters generated without it were discarded.

Table 1. Alias levels and the forms each allows.

3.2 Questions

Each matter has 16 questions answered by a date, an amount, a reference code, or a person or business. At A2 and A3, eight questions are link-dependent: the fact is in a document that mentions the subject only informally (for example a nickname), while the question uses the full name, so answering requires linking the two. The other eight are not. Distractor dates, amounts and codes are included so answers cannot be guessed.

3.3 Automatic scoring

Dates match by day (a year left out is accepted, a different year is not; a time must match if the key has one); amounts by number; codes ignoring case and spacing, with other codes making the answer wrong; and person or business answers must name exactly one entity and the right one, where a form shared by two entities names nobody. The scorer was tested with 937 checks, including that every key scores 100% against itself, that wrong answers of each kind are rejected, and that reasonable formats are accepted. The tests found two gaps before any model run (year-less dates and times of day).

3.4 Name-swapping conditions

Twelve conditions were built: no change; redaction (every name becomes [REDACTED]); numbered placeholders, short codes, and realistic invented names, each with perfect linking taken from the answer key; one unrelated replacement per distinct spelling (a stand-in for tools without coreference); and our pipeline in three name styles. The pipeline asks a model (Claude Haiku 5.5) to list names, then uses our name-linking engine unchanged to link nicknames and initials to people and to build the table. As in real use, names found only in the lawyer's instructions (here the questions) are detected separately and added. Answers are converted back to real names before scoring. The pilot reported in Section 5 uses four conditions: no swap, pipeline with placeholders, pipeline with invented names, and redaction. The name-finding step here calls Anthropic directly and does not apply the Australian-hosting rule that applies to client data, because the data here are invented.

3.5 Models and settings

The swapping pilot used two models: Claude Sonnet 5.5 (Anthropic API) and GLM 5.3 Flash (served by Fireworks). Claude Haiku 5.5 (Anthropic API) was the name finder. Reasoning was switched off and temperature set to 0 where the model accepts them (Haiku 5.5 and Sonnet 5.5 do not accept a temperature setting, so their repeated runs vary; GLM 5.3 Flash reasons regardless). The output limit was 12,000 tokens; an empty reply was retried once. Cost is computed from recorded token counts at list prices.

3.6 Analysis

Accuracy is averaged over repeats within each matter, and the change against no swap is the mean of the eight paired differences with a 95% interval from the t-distribution with seven degrees of freedom. Counts are small and no other significance testing was done.

4. Experiments

E1. Name-swapping pilot: 2 models (Claude Sonnet 5.5, GLM 5.3 Flash), 4 conditions (no swap, pipeline with placeholders, pipeline with invented names, redaction), 8 matters, 2 repeats (128 runs). E2. Name-detector reliability: repeated calls on the same text. The fact questions on unswapped matters were also given to five models; that comparison is not reported here. All experiments used invented data. Generating the matters cost roughly $4 to $5 (estimated).

5. Results: Name Swapping Costs Little Accuracy; Redaction Costs a Great Deal (E1)

Table 2. Fact-question accuracy by condition (16 runs each) and the paired change against no swap over eight matters, in percentage points. *One GLM run produced no answer even after a retry; the cell is 98% over runs that answered.

With our pipeline, accuracy fell by one to two points for both models in the placeholder style and for Sonnet 5.5 in the invented-name style, and every interval includes zero. The loss is concentrated at the higher alias levels (A2 and A3) and is absent at A0 and A1 (Figure 1). Placeholders and invented names did not differ reliably. Redaction lost 21 to 25 points on all eight matters, driven entirely by questions about people (every answer to a "who" question was lost), consistent with the published finding that redaction is worse than entity-preserving replacement [1]. No answer that was right in the swapped names became wrong when converted back.

After the pipeline's name finding was corrected (Section 6), the only real name left in what was sent was a first name in one matter's documents, with placeholders; none was left with invented names. GLM 5.3 Flash wrote no answer on its first try in 5 of 64 runs (8%) and none after a retry in 2 of 64 (3%), including in runs with no swap, so this is a property of the model and not of the swap.

6. A Reliability Fault in Name Detection (E2)

Our first version of the pilot swapped the questions using only the table built from the documents, and appeared to leave a person's name in the swapped text in five of eight matters. Inspection showed the names were in the questions, which the pipeline handles by also detecting names in the lawyer's own instructions and extending the table. When we added that step, the detector returned an empty list for some question sets. Repeated calls on identical text showed the fault clearly (Table 3): on full documents the detector never returned an empty list in 40 calls and found every person and business, whereas on short instruction text it returned an empty list in 10 of 40 calls (6 of 10 on one matter's questions).

Table 3. Claude Haiku 5.5 as the name detector, ten calls per text with the pipeline's prompt (Appendix A.2).

In the pipeline a single call is made and an empty result is treated as "nothing to swap", so the names in such an instruction would be sent unswapped with no error. A retry when the result is empty for text that contains capitalised words that are not sentence starts (up to two more attempts) removed all but one leaked name in the pilot. This change has been made to the pipeline, with unit tests. A second scan of the swapped text for names that are not our own replacements found the one remaining leaked name with no false alarms for placeholders; its first version raised false alarms for invented names and needs a proper test.

7. Discussion

Name swapping looks cheap for fact lookup, but the effect we can resolve is small. No-swap accuracy was already about 98%, so we could detect large losses but not small ones, and the intervals are wide. Earlier work on a longer, messier matter found larger losses, so we do not extrapolate. The consistent pattern is that loss appears with alias level and that the style of replacement does not matter.

Evaluation can find faults in the system under test. The detector fault was found only because the harness was made to match real use more closely, and a first reading of the results (a privacy leak in five of eight matters) was wrong in cause. We kept the superseded runs and recorded the correction.

Swapping matters because of where the model runs. Name swapping is what would make an offshore model usable on client matters, so its reliability, and its cost on messier matters, decide whether offshore models are an option at all.

8. Limitations

• Invented, short, easy documents. Matters are about 1,000 words and the no-swap baseline is near ceiling; real matters are longer and messier.
• Generated by a model. The matters, questions and answer keys were drafted by Claude Sonnet 5.5 and spot-checked on one document only; models from the same lab may be advantaged, and an error in a key would affect all models.
• Small samples. The pilot has 16 runs per cell. Differences of a few points are not firm.
• Two models. Only Claude Sonnet 5.5 and GLM 5.3 Flash were used; other models may lose more or less.
• Replication limits. Haiku 5.5 and Sonnet 5.5 do not accept a temperature setting, so repeated runs vary. The evaluation code and data have not yet been published (Section 10).
• The pipeline in the study differs from a deployed one in the finding step (direct API, no hosting rule) and by the retry we added.

9. Provisional Recommendations, Conclusion and Next Steps

9.1 Provisional recommendations

These follow from the evidence above and carry its limits; they are not a substitute for testing on real, longer matters.

Table 4. Provisional recommendations.

9.2 Conclusion and next steps

On invented Australian matters, name swapping with our pipeline cost about one to two points on fact questions while redaction cost more than twenty, and a retry on empty name detection is needed before such a system is trusted with instructions. Planned next: longer and messier matters, where earlier work found larger losses; the full alias-density grid with more models and repeats; a stronger privacy attacker; a lawyer-scored sample; and a proper test of the pre-send scan for leftover names.

10. Reproducibility and Availability

This section is written for a reader who does not have access to the authors' systems. Where something that a reader needs does not yet exist, it is marked to be added.

10.1 Harness and status of the artefacts

The experiments run on Mike (MikeOSS), the open-source legal AI application from Open Legal Products (github.com/Open-Legal-Products/mike), used as the test harness: its model layer makes the model calls, and our study scripts, scorer and swap conditions run on top of it. The name-linking and swapping engine used in the pipeline condition was written by Foundry Labs and is not part of Mike. Our study code and data have not yet been published, so the results are not yet independently reproducible from a public source. We intend to release the invented matters, the cached detector outputs and the per-run results with a pinned version and an archival identifier: repository to be added, version to be added, archive DOI to be added. Until then, Appendices A to C give the prompts, settings, models and data formats needed to re-implement the experiments.

10.2 Licensing

Mike is released under the GNU Affero General Public Licence v3.0. Study scripts that import its model layer would likely carry AGPL-3.0 terms if released as they stand. The data (invented matters and per-run outputs) contain no third-party or client material and can be released separately; the licence for the data is to be decided (a Creative Commons attribution licence is the usual choice).

10.3 Software and access needed

• Runtime. Node.js 22 or newer (the pilot ran on Node 24; a dependency, undici 8, does not load on Node 20). TypeScript executed with tsx 4.x; tests with vitest 5.0.3.
• Libraries. Vercel AI SDK ai 7.0.127 with @ai-sdk/anthropic ^4.0.65 and @ai-sdk/openai-compatible ^3.0.57; exact versions are in the application's lockfile.
• API keys. An Anthropic key (Claude Haiku 5.5 and Sonnet 5.5) and a Fireworks key (GLM 5.3 Flash). The scorer self-test, the swap tests that do not use the pipeline conditions, and every report script need no network or keys.
• Cost to repeat. A few dollars for the pilot at the prices in Table C1, plus roughly $4 to $5 (estimated; token use was not recorded) to regenerate matters, which will not reproduce the same text (below).

10.4 Determinism and what can be checked

• Matters. People and businesses are generated from seeded pools (seed 1000 + the matter number), but the documents and questions were written by a language model, so they cannot be regenerated identically. The generated matters themselves are therefore the released artefact (Appendix B).
• Swap conditions. Given a matter and the released detector outputs, every condition is deterministic (seeded). The detector output for each matter was cached and is released, so the pipeline conditions can be rebuilt without calling a model.
• Model outputs. Not deterministic. Reasoning was switched off and temperature set to 0 where a provider accepts them, but Claude Haiku 5.5 and Sonnet 5.5 do not accept a temperature setting and GLM 5.3 Flash reasons regardless. Hosted models may also change after the dates of our runs (24 and 25 October 2026); we did not find a published snapshot identifier for the models we used.
• Re-scoring. Every reported accuracy can be recomputed from the released per-run files with the scorer, without calling any model.

10.5 What is still needed before release

1. Publish the study code and data, with a pinned version.
2. Choose the licences for code (AGPL-3.0 if the scripts keep importing the application) and for data.
3. Deposit the data with an archival DOI and record a checksum of the archive.
4. Decouple the scorer, swap conditions and runner from the application's model layer, so that a reader can run them with the AI SDK alone (the pipeline condition would still need our name-linking engine, or a stated re-implementation of its linking rules).

10.6 Steps to repeat the experiments (once released)

1. Install Node 22 or newer, install the released code at its pinned version, and set the two API keys.
2. Run the scorer self-test (937 checks) and the swap test with the pipeline conditions (368 checks).
3. E1: run the swapping pilot (defaults: two models, four conditions, two repeats), then its report script.
4. E2: run the detector reliability test with ten calls per text on matters m04, m05, m06 and m07.

The scripts skip work whose output file already exists, so an interrupted run can be resumed.

References

1. T. Deußer, M. Hahnbück, L. Sparrenberg, T. Uelwer, C. Bauckhage, R. Sifa. On the Impact of Anonymization on the Performance of Large Language Models. arXiv:2609.11335, September 2026. arxiv.org/abs/2609.11335 full text read
2. C. Lothritz, B. Lebichot, K. Allix, S. Ezzini, T. Bissyandé, J. Klein, A. Boytsov, C. Lefebvre, A. Goujon. Evaluating the Impact of Text De-Identification on Downstream NLP Tasks. NoDaLiDa 2023, pp. 10–16. aclanthology.org/2023.nodalida-1.2 abstract page read
3. J. Serenari, S. Lee. Semantically-Aware LLM Agent to Enhance Privacy in Conversational AI Services (LOPSIDED). arXiv:2510.27016, IEEE Big Data 2025. arxiv.org/abs/2510.27016 abstract read
4. S. Hou, R. Shang, Z. Long, X. Fu, Y. Chen. A General Pseudonymization Framework for Cloud-Based LLMs. arXiv:2502.15233, February 2025. arxiv.org/abs/2502.15233 abstract read
5. PSILENCE: A Pseudonymization Tool for International Law. CALD-pseudo 2024. aclanthology.org/2024.caldpseudo-1.4 search summary only
6. CON-QA: contract question answering with local anonymisation. arXiv:2509.19925. arxiv.org/html/2509.19925v1 search summary only

Tags show how closely each source was read for this draft; sources marked "search summary only" should be read before they are relied on.

Appendix A. Prompts

Verbatim, with {placeholders} for inserted text. All experiments used these prompts exactly.

A.1 Fact questions (E1)

System:

You answer questions about legal documents. Use only the documents provided. Reply with JSON only.

User:

DOCUMENTS

{the eight documents, each headed "=== DOCUMENT n ==="}

QUESTIONS
{F1}: {question}
{F2}: {question}
...

Answer every question from the documents. Reply with ONLY a JSON object mapping each question id to a short answer. For dates write the date as in the documents. For amounts write the amount with its $ sign. For reference codes write the code. For "Who" questions write the person's or business's name as written in the documents. Example: {"F1": "12 March 2024", "F2": "$1,200", "F3": "INC-1042", "F4": "Ms Smith"}

A.2 Name detector (E1, E2)

System prompt of the pipeline's name finder, used unchanged; the user message is the text to scan.

You find the details in a legal document that identify a private person or business: names of people, names of businesses and organisations (not courts, government bodies or well-known public companies), street addresses, email addresses, phone numbers and identifiers (ABN, ACN, TFN, booking or reference numbers). Do not list places such as cities, states or countries, courts, statutes, job titles or generic words. Also list the names of vessels, vehicles and properties. List every one, exactly as written, including each short form the document uses for the same person or business (a surname alone, a first name alone, a nickname, initials such as "J.P.N.", a short business name). For a nickname, short form or initials, add "same_as": the full name, exactly as you list it, of the person or business it stands for, when this text makes that clear (for example "Marguerite Pemberton (Maggie)"); leave it out when unsure. Answer with one JSON object and nothing else: {"entities": [{"kind": "person|org|address|email|phone|id", "value": "...", "same_as": "..."}]}.

A.3 Matter generator

One call to Claude Sonnet 5.5 per matter (repair attempts send the best draft back with the validator's complaints). The user message lists the invented cast with every name form each entity may use at the matter's alias level, states the premise, and requires: eight short documents of fixed types (agreement, email thread, chat log, sworn statement, claim letter, record, internal note, file note or reply letter); every name written only as a token {{ID:form}} in the listed forms, never as plain text; sixteen facts, each a question with one short answer of type date, amount, id or entity, with distractors; at A2 and A3 eight link-dependent facts whose document never uses the subject's full form; and the JSON output format. The full text, and the validator rules (token forms, plain names, answer text in document, one entity per question, question style), will be released with the matters (Appendix B).

Appendix B. Data to be released

Table B1. Planned data release. Sizes and checksum to be added when the archive is created.

Appendix C. Models, access and settings

Table C1. Models used in this report. Prices are the list prices used to compute every cost here, from the application's price table.

Settings. One user message and a system message per call; reasoning: "none" and temperature: 0 requested, and dropped for a model that refuses either; up to three SDK retries on transient errors; one further call if the reply was empty. Output limits: 12,000 tokens (E1), 6,000 (name finder). Concurrency 4. The reported runs took place on 24 and 25 October 2026; earlier matter sets, since discarded, were generated on 22 and 23 October.