If you don't work in security, here is the short version.

Some organizations send working documents, here a security team's incident tickets, to outside AI services: the large language models (LLMs) behind chatbots. Such documents carry personal details, which privacy people call PII (personally identifiable information): names, ID numbers, bank details, addresses. One fix is to swap each detail for a placeholder such as [PERSON_1] before the text leaves, keep the key to the swap in a locked local file (the "vault"), and put the real details back into the AI's answer when it returns. Masking on the way out, restoration on the way back: a round trip.

pii-roundtrip

is a free, open-source tool that does this for UK English and Russian documents. Its author, Vladimir Beskorovainyi, published it in 2026 with a benchmark paper reporting leak rates below 1% 1, 2. Those rates were measured on test documents made by the same program that made the documents the tool was developed on. This essay asks whether they hold on documents from a different source and sets out how to test that. Nothing has been run yet.

Why it matters to you: the details in documents like these belong to ordinary people, and the question is whether they stay behind.

This essay proposes an independent-generator evaluation of pii-roundtrip v0.5.0. None of it has been built or run yet: there is no second generator, no corpus and no results. What follows is the plan, and a reading of the numbers that already exist. The objective would be to measure pii-roundtrip masking fidelity and round-trip restoration accuracy on an independently generated synthetic corpus of 600 documents, distinct from the engine- development and held-out document sets of the original research. The method would freeze the pii-roundtrip v0.5.0 release and patched versions, run baselines (existing masking tools, for comparison) including pii-guard RU, GLiNER Guard Omni UK and prespecified Presidio configurations with identical wrappers, and use the released leak scorer for comparability. Prior work measured round-trip restoration on held-out data from the same generator, reporting sub-1% leak claims, but this evaluation would test generalization under structural distribution shift. The legal context (the UK GDPR, Britain's data-protection law) distinguishes an Article 6 lawful basis for personal-data processing from the additional Article 9 condition required when special-category data is processed, noting that leak rate is a risk-assessment input rather than a statutory threshold. The proposal specifies decision rules for testing whether the benchmark's sub-1% leakage results transfer to independently generated SOC-style synthetic documents (made-up documents shaped like the incident tickets of a security operations center, or SOC); production-ticket generalization would require a separate evaluation.

• Status: A plan, not a result. Nothing here has been built or run; every number below comes from the original author's benchmark 1, 3.
• The Benchmark Trap: The engine's 0.2% UK and 0.3% RU document-leak estimates come from its development generator's held-out Set 7 (a batch of test documents kept

back from development, made by the same program as the development data). The external 0.7% RU and 4.0% UK figures describe baseline tools, not the engine 1, 3.

• The Restoration Reality: Russian name restoration (94.6% accuracy) fails mainly through deterministic morphological inflection in pii-guard, not LLM hallucination corrupting the mapping (in plain terms: the tool puts a name back with the wrong grammatical ending, since Russian names change form with their role in a sentence, rather than the AI making things up and scrambling which placeholder stands for whom). That is the benchmark author's own failure audit, not yet replicated 1, 3.
• The Compliance Gap: No UK GDPR leak-rate threshold exists; reversibly pseudonymized data remains personal data when keys are retained (Recital 26), requiring risk-appropriate security under Article 32 8.
• The Evaluation Plan: We propose 300 documents per language as a pilot, not a powered confirmation of sub-1% leakage. Even zero leaks gives a two-sided 95% Wilson upper limit of about 1.26% (a standard ceiling on what the true rate could be, given the sample size); a confirmatory comparison requires a power analysis and uncertainty for both corpora.

• SOC Analysts: Whether a number is a document-level leak rate or a span-level masking precision decides how you read a summary claim of a 99.8% non-leak rate 2, 1.
• Compliance Officers: UK GDPR guidance sets no numerical leak-rate threshold, so no specific percentage guarantees lawful processing 8.
• Security Engineers: The author's failure audit traces Russian name restoration failures mainly to deterministic inflection logic rather than LLM hallucination, which tells you where remediation effort belongs 1, 3.
• Researchers: Generator transfer effects and real-world performance are different things, and a synthetic evaluation corpus has to be designed knowing which one it measures 9, 1.

Summary claims about pii-roundtrip masking fidelity tend to blur one distinction: held- out synthetic documents versus external datasets. The reported sub-1% engine leak estimates come from Set 7. The original acceptance thresholds were at most 2% document leakage and at least 99% restoration; this proposal's 1% leakage target is a new, stricter criterion. On held-out Set 7, the baselines recorded 60.8% leak (RU pii-guard ) and 58.3% leak (UK GLiNER ) 1. Those figures describe the tools on data generated by the same implementation that produced the training set or the specific synthetic set used for evaluation.

The external datasets tell a different story from the held-out Set 7 figures. RU pii-guard showed 0.7% leak on the hivetrace dataset, and UK GLiNER showed 4.0% leak on theNemotron-PII dataset1. The gap shows how sensitive the masking tools are to the distribution of their input. GLiNER Guard Omni recorded 99.6% leak on Russian Set 7, but 58.3% on UK Set 7 1. The 99% figure in the project's documentation is a different animal: a restoration threshold, not a leak rate. The "99% restoration threshold" describes how accurately masked placeholders are replaced with original values during the round trip, not the percentage of PII successfully hidden during the initial masking phase 1.

Confirming these numbers means checking the paper's Section 3.3 (External datasets) and Section 4 (Held-out results). The paper discloses that the engine's checksum rules and the generator share one implementation, and that 2,696 of 3,249 (83%) UK external identifiers in Nemotron-PII were injected by the author's generator 1. So even the "external" datasets may share structural artifacts with the development generator, which complicates any assessment of true generalization 1. External, here, may not mean what you think it means. To separate the specific benchmark numbers from general performance claims, run bench/eval/leak_breakdown.py on Set 7 against pinned v0.5.0 code to check whether it reproduces the 0.2% and 0.3% rates 1, 6.

Then there's the 99.8% that turns up in summaries of the tool's results. Two different metrics happen to round to 99.8%: the UK document-level non-leak rate, corresponding to 998 of 1,000 documents, and the UK fragment-level masking precision. The documentation labels the document count explicitly; readers should not equate it with masking precision 1, 2. The document-level count corresponds to the 0.2% document leak rate on held-out Set 7, whereas fragment-level masking precision is the fraction of predicted masking spans that overlap any annotated PII (99.8% for UK English; 98.4% for Russian); it does not measure how much non-PII text each span also covers 1, 2. Same digits, different denominators. Blending them obscures how the tool performs on individual entity instances, so keep document-level success rates and span-level precision apart when reading the risk profile 1, 2.

The "Benchmark Trap" is the habit of relying on held-out data that shares the distributional properties of the training data. In machine learning, generalization error is generally expected to increase when the test distribution shifts away from the development distribution. The engine was developed over six earlier held-out sets from the same generator; the baseline tools were never tuned on that generator. If a production environment uses a different document structure, the leak rate may rise significantly above the sub-1% benchmark results. We hypothesize, without supporting data, that the baseline tools (pii-guard for Russian, GLiNER Guard Omni for UK English, plus any prespecified Presidio configurations) could show ≥ 40% leak on an independent corpus if structural features (headers, JSON) differ. Mind the direction, though: the external-dataset figures (0.7% RU, 4.0% UK) are lower than the Set 7 baseline figures (60.8% RU, 58.3% UK) 1. The ≥ 40% baseline prediction for the independent corpus is the essay's own hypothesis, not derived from these external results.

Testing generalization under structural distribution shift needs a test set that mimics incident tickets without sharing checksums, morphology rules, or generator logic with pii-roundtrip. We propose 600 synthetic documents, 300 UK English and 300 Russian, from a separately implemented generator. Neither the generator nor the corpus exists yet. The aim is to simulate the structural distribution of production incident tickets with no direct lineage to the generator behind the original benchmark data.

We propose a two-sample classifier test on prespecified structural features rather than t- SNE visualizations. Classifier AUC (a score from 0.5, a coin toss, to 1, perfect, for how well a program tells two sets of documents apart) puts a number on distributional difference; t-SNE is primarily a visualization technique and may not reliably quantify shift. An AUC above 0.8 would meet our chosen separability target: a classifier can tell the original generator's documents from the independent corpus. It would not establish that the differences are causal, practically relevant, or adequately isolated, and 0.8 is a working diagnostic threshold to be justified before the run, not a universal significance gate.

TAB (Text Anonymization Benchmark) is an annotated corpus of European Court of Human Rights judgments 9. Seeding from it, or from external corpora like it, should in our view be preceded by a privacy audit, because those published judgments contain real personal information annotated for masking 9. Real PII in testing needs strict controls against accidental exposure or misuse. The corpus generation script must be validated as not sharing checksum logic with the author's generator, and the evaluators must check that independence. A hand-audit of 50 documents per language is planned, to check for missed PII spans and correct entity density.

The proposed corpus is synthetic and must not be called real incident tickets. Synthetic generators may not capture the full complexity of human-written incident reports; the goal is to mimic the structural distribution of such tickets without the legal and privacy risks that come with real data. We will audit generator provenance and implementation independence while retaining valid identifier standards and grammatical forms. To isolate structural effects, we will also compare paired layouts with entity values, density, and noise held constant. A second generator on its own changes too much at once: entity prevalence, identifier formatting, Russian grammatical case, document length, noise, JSON serialization, token truncation, and LLM competence can all move together. So the paired comparison renders the same fictional entities and annotations twice, once in an original- style layout and once in an independent SOC-style layout, with values, density, and processing settings fixed, and measures the paired leak-rate difference. Separate strata then vary entity density, morphology, document length, and noise. Generator seeds and software hashes would be recorded. Verification would include inspecting generator provenance to establish implementation independence and independently validating Russian name forms; the paired structural comparison keeps the same valid entity forms in both layouts.

Once generator B exists, a structural layout comparison between it and the original generator would document the shift. Separability would be evaluated on held-out documents, with related template or seed groups kept from crossing classifier training and test splits. AUC above 0.8 meets our chosen diagnostic target; AUC below 0.8 means only that this classifier and feature set did not meet it. Neither result predicts the direction of leakage change. That's the paired masking experiment's job.

Synthetic evaluation data sits in a different legal context from production data. UK GDPR Article 9 requires conditions for processing special category data, and leak rate informs the risk assessment under Article 32 8. Processing wholly fictional or effectively anonymous evaluation data does not require an Article 6 lawful basis for that data. The test is whether it relates to identified or identifiable living people, including through reasonably likely linkage or inference—not merely whether a reversible mapping exists. Processing real seed or audit data still requires an applicable lawful basis and safeguards. Reversibly pseudonymized data, however, remains personal data for the key-holder (Recital 26) 8, 2. For wholly fictional data that cannot be related to identifiable people, production-like vault controls are an operational choice rather than a GDPR obligation. Any real personal data used to seed or audit the corpus requires its own safeguards.

The question for this section: does masking leakage on the independent corpus rise significantly above the engine's sub-1% Set 7 results? Those are measured results, not the paper's acceptance criterion, which was at most 2% document leakage; this proposal's 1% target is new and stricter. Our untested hypothesis is that the independent corpus increases the underlying document-leak probability: elevated leak, not a fixed rate above 50%. We will estimate the rate difference and its uncertainty rather than read a high upper bound as observed degradation. The comparison point is the historical benchmark of 0.2% (UK) and 0.3% (RU) on held-out Set 7 1.

Once there are results, a bar chart would show the gap between the benchmark and the new evaluation, with 95% CI error bars to help judge whether any increase in leak rate is statistically significant. The decision turns on the confidence bounds, not the point estimate. An upper confidence bound at or above 1% means the experiment has not established leakage below 1%. An interval spanning 1% is inconclusive; evidence of exceeding the target requires the lower bound to exceed 1%. (Yes, a perfect run can land in that middle ground too; the zero-leak arithmetic in the TL;DR says so.) Getting the upper bound below 1% with zero leaks takes about 381 independent documents per language, and shared templates or seeds shrink the effective sample further. So the 300-document run is a pilot: it finds failure modes and verifies the scoring harness. Configurations, features, and endpoints would then be frozen before a separate confirmatory corpus is scored. Layouts picked because they failed in the pilot would not be reused to estimate the generalization penalty; that would be afterwit passed off as foresight.

The evaluation script bench/eval/leak_breakdown.py must be run on the new corpus with the released scorer, which keeps the method consistent with the original. Pin the v0.5.0 source and the resolved runtime settings rather than relying on current main or later patches, and document how the leak_breakdown.py entry point relates to the scoring module cited here 6. The released scorer detects specified annotated value leaks using type-specific matching rules, not every possible disclosure. Report its benchmark- compatible score separately from extended SOC coverage, including IP addresses, hostnames, and usernames. The leak-breakdown script reads saved detector predictions and corpus annotations to calculate leakage; running detectors, generating LLM answers, and scoring restoration are separate stages.

Knowing the tool's limits starts with pinning down what causes its restoration errors. Russian name restoration (94.6% accuracy) fails mainly through deterministic morphological inflection, not LLM hallucination corrupting the mapping 1, 3. The mechanics are simple enough. The LLM outputs placeholders (e.g., [PERSON_1]), the local restorer matches them to the vault mapping, and the failure occurs when it applies the wrong morphological inflection while re-inserting the value 1. The benchmark author's manual failure audit flagged 199 occurrences of correctly stored names restored incorrectly: 188 involved pii-guard inflection and 11 the engine's proxy-name handling. pii-guard is on both sides of this comparison: it is one of the baselines, and the engine passes stored names to its inflector 3. A further 82 involved incorrectly stored names, and 8 were evaluator errors 3. That is the author's diagnosis; independent replication remains necessary 1, 3.

Deterministic means predictable and reproducible, given the same input and logic. The restorer is deterministic for a given input and vault state 5, so restoration accuracy should hold steady across runs unless the vault or input text changes. It may still vary when generated answers, sampled documents, configuration, or session state change, and that variability alone does not establish hallucination.

Replicating the author's error breakdown (inflection vs. storage vs. evaluator errors) would show which failure mode to fix first. If inflection errors dominate, the fix lives in the restorer's morphological logic. If incorrectly stored name forms dominate, investigate name parsing, normalization, and session-mapping construction before vault persistence. If evaluator errors dominate, the fix lives in the metric calculation.

A critical security boundary: in the intended deployment, the remote model has no direct access to the local vault 1. The mapping is kept in the local session and used only by the restorer; the LLM processes the masked text without the key-value pairs 5. Recognized placeholders in model-controlled output can nevertheless cause the local restorer to reproduce mapped values, and whether that becomes unauthorized disclosure depends on the surrounding workflow and output boundaries. Nor does the boundary establish end-to- end immunity to disclosure: the LLM still sees surrounding context, and residual direct or indirect identifiers can support inference or re-identification even without access to the vault 1, 10. Four events need keeping apart: a masking failure before the document reaches the external model; a restoration-integrity error on the way back; contextual inference from indirect identifiers left unmasked; and an actual unauthorized release of restored values. A mismatched Russian surname ending is not automatically a breach, and a scorer that detects whole-value leaks does not measure every privacy failure.

An attack surface remains on the restoration side: the LLM outputs placeholders in new contexts that the local restorer fills with real PII 1. If the LLM emits a placeholder [PERSON_1] in a context that suggests a different entity than the one mapped in the vault, the local restorer may insert the wrong PII value 1. Substitution occurs locally, but the security analysis must cover attacker influence over model output, restoration behavior, and downstream recipients. Local restoration does not itself send the recovered values back to the remote model.

Whether the tool is "really sure" about its restoration accuracy depends on verifying those restoration-side attack surfaces. A successful restoration-side disclosure attack requires an end-to-end demonstration of attacker control, recognized placeholders, local restoration, and an unauthorized downstream recipient. Mapping substitution alone does not establish exfiltration or contextual re-identification. So a restorer error does not automatically establish a personal-data breach; however, a breach of security leading to accidental or unlawful destruction, loss, alteration, or unauthorized access can also qualify without disclosure.

Confirming the mapping lookup mechanism requires a code audit of pii_roundtrip/core.py5. The v0.5.0 pii_roundtrip/vault.py implementation uses AES-256-GCM with scrypt-derived keys (standard strong encryption, with the key derived from a password) and explicitly limits this protection to data at rest; sessions and restored text remain plaintext in memory. The stored vault is encrypted, but the PII exists in plaintext memory during restoration 5. That is by design, so testing the hypothesis that local user-space memory scraping causes leaks is orthogonal to the document-generator distribution shift.

Compliance framing has to keep risk inputs and statutory thresholds apart. UK GDPR Article 9 requires conditions for processing special category data, not a specific leak rate 8. Ordinary technical incident records are not inherently special-category data; Article 9 applies when such data actually appears. Leak rate informs the risk assessment under Article 32, but no numerical threshold exists in the regulation or guidance 8. A measured 5% leak rate is not a statutory pass/fail line. It may indicate that security measures are not appropriate to the risk under Article 32 8. The ICO guidance (from the Information Commissioner's Office, the UK body that upholds information rights and data privacy) distinguishes lawful basis, special-category conditions, and risk-appropriate security 8.

Reversibly pseudonymized data remains personal data for the key-holder (UK GDPR Recital 26; ICO pseudonymisation guidance). Masked or not, the entity holding the keys is considered to be processing personal data 8. Article 32 security controls therefore apply to the whole lifecycle, key management included 5. The keys must be stored securely, with access restricted to authorized personnel 5.

For this proposed SOC evaluation, we define utility to include chronology and correct entity-to-account associations, measured separately from leakage and placeholder restoration. A tool that removes all PII but destroys the relationships between entities may be useless for operational purposes. In incident response, knowing that two entities are related matters as much as knowing who they are. pii-roundtrip aims to preserve that utility by allowing restoration of the original values 1. If restoration accuracy is low (e.g., 94.6% for Russian names), the utility is compromised 1.

Once measured, the decision to deploy or not would weigh leak rate against risk tolerance. If the leak rate exceeds the risk tolerance, the tool should not be deployed, or additional controls should be added. If restoration accuracy falls below the operational requirement, the tool should not be deployed for use cases requiring high fidelity. Deployment criteria must account for risks to individuals' rights and freedoms, data sensitivity, processing context, and mandatory safeguards; organizational risk appetite cannot override those obligations.

We recommend consulting legal counsel on Article 32 measures versus benchmark pass rates 8; counsel can interpret the risk assessment in the context of a specific deployment 8. Benchmark pass rates are technical metrics, not legal compliance certifications 8. A technical pass does not guarantee legal compliance, and a technical fail does not guarantee legal violation 8. The risk assessment sits between the two 8.

Operational utility also depends on which identifier types are supported. IP addresses, hostnames, or usernames may be supported by the tool, but this must be verified 1. The checked Russian adapter maps IP types, so IP addresses are not universally unsupported 1. Benchmark coverage is not the same thing as implementation support: a tool may claim to support certain types and still fail to mask them in practice.

RAT-Bench is a re-identification-risk benchmark for text anonymization tools, evaluating tools on synthetic text with direct and indirect identifiers across domains and languages 10. It provides general context for anonymizers but does not measure these tools on this corpus. Performance on the independent corpus has to be measured empirically. In 1, the baselines leaked far less on external datasets than on the engine's Set 7, so the direction of shift effects is tool- and corpus-specific; 1 does not by itself predict that the engine's leak rate will rise on the independent corpus.

1. H1 (Leak Rate):pii-roundtrip 's benchmark-defined document-leak rate is higher on the independent corpus than on held-out Set 7.
   • Decision rule: Support H1 if the lower 95% confidence bound of the rate difference exceeds zero. Report the independent-corpus rate against this proposal's 1% target separately: below 1% only if the upper bound is below 1%, above 1% only if the lower bound exceeds 1%, otherwise inconclusive. Prespecify sample size through power analysis.
2. H2 (Restoration): Russian name restoration accuracy is ≥ 95% on independent corpus (v0.5.0 and patched releases scored separately).
   • Decision rule: Find evidence against accuracy of at least 95% if the upper 95% confidence bound is below 95%; support the target if the lower bound is at least 95%; otherwise report inconclusive. Classify error causes separately.
3. H3 (Baselines): On the same independent-corpus documents, each prespecifiedPresidio and GLiNER configuration leaks more than pii-roundtrip. The ≥ 40% level is a secondary, untested prediction.
   • Decision rule: Support H3 for a configuration if the lower 95% confidence bound of its paired leak-rate difference against pii-roundtrip exceeds zero. Assess the 40% level as H1 assesses the 1% target: a lower bound above 40% supports it, an upper bound below 40% counts against it, otherwise inconclusive. Account for multiple configuration comparisons.
4. H4 (Attack): Attacker-influenced LLM responses cause local restorer to emit real PII in new contexts.
   • Protocol: Fictional identities and controlled recipients only. Count placeholders as in How to, below. Call an event exfiltration only with demonstrated attacker influence, restoration behavior, and an unauthorized destination.
   • Not demonstrated by: Zero successes in 1,000 trials. For independent trials from a fixed attack distribution, this gives a one-sided 95% upper success-probability bound of about 0.30%, not proof that no successful attack exists. An earlier memory-scraping hypothesis, local user-space scraping of the vault, is dropped as orthogonal to document-generator distribution shift; see §4.

Evaluation Harness Logic (Pseudocode only)

# Evaluation Harness Logic (Pseudocode only)
# Purpose: Measure leakage and restoration accuracy on independent corpus
# Interfaces are illustrative, not pii-roundtrip's API; every intermediate artifact is saved per document.

def run_evaluation(corpus, mask_tool, restore_func, local_llm):
    results = []
    for doc in corpus:
        # 1. Ingest Ground Truth
        ground_truth_spans = doc.get_pii_spans() # Known PII locations
        clean_text = doc.raw_text

        # 2. Masking Phase
        masked_doc = mask_tool.predict(clean_text)
        # Check for leakage immediately using released scorer
        leaked_spans = check_leakage(masked_doc, ground_truth_spans)

        # 3. LLM Simulation (External Service)
        llm_response = local_llm.generate(masked_doc.text) # Illustrative text-only field; keep mapping and session state local

        # 4. Restoration Phase (Local Vault)
        final_doc = restore_func(llm_response, masked_doc.mapping)

        # 5. Measure Integrity
        # Check if placeholders were replaced with original values in correct role
        # Score issued-placeholder restoration against local session state.
        restoration = score_restoration(final_doc, masked_doc.mapping, ground_truth_spans)
        # Evaluate role fidelity and intentional omissions with a separate task-specific rubric.

        # 6. Capture Intermediate Artifacts for Attack Analysis
        # Record if LLM emitted unissued placeholder IDs

        results.append({
            "leak_count": len(leaked_spans),
            "restoration_errors": restoration.errors,
            "llm_placeholder_inventions": count_new_placeholders(llm_response)
        })

    # Retain document, template/seed-group, layout-pair, and configuration IDs in results.
    # Report ordinary Wilson intervals for benchmark comparability separately from
    # prespecified cluster-aware intervals and paired rate-difference intervals.
    return aggregate_metrics(results, analysis_plan=prespecified_analysis_plan)

• Do: Report ordinary document-level Wilson intervals for benchmark comparability. Separately prespecify dependence-aware inference for document leakage, span metrics, restoration, and paired comparisons wherever the sampling design induces clustering, including treatment of zero failures.
• Do: Verify runtime config (max_length, stride) matches model card (512/128) or document divergence 7.
• Do: Record separately, per document: the unmasked original, masking predictions, the sanitized outbound document, the model response before restoration, the restoration output, scorer findings, and an immutable configuration manifest. Specify how span precision, recall, document-level leakage, role fidelity, and restoration correctness are each calculated.
• Do: Count placeholders the masker never issued separately from recognized placeholders reused in unexpected contexts. Unknown placeholders are generally left unresolved; any disclosure test must also establish an unauthorized recipient of restored output.
• Don't: Conflate document non-leak count (998/1000) with masking precision (span- level) 1, 2. Low document leakage can coexist with poor precision when the tool over- masks non-PII text; missed PII is a recall problem; and a document leaks when any one value meets the leak criterion 6.
• Don't: Use t-SNE for distribution shift; use classifier AUC or MMD.

• "The tool is 99.8% safe."
  • Check: Look for the denominator. Is it document-level (0.2% leak) or span-level (precision)? Is it same-generator or external? 1, 2.
• "Real incident tickets were used."
  • Check: Verify if documents are from the author's generator (Set 7) or external public datasets (hivetrace, Nemotron-PII). 1.
• "Russian names fail because of LLM hallucination."
  • Check: Read REPORT.en.md failure breakdown (188 of 199 stored-name restoration failures are inflection errors, alongside 82 storage errors and 8 evaluator errors). 1, 3.
• "5% leak rate violates UK GDPR."
  • Check: Search ICO guidance for numerical thresholds (none exist). Check Article 6 basis and Article 9 conditions. 8.
• "Baseline tools work on external data."
  • Check: Verify if baselines were run on Set 7 (same-gen) or external sets. 58%/61% are Set 7 numbers. 1.
• "IP addresses are unsupported."
  • Check: Inspect pii_roundtrip/adapters/pii_guard.py (RU adapter maps IP types). 1.

• Production Reality: The plan uses a second synthetic generator, not lawfully sampled production tickets. Its results would show generator transfer, not real-world performance 1, 9.
• Runtime Config: The pinned pii-guard dependency defines ner_max_length=512 and ner_stride=128. The v0.5.0 adapter does not explicitly override them, but environment or YAML settings can; record the resolved configuration for every run.
• Version Drift: Version 0.5.1 (release date confirmed as October 25, 2026 from CHANGELOG), which postdates the v0.5.0 results in the paper and is followed by further patches up to at least 0.5.4, fixes the Russian name errors found on held-out Set 7; we must compare v0.5.0 against patched releases separately 1, 4.
• Legal Interpretation: We analyze compliance framing (Article 9 conditions), but cannot provide binding legal advice on specific deployments 8.
• LLM Model: Local LLMs (the original benchmark used qwen3.5:9b and gemma4:12b; the models used here must be named and pinned) may not perfectly simulate external services; competence in Russian is a confound 1.

That's the whole plan. Now somebody has to build generator B.