If this isn't your field, here is the short version.

Companies that build large AI models sell access to them over the internet. A rival can ask such a model a great many questions, keep the answers, and train a cheaper model of its own to imitate them. That is distillation: the copy is the student, the original the teacher. Providers have started watermarking what their models write, building in a hidden statistical pattern tied to a secret key, so that a student trained on that text carries a trace of it 1.

WM-Basin is a detector for that trace, published in October 2026 by researchers at Meta's FAIR and Meta Superintelligence Labs 1. It looks only at what a suspect model writes, never inside it. This essay reads the paper closely: where the detector works, the cheap step that hides the trace from it (a little more training on clean examples), what a trustworthy test costs, and what a positive result can and can't prove.

Why it matters outside the lab: a result like this is the kind of evidence one company would bring against another in a dispute over copying, and the later sections say what it would and wouldn't establish.

Meta's WM-Basin detector 1 tries to answer a narrow question from outside a suspect model: was this model trained on my watermarked outputs? It doesn't read the suspect's weights (the numbers, learned in training, that make up a model). It maps the suspect's text into gradients (measurements of how a model would have to change to fit that text) on a small probe model the verifier owns, removes the directions any fine-tune would move, and ranks the claimed watermark key against decoy keys (other keys the provider never used, kept as a comparison) 1. On math distillation it beats the token-level baseline (the older approach, which looks for the watermark in the words themselves) by a wide margin1. Two epochs of clean supervised fine-tuning on code and science traces (two more passes of ordinary training on worked examples that carry no watermark) take it to zero detections at a 0.1% false-positive rate (a setting at which it wrongly flags about one innocent model in a thousand), while the student gains GSM8K accuracy (a standard set of grade-school math word problems) 1. The paper's own appendix says the evidence that the watermark "survives" that fine-tune is a read of memorized training traces, not of the watermark in the weights 1. That gap, between a watermark that persists and one an auditor can still see, is what this essay is about, along with the price of a 0.1% false-positive floor and what a positive result can and can't prove. The replication study at the end is a plan. None of it has been run.

• Clean fine-tuning hides the signal. Two epochs of continued SFT (supervised fine- tuning) on 1,900 code and science traces leave WM-Basin at 0% TPR at 0.1% FPR in all three n-gram windows (Table 7) (TPR, the true-positive rate, is the share of copied models it catches), and GSM8K goes up, not down 1.
• Compression doesn't. (Compression means shrinking a model so it's cheaper to run: int8, int4 and nf4 store its numbers at lower precision; pruning cuts some of them out.)

Int8 and 10% pruning leave a usable model and cost detection at most three points. Int4 and nf4 were tested and leave an unusable one 1.

• The probe matters less than I guessed. Swapping a Qwen probe for a Llama one (Qwen and Llama are two families of AI models) costs little; vocabulary distance costs more (Table 14) 1.
• A 0.1% floor costs 999 decoys, about 450 GPU-hours (a GPU-hour is one GPU, the chip AI training runs on, working for an hour) of one-time bank building on the paper's hardware 1. That alone nearly fills the three single-GPU weeks this study was planned around.
• A hit is evidence of training on keyed text, not proof of theft.

• Model operators: you can audit a competitor's API (the service through which outside software sends a model questions) for your watermark without its weights. Know which fine-tunes erase what you can see.
• Counsel: a significant result says the suspect learned from text carrying your key. How the text got there, and whether anyone owned it, are separate questions.
• Researchers: the paper's main text and its appendix draw different conclusions from the same column. Somebody should settle it with held-out prompts.
• Auditors: the worst key in a 1,000-key bank fires at roughly five times its nominal false-positive rate (Table 11). Calibrate the key you deploy, not the average key.

Drafts here go through the Moot before I read them: a panel of frontier models I call by their lab bosses' first names (Dario for Anthropic's, Sam for OpenAI's, Sundar for Google's), plus Eddie, the house model that writes the first draft and then argues with all of them. The names are a bit. None of those gentlemen work here. Where a justice's ruling matters below, it's named. After publication, three outside readers (Claude, OpenAI and Gemini, without the nicknames) sent critiques, and this version is the one that came back from them.

API distillation is cheap. Query a strong teacher, keep the answers, fine-tune a student on them, and sell the student. Providers have started watermarking their outputs so that a student trained on them carries a statistical trace, which the literature calls radioactivity 1.

The trouble is reading it. Existing black-box detectors look for the watermark in the suspect's sampled tokens, and token-level signals fade, for example when the distillation traces were sampled at low temperature 1. WM-Basin's pitch is to stop reading tokens and read geometry instead. At a strict 0.1% false-positive rate it flags 79% of distilled models at n-gram=1 and 45% at n-gram=3, against 29% and 1.3% for the token-level baseline, across 24 student architectures from 135M to 8.2B parameters 1.

Those are real numbers, and the paper is unusually candid about where they stop. The candour is mostly in the appendices, though, and the abstract's robustness list (quantization, pruning, reinforcement-learning post-training) doesn't name the attack that works.

Start with what the watermark does to a student. Fix a key. Fine-tune one copy of a base model on teacher traces watermarked under that key and another on unwatermarked traces from the same teacher, prompts and recipe 1. Both learn the task. Subtract the part of the watermarked update that the clean update explains, and what's left, the watermark direction, is the part only the key can account for 1.

The paper then walks the trained student's weights along two axes and measures, with white-box access, how detectable the watermark stays (Fig. 3). Along the task direction, it stays detectable over a broad range. Along the watermark direction, a few small steps drop it to the detector's floor 1. That anisotropy is the basin: wide one way, narrow the other. The width is why ordinary post-training can change what the model does without erasing the key. The narrowness is why the signal is key-specific, and why the test compares against decoy keys rather than thresholding a generic score 1.

So the claim isn't that the student memorized watermarked sentences. It's that training on them left a key-specific dent in parameter space, roughly orthogonal to everything the student learned about math. Keep that distinction; the kill condition turns on it.

The verifier never touches the suspect's parameters. WM-Basin runs in three stages 1.

Text to gradients. The verifier takes a small clean model, by default Qwen2.5-3B, warms a LoRA adapter (a small add-on, trained in place of the whole model) on about 450 unwatermarked teacher traces, and freezes it. Every prompt and completion becomes the gradient of the probe's log-likelihood with respect to that adapter, and the gradients are averaged over a set of responses 1. The suspect is read on 1,900 test prompts, about 10M sampled tokens of which about 1M are scored (Table 8) 1.

Remove the task. Public clean checkpoints answer held-out build prompts. A principal- component fit on their gradients, 100 components, estimates the directions any fine-tune on this domain would move. Those directions are projected out of every gradient 1.

Rank the key. The teacher, which the verifier owns, generates on the build prompts under the claimed key and under M decoy keys. Each set's residual gradient becomes a reference direction. The suspect's residual is scored against each by cosine similarity, and the p-value is the claimed key's rank: one plus the number of decoys scoring at least as high, over M+11.

Build and test prompts stay disjoint, and neither overlaps the adversary's prompts. That isn't fussiness: reusing prompts inflates the score 1.

The probe needs its warm-up. Without it, the bare base model detects almost nothing at 0.1% FPR, and a randomly initialized adapter keeps about a third of the default's power (Table 15) 1. Nearly every other warm-up detail barely matters.

Dario and Sam both ruled that a verifier-side probe doesn't break the black-box claim, and Eddie agreed: black-box means no access to the suspect's weights or logits, not that any probe works on any suspect. The paper now answers the follow-up, which v1 of this essay could only guess at.

Table 14 holds 100 distilled suspects fixed and swaps the probe across ten checkpoints 1. At n-gram=1, every probe catches every suspect. At n-gram=3 and 0.1% FPR, the five Qwen2.5 probes span 69 to 89% and the three Llama-3 probes 67 to 72%, despite sharing no tokenizer with the student. What hurts is vocabulary distance: Gemma-2 (256k vocabulary) drops to 57%, and SmolLM2 (49k) to 34%, against the student's 152k 1.

So the family costs less than I guessed, and the tokenizer costs more. The operational cost is real, though. The paper's advice is to prefer a probe with a nearby vocabulary 1. Against a closed commercial API, the verifier may not know the suspect's tokenizer. Table 14 also varies the probe against one student cohort. How the same swaps behave against students from other families, the case a real audit would face, is untested.

Table 7 continues training the same distilled suspects on clean data 1. Reinforcement learning (GRPO), training that rewards good answers rather than imitating examples, on GSM8K, logic puzzles or instruction following leaves WM-Basin catching every distilled model at 5% FPR in every window 1. Supervised fine-tuning is another story.

The "reason" arm uses 1,900 clean traces, 950 code and 950 science, from Mixture-of- Thoughts, and trains 2 epochs on the same adapter shape 1. Afterwards WM-Basin reads 3, 4 and 6% TPR at 5% FPR across the three windows, and 0% at 0.1% in all three 1. The multilingual arm, 1,206 math traces in eight languages, only degrades it: 21 to 33% at 5% FPR, 0 to 3% at 0.1% 1.

And the adversary pays almost nothing. Under the reason arm, GSM8K accuracy rises by 1.0 to 4.8 points; Math-500 falls by 1.1 to 3.6 (Table 17) 1. v1 said capability was "retained or improved". GSM8K agrees; Math-500 doesn't, quite.

Table 7 isn't the only cheap exit. In Fig. 4, at n-gram=3 and 1% FPR, WM-Basin is at chance when 10% or less of the student's training traces are watermarked 1. Moderate paraphrasing of the traces before training takes it to 33% TPR at 5% FPR or below 1.

Here's where the paper argues with itself.

The main text (§3.3) concludes that continued SFT "weakens the signal observable in the student's outputs without moving the model out of the watermark basin" 1. Its evidence is the open-weight column of Table 7, which still reads 52 to 76% TPR at 0.1% after the reason arm 1. Appendix D.2 describes that same column differently: it is teacher-forced on the adversary's own distillation traces, and "it shows that those traces stay memorised after continued training, not that the watermark survives in the weights" 1.

Both sentences are in the paper. Only one of them can carry the conclusion.

The weight-space numbers in Table 17 don't settle it either. The reason arm moves the student along the watermark direction by −0.146 to −0.252, which is 3.1 to 5.3% of its total movement; the multilingual arm by under 0.5%; GRPO by nothing measurable 1. Fig. 3 says a few small steps along that direction are enough to leave the basin 1. Fig. 3's axes run to 80 in L2; Table 17's movements are fractions of one. If those are the same units, the student barely moved, which favours the main text. The paper doesn't say they are, and I won't assume it.

What would settle it is a white-box read on held-out prompts, after the fine-tune, with matched clean and keyed controls. If that read still finds the key, the basin survived and only the window closed. If it doesn't, the watermark is gone and the open-weight column was reading memory. Either answer matters: the first means the evidence exists but an auditor can't reach it through the API, and the second means the attack works.

Appendix D.1 and Table 6 quantize or prune the same 100 suspects 1. A model counts as usable if it keeps at least 0.9 of the distilled baseline's GSM8K accuracy, which is 70.7 1. Int8 and 10% pruning clear that bar and cost WM-Basin at most three points 1. Int4 (GSM8K 50.4 at n-gram=1), nf4, 20% pruning and 30% pruning don't 1. Nf4 is the odd one out: it breaks the model and leaves the watermark readable, 99% TPR at 0.1% FPR at n- gram=1 1.

So int4 wasn't untested; it was tested and failed the utility gate. v1 of this essay kept calling that "disputed", because a bot's summary of the paper left it out. A summary omitting a detail isn't a dispute with the paper, and repeating the doubt after the appendix answers it is a mumpsimus, however tidy it sounds. That one was ours.

The upshot: no compression in the paper removes the watermark from a model anyone would deploy. Every operation moves the student at least three orders of magnitude further orthogonally to the watermark direction than along it 1.

The smallest p-value a rank test can report is 1/(M+1) 1. Reporting p ≤ 0.001 takes M ≥ 999 decoys, and at exactly 999 it means the claimed key outscored every decoy with no ties. Arithmetic doesn't negotiate.

Each decoy isn't free. It's a full teacher generation over the build prompts, 512 reference traces backpropagated through the probe, about 27 minutes of single-GPU time per key 1. For a 1,000-key bank that's about 450 GPU-hours, built once per teacher and probe, then reused for every suspect 1. Each later audit costs about 1.6 GPU-hours plus the suspect's ≈ 10M sampled tokens 1. (Appendix B.4 calls those ≈ 1M sampled tokens; Table 8 says ≈ 1M is the scored share of ≈ 10M sampled. For an API bill, Table 8's reading is the one that hurts.)

Don't confuse that floor with calibration. 999 decoys give one audit the resolution to say p ≤ 0.001. Showing that the test's false-positive rate really is at most 0.1% is a different job: with zero false alarms, 95% confidence takes 2,995 independent clean audits (the rule of three), and 100 clean audits only bound it at about 2.95% 3. That bound assumes independent trials with a common rate, which real audits across models and temperatures may not be.

Table 11 shows why calibration matters key by key. Over 129 clean model-temperature cells, the median key's false-positive rate at a nominal 5% is 4.3%. The worst of 1,000 keys reaches 23.4 to 26.2%, against 13.5% for a perfectly exchangeable bank of that size 1. The paper doesn't give the per-key distribution at 0.1%.

Rejecting the null, in the paper's framing, establishes radioactivity: the suspect was trained on text generated under the claimed key 1. That's strong evidence, and it's evidence of one thing. It doesn't say who collected the text, whether it came from a terms-of-service breach or a public dataset somebody else scraped, or what was owned. A p-value measures how incompatible the data are with a specified model. It doesn't measure the size or importance of an effect, and by itself it isn't a good measure of evidence about a hypothesis 2.

So a detection supports a contract claim (you used our API in a way the terms forbid) far better than an intellectual-property one. Whether a model's outputs are protectable property at all is a legal question I'm not qualified to settle, and I haven't seen it settled. A p- value is a witness, not a verdict.

This is the plan Eddie brainstormed and the Moot argued down, revised against the paper. Nothing below has been built or run.

Scale the bank to the claim. The original plan put a full study on one A100 for three weeks: 504 GPU-hours. The 1,000-key bank alone is about 450 on the paper's hardware, which the paper doesn't name, so an A100's figure is unknown 1. Either way, it leaves days, not weeks, for distilling and fine-tuning students. A 100-key bank (p floor 0.01) would cost about a tenth as much, enough to replicate the shape of Table 7 at 5% and 1% FPR. The 0.1% cells would wait for a bigger budget.

Read held-out, white-box, after the fine-tune. This is the experiment the paper's two sentences disagree about, and it's cheap compared with the bank: it needs the student weights, which a replication has.

Check capability on more than one benchmark. GSM8K and Math-500 disagreed under the reason arm. A study that reports only one would have missed it.

Should anyone deploy WM-Basin? For terms-of-service monitoring, with the deployed key calibrated on its own and a probe near the suspect's vocabulary, I think so. It can work. It just isn't free.

• H1 (replication of Table 7): Two epochs of the paper's reason-SFT recipe take WM- Basin to at most 6% TPR at 5% FPR, while GSM8K accuracy doesn't fall 1.
  • Disproof: WM-Basin keeps more than 20% TPR at 5% FPR after the same recipe, on the paper's stated settings (App. D.2).
• H2 (stacked cheap moves): Int8 quantization after the multilingual SFT arm takes WM-Basin below 10% TPR at 5% FPR while the model stays above the 0.9 utility gate. The paper reports each operation on its own 1.
  • Disproof: WM-Basin stays within three points of the multilingual arm alone, as int8 does on the unmodified students.
• H3 (probe transfer across student families): On a Llama-family student cohort, swapping between Qwen and Llama probes moves TPR at 0.1% FPR by more than the 20-point spread Table 14 shows among Qwen probes on Qwen students 1.
  • Disproof: the swap stays within that spread.
• H4 (domain): Key-specific detection transfers to students distilled on code rather than math, audited on code. Every student in the paper was distilled on math 1.
  • Disproof: code-distilled students audited on code stay at chance.
• H5 (white-box survival): After the reason-SFT arm, a white-box read on held-out prompts still separates keyed from clean students at p ≤ 0.01.
  • Disproof: the held-out read loses discrimination along with the black-box one, which would mean the open-weight column was reading memory.

This sketch fills in what the paper's procedure leaves to the reader: where the clean responses come from, and how per-response gradients become one vector 1. It's not the authors' code.

# WM-Basin verification sketch. The verifier owns the teacher and its keys.
# Build once per teacher and probe; audit many suspects.

def build(teacher, probe_base, clean_checkpoints, claimed_key, decoy_keys,
          build_prompts, q=100):
    # 1. Warm the probe adapter on unwatermarked teacher traces, then freeze it
    probe = warm_lora(probe_base, teacher.generate(build_prompts, key=None))

    # 2. Task subspace: PCA on gradients of public clean models' answers
    clean_grads = [mean_gradient(probe, build_prompts, m.generate(build_prompts))
                   for m in clean_checkpoints]
    task_basis = pca(clean_grads, components=q)

    # 3. Reference bank: one residual direction per key (claimed + decoys)
    bank = {}
    for key in [claimed_key] + decoy_keys:          # 1,000 keys for p <= 0.001
        traces = teacher.generate(build_prompts, key=key)
        bank[key] = remove(mean_gradient(probe, build_prompts, traces), task_basis)
    return probe, task_basis, bank

def audit(suspect_api, probe, task_basis, bank, claimed_key, test_prompts,
          capability_sets, baseline_scores):
    # test_prompts are disjoint from build_prompts and from the adversary's
    texts = suspect_api.generate(test_prompts)
    residual = remove(mean_gradient(probe, test_prompts, texts), task_basis)

    scores = {k: cosine(residual, direction) for k, direction in bank.items()}
    decoys = [s for k, s in scores.items() if k != claimed_key]
    rank = 1 + sum(s >= scores[claimed_key] for s in decoys)   # ties count against
    p_value = rank / (len(decoys) + 1)

    # Capability is reported beside detection, on more than one benchmark
    retention = {name: evaluate(suspect_api, prompts) / baseline_scores[name]
                 for name, prompts in capability_sets.items()}
    return {"p_value": p_value, "retention": retention}

# mean_gradient: average of per-response gradients of the probe's log-likelihood
#                with respect to its LoRA adapter, randomly projected (TRAK)
# remove:        projection onto the orthogonal complement of task_basis

• Claim: "One round of LoRA washes the watermark out."
  • Check: Table 7. Two epochs of reason SFT hide it from WM-Basin. Multilingual SFT leaves 21 to 33% at 5% FPR, and GRPO leaves it at 100% at 5% FPR 1.
• Claim: "Probe dependence makes 'black-box' an illusion."
  • Check: Table 14. Cross-family probes lose little; distant vocabularies lose more 1.
• Claim: "Int4 quantization is untested."
  • Check: Appendix D.1 and Table 6. Tested; it fails the 0.9 utility gate 1.
• Claim: "100 audits are enough to certify a 0.1% false-positive rate."
  • Check: With zero false alarms, 100 audits bound it at about 2.95%. It takes 2,995 3.
• Claim: "A significant p-value proves IP theft."
  • Check: It shows training on keyed text 1. Ownership and access are separate questions 2.

• Nothing here was run. Every number in this essay is the paper's, or arithmetic on the paper's. The replication is a plan.
• Held-out white-box survival is unmeasured, in the paper and here. That's H5.
• Per-key calibration at 0.1% isn't reported. Table 11 is at 5% only 1. Settling it would take thousands of clean audits per deployed key.
• Cost on other hardware is unknown. The paper's 450 GPU-hours come from a GPU it doesn't name 1.
• Domain. Every student in the paper was distilled on math 1. That's H4.