A query-time filter for poisoned retrieval is judged in the lab by how well it ranks, and used in production by what it blocks at one fixed threshold. Those are different questions. A nine-day-old preprint, Ranking Is Not Deciding (Kumar and Sahaym, arXiv 2610.33849), argues that the field mostly answers the first and ships on the second 1. This essay is a re- examination of that preprint, not original research: we ran nothing. We checked its numbers against its own tables, against the defense paper it tests 2, and against one open benchmark that measured something close 7. Two things govern how much weight its numbers can bear, so they come first. The filter it tests is the authors' own port, not the original code, and the authors call its separability numbers an upper bound 1. And the preprint's first author also coauthors a competing defense with the same name 3, which the preprint does not declare as a conflict 1. What survives is still worth a SOC's attention. In the authors' port, deleting one sentence from each poisoned passage took the miss rate from 0.191 to 0.763 at an almost unchanged false-positive rate 1. An independent benchmark shows the same cliff in a different detector 7. At the operating point the preprint reports, our own arithmetic says fewer than 4 in 1,000 blocked passages would be poison.

In plain terms: a RAG system (retrieval-augmented generation) answers questions by first fetching passages from a document store and then handing them to a language model. Poisoning means planting passages in that store so the model gives the answer the attacker wants. A query-time filter sits between the fetch and the model and throws out passages that look planted. A SOC (security operations center) is the team that has to live with whatever that filter lets through, and with everything it wrongly blocks.

• Ranking isn't deciding. AUROC says how well a filter orders poison above clean text across every possible threshold. A SOC runs one threshold. Ask for the miss rate and the false-positive rate at that one.
• The cliff is real in two places. In the preprint's port of the RAGuard filter, removing the copied question from each poisoned passage raised the miss rate from 0.191 to 0.763 1. In AgentDetox's own detector, the same edit took detection at its strict setting from 100% to 0.0% 7.
• At low prevalence, even a good filter is a noisy alarm. That's arithmetic, not a finding about any product, and Section 5 shows it.

• If you run a SOC: a filter's dashboard can be accurate and useless at once. The number that tells you what lands in your queue is precision at your prevalence, and it moves when prevalence moves. The false-positive rate doesn't.
• If you buy these filters: a vendor who reports only AUROC, or only accuracy, hasn't told you what the product does at the threshold you'll run.
• If you build them: an evaluation that never removes the construction cues of its own test data may be measuring the cues.

The limits come first, because they set the weight of everything after. Every empirical number in this essay that isn't ours comes from one preprint, submitted on 24 October 2026 and not peer reviewed 1. An earlier version was rejected from HICSS-60, and the authors print both reviews in an appendix, which is more candor than most papers manage1. We didn't rerun anything. The RAGuard the preprint tests is the authors' port of Cheng et al.'s filter 2, because Cheng's code isn't public; the preprint says RAGuard "ran only as our ports" and that every separability number on the regenerated data is an upper bound 1. The port's setup also differs from Cheng's: a different retriever, ten passages per query, and a smaller calibration set [1, 2].

Then there's the name. Two defenses are called RAGuard: Cheng et al.'s query-time filter 2, and a layered framework, RAGuard-ZKIP, whose authors include Pushkal Kumar 3, who is also the preprint's first author 1. The preprint tests Cheng's filter, not Kumar's, and it codes Kumar's own framework in its survey as not reporting an operating-point false-positive rate 1. That's a point against interest. It still has no conflict-of-interest statement, so we're stating the overlap here.

So read what follows as the answer to one question: if the preprint is right, what should a SOC do differently? Where we only have the preprint's word, the claim register at the end says so.

A filter gives every passage a score. Ranking asks whether poison scores higher than clean text. Deciding asks which passages get blocked once you pick a cutoff. AUROC, the area under the ROC curve, summarizes ranking across every possible cutoff at once 6. Production runs exactly one.

Can a filter rank perfectly and still let everything through? Yes. If the three poisoned passages score 1st, 2nd and 3rd, and the block threshold sits above all three, the ranking is flawless and the filter blocks nothing. The dashboard isn't lying, exactly; it's answering a question nobody on shift asked.

The numbers that answer the shift's question are the miss rate (the share of poisoned passages that get through) and the false-positive rate (the share of clean passages that get blocked), both at the deployed threshold. Each is conditioned on the passage's true class. So if the scores of poisoned and clean passages keep the same distributions and the threshold stays put, changing how much poison there is (prevalence) changes neither of them. What changes is precision: the share of blocked passages that were actually poison. That follows from the definitions 6; it's our derivation, not the preprint's claim. It matters because it tells you where to look when production goes wrong. If the false-positive rate moves, something other than prevalence moved: the threshold, or what the passages look like.

The survey. The authors coded 21 published evaluations of RAG poisoning defenses. Twelve report no false-positive rate at the operating point they deploy, and only one reports a ranking metric at all 1. That second number corrects our first version, which said the field leans on AUROC. It mostly doesn't report AUROC either. The gap the survey documents is missing operating-point evidence, not a fondness for the wrong metric. Our count of the survey table (its Table II) splits the twelve as ten "No", one "Partial" and one "Nominal only"; the preprint gives only the total 1. Cheng et al.'s paper is one of the evaluations that does report the rates: a false-positive rate of 0.028 and a miss rate of 0.038 on Natural Questions with PoisonedRAG, with the threshold set at a 2.5% percentile of scores on 1,000 texts sampled from the knowledge database 2.

The cue. PoisonedRAG, the standard attack these defenses are tested against, builds each poisoned passage by putting the target question, word for word, at the front 4. The preprint documents the line of code that does it 1. A filter can learn to spot that copied question instead of the poison. The authors' test was simple: delete the question and leave the rest of the passage alone. On Natural Questions, their RAGuard port went from a miss rate of 0.191 at a false-positive rate of 0.065 to a miss rate of 0.763 at 0.061, and its AUROC fell from 0.953 to 0.758 1. Meanwhile 0.93 of the edited passages were still retrieved for their target question, against 0.99 before 1. Most of the poison still reaches the filter, and three quarters of it now gets past.

That's the headline, and it needs its footnote. Cheng's own paper reports a miss rate of 0.038 on the same dataset, and the port's 0.191 with the cue intact doesn't match it [1, 2]. Either the port is weaker than the original, or the setups differ enough to matter, or both. The cliff is measured; how much of it belongs to Cheng's actual filter is unknown.

The learned scorer. The authors also trained their own scorer, which turned out to lean on passage length. With the length cue intact, its AUROC was 1.000 and it missed nothing 1. When they padded the poison to clean length with shuffled filler words, the miss rate at its deployed threshold rose to 0.986 while AUROC was still 0.953 1. Recalibrating the threshold to a 5% false-positive rate brought the miss rate back to 0.321 1. So the scores still carried most of the information, and the deployed threshold was what broke; getting it back cost a 5% false-positive rate. A filter with 0.953 AUROC blocking almost nothing is the essay's whole argument in one row of one table.

What it didn't measure. Whether the poison changed the model's answer. The preprint calls generator-side harm "secondary": one small model, substring matching, one condition1. A passage that slips past a filter is a necessary step toward a bad answer, not proof of one.

The AgentDetox benchmark repository ran its own detector against the same PoisonedRAG passages, 100 target questions with five poisoned passages each, mixed into 5,225 clean SQuAD paragraphs, with and without the copied question 7. At its strict setting, detection was 100% with the question and 0.0% without it, on all three datasets it tested, with no clean passages blocked 7. At a looser setting, detection without the question was 89.2%, 98.0% and 92.0%, at a clean false-positive rate of 2.6% 7. The repository itself says the no-prefix columns "are the more meaningful" 7.

It's a different detector, so it can't confirm the preprint's RAGuard numbers. What it confirms is the mechanism: remove the cue and a strict threshold goes blind, and getting the detection back costs false positives. (It's also a reminder that the preprint's experiment isn't the first to run the no-prefix condition, which matters for Section 7.)

What does a 0.763 miss rate at a 0.061 false-positive rate look like on a shift? Here's an illustration, not a measurement. Take 1,000,000 retrieved passages, of which 1,000 are poisoned. That 0.1% is a scenario, not a known rate: nobody publishes real enterprise poisoning prevalence. Treat each passage as one independent decision.

The first two rows use the preprint's measured rates 1, the third Cheng's 2; the arithmetic is ours. Even the best row means about 30 clean blocks for every poisoned one. That's not a defect in any filter. It's what a 2.8% false-positive rate does to a 0.1% base rate.

Two caveats keep it honest. For a query the attacker actually targeted, the poison isn't 0.1% of what comes back. In the benchmark design, up to five of the ten passages retrieved for that query are poison, and precision on those queries looks very different. And a per- passage false-positive rate understates what a user loses. If ten passages come back per query and each clean one has a 6.1% chance of being blocked, about 47% of queries lose at least one good passage. That assumes independence, which real passages don't have, so treat it as a rough guide.

Which threshold is right depends on what a missed poisoning costs you, what a blocked answer costs you, and whether a block goes to an analyst or silently disappears. The preprint doesn't set those costs, and neither do we. A vendor who sets them for you is choosing your risk appetite.

It depends on which number you're trying to pin down. If a filter misses none of n poisoned passages, an exact one-sided 95% upper bound on its true miss rate is 1 − 0.05^(1/n). Want to know what zero misses in a small test proves? Not much.

Two consequences. First, a claim of a false-positive rate below 0.1% needs around 3,000 representative clean passages with no false blocks, not a few hundred poisoned ones. Our first version said "500+ samples" as if that settled every metric; it doesn't. Second, "independent" is doing real work. The benchmark's 500 poisoned passages come from 100 target questions, five each, and passages written for the same question tend to pass or fail together. That's closer to 100 independent units than 500, and the confidence interval should be computed over questions.

This is a plan. None of it has been run, and nothing in this essay depends on it.

AgentDetox already runs the prefix and no-prefix conditions 7, so repeating them would add nothing. What would be new is an independent implementation of Cheng's filter, checked against Cheng's published rates before anything else 2. Thresholds would be set on a held-out calibration set, because a threshold tuned and graded on the same data is afterwit. Confidence intervals would be computed over target questions, not passages. The generator's answer would be measured, so that "slipped past the filter" and "changed the answer" are counted separately. And one worked deployment would carry stated costs, so the threshold is chosen rather than inherited.

If we run it, the results will be a separate issue with its own ledger. If we don't, this section stays labeled as a plan.

For anyone selling a filter, the ask is smaller: the miss rate and the false-positive rate at the deployed threshold, how much of the poison still gets retrieved, and how often it changes the answer. Four numbers and some error bars. That's the whole ask.

1. H1 (the cue): On an independent implementation of Cheng's filter, deleting the copied question from PoisonedRAG passages raises the miss rate at the deployed threshold by at least 10 percentage points, while at least 90% of the edited passages are still retrieved.
   • Disproof: the miss rate rises by less than 10 points, or retrieval of the edited passages falls below 90%.
2. H2 (the threshold): For the preprint's learned scorer, most of the cue-removal collapse is the threshold, not the scores: recalibrating on held-out cue-free data recovers most of the lost detection.
   • Disproof: at a matched false-positive rate on held-out data, the recalibrated miss rate stays above 0.5.
3. H3 (the company a passage keeps): For a filter that judges passages as a set, such as one that clusters the retrieved texts, the false-positive rate on the same clean passages changes when the share of poison in the retrieved set changes.
   • Disproof: the false-positive rate on a fixed set of clean passages stays within its confidence interval across retrieved-set compositions from no poison to half poison.
4. H4 (the survey): At least 12 of the 21 surveyed evaluations report no false-positive rate at their deployed operating point.
   • Disproof: an independent recount, using a published coding rule, finds fewer than twelve.

• Do report the miss rate and the false-positive rate at the threshold you deploy, with confidence intervals computed over independent units.
• Do report precision under stated prevalence scenarios, and say they're scenarios.
• Do test with the construction cues of your evaluation data removed.
• Don't expect prevalence alone to move the false-positive rate. If it moved, look at the threshold and the inputs.
• Don't read zero failures in a small test as a low failure rate.

• Claim: "Our filter has 95% accuracy."
  • Check: At 0.1% poison, a filter that blocks nothing scores 99.9%. Ask for the miss rate and false-positive rate at the deployed threshold.
• Claim: "AUROC of 0.95."
  • Check: That's ranking. The preprint's learned scorer had AUROC 0.953 while missing 98.6% of padded poison at its deployed threshold 1.
• Claim: "Zero false positives in testing."
  • Check: Out of how many clean passages, and were they like yours? Zero out of 100 bounds the rate at 3.0%, not zero.
• Claim: "Tested against PoisonedRAG."
  • Check: With the copied question removed? Without that condition, the test may be measuring the question.

• One preprint. Every number not marked as ours comes from a single, unreviewed source 1, checked only against its own tables, Cheng's paper 2 and one repository 7.
• A port, not the original. The RAGuard numbers belong to the authors' implementation, which doesn't reproduce Cheng's published rates [1, 2].
• Prevalence is assumed. The 0.1% in Section 5 is a scenario, and the independence assumption in the per-query figure is wrong in a known direction.
• Retrieval, not harm. Neither the preprint nor this essay measures whether poison that gets through changes the model's answer.
• The ingestion question is open. Pathak and Sharma argue that filters applied when documents enter the store can't contain coordinated poisoning 5. We haven't checked their argument, and it doesn't bear on the query-time results here.

The first version of this issue went up on 2 November 2026 and was rewritten the same day after reviews by Claude, Gemini and Astra. It said the field emphasizes AUROC, which the preprint's own survey contradicts; it suggested assuming a vendor tuned its threshold to the attack, which no metric supports; it offered "500+ samples" as a general rule; it presented an unrun plan as "Our Contribution"; it cited the preprint for our own arithmetic and plans; and it printed several of our reviewers' internal notes as if they were findings. All of those are gone. The first version is kept, unchanged, in the issue's source folder.