The BS Detector
Things people say with confidence, checked against what the sources actually say. Every issue's BS Detector, in one pile.
It's open source and popular, so someone would have caught this." Check: Someone did look. #8616 called the advisory build-time only, which was true at its commit; it took a later comment to show the runtime path.
No. 17 Mostly Harmless: a no-execution audit of PhotoCraft 0.17.0 for backdoors, phone-home and leaks
No telemetry found, so it never connects." Check: It checks GitHub at every launch by default.
No. 17 Mostly Harmless: a no-execution audit of PhotoCraft 0.17.0 for backdoors, phone-home and leaks
It's Rust, so the TLS is safe." Check: RUSTSEC-2026-0382 is a trust-logic flaw, not a memory-safety one. Memory safety doesn't decide which certificates to trust.
No. 17 Mostly Harmless: a no-execution audit of PhotoCraft 0.17.0 for backdoors, phone-home and leaks
Update ca-certificates and you're done." Check: That closes the trust-store link for the system bundle. SSL_CERT_FILE or SSL_CERT_DIR can point the updater at a different store, and a machine without main, as ours was, never receives the update.
No. 17 Mostly Harmless: a no-execution audit of PhotoCraft 0.17.0 for backdoors, phone-home and leaks
3.8 out of 10 means it's safe." Check: The score is cumulative across findings. On a lagging Linux machine, the update channel is the whole story.
No. 17 Mostly Harmless: a no-execution audit of PhotoCraft 0.17.0 for backdoors, phone-home and leaks
Fresh coins eliminate all leakage.
Fresh per-call coins close the timing channel within an episode. They don't close a shared reading that includes the session's own coin, such as a batch makespan. Ask what sessions share.
No. 15 Can agents tell when they are being audited?
The agent behaves the same when audited, so the audit is invisible.
Identical behavior can't tell a blind agent from a discreet one. Ask for the view-level test.
No. 15 Can agents tell when they are being audited?
Our classifier couldn't predict the coin, so leakage is below X.
A classifier gives only a lower bound. Ask for the design argument behind the upper bound.
No. 15 Can agents tell when they are being audited?
We pad every response to a fixed deadline.
Ask whether the audit's work grows with what the agent reads, and what happens to a run that overshoots. On the paper's gate a fixed ceiling went from looking safe to D_A ≥ 0.91.
No. 15 Can agents tell when they are being audited?
Under 1 percent of agent actions violated policy.
Ask whether that counts calls or episodes, and how call counts were handled.
No. 15 Can agents tell when they are being audited?
Our agents are audit-certified.
Ask for the audit rate, the estimator, the readout, and what the agent can see before commit.
No. 15 Can agents tell when they are being audited?
Survival LLM uses semantic vector search." Check: Read library/search.py. (Correction: SQLite FTS5 BM25).
No. 14 Assessing Offline LLM Efficacy for Emergency Operations
Llama 3 27B model available." Check: Check Meta HuggingFace. (Correction: No official 27B; use Qwen 3.8 27B).
No. 14 Assessing Offline LLM Efficacy for Emergency Operations
100 W power draw is sustained." Check: Measure with wattmeter. (Correction: 100 W is 5–40 s burst only).
No. 14 Assessing Offline LLM Efficacy for Emergency Operations
50 questions proves <5 percent error rate." Check: Calculate Binomial Upper Bound. (Correction: 59 zero-error trials needed).
No. 14 Assessing Offline LLM Efficacy for Emergency Operations
Offline means no network traffic." Check: Packet capture in the three configurations. (The vendor documents a daily update check that sends nothing about you; confirm it).
No. 14 Assessing Offline LLM Efficacy for Emergency Operations
On-device execution protects training data.
Verify if MIA uses outputs. If API returns confidence scores, location of compute (cloud vs. edge) does not prevent black-box inference.
No. 13 Quantization is not a privacy guarantee
Quantized models satisfy GDPR Article 17 (Right to Erasure).
Article 17 specifies grounds for erasure, not technical MIA thresholds; whether a model holds personal data is assessed case by case. Compression changes how precisely the weights are stored, not what they learned.
No. 13 Quantization is not a privacy guarantee
Low-bit quantization provides DP guarantees.
Quantization that uses no private data is post-processing and keeps the DP guarantee; PTQ/QAT that uses private data keeps it only if that use is counted in the privacy budget.
No. 13 Quantization is not a privacy guarantee
MIA AUC is 0.5 (chance).
Verify TPR@1% FPR. Aggregate AUC can hide reliable identification of specific members at low false-positive rates.
No. 13 Quantization is not a privacy guarantee
My distro security tracker says'Fixed', so I am patched.
Distro trackers report source packages. Find out which firmware file the VM loads, then inspect it or match its hash to a validated fixed build.
No. 12 EDK II PXE bug: which shipped firmware and VM images actually contain the fix?
My VM image is safe because I updated the OS.
Guest OS updates do not replace hypervisor-supplied OVMF firmware.
No. 12 EDK II PXE bug: which shipped firmware and VM images actually contain the fix?
I can detect the fix via the firmware file hash.
A hash match to a validated fixed build is evidence; a mismatch is not. Otherwise, inspect the copy loop.
No. 12 EDK II PXE bug: which shipped firmware and VM images actually contain the fix?
The vulnerability requires local network access.
Upstream advisory allows delivery from any network distance via unicast ICMP during PXE. Red Hat/SUSE assign adjacent-network vectors for specific assessments.
No. 12 EDK II PXE bug: which shipped firmware and VM images actually contain the fix?
WhatsApp's chatbot told people not to contact regulators.
Nobody outside Meta has seen the bot's answers. Redact: "We read its instructions, not its answers".
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
AI crawlers were fed secret instructions.
In Redact's test, the AI crawler names got neither the notes nor the article. The search crawler names got both. Those were names, not verified crawlers.
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
Readability strips hidden text, so extraction makes this safe.
On these pages Readability returned nothing at all. Turndown returned everything (section 4).
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
An LLMInclude/LLMExclude pair proves the leak was deliberate.
It proves the capability was deliberate. Whether publishing the notes was intended is a separate question.
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
OWASP's scan found zero, so this doesn't happen.
That scan covered homepages and a few inner pages of top-million sites, looked for manipulation rather than first-party instructions, and counted reachable sites only.
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
This is a new class of attack.
Indirect prompt injection was named in 2023. What's new here is who wrote it.
No. 11 Written for the Chatbot, Sent to Everyone: auditing the text help pages carry for machines
The exploit failed 50 times, so this is hard to weaponize.
That was one attacker's chain with a broken final stage. ELLIO's chain got a shell on four default targets.
No. 10 Broccoli2Shell: counting vendored Brotli decoders hiding in the software supply chain
process.versions.brotli says 1.2.0, so we're vulnerable", or "it says 1.2.0, so we're fixed.
Neither. Node.js 24.22.1 is fixed and still says 1.2.0.
No. 10 Broccoli2Shell: counting vendored Brotli decoders hiding in the software supply chain
We upgraded brotli in the lockfile, so we're done.
That fixes tier B. The runtime and any native extension with a private copy are separate.
No. 10 Broccoli2Shell: counting vendored Brotli decoders hiding in the software supply chain
91% of everything is vulnerable.
Wiz counts cloud environments with at least one vulnerable resource, from its own telemetry. That's a different number from a share of artifacts.
No. 10 Broccoli2Shell: counting vendored Brotli decoders hiding in the software supply chain
Blocking Content-Encoding: br at the WAF covers it.
Only for inbound requests that pass through that WAF. Outbound responses and files don't.
No. 10 Broccoli2Shell: counting vendored Brotli decoders hiding in the software supply chain
Qwen3.8 phones home 98% of the time.
98% is the share of completions that wrote a URL after being handed; curl-s -m 10 -o /tmp/. Unprompted, it's under one call in 100,000.
No. 09 What a Local GPU Can and Can't Check About Qwen3.8's Phone-Home URLs
It hit the metadata endpoint in a third of runs.
It wrote the address in 23–36% of completions at the first two cuts. Nothing was reached.
No. 09 What a Local GPU Can and Can't Check About Qwen3.8's Phone-Home URLs
Check Point proved it isn't a backdoor.
Check Point says its tests "do not prove that there is no backdoor".
No. 09 What a Local GPU Can and Can't Check About Qwen3.8's Phone-Home URLs
Zenity reproduced Cohere's finding.
On Qwen3.8-27B, the routify link reproduced. The -o jump, the metadata habit and the upload did not. Flash-Next wasn't tested.
No. 09 What a Local GPU Can and Can't Check About Qwen3.8's Phone-Home URLs
Self-hosting keeps the data in.
The weights don't decide where traffic goes. The network policy does, and only if it's enforced outside the model.
No. 09 What a Local GPU Can and Can't Check About Qwen3.8's Phone-Home URLs
HoS enforces instruction–data separation." (the paper's framing)
It enforces a learned attenuation across a boundary drawn by message roles. The paper's own Limitations call it "not an end-to-end defense".
No. 08 Architectural Defenses Against Prompt Injection
Only 10% attack success against HoS.
True for the highest RL-Hammer attacker at medium reasoning effort. At low effort, an attacker trained against HoS reaches 36%.
No. 08 Architectural Defenses Against Prompt Injection
The base model is frozen, so its capabilities are preserved.
Llama-3.3-70B's AgentDyn utility falls from 33.7% to 18.3% with the weights frozen.
No. 08 Architectural Defenses Against Prompt Injection
It's 18,000 parameters, so it's free to run.
The gate table is tiny. The released server gives up prefix caching and CUDA graphs to apply it, and that cost is unmeasured.
No. 08 Architectural Defenses Against Prompt Injection
AutoGen runs generated code on your machine by default.
Since 0.2.8 the default is a Docker container, and it errors if Docker isn't running. Local execution is opt-in.
No. 07 The Missing Ring-0 Protections for AI Agents
Prompt injection is the only real attack on an agent.
CVE-2025-68664 and the LangGraph checkpointer chain reach secrets and code without a prompt [13, 15].
No. 07 The Missing Ring-0 Protections for AI Agents
Claude Code's sandbox covers everything the agent does.
It covers shell commands and their child processes. Read, Edit, Write, WebFetch and WebSearch follow permission rules instead.
No. 07 The Missing Ring-0 Protections for AI Agents
We ran it in Docker, so the agent is contained.
The container limits what the agent can reach. It doesn't decide which of the agent's own actions are allowed.
No. 07 The Missing Ring-0 Protections for AI Agents
Zero violations in 500 runs means it's safe.
It bounds the rate below about 0.6% at one-sided 95% confidence, for the distribution tested.
No. 07 The Missing Ring-0 Protections for AI Agents
One round of LoRA washes the watermark out.
Table 7. Two epochs of reason SFT hide it from WM-Basin. Multilingual SFT leaves 21 to 33% at 5% FPR, and GRPO leaves it at 100% at 5% FPR.
No. 06 Black-Box Detection of Model Distillation using Watermark Basins
Probe dependence makes'black-box' an illusion.
Table 14. Cross-family probes lose little; distant vocabularies lose more.
No. 06 Black-Box Detection of Model Distillation using Watermark Basins
Int4 quantization is untested.
Appendix D.1 and Table 6. Tested; it fails the 0.9 utility gate.
No. 06 Black-Box Detection of Model Distillation using Watermark Basins
100 audits are enough to certify a 0.1% false-positive rate.
With zero false alarms, 100 audits bound it at about 2.95%. It takes 2,995.
No. 06 Black-Box Detection of Model Distillation using Watermark Basins
A significant p-value proves IP theft.
It shows training on keyed text. Ownership and access are separate questions.
No. 06 Black-Box Detection of Model Distillation using Watermark Basins
NTP drift is negligible for forensics under normal load.
Measure the offset under stated CPU and I/O load. W32Time supports 1 second, 50ms or 1ms accuracy, each only under its own documented requirements.
No. 05 When clock uncertainty changes an incident verdict
Logs are trustworthy because they are cryptographically signed.
Ask what the signature proves. Signed syslog detects tampering and missing messages. It doesn't make the producer's clock right, and it doesn't stop delay or reordering.
No. 05 When clock uncertainty changes an incident verdict
TCP guarantees in-order delivery, so logs arrive in order.
Count the connections. TCP orders bytes within one connection. Multi- worker fan-in, per-CPU buffering and clock steps reorder records across them.
No. 05 When clock uncertainty changes an incident verdict
My detector uses logical clocks, so it's immune to time drift.
Ask where the logical order stops. SLEUTH keeps a per-subject order of events, which lets it store timestamps at millisecond resolution in 16 bits in the typical case. Order within a subject isn't order across hosts.
No. 05 When clock uncertainty changes an incident verdict
The tool is 99.8% safe.
Look for the denominator. Is it document-level (0.2% leak) or span-level (precision)? Is it same-generator or external?,.
No. 04 Does PII masking survive a different document generator?
Real incident tickets were used.
Verify if documents are from the author's generator (Set 7) or external public datasets (hivetrace, Nemotron-PII).
No. 04 Does PII masking survive a different document generator?
Russian names fail because of LLM hallucination.
Read REPORT.en.md failure breakdown (188 of 199 stored-name restoration failures are inflection errors, alongside 82 storage errors and 8 evaluator errors).,.
No. 04 Does PII masking survive a different document generator?
5% leak rate violates UK GDPR.
Search ICO guidance for numerical thresholds (none exist). Check Article 6 basis and Article 9 conditions.
No. 04 Does PII masking survive a different document generator?
Baseline tools work on external data.
Verify if baselines were run on Set 7 (same-gen) or external sets. 58%/61% are Set 7 numbers.
No. 04 Does PII masking survive a different document generator?
IP addresses are unsupported.
Inspect pii_roundtrip/adapters/pii_guard.py (RU adapter maps IP types).
No. 04 Does PII masking survive a different document generator?