
No. 26 · 16 November 2026 · Updated 22 November 2026 · Security · Forensics
How we do forensics, and where it breaks: the Hawthorne method, open for argument
Three AI reviewers together caught 8 of 10 errors planted in one essay, no more than the strongest reviewer alone. Hawthorne reports that all three caught the two missed errors in a follow-up that supplied two relevant records.
Every audit asks the reader to trust a process they cannot watch [1]. That puts the verification burden on the reader. A self-audited forensic process needs logs the reader can check; without external validation of the evidence bundle, it still relies on the integrity of the people reporting the results. The evidence bundle for t0035, the house’s audit of PhotoCraft, is not yet public: site deployment is pending [19: N3].
The abstract, the TL;DR and a note on who built the method are in The Short Version, after the conclusion.

The Author’s house publishes its own process failures next to the target’s, and every publication goes through him [19: F29] [19: F02]. We publish those alongside the target’s vulnerabilities so readers can decide how much trust to place in the audit [1]. No medals for hiding the mess.
The distinction matters [1]. If we miss a vulnerability because a tool broke, that is a process failure. If the target has a vulnerability that stays unpatched, that is a target vulnerability. Scoring is only accurate if the two stay separate. Some of the bugs come from inside the house, and those go in the log too.
Our standard for reader verification includes an error log and pre-registered control results. It does not guarantee completeness or truth [1]. A process can be transparent and still produce false negatives if its method has blind spots. Verification also means checking whether the published flowchart matches the live essay’s version history and the Gate 2 approval records [1].
The pipeline has specific control points where deviations can occur. Gate 2 is the approval step where corrections are validated. The Moot Bypass (v4-v7) is a gap where edits occurred without full review by the Moot, the house’s review panel [1]. Gaps like that put uncertainty into the historical record, so the reader should assume that any unverified ledger entry is potentially subject to modification until an external checkpoint anchors it [4].
That is the foundation: self-audited processes need reader-verifiable logs, and we publish our own failures so readers can set their trust accordingly. Nothing in this section depends on a later one. The technical sections that follow depend on it.
The Author set rules for how we work: zero guess laundering, no overwrites and honest fetching [19: F15] [19: F25] [19: F26]. The audit also keeps a hash-chained ledger, whose limits are stated below [14]. They are not a guarantee that the results are right. They are how we leave the argument open to inspection.

Web Capture Policy: Web capture (collecting pages) is passive: the tool identifies itself, obeys robots.txt (a site’s instructions to automated visitors) as policy, and performs one GET (a request for a page) per URL (page address) [1]. On 1 November 2026, after a government site returned a 403 (access forbidden), the Author said, “yeah, we don't bypass government warnings.” [19: F26] The stop rule applies to every host: no other user agent, route or mirror around a block [19: F26]. Robots.txt is not authorization; obeying it is a policy choice, not a security control [15].
Score Integrity: “Never overwrite” went into the house plan on 28 October 2026; on 1 November, the Author identified a versioning problem and ordered the current versions advanced to the same next integer [19: F25]. Scores are appended per target version and rubric version, never overwritten [1], so historical performance can be tracked. The old score doesn’t get quietly escorted out the back.
The ledger (the audit’s record of events) is a hash chain: each entry includes the previous entry’s hash (a digital fingerprint of its data) [14]. Its entry schema places a 0x1f separator (a byte marking a field boundary) after every field, including the last, detail_str [1] [14]. Re-serialization (turning data back into a stored string) can change whitespace, escaping or key order and therefore the hash; verification should reproduce the specified original bytes, preferably from the stored string [14].
Guess Labeling: Zero guess laundering is a rule with an origin story we would rather not need: the Author adopted it on 30 October 2026, after No. 04 went live reading as though a study had been built and run when it was only a plan [18] [19: F15]. A guess, inference, hypothetical or unverified claim keeps its label through rewrites until a primary source verifies it; dropping the label is a blocking defect [18] [19: F15]. No new factual claims without source; guesses are labeled [1]. Disputed claims are stated as disputed, with both sides and what would settle it [1].
For example, the cost audit is incomplete because usd is NULL for sam/sundar calls [1]. This is a recording limitation, not a hidden cost [1]. (NULL means the number wasn't recorded, not that it was zero.)
The ledger entry, field by field. The hash covers six fields in this order, and a 0x1f byte follows every one of them, the last included [14].
| Order | Field | Holds |
|---|---|---|
| 1 | prev_hash | previous entry's hash |
| 2 | ts | timestamp |
| 3 | actor | who acted |
| 4 | action | what was done |
| 5 | subject | what it was done to |
| 6 | detail_str | the stored detail string |
The entry's hash is the SHA-256 of those bytes [14].
Verification: The reader reconstructs a ledger entry from the stored data and confirms its hash matches the next entry’s prev_hash (previous-entry hash) [14]. If the hash does not match, the chain is broken [4].

All of this assumes the ledger is append-only (entries are added, not altered or deleted). Without a signed commitment (external checkpoint) after the latest entry, the owner can rewrite the suffix [4]. Publishing head entry 288 in t0035 does not authenticate later entries 314, 321, 322, and 324 [4]. That’s structural: a limitation of the current implementation. The ledger doesn’t get to provide its own alibi.
The repair (proposed, not yet adopted): The hole has a known fix, and it is a matter of order: publish the commitment outside the house before the work it is meant to protect. For the next audit, that means three steps. First, before the target is opened or a control is run, publish the hash of the pre-registration somewhere the house cannot quietly edit: a public registration that is timestamped and read-only [5], or a signed tag in a public repository. Second, publish a checkpoint of the ledger’s head at each gate and again after the final entry, so no tail is left unanchored [4]. Third, commit the pre-registration and the results separately, so their order does not rest on the ledger’s word alone. None of this was done for t0035 or t0036. Entries 314–324 stay unanchored, and a checkpoint published now cannot prove old timestamps [4]. This is a rule for the next audit, not a repair of this one.
The Author doesn’t need the failures to disappear for the method to look good. He keeps the receipts and changes the rules: a grading bar, a paid-video mishap and even a wrong commit message are all on the record. [19: F19] [19: F25] [19: F30]
The publication gate. All 28 tests failed one draft, Volume 2 v9, whose grades averaged 3.65; on 6 November 2026, the Author changed the rule with the instruction “relax the A on every test, keep rigor absolute, accept that outliers happen.” [19: F19] The current bar has 33 named-personality tests—each a labeled best educated guess about how that person would grade—and requires two independent runs with zero open blocking problems. [19: F19] No heroic average gets a blocking problem off the hook. [19: F19]
The prompt budget. On 30 October, the house measured dario’s input at a median of 130,000–160,000 tokens per call, with a maximum of 348,000, because search results were being resent every turn. [19: F21] A token is a small chunk of text, the unit model input is measured in. The rule now sends long text to the justices in chunks and resends history conservatively, rather than putting roughly 5,000 words into one prompt. [19: F21]
The receipt. One incident produced 90 paid videos instead of 30. [17] [19: F25] The rule that followed: save a generation’s ID before downloading the goods. [17] [19: F25] Paying for the same clips three times is a memorable way to learn that lesson. [17] [19: F25]
Rule -1. When a training start ran out of memory because idle image servers were holding cached models, the Author’s rule, on 15 November, was: “optimize memory when needed, we must finish these work products.” [19: F33]
The prediction and its correction. On 28 October, the Author predicted that about half the scouted topics would fail the next step and had the prediction registered, with the test written, before that step ran. [19: F30] The result was 11 of 27, or 41%, with a 95% Wilson interval—a range expressing statistical uncertainty—of 25–59%; a hand-typed commit message said 10, and the next commit corrected it: “11 of 27 failed, not 10”. [19: F30] The correction is part of the record, not an embarrassment to hide. [19: F30]
The model’s job. On 5 November the Author put it this way: “we are assigning roles with measurements and not just guesses.” [19: F08] On 4 November, he had Nemotron run the same two research tasks as the previous eddie before any swap: Nemotron had 18 of 18 quotations verified word for word; the previous eddie, Qwen, had 10 of 16. [19: F07] Nemotron became eddie on 5 November. [19: F07] That’s an audition with receipts, not a warranty.
Rules are only as good as the next failure they catch.
Severity scores shape the risk assessment, so the arithmetic had better work. Ours didn’t at first: the v1 pseudocode left out the confidence multiplier, turning 0.64 points into 2.56 for severity 8 and confidence 0.25 [13]. That is a lively difference.
The scoring formula is influenced by CVSS (the Common Vulnerability Scoring System) [3] and SSVC (Stakeholder-Specific Vulnerability Categorization), with new axes introduced for internal consistency.

The Author’s Hawthorne Classification gives each finding a three-word verdict: Species · Immediacy · Devastation, each on a four-rung ladder; the 0–10 score is only a sort key [16] [19: F32]. It describes rather than prescribes: “We describe; the reader prescribes.” [16] [19: F32] Version 0.3 is a pre-validation draft, not a frozen instrument; no inter-rater exercise (a test of agreement among reviewers) has run yet, and freezing awaits an exercise that clears a stated bar [16] [19: F32] [19: N5].
Scoring Formula: Per-finding points = 4 × (severity/10)² × confidence [13].
- Example calculation:
- Severity: 8.0
- Confidence: 0.25
- Calculation: 4 × (8/10)² × 0.25 = 4 × 0.64 × 0.25 = 0.64
- Without the multiplier: 4 × 0.64 = 2.56 (incorrect)
[13].
Floors: Floors establish policy constraints on the audit score [13].
- 9.0 for confirmed malicious code, CWE-506 or CWE-912 (Common Weakness Enumeration identifiers), or covert exfiltration (secretly sending data out)
[13]. - 6.0 for confirmed CVSS ≥ 9
[13]. - 0.1 if anything is unresolved
[13].
They are policy constraints, not calibration or a guarantee against real-world under-scoring [13, 16]. A finding may still be under-scored if the Classification spec is misapplied [13, 16].
Classification Spec vs. Rubric: Classification Spec v0.3 and Scoring Rubric v0.3 must be kept distinct [13, 16]. The v1 pseudocode’s floor conditions did not match the rubric; the corrected code in “How to” follows rubric v0.3 and score.py, with one gap in score.py itself: its 9.0 floor checks only CWE-506 and CWE-912, not covert exfiltration, which the rubric includes [13] [19: N5].
Verification: Enter severity 8 and confidence 0.25 and confirm per-finding points of 0.64, not 2.56; for an audit containing only that finding, with no higher floor triggered, score.py reports the audit score rounded to one decimal, 0.6 [13]. If the calculation differs, the implementation does not conform to the adopted rubric [13].
The floors are applied to the audit's total, not to each finding: rubric v0.3 floors the score, and score.py applies them after summing the points [13]. That matters whenever an audit has more than one finding. The policy floors alone do not supply calibration [13, 16].
This is what went wrong in t0035 (PhotoCraft). The house publishes its own process failures next to the target’s, and every publication goes through the Author [19: F02] [19: F29]. The receipt: seven false positives, two reviewer errors and one process deviation— Claude wrote draft v2 [1]. The log makes this essay more useful: readers get the mistakes as well as the findings.
False Positives and Errors: False positives mean the classification process is not perfect. Reviewer errors are a different category from process deviations, which include third-party package execution and database updates.
CVE Labeling: t0035 mislabeled CVE (Common Vulnerabilities and Exposures, the public catalogue of known vulnerabilities) initially, then over-corrected [1]. That indicates instability in the automated classification pipeline; the over-correction suggests a feedback loop that was not properly managed.
Version Control: The t0035 corrections—voiced versions 8, 9 and 10—are unapproved, awaiting the Author’s Gate 2 (his fact-approval step) [1] [19: F17]. The live essay requires visible correction [1]. The Moot did not review t0035 after v3, and the SQL UPDATE (a database change) to finding #32 was logged after the fact [1]. That log entry is afterwit: the record shows up, but only after the change it records. The public version of the audit therefore contains unreviewed changes.
What happened to t0035, in the order the record gives it [1].
| When | Event | Standing |
|---|---|---|
| 30 October 2026 | Third-party packages ran | Disclosed |
| 31 October 2026 | The venv was deleted | Disclosed |
| After v3 | Moot review stopped | v4–v7 unreviewed |
| Undated here | Finding #32 changed by SQL | Logged late, entry 322 |
| Pending | Corrections v8, v9, v10 | Await Gate 2 |
Verification: Once the bundle is public, a reader can check audit.db for entry 322 and the version history for v3–v7 Moot review status [1]. If entry 322 is missing or the timestamp is inconsistent, the log’s integrity is compromised [4]. The evidence bundle is in the private repository and on the shared drive; site deployment is still pending, so the public cannot yet perform that artifact check [19: N3].

Third-party package execution occurred on October 30, 2026, during an audit meant to execute nothing [1] [19: F29]. The failure log also discloses that the virtual environment (venv, an isolated software environment) was deleted on October 31, 2026 [1]. A deleted venv is not an alibi. Deletion does not retrospectively prove the absence of earlier effects on the host computer or file access [1], and it removes the ability to verify the exact environment state used during the audit [1].
The Author had his own reviewers tested: the pre-registration was written on 31 October 2026, and the house's audit ledger logs its hash before the result [19: F31]. That order is the ledger's own claim: both entries come after the last public checkpoint, and the commit that holds the pre-registration also holds the results, so git cannot show which came first [19: F31]. What follows is an author-reported pilot, not a victory lap.
Initial control:
- The drift checker caught 10/10
[11]. - The panel union—the errors caught by at least one of the three reviewers—caught 8/10, missing R5 and R8.
- The bundle’s results file records dario at 7/10, sam at 8/10 and sundar at 8/10
[11].
That file exists in the bundle but is not yet public, so readers cannot independently verify the counts [11] [19: N3]. The initial misses indicate a blind spot for planted errors involving internal evidence when the relevant records are absent [11].
Control C (targeted appendix):
- All three reviewers caught R5 and R8.
- Dario caught 9/10, missing R9; sam and sundar each caught 10/10
[11]. - Control C was designed post-hoc after observing misses; it is a follow-up, not an independent discovery test
[11].

This suggests that targeted records improve detection, but it was a follow-up test, not a randomized controlled trial [11].
Panel union: Panel union recall equals the best individual in both conditions; no incremental benefit demonstrated in this n=1 pilot [11]. That challenges the assumption that cross-lab AI reviewers (models from different labs) add value beyond one strong reviewer [11]. The Author's own three-lab panel has not yet been shown to beat one [19: N7]. Three reviewers, one result to beat, and no bonus points for looking impressive.
Access vs. competence: Control C used a targeted two-record appendix. It does not prove that access alone caused the initial misses: the records' prominence and the difficulty of finding them also changed [11]. Testing the access hypothesis calls for a follow-up with the full 3.5 MB bundle, not the targeted appendix [11].
Who caught which planted error. The ten factual plants are R1 to R10; the reviewers are the three justices [11].
| Initial control | dario | sam | sundar |
|---|---|---|---|
| R1 wrong date | caught | caught | caught |
| R2 wrong version | caught | caught | caught |
| R3 wrong PR number | caught | caught | caught |
| R4 wrong CVE issuer | caught | caught | caught |
| R5 wrong rubric floor | missed | missed | missed |
| R6 wrong bug number | caught | caught | caught |
| R7 wrong notice number | caught | caught | caught |
| R8 wrong binary sizes | missed | missed | missed |
| R9 fabricated quote | missed | caught | caught |
| R10 score mismatch | caught | caught | caught |
In Control C, with the two records supplied, the only miss left was dario's on R9 [11].
| Control C | dario | sam | sundar |
|---|---|---|---|
| R5 wrong rubric floor | caught | caught | caught |
| R8 wrong binary sizes | caught | caught | caught |
| R9 fabricated quote | missed | caught | caught |
| The other seven | caught | caught | caught |
Verification: A reader can replay the control with the full 3.5 MB bundle, if it is released, to test the access hypothesis [11]. Until the bundle is released, that replay cannot be done by the reader [11].
Limitations: Control C was an n=1 pilot: one essay, ten planted errors per checker, easy mode [11]. It detects a specific class of error; it does not establish general effectiveness or a general failure rate. Nor does it prove that access alone explains the misses [11].
What we don't know yet: the places where the record is incomplete or disputed.
Prior Art Survey: Q11: Prior art survey for AI-assisted security research is not yet done [7, 12]. The universal novelty claim is dropped until it is [7, 12]. Schroter et al. 2008 establishes relevant prior art in peer review [7]. NIST SATE VI establishes relevant prior art in static analysis [6].

Evidence Bundle: Released to the private repository and shared drive, site deploy pending [1] [19: N3]. The README address returned homepage HTML during this check; that response does not supply the promised artifact verification [1]. The live essay publishes the stated head hash, calls its evidence record unpublished, and retains broad no-downloaded-code-execution wording [1].
Cost Audit: Cost audit incomplete: usd is NULL for sam/sundar calls; token counts (80 input vs 59,827 output) need telemetry explanation [1]. Explaining the 80 input vs 59,827 output discrepancy across seven dario calls requires inspecting raw ledger tables and API logs [1]. Neither public methodological documentation nor agreement among reviewers authenticates local entry numbers, execution dates, approval decisions, control totals or database changes [1].
Verification: Reader monitors site deploy and future control runs [1]. Until the bundle is published with hashes, the counts remain unverified [1].
What isn't done yet: Here's the part where the receipts come for us.
- The scheduled fact watch last ran cleanly on 3 November 2026. Every scheduled run from 7 November to 16 November failed, and it needs repair
[19: N1]. - The external-checkpoint repair in §2 is proposed, not adopted; nothing after entry 288 is anchored
[4][19: N3]. - The Classification is a pre-validation draft; no inter-rater exercise has run
[16][19:N5]. - No publication-bar grades are recorded for this essay
[19: N8]. - OpenAI's gpt-6.1-terra wrote this v3 text at the Author's request
[19: T5]. It has not been through his voicing system, hthompson, which house rules make the last edit[19:F18] [19: N9].
Scores are versioned; check the version [1]. Do not trust the score; trust the evidence bundle enough to examine it [1]. Transparency enables verification. It does not guarantee truth.

The ledger verifier is in the t0035 evidence bundle, but the site deploy is pending; the control bundle is not yet public either [19: F23] [19: N3] [19: F31]. Once the artifacts are available, readers can use the verifier and control data to assess the audit. Until then, the reported results remain claims awaiting independent inspection.
Hawthorne is one operator, and everything above went through him [19: F02]. We have a hash-chained audit ledger, a pre-registered control and a failure log kept next to the target’s findings; the control’s order is the ledger’s claim, not externally anchored [19: F23] [19: F31] [19: F29]. On 30 October 2026, the Author adopted zero guess laundering after No. 04 made a plan read like a completed study [19: F15]. On 6 November, he revised the publication gate after all 28 tests failed one draft under the old rule [19: F19]. The operation publishes its own failure log [19: F29]. His question, roughly: who else in the world could do this? [19: F03] Check the receipts.
The summary, moved behind the argument: what the essay claims, what it admits, who it is for and who built the method.
Abstract
This essay sets out how Hawthorne audits software, and where that method has failed. It covers two pieces of work: t0035, an audit of PhotoCraft, and t0036, a control in which errors were planted in an essay to see whether the house’s AI reviewers would catch them [1, 11]. The Author designed the method, runs Hawthorne and sets its rules [19: F02, F37].
Our own audit gets an error log, not an exemption. A process failure (our tool or our procedure broke) is recorded separately from a target vulnerability (the software under audit has a flaw) [1]. Three failures are disclosed here. Third-party packages ran on 30 October 2026, in an audit meant to execute nothing. A database change (an SQL UPDATE) was made without a ledger entry and logged only afterwards, at entry 322. And the author edited the text after the Moot, the house’s review panel, had finished; those edits still await approval [1] [19: F29].
The control’s result: reviewers from different AI labs missed planted errors that could only be checked against private records, and the panel caught them once two relevant records were supplied. That is one essay (n=1), reported by the author, with the follow-up designed after the first results were in [11]. We have dropped the claim that the method is new everywhere until a survey of prior work (Question 11) is done [7, 12].
TL;DR
- One operator, rules with receipts. The Author is Hawthorne’s one operator, and several house rules arrived after something broke. Zero guess laundering followed an issue that went live reading as though a study had been built and run, when it was only a plan. The receipt rule for paid generation followed 90 paid videos instead of 30 (tuition paid; lesson filed)
[19: F02, F15, F25]. - Transparency is a house standard, not a guarantee. We publish our error logs and control results so a reader can check the work. That does not make the work true
[1]. - Two corrections to our own tooling. The scoring formula has a confidence multiplier that the first pseudocode left out. The ledger hashes strings as stored, not objects written out again, and only an external checkpoint protects the history it covers
[13,14]. - The control. At first the cross-lab reviewers missed 2 of 10 planted errors, both involving internal evidence. In a follow-up that added a two-record appendix, the panel together caught all 10 (Dario 9, Sam and Sundar 10 each). One essay, one pilot
[11]. - The execution admission. Third-party packages ran on 30 October 2026, and the virtual environment (venv, an isolated software environment) was deleted on 31 October. The live t0035 essay still needs a visible correction
[1][19: N4].
Why you should care
- If you research security: this is a baseline for reproducing an audit when AI tools collect the evidence. It keeps process errors (a gap in the ledger, say) apart from flaws in the target.
- If you audit: the ledger’s construction (a hash chain with a
0x1fseparator) is a template for checking an append-only record, and the failure log is a reference for the common ways an implementation drifts from its specification. - If you read to decide what to trust: the process errors and the unapproved edits are printed, so you can see what the trust rests on. You can check the essay against the error log; full validation means inspecting the artifacts yourself.
- If you build AI: the control reports how models performed with and without the targeted records, which bears on prompt design and on retrieval (RAG, giving a model documents to consult).
Who built this, and where
From a 100-square-foot cottage in an orchard without running water, the Author runs Hawthorne. [19: F01] Every pick, ruling and publication passes through him. [19: F02] By 16 November 2026, the release shelf held 25 numbered Hawthorne issues, and the publication bar had 33 tests guessing how named people would grade sections; those grades are labeled best educated guesses. [19: E] [19: F19] The house “we” is a method. No outside party has audited the system yet. [19: N6]
No publication-bar grades are recorded for this essay yet. [19: N8]
The work ledger does not get to quietly tidy up after itself: it is append-only, designed to accept additions, not updates or deletions. Since 30 October, it had recorded 31,565 events across 64 sessions by 16 November. [19: F22] The separate t0035 audit ledger has 324 entries in a hash chain: each entry’s hash, a digital fingerprint, incorporates the previous one. [19: F23] Its public checkpoint reaches entry 288; later entries are not anchored by a published checkpoint, and the evidence bundle’s site deploy is pending. [19: N3]
The local research model, eddie, does the research and coordinates the work; local models also make the images, video and voice. [19: F04] After a measured quiz, Gemma was given fast triage (quick first checks, never admitting a fact). [19: F08] Frontier models from three labs argue as justices: disagreement is a signal, while agreement is not proof. [19: F10] A court with a standing order not to believe its own applause.
The Author’s challenge, in cleaner clothes: who else could do this? [19: F03] That is a question, not a finding of exclusivity. Here are some receipts: the t0035 failure log records seven false positives, two reviewer errors and a process deviation; the record also includes a third-party package run on 30 October during an audit meant to execute nothing, and an after-the-fact database update, logged in the audit ledger (entry 322). [19: F29] The essay above is the receipts, failures included. The counts show work recorded, not that every result is right.
- H1: Private-evidence blind spot. Reviewers miss private-evidence errors when records are absent, and catch them when provided.
- Falsifier: A follow-up experiment with a full 3.5 MB bundle (not targeted appendix) shows union recall <80% on internal-evidence plants, or failure with full access explains the misses
[11].
- Falsifier: A follow-up experiment with a full 3.5 MB bundle (not targeted appendix) shows union recall <80% on internal-evidence plants, or failure with full access explains the misses
- H2: Prompt transparency. Publishing original generation prompts increases external reviewers' supported error detection by >10% when draft and evidence access are held constant.
- Falsifier: An equivalence test with a prespecified margin (10%) shows the confidence interval includes zero or favors the null, and the study design distinguishes prompt visibility from prompt causation
[11].
- Falsifier: An equivalence test with a prespecified margin (10%) shows the confidence interval includes zero or favors the null, and the study design distinguishes prompt visibility from prompt causation
- H3: Rubric floors. The rubric v0.3 floors prevent under-scoring of critical issues.
- Falsifier: An audit with a confirmed CVSS 10.0 finding scores <6.0 under v0.3 rules
[1].
- Falsifier: An audit with a confirmed CVSS 10.0 finding scores <6.0 under v0.3 rules
- H4: Cross-lab value. Cross-lab AI reviewers add value beyond one strong reviewer or three independent calls to the same model.
- Falsifier: At matched budget, a cross-lab panel does not improve union recall by >5% over the strongest single-model comparator, or increases false accusations
[11].
- Falsifier: At matched budget, a cross-lab panel does not improve union recall by >5% over the strongest single-model comparator, or increases false accusations
- H5: Ledger conformance. The implementation conforms exactly to the adopted rubric's confidence weights, aggregation rules and floor predicates.
- Falsifier: Any valid independently constructed input produces a result inconsistent with the adopted specification
[13].
- Falsifier: Any valid independently constructed input produces a result inconsistent with the adopted specification
- H6: Ledger completeness. Every score-relevant database mutation in the audited interval is reconstructible from the ledger.
- Falsifier: One unreconstructible mutation, an inconsistent reconstruction, or a claimed pre-review event outside the relevant commitment (e.g., finding #32 UPDATE at entry 322)
[1].
- Falsifier: One unreconstructible mutation, an inconsistent reconstruction, or a claimed pre-review event outside the relevant commitment (e.g., finding #32 UPDATE at entry 322)
- H7: Cost efficiency. AI-assisted review has lower total cost than human-only review while meeting a preregistered noninferiority margin for supported detection.
- Falsifier: It demonstrably fails the quality margin, or its total cost at comparable quality is not lower (requires explicit accounting boundaries)
[1].
- Falsifier: It demonstrably fails the quality margin, or its total cost at comparable quality is not lower (requires explicit accounting boundaries)
Purpose: Demonstrate the ledger hash chain construction and scoring formula (corrected).
# Ledger Entry Hash Construction (Corrected)
# Source: local:hwtn.io/evidence/t0035-photocraft-0.17.0/verify_ledger.py
def _entry_hash(prev_hash, ts, actor, action, subject, detail_str):
# 1. detail_str is ALREADY SERIALIZED (stored string, not object)
# 2. Separator 0x1f after EVERY field including the last (detail_str)
raw = prev_hash.encode() + b'\x1f'
raw += ts.encode() + b'\x1f'
raw += actor.encode() + b'\x1f'
raw += action.encode() + b'\x1f'
raw += (subject or "").encode() + b'\x1f'
raw += detail_str.encode() + b'\x1f' # Trailing separator included
# 3. SHA256
from hashlib import sha256
return sha256(raw).hexdigest()
# Scoring Formula (Corrected)
# Source: local:reveng/RUBRIC.md (v0.3) and local:reveng/score.py
def calculate_audit_score(findings):
# Each finding: severity 0-10 (float); confidence 1.0 (confirmed),
# 0.5 (plausible) or 0.25 (speculative).
# An ambiguity (a surface we could not clear) has no severity and a
# flat cost: 0.1 minor, 0.3 major.
total, floor = 0.0, 0.0
for f in findings:
if f.get("ambiguity") is not None:
total += f["ambiguity"]
floor = max(floor, 0.1) # unresolved
continue
# FIX: Multiplier included
total += 4.0 * (f["severity"] / 10.0) ** 2 * f["confidence"]
confirmed = f["confidence"] == 1.0
if confirmed and (f.get("is_malicious") or f.get("is_covert")):
# confirmed CWE-506/CWE-912 or covert exfiltration
# (score.py checks only the CWEs)
floor = max(floor, 9.0)
if confirmed and f.get("is_cvss") and f["severity"] >= 9.0:
floor = max(floor, 6.0) # confirmed CVSS >= 9.0
if not confirmed:
floor = max(floor, 0.1) # unresolved
# FIX: Floors apply to the audit's total, not to each finding
# (rubric v0.3, score.py). score.py reports it rounded to one decimal.
return min(max(total, floor), 10.0)What it shows: The ledger is append-only relative to a checkpoint; the scoring is deterministic with confidence weighting [13, 14]. What it leaves out: Actual JSON schema complexity and drift check logic. The code models covert exfiltration as a flag; score.py itself checks only CWE-506/CWE-912 for the 9.0 floor [13].
The code block shows the correct separator placement. The trailing separator is part of the specified construction; leave it out and the hash will not match the stored chain [1]. The scoring formula shows what the confidence multiplier does. Before floors, omitting confidence multiplies the base points by 1/confidence: fourfold at 0.25 and twofold at 0.5. The final score may be unchanged if the same floor dominates both calculations.
- Claim: "The audit executed no code."
- Check: Compare the 30 October creation logs for the virtual environment with the 31 October deletion logs. Third-party packages ran on 30 October, and installing a package can run its code, for example when it is built from source
[1, 10].
- Check: Compare the 30 October creation logs for the virtual environment with the 31 October deletion logs. Third-party packages ran on 30 October, and installing a package can run its code, for example when it is built from source
- Claim: "The ledger proves nobody can tamper with the history."
- Check: Look for a checkpoint published outside the house after the latest entry. Only entry 288 is published, so entries 314–324 are unanchored, and finding #32 was changed by SQL and logged afterwards at entry 322
[1, 4].
- Check: Look for a checkpoint published outside the house after the latest entry. Only entry 288 is published, so entries 314–324 are unanchored, and finding #32 was changed by SQL and logged afterwards at entry 322
- Claim: "This is the only public methodology with a planted-error control."
- Check: Read Schroter et al. (2008), who inserted errors into manuscripts to test peer reviewers, and NIST's SATE VI report on bug injection. Both came first
[6, 7].
- Check: Read Schroter et al. (2008), who inserted errors into manuscripts to test peer reviewers, and NIST's SATE VI report on bug injection. Both came first
- Claim: "The reviewers caught every error with full access."
- Check: Ask whether the full 3.5 MB bundle was used in Control C. It was not: the reviewers got a targeted two-record appendix
[11].
- Check: Ask whether the full 3.5 MB bundle was used in Control C. It was not: the reviewers got a targeted two-record appendix
- Claim: "The control proves access, not competence, caused the misses."
- Check: Ask whether access was the only thing that changed. The appendix also made the two records prominent and easy to find
[11].
- Check: Ask whether access was the only thing that changed. The appendix also made the two records prominent and easy to find
The Moot is the house's review panel. Models from three labs, called dario, sam and sundar here, argue over a draft, and the research model, eddie, rules on each objection [19: F04, F10]. This is the short version of what that argument settled and what it left open. The item-by-item rulings are in the essay's working folder, which is not public.
Settled
- The first scoring pseudocode left out the confidence multiplier: the worked example gives 2.56 without it and 0.64 with it. It also misstated the 9.0, 6.0 and 0.1 floor conditions. Settled by arithmetic and by rubric v0.3, whose floors apply to the audit's total (see §3)
[13]. - The first hash pseudocode serialized the detail a second time inside the hash function and left out the trailing separator. The verifier hashes the stored string, serialized once when the entry was written
[14]. - An unanchored hash chain does not stop the log's owner rewriting its tail. Publishing head entry 288 in t0035 does not authenticate later entries 314, 321, 322 and 324
[4]. - Panel-union recall adds nothing over the best individual reviewer in either condition. That holds on the supplied counts, and the counts themselves remain unverified
[11]. - The control results must be scoped to the n=1 pilot (8/10 initial union; dario 9/10, sam and sundar 10/10 in Control C with a targeted two-record appendix). They support no general failure rate and do not prove that access alone explains the misses
[6, 11]. - Universal novelty cannot be claimed. Q11 is unresearched, and planted-error evaluations exist in peer review (Schroter et al. 2008) and in static analysis (NIST SATE
VI). That establishes relevant prior art; it does not disprove every narrowly specified novelty claim [6, 7].
- The failure log and the pipeline description must disclose the 30 October 2026 third-party package execution, the SQL UPDATE to finding #32 that was logged afterwards at entry 322, the Moot bypass after v3 (v4–v7), and the unapproved status of t0035 voiced v8–v10
[1]. - robots.txt is not authorization
[15]. Installing a package with pip can run its distribution code[10]. An OSF preregistration is a timestamped, read-only public plan[5]. - CVSS communicates vulnerability severity
[3]; SSVC prioritizes vulnerability-management actions (CISA/CERT SSVC specification). The respective primary specifications support the distinction. - The README address returned homepage HTML during this check, which does not supply the promised artifact verification. The live t0035 essay publishes the stated head hash, calls its evidence record unpublished, and retains broad no-downloaded-code-execution wording. Those are verified statements about the page, not verification of the underlying audit
[1]. - Nonsignificance alone does not demonstrate equivalence. Amrhein, Korner-Nievergelt and Roth (2017) support interpreting estimates and their uncertainty instead of treating a nonsignificant result as no effect
[21].
Still Disputed
- Whether the reported control counts, bundle size and token and cost figures match the raw artifacts. This needs the evidence bundle and control outputs to be published, with hashes
[1, 11]. - The per-reviewer counts. The bundle's result files now report them (dario 7, sam 8, sundar 8 initially; dario missed R9 in Control C), but those files are author-reported and not yet public
[11]. - Whether entries 314–324 can be authenticated. This needs the hashing and verification code, the stored bytes, a witnessed checkpoint published after entry 324 and a reconciliation of the mutable tables. A newly published checkpoint alone cannot prove old timestamps
[4]. - The 175 versus 183 claim-count discrepancy, and the 80 input tokens against 59,827 output tokens across seven dario calls. These need the extraction method and version for each count, the raw ledger tables and the API logs
[1]. - The exact text of seven excerpts that were not found word for word (FIRST, the Python documentation, RFC 9162, SATE VI, Amrhein et al., SSVC and the README). Each needs a fresh archived fetch, and none may be quoted as a quotation until then
[1, 3,4, 6, 9]. - The archived quotation-check count remains unreproduced, because the archived captures and the matching procedure were not supplied. A check against current sources can establish present support; it cannot certify what eddie retrieved at an earlier time
[1]. - Control C's causal interpretation. A matched prospective design that separates evidence presence, targeted presentation, full-bundle retrieval and reasoning would distinguish access effects from the alternatives
[11]. - Scoring conformance and validation. These need separately identified classification and rubric versions, precise predicates and aggregation rules, executable tests and validation evidence; policy floors alone do not supply calibration
[13, 16]. - Generalizability. A bounded prior-art comparison can support bounded novelty, while independently sampled audits and prespecified endpoints are needed for broader performance claims
[6, 7].
Where the review itself slipped
The review made its own mistakes, and they are on the record as well. While ruling on this essay's draft, eddie put Control C's per-reviewer totals into the initial control, contradicting an earlier ruling of his own [11]. Two rulings stated as done things the record does not show: a completed adjudication with separate scoring of unplanted discoveries, and a completed amendment to the failure log [1]. One ruling reversed an inference: failure with full evidence counts against an access-only explanation, not for it [11]. One attached an excerpt from an unrelated paper, Lost in the Middle, to the peer-review prior-art point [7, 8]. Several kept merging the Classification spec with the scoring rubric, and one described the proposed corrected scoring formula as existing behaviour [13, 16].

The quotations were the largest problem. Seven of twelve excerpts failed a word-for-word check, because non-contiguous sentences had been spliced with ellipses or words had been altered, even though the underlying pages supported the objections [8]. The other reviewers caught each of these slips from the supplied record.
Artifact verification: The reported control counts, failure counts, 3.5 MB bundle size and model-call costs remain author-reported pilot observations pending independent artifact inspection. We could not fetch the README or control outputs via public tooling in this check; the evidence bundle’s site deploy is pending [1] [19: N3]. A reported result is not an independently verified one.
Control C generalizability: The control was an n=1 pilot: one essay, ten plants per checker, easy mode. It does not prove general effectiveness; it tests a specific class of error [11].
Execution history: The virtual environment was deleted on October 31, 2026. That does not retrospectively prove the absence of earlier host effects or file access [1]. Deletion is not a time machine.
Novelty claim: Question 11, the prior-art survey, is not yet researched. The universal novelty claim is dropped until it is completed [7, 12].
Approval status: Corrections in voiced v8, v9 and v10 await the Author’s Gate 2 approval. The live page is an earlier version [1] [19: N4].
Cost data: usd is NULL for sam/sundar calls, and voicer and image costs are not in the ledger. The total cost is incomplete [1].
Builder claims: Claims about the Author, his tools and his process are meant to map to entries in the internal builder’s fact file, which includes his own words with dates and is not public [19].
Voicing and checks: This version was written by OpenAI’s gpt-6.1-terra and was not voiced by hthompson [19: T5] [19: N9]. Its changed sections were accuracy-checked by one lab only, Anthropic’s claude-opus-5-5; a Gemma model quick-checked a handful of load-bearing claims as triage, not evidence. Google’s Gemini did not take part, and the OpenAI seat is conflicted as the writer’s lab [19: F36].
This revision: Version 10, 22 November 2026, reorders the essay. It now opens with the House of Cards; the abstract, TL;DR, “Why you should care” and the builder’s note follow the conclusion as The Short Version, and the first three were rewritten for plainer reading with the same facts and sources. In the body, the builder’s name is replaced by “the Author”. Version 9, 21 November 2026, answered an outside reader's critique: it merges the two BS Detector sections into one, replaced three figure captions that had no figure with tables built from the control's result files and the ledger verifier, rewrote the Moot record for a reader who was not in the room, stated a proposed repair for the unanchored ledger tail, reconciled the fact-watch dates, relabelled the pilot counts in the claim register, added reference 21 and rewrapped three code comments that ran off the page. The edits in both versions were made by Anthropic's Claude at the Author's request. They were not re-voiced and the panel has not reviewed them.
Tools that testify
The Author keeps a release shelf that can testify against him. [20: PW06] On 16 November 2026, it held 41 read-only golden-master releases (reference copies of what shipped), containing 138 files, and three append-only hash chains (logs added to rather than changed, with each entry’s digital fingerprint linked to the previous one); the quick verifier passed. [20: PT11] Release uploads use --immutable to prevent overwriting; mirrors copy rather than sync and move replaced files into a backup folder. [20: PT06] The half-hourly shelf job skips itself while an earlier upload is still running. [20: PT06] That is a tool with instructions for what not to do, which is a useful kind.
His research model got its job by audition: on 4 November 2026, 18 of 18 of Nemotron’s quotations checked out word for word, against 10 of 16 for Qwen. [19: F07] He chooses the tools, sets the rules, rules on the output and decides what ships. [19: F02, F37]
Workflows with a failure log
The Author’s workflow has a review trail long enough to trip over, and we can count it. On 16 November 2026, the house found 24 essay folders with a long-Moot record, 37 justice review files and 47 fix-desk files. [20: PW01] After voicing, a model from a different lab checks for drift and reports rather than edits; 22 essay READMEs reported checks covering 2,726 claims, with 91 flagged. [20: PW05] Those are records of checks, not a verdict that every claim survived one.
The house also checks whether work that looks finished is actually there: read the log, compare expected outputs with files on disk, redo what is missing and check the remote service. [20: PW09] In the animation work, code-drawn blocking boards come before the Author’s approval and any ink keys; by the evening of 15 November, 244 shots had been blocked across eight pieces. [20: PW07] Stick figures first, production later.
Environment, minus the scenery
The Author’s memory rule, set on 15 November 2026, is “optimize memory when needed, we must finish these work products.” [20: PE03] When training needed room that day, idle image servers were cleared under that rule and training started. [20: PE03]
Under the Author’s standing rule, the shelf sync sends smaller files first. [20: PE04] In his words, auto-refill “is turned off on everything” and he has “done this on purpose”; the rule is to display spend estimates, stop a provider cleanly and say which account stopped, never switching models silently. [20: PT03]
| Claim | Receipt | Entry |
|---|---|---|
| Release verification | Three hash chains; 41 golden-master releases; quick verifier passed on 16 November 2026 | [20: PT11] |
| Review trail | 24 long-Moot folders, 37 reviews and 47 fix- desk files, counted 16 November 2026 | [20: PW01] |
| Drift checked | 22 reported checks, 2,726 claims and 91 flags, counted 16 November 2026 | [20: PW05] |
| Memory managed under his rule | Idle image servers were cleared so training could start, 15 November 2026 | [20: PE03] |
| Spend controlled on purpose | Auto-refill off on every account; no silent model switches | [20: PT03] |
evidence/t0036-control/RESULTS.md
and
RESULTS-C.md
, 2026-10-31. Released to the repository and the shared drive; site deploy pending, so not yet public (checked 2026-11-16).
essays/t0035-mostly-harmless- no-execution-audit/voiced/t0035-mostly-harmless-no-execution- audit.voiced.v10.md
, 2026-10-31. Awaits the Author's gate 2; not public.
reveng/RUBRIC.md
) and its scoring script (
reveng/ score.py
). Internal documents, adopted 2026-10-30. Not yet public.
evidence/t0035- photocraft-0.17.0/verify_ledger.py
), 2026-10-31. Released to the repository and the shared drive; site deploy pending, so not yet public.
docs/zero-guess-laundering-v1.md
). Internal policy, 30 October 2026. Not yet public.
essays/BUILDER-FACTS.2026-11-16.md
). Internal record, 16 November 2026: every claim about the Author, his tools and his process, with its evidence and a label (from the record, his own words, published, or inference). Cited as [19: entry]. Not yet public.
essays/BUILDER-FACTS.proficiency. 2026-11-16.md
). Internal record, 16 November 2026: tools, workflows and environment, each with its evidence (path, commit, config key, command and output, or his dated words) and a label. Cited as [20: entry]. Not yet public.
| Claim | Footnote(s) | Source type | Status |
|---|---|---|---|
| Self-audited processes require reader-verifiable logs. | [1] | Primary | Established |
| t0035 PhotoCraft audit includes error log. | [1] | Primary | Established |
| t0036 control experiment is n=1 pilot. | [11] | Primary | Established |
| Third-party packages executed on Oct 30, 2026. | [1] | Primary | Established |
| venv deleted Oct 31, 2026. | [1] | Primary | Established |
| Scoring formula includes confidence multiplier. | [13] | Primary (internal) | Established |
| Floors: 9.0 for malicious code. | [13] | Primary (internal) | Established |
| Floors: 6.0 for CVSS ≥ 9. | [13] | Primary (internal) | Established |
| Floors: 0.1 for unresolved. | [13] | Primary (internal) | Established |
| Floors apply to the audit's total, not to each finding. | [13] | Primary (internal) | Established |
| score.py's 9.0 floor checks CWE-506/ CWE-912 only, not covert exfiltration. | [13] | Primary (internal) | Established |
| Initial control: 8/10 recall (union). | [11] | Primary | Author-reported pilot (n=1) |
| Control C: 10/10 recall (targeted). | [11] | Primary | Author-reported pilot (n=1) |
| Repair for the unanchored ledger tail: publish the pre- registration hash and ledger checkpoints outside the house before the work they protect. | [4, 5] | House proposal | Proposed, not adopted |
| Novelty claim dropped (Q11). | [7, 12] | Primary | Established |
| Ledger entry 288 does not authenticate 314-324. | [4, 14] | Primary | Established (v3: was Disputed; the Moot's Settled list and the hash-chain structure both settle it, and no checkpoint after 288 had been published as of 2026-11-16) |
| Cost audit incomplete ( usd NULL). | [1] | Primary | Established |
| v8-v10 corrections unapproved. | [1] | Primary | Established |
| Schroter et al. 2008 establishes planted- error prior art. | [7] | Secondary | Established |
| NIST SATE VI establishes static analysis prior art. | [6] | Secondary | Established |
| Robots.txt is not authorization. | [15] | Primary | Established |
| pip installation runs code by default. | [10] | Primary | Established |
| Hash separator 0x1f after every field. | [14] | Primary (internal) | Established |
| The Classification's verdict is a Species · Immediacy · Devastation tuple; the score is a sort key; v0.3 is pre- validation. | [16] | Primary | Established |
| The method's rules are set by one person, the Author, who runs Hawthorne from a 100- square-foot cottage in an orchard without running water. | [19: F01, F02, F37] | His own statement (2026-11-16); published page [17] | His statement |
| Model roles were assigned by measurement (bake-off 18/18 vs 10/16 verified quotes; quiz 20/20 vs 19/20). | [19: F07, F08] | Internal record | Established |
| House rules were written after named failures (No. 04; the 28-test failure; 90 paid videos instead of 30; a 403; an out-of- memory crash). | [17], [19: F15, F19, F25, F26, F33] | Internal record; published page | Established |
| The fact watch last ran cleanly on 2026-11-03 and failed every scheduled run from 2026-11-07 to 2026-11-16. | [19: N1] | Internal record (run logs) | Established |
| Scale counts (25 numbered issues, 33 tests, 31,565 ledger events, 324-entry audit ledger) as of 2026-11-16. | [19: E] | Internal record, counted with the commands listed there | Established |
| "Who else in the world could do this?" | [19: F03] | His own question | A question, not a claim |
| Release shelf: 41 read- only golden-master releases (138 files) and three append-only hash- chained logs; quick verify passed 2026-11-16. Release uploads are immutable; mirrors copy, never sync, with a backup folder. | [20: PT11, PT06, PW06] | Internal record (command output; script); his own words | Established |
| Review trail on 2026-11-16: 24 essay folders with a long-Moot record, 37 justice review files, 47 fix-desk files; 22 drift checks covering 2,726 claims, 91 flagged. | [20: PW01, PW05] | Internal record (counts with the commands) | Established |
| The house checks finished work against the log and the files on disk; animation is blocked by code before approval and ink (244 shots by 2026-11-15). | [20: PW09, PW07] | His own words; internal record | Established |
| Under his Rule -1, idle image servers were cleared on 2026-11-15 so training could start. | [20: PE03] | His own words; internal record | Established |
| The shelf sync sends the smallest files first, under his standing rule. | [20: PE04] | Internal record (session report, rclone output) | Established |
| Auto-refill is off on every account on purpose; a credit error stops that provider cleanly with no silent model switch. | [20: PT03] | His own words (2026-11-16) | His statement |
References
- 01hwtn.io. "t0035 PhotoCraft Evidence." Evidence, n/a. https://hwtn.io/evidence/t0035-photocraft-0.17.0/README.md (accessed 2026-10-31; on 2026-11-16 this address still returned the site's homepage, see §6).
- 02hwtn.io. "No. 03 Moot of Doom." Essays, 2026-10-29. https://hwtn.io/essays/moot-of-doom-essay-and-other-fun-facts (accessed 2026-10-31).
- 03FIRST. "CVSS v4.0 Specification Document." Specification, n/a. https://www.first.org/cvss/v4.0/specification-document (accessed 2026-10-31).
- 04RFC Editor. "RFC 9162." RFC, n/a. https://www.rfc-editor.org/rfc/rfc9162.html (accessed 2026-10-31).
- 05OSF. "Welcome to Registrations." Help, n/a. https://help.osf.io/article/330-welcome-to-registrations (accessed 2026-10-31).
- 06NIST. "SATE VI Report: Bug Injection and Collection." NIST SP 500-341, June 14, 2023. https://www.nist.gov/publications/sate-vi-report-bug-injection-and-collection (accessed 2026-10-31).
- 07Schroter, S., et al. "What errors do peer reviewers detect, and does training improve their ability to detect them?" J R Soc Med 101:507–514, 2008. doi:10.1258/jrsm.2008.080062. https://pmc.ncbi.nlm.nih.gov/articles/PMC2586872/ (accessed 2026-10-31).
- 08Liu, N. F., et al. "Lost in the Middle: How Language Models Use Long Contexts." arXiv:2307.03172, submitted July 6, 2023; revised November 20, 2023. https://arxiv.org/abs/2307.03172 (accessed 2026-10-31).
- 09Python Software Foundation. "Python json." Documentation, n/a. https://docs.python.org/3/library/json.html (accessed 2026-10-31).
- 10PIP. "Secure Installs." Documentation, n/a. https://pip.pypa.io/en/latest/topics/secure-installs/ (accessed 2026-10-31).
- 11Hawthorne. "t0036 Control Results." Evidence bundle files
- 12Hawthorne. "No. 17, t0035, voiced v10." Unpublished correction,
- 13Hawthorne. "Hawthorne score: rubric v0.3" (
- 14Hawthorne. "verify_ledger.py," the t0035 evidence bundle's ledger verifier (
- 15IETF. "RFC 9309: Robots Exclusion Protocol." RFC, September 2022. https://www.rfc-editor.org/rfc/rfc9309.html (accessed 2026-11-16).
- 16Hawthorne. "The Hawthorne Classification," spec v0.3, pre-validation draft, 31 October 2026. https://hwtn.io/classification (live since 2026-11-16).
- 17Hawthorne. "How Hawthorne Runs," v1, 1 November 2026. Projects page, https://hwtn.io/about.
- 18Hawthorne. "Zero guess laundering, v1" (
- 19Hawthorne. "The builder's fact file" (
- 20Hawthorne. "The builder's fact file, part two: proficiency" (
- 21Amrhein, V., Korner-Nievergelt, F., and Roth, T. "The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research." PeerJ 5:e3544, 2017. doi:10.7717/peerj.3544. https://pmc.ncbi.nlm.nih.gov/articles/PMC5502092/ (accessed 2026-11-21).