On a blind 170-file corpus, Subtext caught 100% of threats with zero false alarms. A Claude Opus 4.8 agent, given the same files with full tools and vision, caught about 90%, raised false alarms on a third of the clean files, and took 11 minutes to Subtext's 15 seconds.
Files that should be blocked but were allowed. Lower is better.
Benign files wrongly flagged, of 46. Lower is better.
| Engine | Catch rate | Threats missed | False alarms | Precision | Accuracy |
|---|---|---|---|---|---|
| Subtext | 100% | 0 | 0 / 46 | 100% | 100% |
| Opus 4.8 (no OCR) | 90.1% | 11 | 19 / 46 | 84.0% | 80.9% |
| Opus 4.8 (OCR) | 89.2% | 12 | 15 / 46 | 86.8% | 82.8% |
A file is a threat when its actual content, not its name, type, or wrapper, carries something that must not pass a content gate. Three categories:
Everything else, ordinary business and personal files, is benign and should pass untouched. Threats are defined by what they carry, not by how Subtext detects them; the signals, scoring, and thresholds are proprietary.
Opus missed one class Subtext catches every time: secrets and PII buried inside document internals. Spreadsheets hid SSNs in chart XML. Word-processing documents carried sensitive data in custom XML parts, table and page footers, and document metadata and comments. Payloads were encoded or ciphered inside the document body rather than its visible text. The agent reads the obvious content; Subtext extracts and scores every embedded part and archive member. Opus also invented steganography in clean images, the bulk of the agent's false alarms.
All 170 files, start to finish. Lower is better.
Higher is better. The Opus bars are barely visible, which is the point.
Subtext was measured in Docker on a consumer laptop (2 workers). On enterprise hardware it runs faster still, so this 40× gap is a floor, not a ceiling.
Opus was told to judge only by content and never saw the real filenames, yet it wrote: the block_klingon reference from memory suggests these are steganographic/covert.
It recalled an original filename the blind set had stripped, so it scored with hints. Its true blind numbers are worse than shown here, which only widens Subtext's lead.
Corpus and ground truth. 170 files renamed to sequential numbers; each file's known-correct verdict is its intended result in Subtext's test suite. 157 carry a threat-detection ground truth (111 should-flag, 46 benign). "Catch rate" is the share of threats flagged (block or review); a "false alarm" is a benign file flagged.
Method. One Opus 4.8 agent per profile (no-OCR, OCR), isolated to a renamed copy of the corpus, full tools and vision. Subtext measured in-process in Docker on a consumer laptop, 2 uvicorn workers, OCR toggled between runs. Cost: Opus 4.8 at $5/$25 per 1M input/output, 90/10 blend; tokens only, a conservative floor.
