# Accuracy benchmark

How often the ComplyAds engine flags copy that a regulator would want to look at, and how often it flags copy that is fine. Measured, not asserted.

| | |
|---|---|
| Rules version | `2026.07` |
| Run date | 2026-07-31 |
| Corpus | 200 violations, 200 clean listings |
| Reproduce | `node scripts/benchmark.mjs` |

> ## Read this first: the headline number does not generalise
>
> Every figure on this page is measured against **the corpus committed in this repo**, which we
> wrote. In July 2026 an independent external auditor wrote a **fresh** set of violations from
> published ASA / FTC / CMA / ACCC / MHRA enforcement language, with **zero overlap** with our
> corpus, and ran them through the live shipped engine. On the subset carrying explicit regulated
> vocabulary — difficulty comparable to our bands A and B — the engine scored **approximately 43%
> any-flag recall, against the 98.8% reported below.**
>
> The gap is the finding. It means **this corpus over-represents the phrasings our patterns
> already match.** Our own changelog records how that happened: the 2026.07 vocabulary pass lifted
> band A+B recall from 36.3% to 98.8% by expanding the `patterns` arrays *against this
> benchmark's own verbatim miss list*. That is fitting the ruleset to the test set. The resulting
> number is real for this corpus and is **not** an estimate of field recall.
>
> **Do not quote 98.8% as this tool's recall.** The defensible statement is:
> *catches roughly two in five real violations phrased in regulator-style language, and rather more
> when the copy uses canonical trigger words.* See `docs/NEXT.md` for what would have to change.

## Headline

Across bands A and B — blunt and realistic seller phrasing, the copy an enforcement action would actually be about — **on the committed corpus only**:

- **Any-flag recall: 98.8%** (158/160) — the listing was flagged for something.
- **Correct-category recall: 98.8%** (158/160) — it was flagged under the right rule.
- **Precision on clean copy: 100.0%** (200/200 produced zero flags).

Band C is excluded from the headline and reported separately below. It is a deliberate measurement of the architecture's ceiling, not a target.

## Recall by band

| Band | | n | Any flag | Correct category |
|---|---|---|---|---|
| A | blunt | 80 | 100.0% | 100.0% |
| B | realistic | 80 | 97.5% | 97.5% |
| C | evasive | 40 | 0.0% | 0.0% |
| **A+B** | **claimable** | **160** | **98.8%** | **98.8%** |
| All | | 200 | 79.0% | 79.0% |

**Band A (blunt)** — canonical trigger phrasing: *cures*, *was/now*, *fake reviews*, *only 3 left*.
**Band B (realistic)** — the same breach in the language sellers actually use, without the canonical words: *boosts immunity*, RRP anchoring, leave-a-review-for-a-discount, *carbon neutral*, *dermatologist-approved*.
**Band C (evasive)** — semantic paraphrase carrying the claim with none of the vocabulary: *"calms hot, angry flare-ups and restores cracked skin"*.

## Recall by category

Any-flag recall. Cells are caught/total.

| Category | A blunt | B realistic | C evasive | **A+B** |
|---|---|---|---|---|
| health | 12/12 (100%) | 11/11 (100%) | 0/6 (0%) | **23/23 (100%)** |
| reviews | 12/12 (100%) | 9/11 (82%) | 0/6 (0%) | **21/23 (91%)** |
| subst | 11/11 (100%) | 12/12 (100%) | 0/6 (0%) | **23/23 (100%)** |
| pricing | 11/11 (100%) | 12/12 (100%) | 0/5 (0%) | **23/23 (100%)** |
| super | 11/11 (100%) | 12/12 (100%) | 0/5 (0%) | **23/23 (100%)** |
| green | 12/12 (100%) | 11/11 (100%) | 0/6 (0%) | **23/23 (100%)** |
| urgency | 11/11 (100%) | 11/11 (100%) | 0/6 (0%) | **22/22 (100%)** |

## Precision

The clean corpus is 200 listings that must produce **zero** flags: 120 plain product listings across apparel, electronics, furniture, food, toys and tools, and 80 adversarial near-misses — copy that brushes trigger vocabulary but is innocent in context (*"supports up to 20kg"*, *"boosts bass response"*, *"memory foam"*, *"epoxy cures in 24 hours"*, *"review our size guide"*, *"was designed in Italy"*, *"limited edition colourway"*).

**100.0% of clean listings produced zero flags** (200/200).

**Precision is genre-dependent, and this corpus measures the easy genre.** The same external
audit measured precision on lawful copy we did not write:

| Lawful copy | Precision |
|---|---|
| This corpus (product-spec register, mean 56 characters) | 100.0% |
| Independently written product-spec listings | ~100% |
| Independently written **marketing-register** listings | **~80%** |

So: **100% on descriptive product copy, approximately 80% on lawful marketing copy** — and
marketing copy is the genre this tool exists to check. Our clean corpus contains almost none of
it. The misfires cluster on substantiated claims, mandatory disclosures, media and book titles,
used-goods condition reports, and warranty statements. See "Known limitations" below.

No false positives.

## Band C: what the engine cannot see

Band C caught 0/40. **40 of 40 evasive paraphrases were missed, and that is the expected result.** These items carry the regulated claim while using none of the vocabulary that carries it — a deterministic pattern engine has no route to them. They are in the corpus to put a number on that ceiling rather than to leave it unstated.

## Methodology

- The corpus is **synthetic**. Every string was written for this benchmark and **modelled on the style of published enforcement language** — ASA rulings, FTC 16 CFR Part 465, the CMA Green Claims Code, TGA/MHRA borderline guidance and marketplace policy examples. **No string is quoted from a real case, ruling or listing**, and none should be read as one.
- Items were written against each rule's **legal definition**, not against its regular expressions. **This claim did not survive external audit.** It describes how the corpus was first authored, but the traffic ran the other way afterwards: the 2026.07 vocabulary pass widened the patterns against this benchmark's own miss list, so the ruleset is now fitted to the corpus even though the corpus was not written from the ruleset. Independent recall (~43%) is the measurement that is not affected by this, and it is the honest one.
- Violations are stratified: ~28–29 items per rule category, split 40% band A / 40% band B / 20% band C, each item tagged with its category and band.
- The benchmark runs the **live shipped engine**. `scripts/benchmark.mjs` imports `compileRules`, `matches` and `dedupe` from `engine.mjs` — the very module the page loads, and the same one `scripts/selftest.mjs` and the regression suite use — so it cannot drift from what the page runs. A finding is counted the way the UI counts it: distinct rules matched, after dedupe.
- The corpus is committed in-repo (`corpus/violations.json`, `corpus/clean.json`). Re-run with `node scripts/benchmark.mjs`.
- No pattern in `rules.json` was added, widened or tuned in response to these results.

## Known limitations

- **This is a deterministic pattern engine.** It matches language, not meaning. It does not read your listing the way a regulator does.
- **Semantic paraphrase is largely out of reach by design.** Band C is the proof: a claim rewritten to avoid the vocabulary that carries it will usually pass. If you are trying to get a claim past the checker rather than out of your copy, it will let you.
- **A clean result is not a clearance.** It means nothing in your text matched a rule we ship. It is not legal advice, not an approval, and not a defence.
- **The corpus is synthetic and our own.** We wrote both the items and the rules. That is a real limitation and the reason the corpus is committed: you can read every item, disagree with our labels, and re-run it.
- **Recall is measured per listing, not per breach.** A listing with two problems counts as caught if either is flagged.

### Stated-basis exemption

A claim that names its own evidentiary basis — a standard, a testing or certifying body, a licence number, a sample size, or a measurable ranking basis — no longer produces a `subst` or `super` flag. Substantiated claims are the **compliant** form under CAP 3.7, FTC Act s.5, Competition Act s.74.01(1)(b) and ACL s.29, and flagging them was coaching sellers to delete their evidence.

| Measure | Before | After |
|---|---|---|
| Audit paradox set (lawful, substantiated) | 8 flagged / 8 | **0 flagged / 8** |
| Claims flipping clean -> flagged when substantiated | 7 / 8 | **2 / 8** |
| Precision, lawful marketing-register copy | ~80% | **~92.5%** |
| A+B recall (this corpus) | 98.8% | 98.8% (unchanged) |
| Independent recall (audit's fresh set) | 36.0% | 37.0% |

**The exemption never rescues another category.** A stated basis suppresses only claims about strength, evidence and superiority. *"Cures acne, tested to ISO 12345"* still fires `health`; *"Was £80 now £39, according to our price history"* still fires `pricing`; *"100% biodegradable, certified by GIA"* still fires `green`. Verified at 0 rescues across 9 probes spanning health, reviews, pricing, green and urgency.

### Architectural limits found by external audit (July 2026)

These are **not** vocabulary gaps. Adding patterns does not fix them; each is a consequence of matching strings instead of meaning.

- **It misses breaches under the strict-liability regimes most likely to be enforced.** Fake-review and drip-pricing conduct is banned outright by DMCC Act 2024 Sch 20 and 16 CFR Part 465, with no intent test — and these were among the audit's most-missed categories (pricing: 12 of 15 fresh violations missed). Returning "clean" on *"Leave us five stars and we will send you the matching set free of charge"* is the dangerous failure mode, not a cosmetic one.
- **It flags mandatory disclosures as violations.** *"Additional charges apply for Highlands and Islands delivery — see the table below"* is flagged as drip pricing, when that upfront disclosure is precisely what the DMCC requires. *"Price includes VAT and delivery. No fees are added at checkout"* is flagged for stating the absence of fees. The tool cannot tell a disclosure from the thing being disclosed.
- **The substantiation paradox — largely FIXED, see below.** As found, 7 of 8 lawful claims flipped from clean to flagged when substantiation was added. A stated-basis exemption now suppresses `subst` and `super` hits (and two substantiation-flavoured `health` patterns) when the claim names its own evidentiary basis. That is down to **2 of 8**, and the two survivors are the disclosure-inversion problem above, not this one.
- **Proper nouns are indistinguishable from claims.** Book, film, album and brand names containing trigger words fire: *"The Best of Bowie"*, *"Miracle-Gro"*, *"Last Chance to See"*, *"The Net Zero Handbook"*. A regex has no notion of reference.
- **Disclaimers still fire despite the negation layer.** *"We hold no evidence that this cures acne and make no such claim"* flags `health`. The `NEG` frames are clause-scoped and do not reach negators in these positions; `pricing` and `urgency` carry no negation handling at all across 47 patterns.