How we benchmark detection and false alarms

Every security tool claims to find more. Very few say how many of the things they report are wrong, and fewer still say what they missed. Those two numbers decide whether a tool saves a reviewer time or costs it, so they are the two we publish first.

The corpus

We benchmark on apps where the answers are already known: intentionally vulnerable training apps, and real apps with previously disclosed vulnerabilities whose details are public. The corpus covers Android, iOS, Windows and firmware targets. Before each run, every known vulnerability in each target is written down and the list is frozen. Nothing is added to the answer key after we see the output.

Known-answer corpora have an obvious weakness: the vulnerabilities in them are the kind people already know how to find. A tool tuned to a public corpus can look excellent and still miss the novel issue in your client's app. We do not have a full answer to that. We mitigate it by rotating disclosed targets in from recent advisories, and by reporting results per platform rather than as one flattering average.

The run

The agents get the binary and nothing else. No hints about where to look, no answer key in context, no retries to pick the best of several runs. Each target is analysed once, with the same configuration a customer would get.

The scoring

A human reviewer scores every reported finding against the frozen list. A finding is a hit if it identifies a listed vulnerability with evidence that would let a tester reproduce it. Anything else reported as Confirmed or Likely is a false alarm, including real issues that are not on the list. We count those separately rather than quietly crediting ourselves for them.

Unverified items are excluded from both rates, because they are explicitly not claims. We report how many there were, so you can see whether the system is hiding uncertainty in that bucket.

Three numbers come out of each run:

• Detection rate: listed vulnerabilities found, over listed vulnerabilities
• False-alarm rate: false alarms, over all Confirmed and Likely findings
• Median time to report: from upload to a finished report

What we publish

Everything: per-target results, the misses, the false alarms with the reviewer's note on each, and the scoring rules. The numbers will go up and down between releases, and we will publish the regressions too.

Until launch the results are held back while the corpus and scoring rules settle. They publish at launch, on the benchmarks page, with the method above.