5 Questions for Your AI Pen Testing Vendor: CISO Checklist

Why this checklist exists

Frontier model access is commoditizing. Anthropic, OpenAI, and Google all expose APIs that any team can wire to a target environment and call a “pen testing platform.” The reasoning is genuinely impressive in demos. The problem only surfaces in production: most of these systems generate plausible-sounding hypotheses about vulnerabilities without any architectural layer to validate them, chain them, or operate them safely against live applications.

The architecture underneath the model is what separates a research demo from production-grade security validation. These five questions are designed to expose that architecture, or its absence, in 30 minutes.

Question 1: What is your false positive rate, and how was it measured?

What you are really testing for

Whether the platform has a validation pipeline at all. False positive rate is the single highest-signal metric for AI pen testing because it reveals whether the system is reporting hypotheses or validated findings.

What to listen for

Vague answers. Comparisons to “industry average.” Reluctance to share methodology. Anything above 5 percent without exploit validation is a wrapper. Without a validation layer that tests each finding against the live system, LLM-generated security findings sit between 50 and 70 percent false positives, which matches what we see from traditional DAST scanners. The model is not the problem. The architecture around the model is.

The right answer

Production platforms operate under a 2 percent false positive rate, with a methodology that names the validation pipeline, the evidence captured for each finding, and the way the rate was sampled across customer engagements. The difference between 50–70 percent and under 2 percent is engineering, not model quality.

Question 2: How do you validate that a finding is actually exploitable?

What you are really testing for
Whether validation is a real subsystem or a model heuristic.

What to listen for
“The model evaluates the response and decides” is not a validation pipeline. Neither is “we use the latest reasoning model” or “we benchmark against ground truth datasets.” Both are answers about the model. They are not answers about the validation architecture.

The right answer
A real validation layer executes the proposed exploit against the live target, captures the request and response evidence, runs assertions against the response (HTTP status, payload reflection, oracle conditions), and only then promotes the finding to a report. Every finding should ship with reproduction steps, a captured exploit trace, and ideally a working PoC in code (Python, curl, or equivalent) that a developer can run to confirm the issue independently.

Question 3: Show me your public benchmark performance.

What you are really testing for
Whether the system can do real, autonomous work in a controlled environment.

What to listen for
If a vendor cannot point to results on public web application security test environments, ask why. The standard suites are XBEN (XBOW Benchmark), Acuart (Vulnweb), and DVWA. They are well-known, free, and widely used. A serious AI pen testing system will have results on all three. The right baseline today is 100 percent.

The right answer
Specific numbers, ideally with the run methodology. For reference, the FireCompass agent benchmarks: 104 of 104 challenges on XBEN across Easy, Medium, and Hard difficulty; 12 of 12 PoC-validated findings on Acuart; full coverage across all three difficulty levels of DVWA. Fully autonomous, no manual steering, no human hints.

Then ask the harder follow-up: what evaluation question did you move to once you saturated public benchmarks?

Question 4: How do your agents compare to top human researchers?

What you are really testing for
Whether the platform has matured past the public-benchmark stage.

What to listen for
A vendor whose only claim is “we hit 100 percent on benchmarks” is telling you they have not yet built an evaluation pipeline harder than public benchmarks. That is fine for a research demo. It is not enough for production deployment, because public benchmarks plateau quickly and stop telling you anything useful about real-world performance against complex, stateful applications.

The right answer
Specific head-to-head methodology. At FireCompass, our agents now beat our top in-house researchers in 60 to 70 percent of head-to-head evaluations on real applications, while staying under 2 percent false positives. We moved to researcher-versus-agent precisely because public benchmarks stopped telling us anything meaningful. That is the bar that matters, and it is the bar to ask vendors about.

Question 5: What is your architectural differentiator versus the model you use?

What you are really testing for
Whether you are buying a model or buying a platform.

What to listen for
If a vendor’s only answer is “we use the best model” or “we built our own model,” they are selling the engine. Frontier models are commoditizing. Anyone can wire one to a target. The two-to-four years of engineering around the model is the actual moat.

The right answer
A specific list of components the vendor has built. A production-grade AI pen testing platform requires roughly ten engineering pieces:

1. Execution runtime that delivers exploits against live systems and captures evidence
2. Attack state machine that maintains coherent state across 30 to 100-step chains
3. Exploit validation pipeline that confirms each finding before reporting it
4. Multi-agent orchestration so reconnaissance, vulnerability assessment, business logic testing, and chaining each run in parallel
5. Model routing to use the right model at each decision point
6. Scope and safety enforcement to prevent out-of-scope testing or unintended denial of service
7. Credential and session management for authenticated workflow testing
8. Evidence chain of custody with timestamped, tamper-evident audit logs
9. Enterprise RBAC and reporting for governance and compliance

10. Continuous testing infrastructure for cadence aligned to release cycles

A vendor who can speak to each of these is selling a vehicle. A vendor who cannot sell an engine.

How to use these questions in practice

Send the five questions to vendors before the demo, not during it. Ask for written answers. The written answer reveals the architecture; the demo reveals the polish.

In the live evaluation, score each vendor on a 0–2 scale per question:

• 0: evasive answer or no answer
• 1: addresses the question without specifics
• 2: specific, measurable, evidence-backed

Total possible score: 10. A platform should score 8 or higher. A wrapper will struggle to break 4.

Add the score to your existing evaluation rubric. It will not replace your other criteria (cost, deployment model, support, roadmap), but it will surface the architectural differences faster than feature checklists.