Home
Summary of a closed-door CISO roundtable, hosted by CISO Platform with FireCompass, August 2026. Held under Chatham House convention.
Senior security leaders spent a session on three questions. What does it cost to run an AI pentesting agent at full speed? What has to be true architecturally before you point one at production. And what happens to the testing program you already built once you accept both answers.
Two of the conclusions argue against things this industry has spent five years selling, including a term FireCompass coined “CART” (Continuous Automated Red Teaming).
1. What the HackerOne experiment established, and what it did not
The setup. One agent type was used, the web app pentest agent. HackerOne is a race: two researchers finding the same vulnerability produce one credit, and it goes to whoever files first. So APIs fed each newly published target into the system automatically, with the scan starting without a human trigger. That is the posture a capable adversary takes. Offense is a game of speed, not a treasure hunt.
Context window is the binding technical constraint, and large context drives cost up fast. A root agent crawls the application, establishes its business context, then decomposes the work into sub-agents. A shopping cart triggers a business logic agent. Object references trigger IDOR and BOLA agents. Those classes are precisely what signature-based automation cannot reach, and the decomposition is also what holds the token bill down.
The measured result. Top-three placements across several HackerOne business leaderboards on approximately $5,000 a month in compute. Full methodology, disposition data and limitations are published in our HackerOne post.
The constraint was process, not model quality. HackerOne requires a human to post each finding, so submissions cannot be pushed through an API. One human review shift a day was funded when more was needed. Real findings queued while faster competitors filed the same bug, and most of the gap between what the agent found and what earned credit is duplicates rather than false positives. The disposition data behind that is in the HackerOne post.
What this does not establish. One quarter, one platform, one agent type, self-reported. It is a credible read on unit economics and a poor substitute for a benchmark. Leaderboard rank in particular blends capability with filing speed, which is why the disposition data matters more than the placement.
2. AI finds new attack paths, not new attack techniques
This distinction settled a lot of debate. No AI system, ours included, has invented a new attack technique. What agents contribute is exhaustiveness.
One chain, from a large insurance company. A pre-production asset had a directory traversal. The directory held logs. The logs held a database password. The database was not reachable externally, so the path looked closed. The agent retained the password, compromised a different system, obtained SSH root there, and reached the database from the inside. We have published a comparable chain from external web application to internal foothold (Read here).
Any competent tester could construct that. The question is whether anyone has time to attempt every case.
It changes the report, too. A typical pentest report is a list. An agentic report can show multi-stage attacks with the point at which each stage succeeded or failed, and the failed paths carry assurance value a consultant cannot document at scale. Prioritization then runs on two inputs together: business context and technical context, where technical context means attack paths. If blocking one weakness closes five downstream paths, it outranks a higher-severity issue that leads nowhere.
3. Guardrails have to be deterministic, and that is an architecture decision
Do not use AI to guard AI. Probabilistic controls compound rather than cancel. A model erring at 0.1 behind an AI guardrail missing at 0.1 leaves residual risk, not zero. The final enforcement layer is always rule-based, because rule-based is deterministic.
Do not instruct a model not to act. Remove the capability. Telling a model to avoid something leaves the tool in its hands. Agents that must not do something have no access to the tool at all. In this architecture the models are helpless: they have a brain and nothing else. The component with the brain has no tooth, the component with the tooth has no brain, and a separate orchestrator controls both. A monolithic design cannot enforce that separation, which is why this is an architecture-level decision rather than a policy one.
Reasoning and action have to be recordable. You should be able to see what an agent thought and what it did, with full traceability. When something goes wrong at a large enterprise, the standard is demonstrating that adequate precautions were taken rather than gross negligence. An audit trail is what meets it.
The control stack described in the room:
- Defined agent types with concurrency and peak-load ceilings, one type at 125 concurrent requests, another at 500
- An AI input firewall and an AI output firewall. Chatting with the agents is technically possible and disabled purely for safety
- Action scope constraints and fine-grained capability control, covering not only whether an agent can act but to what extent
- Throttling and rate limiting per target
- Role and application-based access control at data-class granularity, so one person sees the logs and another does not, one sees the exploit code and another does not
- A complete audit trail of every decision and action
- Consistency measured as standard deviation around 10 percent, against tools that commonly sit at 60 to 70 percent
Fast and deep are a trade, not a roadmap item. The fast profile gains speed and safety margin and gives up depth. The deep profile does not compromise on depth. One system cannot be both, so run deep against pre-production before pointing anything at production.
On keeping customer data away from public models, three things in combination: purpose-built small language models with capability layered on top, enterprise API keys under an SLA that the provider will not train on the data, and a gateway that scrubs customer data so only anonymized data leaves. Customers on Bedrock or Azure can bring their own keys, keeping the context window inside their own enclosure while the tools and agents run under the same guardrails.
4. Continuous was the right answer on engines that had no depth
Two conclusions from the room will be unpopular. Both are judgments offered by practitioners, not measured forecasts, and should be read that way.
Standard ASM and CTEM, run in their current form, lose effectiveness over roughly the next twelve months. Not because discovery stopped mattering, but because the layer underneath it is no longer covered. For years, attackers and your consultants tested the same way, so the pentest base held. Attacker AI adoption has jumped recently while enterprise adoption stayed slow. One very large organization’s AI committee spent two months revising its evaluation criteria while the tooling changed underneath it. Attackers pick up new tools and start the same week. We go deeper on what this means for existing programs in our CTEM implementation guide.
Nothing in the AI world will be genuinely continuous, because of token cost. Subscriptions will not stay unlimited. The market moves to consumption pricing, which means you cut the cloth to fit.
FireCompass coined the term CART, continuous automated red teaming, so this argues against our own creation. Continuous makes sense on non-AI engines, and non-AI engines have no depth. Continuously shallow does not reduce risk. We unpack the same tension across the market in our piece on continuous offensive security testing.
5. The program that replaces it
Bucket every asset on two inputs only: business criticality, and how often it changes materially. Adding further criteria makes the model harder to run without making it more accurate. Then run four activities at four frequencies.
| Cadence | Scope | Method |
|---|---|---|
| Daily | Whole estate | Rule-based Day-1 CVE exposure checks. Deterministic matching, no AI in the continuous layer |
| Frequent | P1 business-critical assets | Deep AI pentest: authenticated paths, business logic, authorization, chaining |
| Periodic | P2 and P3 assets | Shallow testing for coverage and regression, not depth |
| Two to three times a quarter | Across applications and into infrastructure | Objective-based red teaming |
Deep testing on one class of assets at one frequency, shallow on the rest at another. Built and governed as a program.
6. Who owns the risk
AI security may be the CISO’s responsibility, but AI risk is the business’s accountability, because the business decides what to adopt. Held by security alone, it reduces to a security metric that never reaches the board as a business decision.
Two working structures were described. A board-level AI subcommittee spanning security, legal, privacy and finance, proposed jointly with legal. Or an AI centre of excellence with a named AI officer, where business units bring use cases and the centre convenes security, legal and compliance around each one.
Acting alone by decree works exactly once, and after that you are the obstacle.
See what the same engine finds on your own web apps and APIs.
Frequently Asked Questions
What does it cost to run an agentic AI pentest at full speed?
In the experiment discussed, one agent type running continuously cost roughly $5,000 a month in compute, less than the salary of a junior penetration tester. That figure covers one quarter on one platform with one agent type, so it is indicative of unit economics rather than a benchmark.
Why should guardrails be rule-based rather than AI-based?
Because probabilities compound rather than cancel. If the model errs at 0.1 and an AI guardrail misses at 0.1, residual risk is 0.01, not zero. A deterministic layer either blocks an out-of-scope action or it does not, which is the only behaviour an audit will accept.
Can AI find vulnerabilities that human pentesters miss?
AI has not invented new attack techniques. It finds new attack paths, because it can attempt every case rather than the most promising ones. The insurance-company chain described above is the pattern: a traversal, a password in a log file, and a route into a database that looked unreachable, assembled by trying combinations no one had time to try.
Why is continuous automated testing the wrong default now?
Because consumption pricing makes genuinely continuous deep testing unaffordable, and continuously shallow testing does not reduce risk. The workable pattern is deterministic daily CVE exposure checks, deep AI pentesting on business-critical assets at a set frequency, and objective-based red teaming two to three times a quarter.
How should assets be bucketed for an AI pentest program?
On two inputs only: business criticality, and how often the asset changes materially. Those two decide whether an asset gets deep testing at a high frequency or shallow coverage testing at a lower one.
Who should own AI risk in an enterprise?
Security owns AI security. The business owns AI risk, because the business decides what to adopt. In practice that means a board-level AI subcommittee spanning security, legal, privacy and finance, or an AI centre of excellence with a named AI officer that convenes those functions around each use case.
Read the full Roundtable Summary here.
