Home
Can you run a real penetration test by pointing a strong AI model at an application and letting it work? It is a fair question, and the honest answer is no, not on the model alone. This article walks through the four things a frontier model cannot do in a live pentest, what has to sit around it to close those gaps, and how that architecture holds up in practice: against benchmarks, against the world’s best researchers on HackerOne, and inside a real enterprise program.
The premise is reasonable on its face: pick the strongest frontier model, point it at the application, and pentest quality follows. A newer model does reason better about targets and write better exploit strategies. None of that is the hard part. A penetration test is a live interaction with running infrastructure, and plausibility is not proof. The model cannot send an HTTP request, hold state across a hundred-step chain, enforce a scope boundary, or produce a tamper-evident log. Those are not things a model does badly. They are things a model does not do at all.
A better model therefore raises the ceiling on reasoning and does nothing for execution, state, validation, or safety. Those are harness properties, not model properties. The model is necessary. It is not sufficient. Harness engineering is the layer that turns model intelligence into validated, safe, auditable penetration testing.
1. Why an LLM cannot do DAST
Frontier models recognize vulnerability patterns, decompose objectives into testable steps, generate proof-of-concept code, and write readable reports. Four gaps stand between that capability and an enterprise-grade result. None is a reasoning problem, so none is solved by a smarter model.
1.1 Execution: intelligence that cannot act
Models generate text. An exploit payload from a model is a string, not an exploit. Delivering it to a live target, observing the response, and deciding whether exploitation succeeded requires a runtime the model has no connection to, with real-time awareness the model lacks: an endpoint that returned 200 now returns 403, a firewall dropped a payload mid-chain, a session token expired. Multi-step chains need that feedback wired directly into the decision loop.
1.2 State: coherence across long chains
Enterprise attack chains run 30 to 100 or more discrete steps: discovery, enumeration, credential extraction, authentication, privilege escalation, lateral movement, data access. This is not a context-window problem that a larger model solves. Inside a conversation context, models hallucinate prior steps, forget discovered credentials, and contradict earlier decisions, and the longer the chain the worse it gets. Coherence requires a structured state machine that records what was tried, what succeeded, what was blocked, and which credentials are held, independent of the model’s conversation history.
1.3 Validation: hypotheses are not vulnerabilities
Models are probabilistic. Asked to find vulnerabilities, they produce confident, syntactically correct findings that are sometimes simply wrong: vulnerabilities that do not exist, paths that are blocked, findings that cannot be reproduced. Without a validation layer that tests each finding against the live system, unvalidated model output and traditional DAST scanners alike land in the 50 to 70 percent false-positive range. Every model output is a hypothesis until execution proves it.
1.4 Safety and governance: bounded, auditable operation
A model given aggressive objectives attempts aggressive techniques, because it optimizes for task completion, not infrastructure safety. Unbounded, that means request rates that cause denial of service, writes that corrupt data, and brute force that locks accounts. It has no concept of scope and will follow attack paths into out-of-scope systems, production databases, and third-party services. And it produces outputs, not logs, while PCI DSS, SOC 2, ISO 27001, and DORA expect retained, auditable evidence of testing. Safety, scope, and auditability must be enforced at the execution layer, below the model, where they cannot be reasoned around.
| Capability required for enterprise testing | Frontier model provides by default |
|---|---|
| Execute real exploits against live systems | No. Text generation only |
| Maintain coherent attack state across 50+ steps | No. Conversation context degrades |
| Validate that each finding is actually exploitable | No. Produces hypotheses, not evidence |
| Enforce test scope boundaries | No. Requires external enforcement |
| Prevent production system disruption | No. Requires platform-level safety controls |
| Generate timestamped, tamper-evident audit logs | No. Produces outputs, not logs |
| Produce consistent, reproducible results | No. Non-deterministic by nature |
2. Bridging the gap: what harness engineering is
Harness engineering is the discipline of building the system around the model that makes a test real. A harness has three moving parts. First, the models: not one but several, each chosen for the task it handles best. Second, the tools those models drive, from crawlers and request engines to browser automation and credential handlers. Third, the orchestration that sequences the tools, holds their output, feeds it back into the reasoning loop, and enforces the boundaries the model itself has no concept of.
Think of an engine and a vehicle. An engine generates power but has no wheels, steering, or brakes, so on its own it goes nowhere. The frontier model is the engine. The harness is the vehicle: the drivetrain that turns reasoning into motion, the steering that keeps it on course, the brakes that stop it before it does harm. A vehicle without an engine is inert. An engine without a vehicle is uncontrollable. FireCompass builds the vehicle and treats the engine as a component it can swap as the market improves.
3. How the harness is built: components
3.1 Multiple routed models, plus purpose-built small models
Code generation benefits from strong coding models, long multi-step reasoning from extended-reasoning models, and rapid surface classification from faster, lower-cost models. FireCompass runs multiple large language models and routes each task to the model that performs best for that task class, maintaining a model registry and evaluating performance continuously against internal security benchmarks. Because intelligence is separated from execution, a new frontier model is integrated without changing a line of execution infrastructure.
Alongside those, FireCompass has built its own small language models for specific, high-volume tasks. This is partly an accuracy decision and largely an economic one: driving every step through frontier-model tokens makes always-on testing prohibitively expensive. Owned small models plus an owned orchestration engine bring the cost down far enough that a continuous cadence is real rather than aspirational.
| Task category | Model selection rationale |
|---|---|
| Initial vulnerability hypothesis generation | Best-performing security reasoning model per current benchmark, evaluated continuously across providers |
| Complex multi-step chain construction | Extended-reasoning model selected for chain coherence and long-horizon planning depth |
| Exploit code generation and payload crafting | Top-ranked coding model with security-domain prompting from the orchestration layer |
| Application context analysis (JS, API docs) | General model with large context window for document understanding |
| High-volume classification and triage | FireCompass small language models, tuned for cost and speed at scale |
3.2 Specialized agents, orchestrated
Execution runs on a distributed network of specialized agents. Each agent owns one capability domain and carries its own scope validation, rate limiting, and audit logging. Agents communicate through a shared state store rather than directly, so attack state stays coherent even when an individual agent fails. The orchestration engine spawns, monitors, and terminates agents, enforces scope before any action is dispatched, controls request velocity globally, and routes agent output back into the state store for the intelligence layer to plan against.
| Agent | Execution responsibility |
|---|---|
| Surface Mapping | Crawls and enumerates subdomains, endpoints, APIs, and authentication surfaces. Discovers shadow assets outside the initial scope definition. |
| Reconnaissance | External intelligence: dark-web credential exposure, leaked-repository scanning, DNS enumeration, technology fingerprinting. |
| Vulnerability Assessment | Runs targeted tests against identified endpoints. Implements OWASP Top 10:2025 for web app targets. Routes all findings to validation before reporting. |
| Authentication and Credential | Manages authenticated sessions: credential injection, session-token handling, OAuth navigation, credential-reuse testing across environment boundaries. |
| Business Logic | Tests workflow logic: multi-step process abuse, state manipulation, authorization-boundary violations, transaction integrity, in authenticated context. |
| Chain and Lateral Movement | Constructs and executes multi-stage chains from validated findings, attempting credential reuse and application pivoting. |
| Evidence Collection | Attached to every confirmation. Captures request and response pairs, generates PoC code validated against the live target, assembles chain-of-custody documentation. |
3.3 The state machine
The state machine is the structural backbone of an engagement: a persistent graph of the target environment, its assets, authentication surfaces, discovered vulnerabilities, tested paths, and accumulated credentials that survives any agent or model context reset. It gives the intelligence layer full current context at each planning step, steers prioritization toward unexplored high-value paths, and is the authoritative source of record for the audit trail. Attack paths are modeled as a directed graph aligned to OWASP Top 10.
3.4 The validation pipeline
Every candidate finding passes through four stages before it is confirmed: the intelligence layer generates a structured hypothesis; the runtime delivers the proposed exploit against the live target under full scope and safety controls, capturing the exact request and response; the response is evaluated against the expected exploitation signature, with ambiguous results escalated for re-testing; and the evidence agent assembles the record, including reproduction steps and working PoC code. Findings that fail exploitability are discarded, not reported as low-confidence noise. This is the mechanism behind the below-2-percent false-positive rate.
3.5 The safety architecture
Safety is a mandatory enforcement layer between orchestration and the execution runtime. No agent action reaches the runtime without passing through it. The governing principle: a non-deterministic LLM is controlled by deterministic, rule-based systems, never by another model. The LLM proposes; deterministic rules dispose.
| Control | Mechanism | What it prevents |
|---|---|---|
| AI Input and Output Firewall | Input firewall drops non-compliant requests; output firewall inspects every model-generated action and dispatches only approved ones | Malicious prompts reaching the model; unvetted model actions reaching a live target |
| Deterministic Enforcement Layer | A final rule-based gate below the model. Enforcement never depends on another LLM | Safety and scope decisions left to probabilistic judgment |
| Scope Boundary Enforcement | Asset whitelist checked against every proposed action before dispatch | Out-of-scope testing, production database access, third-party exposure |
| Fine-Grained Capability Controls | Per-agent capability definitions specifying exactly what each agent can and cannot do | Agents acting outside their role |
| Rate Limit Controller | Token-bucket limiter per target host with a configurable request ceiling | Unintended denial of service; account lockout from brute force |
| Credential Scope Guard | Environment tags on all credentials; cross-environment reuse blocked | UAT credentials used against production; credential leakage across programs |
| Safe Payload Enforcement | Read and Create permitted within scope; Modify, Update, Delete blocked by default and never executed autonomously | Data modification, record deletion, state corruption |
| Kill Switch | Operator-controlled termination signal propagated to all active agents | Runaway tests, unexpected system impact |
| RBAC and ABAC | Role and application-based control over who can run, view, and manage tests | Unauthorized test execution; uncontrolled access to findings |
| Full Action Audit Trail | Chain-of-thought and action logging per agent, captured by the execution layer as it happens, not summarized by the model | Traceability gaps; inability to produce retained evidence for PCI DSS and DORA |
3.6 Enterprise governance and deployment
The governance layer has no model dependency. RBAC scopes permissions across user roles, asset groups, programs, and data classification, with multi-tenancy isolation so findings and credentials from one program are never visible from another. Every platform action is recorded in an append-only audit log with a cryptographic timestamp, supporting DORA, PCI DSS 4.0, SOC 2 Type II, and ISO 27001. One data model produces an executive report on risk posture and an engineering report with endpoints, HTTP evidence, reproduction steps, PoC code, and remediation guidance.
FireCompass deploys as a managed SaaS for external assets and as a customer-deployed virtual appliance for internal assets, sharing the same intelligence layer, agent network, and governance. Continuous here means always-on surface discovery plus scheduled or on-demand testing campaigns, not thousands of agents running in parallel at all times. Delta analysis prioritizes what changed between runs rather than re-testing every surface every time.
4. The FireCompass advantage
The components in Section 3 are what FireCompass has built over several years, combining offensive security expertise with frontier model integration. What separates the platform from a model wrapper reduces to four things.
- Multiple frontier models, routed per task. A model registry, continuous benchmarking, and per-task routing mean the best available model handles each task class, and a new model is integrated without touching the execution layer.
- Purpose-built small language models. High-volume classification and triage run on FireCompass-owned small models tuned for cost and speed, not on frontier tokens. This is what makes always-on cadence affordable.
- An owned orchestration engine and agent network. The state machine, validation pipeline, safety gateway, and audit layer are FireCompass engineering, not model features. They work identically regardless of which model is plugged in.
- A measured multiplier. Take any frontier model, use it as a plain wrapper with tools connected, and call that result X. Across FireCompass testing the full harness produces about 3X to 10X that result depending on task class. For external reference, a new frontier model typically delivers about 2X over its own previous generation. The gain comes from the harness, not from any single model being smarter, and it is the basis for the claim that the harness, not the next model release, is what moves results on dynamic application security testing.
4.1 Where the harness applies: dynamic testing, not static analysis
FireCompass is not a general-purpose replacement for every kind of AI-assisted security analysis. Frontier models alone are stronger at static analysis and code analysis, and FireCompass does not compete there. FireCompass does dynamic testing: it exercises a running application and its APIs, orchestrating the tools and agents required to discover, exploit, validate, and chain findings against live systems. A model can reason about a running application, but it will not stand up the agents, drive the tools, hold the state, and enforce the safety controls that dynamic testing requires. The harness does that.
| Static analysis and code analysis | Dynamic application and API testing | |
|---|---|---|
| A frontier model alone | Strong | Reasons, but will not orchestrate the tools to execute |
| FireCompass harness | Not the focus | Orchestrates tools and agents to execute end to end |
5. Proven against the world’s best: top 3 on HackerOne, ahead of leading security researchers
Benchmarks are controlled environments. HackerOne is not. Between April and July 2026, FireCompass AI agents competed in the open on live, authorized HackerOne programs against the world’s best human researchers and every other AI agent hunting the same production targets. The agents reached top-three positions across multiple HackerOne US Business leaderboards, including No. 1 in OWASP A01 Broken Access Control, No. 1 in the Up and Comers category, No. 2 for Highest Critical Reputation, and No. 3 on the U.S. country board.
The agents ran on the harness described in Section 3, under the same controls a customer engagement runs under: hard enforcement of authorized scope, non-destructive validation, limits on request rate and blast radius, and proof of exploit on every submitted finding. The frontier models behind the agents were available to every entrant on those boards. The ranking is a harness result.
Peak rankings under the US, Business, Web Application filter, captured 17 July 2026. Full methodology and disposition data: firecompass.com/blog-ai-penetration-testing-hackerone-top-3.
6. What this means for your program
6.1 Cost effectiveness
Small-model economics plus delta analysis change the unit cost of a test enough to change the cadence. A Fortune 500 technology company moved a large consulting firm’s manual program to FireCompass across 2,000+ web applications, of which only 200 were being tested annually before.
| Metric | Before | After |
|---|---|---|
| Cost per application test | ~$5,000 per test (2 consultant-days) | Under $1,000 per test |
| Application coverage | 200 of 2,000+ applications annually | Full portfolio, always-on plus on-demand |
| Lead time to test initiation | 2+ weeks scheduling and setup | On-demand, zero lead time |
| False positive rate | ~70% from supplementary DAST scanning | Below 2% |
| Attack chain discovery | Isolated findings; chains scoped out of manual work | Chained paths discovered autonomously |
| Previously untested assets | No visibility | Vulnerabilities found on assets never previously tested |
6.2 Quality
Quality means validated, reproducible findings, not volume. FireCompass agents were evaluated against recognized web application test environments in fully autonomous mode, with no manual steering or predefined exploit sequences. The results are harness outcomes, not standalone model measurements: complete coverage on the XBEN validation suite (104 of 104), every evaluated Acuart finding validated with exploit evidence (12 of 12), and full vulnerability-class coverage across all configured DVWA difficulty levels with multi-step exploit validation confirmed. In production, the false-positive rate is below 2 percent because every finding is proven against the live target before it appears.
6.3 Safety and governance
Every control in Section 3.5 is deterministic and sits below the model, so safety does not depend on how a model interprets an instruction. For your program this means testing can run against production-adjacent surfaces with scope whitelisting, per-host rate ceilings, writes blocked by default, environment-tagged credentials, and an operator kill switch, with every action captured in an append-only, cryptographically timestamped log as it happens. That is the retained, structured evidence DORA, PCI DSS 4.0, SOC 2 Type II, and ISO 27001 auditors ask for.
6.4 Model independence
Because the engine is a swappable component, your organization is not locked to any model vendor’s roadmap. When a stronger frontier model ships, it enters the model registry, is benchmarked against internal security tests, and is routed to the task classes where it wins. Nothing in execution, state, validation, or safety changes.
About FireCompass
FireCompass is an Agentic AI platform for autonomous penetration testing and red teaming across Web, API and infrastructure. It discovers shadow assets and web applications, safely validates what is exploitable, and connects findings into multi-stage attack paths with near-zero false positives. Unlike traditional scanners, it discovers credential reuse, business-logic flaws, privilege escalation, and app-to-app or app-to-network lateral movement. It can operate autonomously or with expert-in-the-loop validation. FireCompass has 30+ analyst recognitions across Gartner, Forrester, IDC, and is trusted by Fortune 100 enterprises.
