How FireCompass AI Agents Reached #1 on HackerOne on $5,000 a Month: Full Methodology, Data, and Limitations

FireCompass Emerging Research. April to July 2026. For one quarter, we ran our AI agent on HackerOne, in the open, competing against human researchers and every other agent hunting the same targets. No lab. No curated benchmark. The agent competed as firecompass-ai on a compute budget of about $5,000 per month.

By the July snapshot, it held top-three positions across several HackerOne business leaderboards, including No. 1 on the OWASP A01 web application leaderboard, ahead of XBOW:

• No. 1 on the OWASP A01 (Broken Access Control) U.S. Web Application leaderboard, Q2 2026, captured June 18, 2026: firecompass-ai No. 1 (Reputation 116, Signal 6.29), XBOW No. 2 (Reputation 18, Signal 4.00), cantina-security No. 3
• No. 1 in the Up and Comers category (current quarter to date, captured July 17, 2026)
• No. 2 for Highest Critical Reputation, ahead of XBOW (current quarter to date, captured July 17, 2026)
• No. 3 on the U.S. country board (last quarter, captured July 6, 2026)

These are point-in-time peaks under the US, Business, Web Application filter, captured on the dates shown. HackerOne boards reset each quarter and recalculate daily, so the live boards have since moved.

The rank is the attention-getter. The number that should hold your attention is the one next to it: $5,000 a month.

Source: hackerone.com/leaderboard, screenshots above. HackerOne is a trademark of HackerOne Inc. HackerOne has not sponsored, verified, approved or endorsed these results.

The economics just changed, and that is the real story

Reaching the top tier of a global bug bounty platform used to require scarce expert talent, real infrastructure, and long timelines. In this experiment, it took a governed AI agent and roughly $5,000 a month in cloud and AI compute.

Two numbers put that in perspective:

• $5,000 a month is less than the salary of a junior pen tester. For the cost of one junior hire, our AI agents reached the world’s top three. Elite offensive capability now carries an entry-level price tag.
• A single manual pen test of one application typically costs $2,400 to $10,000, and complex engagements run to $40,000 or more. That is per application, once. The same $5,000 a month ran continuous testing across all in-scope assets, not a single point-in-time test of one app.

For a security leader, that shift has a direct consequence. The reason most programs test each application once a year is that testing is expensive and testers are scarce. When the cost of finding a real vulnerability drops by an order of magnitude, annual testing stops being a budget constraint and starts being a choice. Continuous testing of everything you ship becomes affordable, not aspirational.

That is the opportunity. The uncomfortable half of it comes later in this post.

Under the hood: the pipeline and the agents

This is where an AI security result usually goes vague. Ours does not, because the architecture is the reason the numbers came out the way they did.

The system under test is the same offensive engine that runs inside the FireCompass product. It is not a single model prompted in a loop. It is a multi-agent framework in which specialized agents own distinct phases of an engagement, and an orchestrator coordinates them toward a goal. At a working level, the agents divide the job:

• Reconnaissance agents enumerate the target: subdomains, endpoints buried in JavaScript, APIs, and forgotten or shadow assets. This is where published scope becomes a real attack surface.
• Exploitation agents probe and attempt validated exploitation across the OWASP Top 10 classes: broken access control (IDOR, BOLA, authorization bypass), injection (SQLi, XSS, command injection), misconfiguration, SSRF, and authentication and session flaws.
• Validation agents confirm exploitability and produce a proof-of-exploit artifact for every candidate. This is what keeps the false-positive rate near zero instead of emitting scanner-style maybes.
• The orchestrator sequences the phases and, where it applies, connects individual findings into multi-stage attack paths the way a real adversary would: initial access, then escalation, then lateral movement.

Around that engine we built a submission pipeline that runs as a continuous assembly line, not a one-shot scan. It has four stages. Scope ingestion pulls newly published in-scope assets automatically and prioritizes them, because fresh scope is where the duplicate probability is lowest. Autonomous testing runs the multi-agent framework above and emits each candidate with its proof of exploit. A human triage gate then confirms exploitability and scope fit before anything is filed, which also satisfies HackerOne’s requirement that AI-assisted findings are human-validated before submission. Finally, submission and disposition tracking files confirmed candidates and record the terminal outcome of every report, which is what lets us do the analysis in this post.

The models: frontier LLMs plus FireCompass purpose-built small language models

The engine is model-plural and model-agnostic by architecture. It pairs multiple frontier LLMs with FireCompass purpose-built small language models trained for narrow offensive-security tasks. No single model is best at everything, so a routing layer sends each task to the model that currently performs best for it: extended-reasoning models for multi-step attack-chain planning, strong coding models for payload crafting, large-context models for parsing API docs and JavaScript, and security-specialized models for CVE matching and business-logic analysis. The purpose-built small models handle the high-frequency, well-scoped decisions inside the loop, where a specialized model is faster and cheaper than a general one.

That separation of thinking from doing is the reason the run held to about $5,000 a month: you do not pay frontier-model rates for work a purpose-built model does better. It is also why the platform stays current. When a better model ships from any provider, it is benchmarked and routed in without re-engineering the execution layer underneath.

Safe by design: a non-deterministic model governed by deterministic controls

Running an autonomous offensive agent against live production systems is not something you do without hard guardrails, and this is the part of the problem the industry most often underestimates. A frontier model is strong at offensive reasoning, but on its own it has no concept of scope, no built-in brakes, and no audit trail. Intelligence is not execution, and a model that optimizes for task completion will happily follow an attack path out of scope, hammer a target into a denial of service, or report a vulnerability that does not exist.

The core principle is simple: the non-deterministic LLM is governed by deterministic, rule-based systems. No agent action reaches a live target without first passing a rule-based enforcement layer, which is what makes the system safe by design rather than safe by convention. The controls, in brief:

• AI input/output firewall: incoming requests filtered, model actions checked before they run
• Scope boundary enforcement: out-of-scope hosts hard-blocked, not deprioritized
• Rate and blast-radius limits: request velocity capped per target host
• Safe-payload controls: writes, updates, and deletes blocked by default
• Real-time kill switch: one signal halts every running agent
• Anomaly detection: unexpected error or latency spikes pause execution
• Full audit trail: every agent action logged and reviewable

Those controls cost raw output, on purpose. A governed agent that respects scope and proves impact non-destructively files less than one that sprays, and it files cleaner. The disposition data is what that trade-off looks like in numbers.

What the agent actually found

Over the primary quarter, the system filed 150 reports, and 205 across the full window, including the July snapshot. Here is where those 150 reports landed:

• Duplicate: 58 (38.7%). Real, valid vulnerabilities that another researcher filed first. On HackerOne, a duplicate is evidence of capability, not an error. More on this below.
• Informative: 49 (32.7%). Real behavior the program judged out of scope for a reward.
• Triaged: 13 (8.7%) and Resolved: 6 (4.0%). Accepted findings, an accepted rate of 12.7 percent.
• Not applicable: 6 (4.0%). Reports the program judged to have no valid in-scope security impact, which can include out-of-scope findings as well as non-issues.
• New/ pending/ in progress: 18 (12.0%). Still under review at the close of the window.

Some of the high-critical issues identified by our agents : 

1. Unauthenticated RCE via command injection 
2. Live cloud credentials via unauthenticated token-minting endpoint — an API meant to issue internal auth tokens had no auth check, handing out live OAuth credentials for a connected ERP system. 
3. Default credentials on an internal auth service — a login endpoint accepted a hardcoded default credential pair, issuing valid session tokens with cross-service privileged access. 
4. Error-based SQL injection in login/password-reset flow — classic injectable username parameter on a password-reset endpoint. 
5. Command injection in a CLI installer generator — a web-hosted script generator embedded unsanitized user input into a shell command executed server-side.
6. Unauthenticated PII/company-scoped data exposure via header manipulation — swapping a company-id header with no authorization check returned another tenant’s full customer PII. 
7. Subdomain takeover via dangling cloud CNAME — a decommissioned app-service/CDN record left the subdomain claimable, enabling phishing/content spoofing. (High/Critical, several instances across different programs)
8. Broken Object-Level Authorization (BOLA) at scale — a recurring pattern across e-commerce/API programs: swapping an ID parameter (cart, basket, order-form, account) in an API call returned or modified another user’s data with no ownership check. (High, multiple instances)
9. Hardcoded secret in shipped JavaScript — a production frontend bundle contained a plaintext signing/API token, exploitable by anyone reading the page source. (Critical/Highs)
10. SSRF chains via internal proxy/MCP-style endpoints — several “agent”/gateway integrations forwarded attacker-supplied URLs server-side with no egress restriction, enabling internal network probing.

• Injection (SQLi, XSS – reflected/DOM, command injection, code injection) – 13
• SSRF – 9
• Broken access control (IDOR, BOLA, authorization bypass, forced browsing) – 48
• Information Disclosure – 28
• Authentication & session flaws (default/hardcoded creds, missing auth, auth bypass) – 28
• Misconfiguration – 23
• Business Logic Errors – 6
• Cleartext Storage of Sensitive Information – 6
• Broken access control (IDOR, BOLA, authorization bypass), injection (SQLi, XSS, command injection), misconfiguration, SSRF, and authentication and session flaws.
• Injection (SQLi, XSS – reflected/DOM, command injection, code injection) – 13
• SSRF – 9
• Broken access control (IDOR, BOLA, authorization bypass, forced browsing) – 48
• Information Disclosure – 28
• Authentication & session flaws (default/hardcoded creds, missing auth, auth bypass) – 28
• Misconfiguration – 23
• Business Logic Errors – 6
• Cleartext Storage of Sensitive Information – 6

Two results matter for a business reader, and both are downstream of the architecture above.

First, the findings were serious, not noise. Critical and high-severity issues made up 64.4 percent of the 101 severity-rated reports: 14 critical (13.9%) and 51 high (50.5%), with no low-severity findings at all. This runs against the common worry that autonomous tools mostly surface trivial findings. The weakness classes concentrated exactly where application logic lives, not where signatures do: Information Disclosure (25), Misconfiguration (19), Improper Access Control (18), IDOR (17), and Improper Authentication (16) led the distribution, with SSRF (7), reflected XSS (6), and business-logic flaws (5) forming the tail. These are the classes that traditional scanners routinely miss and that agents reasoning about application state surface reliably.

Second, the agent was precise. The not-applicable rate, which reports the program judged to have no valid in-scope impact, was 4.0 percent overall, 4 percent among high-severity findings, and zero percent among criticals. That is consistent with the sub-2 percent false-positive rate the engine holds in production, compared with the 40 to 70 percent typical of scanners. The precision comes directly from the proof-of-exploit requirement in the validation agents: the system has to demonstrate a vulnerability is real before it submits. For any team that has drowned in scanner output, that discipline is the difference between a tool that saves time and one that creates work.

The finding behind the finding: speed, not skill, was the limit

Here is the part that surprised us, and it carries a lesson for anyone evaluating these tools.

On the surface, only 12.7 percent of our reports were formally accepted. That looks modest. But when you count every report the programs agreed was a genuine vulnerability, whether or not we got the credit, the number is 51.3 percent. The gap between those two is almost entirely duplicates: real bugs we found, but that another researcher reported minutes or hours before us.

That distinction matters, because a duplicate on a bug bounty platform is not a mistake. It is a correct finding that arrived second. Roughly three-quarters of our genuine findings were duplicates. In plain terms, the agent’s ability to find real vulnerabilities was strong. What held back its rank was speed of submission, not skill at discovery.

And the reason it was sometimes slow was a choice we made, not a limit of the technology. We funded one human review shift a day instead of three, because a person has to validate each finding before it is filed. Real findings waited in that queue while faster competitors filed the same bug. More review capacity would have converted a meaningful share of those duplicates into credited findings. The single biggest change we would make next time is not a smarter agent. It is more review capacity feeding a faster pipeline.

The takeaway for a buyer is simple. When you assess an offensive AI tool, do not judge it on a leaderboard rank alone. A rank blends how good the system is with how fast it happened to file. Ask how many of its findings were real, and you get a far truer picture of capability.

The opportunity, and the responsibility that comes with it

The capability shown here is arriving for defenders and attackers at the same time and at the same low price. A threat actor with a small budget can now reach for offensive capability that used to be out of reach. That is precisely why the controls matter as much as the capability. The advantage will not belong to whoever has the most powerful model. It will belong to whoever wraps that model in the strongest validation, governance, and human accountability.

That is where most of our engineering goes, and it is why the safety layer in this experiment was designed in from the start rather than added at the end.

See what the same engine finds on your own web apps and APIs. Start a free AI pen test.