FireCompass AI Agents vs Open-Source Agent Frameworks

Why the same models produce different results, and what a self-hosting team inherits.

An AppSec engineer can self-host an autonomous pentest agent this weekend. Clone Strix or PentAGI, plug in an API key, and by Monday it is finding vulnerabilities in staging. The code is good, the licence is free, and the models underneath are the same frontier models every commercial platform in this category calls.

So the model is not what separates these options. The engineering around the model is.

What open-source agent frameworks do well

This ecosystem is real engineering. PentAGI runs an orchestrator over a researcher, a developer and an executor in isolated containers. BlacksmithAI maps agents to recon, scan, analysis, exploit and post-exploitation. Strix specialises agents by phase and validates findings with a proof of concept. VulnBot plans long-horizon work through a task graph and runs without a frontier model at all.

Every one of them implements the reasoning loop well: read state, decide, call a tool, read the output, repeat. That loop is now commodity. It was never the bottleneck.

The four jobs no model does

Four things stand between model capability and a result an enterprise can act on. None is a reasoning problem, so none is solved by a better model.

• Execution. A model that produces an exploit payload has produced a string, not an exploit. Something has to deliver it to a live target and determine whether exploitation succeeded.
• State. Enterprise attack chains run 30 to 100 or more steps. Across that length models forget discovered credentials and contradict earlier decisions. A larger context window does not fix it.
• Validation. Unvalidated model output and traditional DAST scanners both land in the 50 to 70 percent false-positive range. Fluent false positives cost more triage time than scanner noise, not less.
• Safety and governance. A model optimises for task completion, not infrastructure safety. It has no concept of a test scope, and it produces outputs rather than logs.

Harness engineering

A frontier model is a car engine: enormous force, no wheels, no steering, no brakes. The harness is the vehicle around it. FireCompass routes each task class to the model that performs best for it, alongside purpose-built small language models for high-volume work, and runs execution across seven specialised agents over a persistent state machine that survives any agent or context reset. Every candidate finding passes a validation pipeline, and findings that fail to reproduce are discarded rather than reported as low-confidence noise. That is the mechanism behind a below-2-percent false-positive rate.

Safety and governance

Safety sits as a mandatory enforcement layer between orchestration and the execution runtime. No agent action reaches the runtime without passing validation. The governing principle is that a non-deterministic model is controlled by deterministic rule-based systems, never by another model. Scope is checked before every dispatch; read and create are permitted in scope while modify, update and delete are blocked by default; request velocity is capped per host; and an operator kill switch propagates to every active agent.

This is where build versus buy gets uncomfortable. When an autonomous agent tests out of scope, locks out a production account, or writes to a live database, someone signs the incident report. With a platform that accountability is contractual and the evidence already exists. With a cloned framework it belongs to the engineer who cloned it.

Side by side

Open-source column reflects the significant public projects as of August 2026, including PentestGPT, Strix, PentAGI, CAI, VulnBot, xOffense and BlacksmithAI. Individual projects vary.

The honest verdict

Clone an open-source framework if the team has genuine offensive engineering capacity, a research mandate and a narrow scope. Anyone who wants to understand this technology deeply should. It is the wrong call for a programme covering thousands of applications with an evidence trail an auditor will accept, because at that point the choice is not adopting a tool. It is staffing a product team.

Frontier models are becoming more capable, more accessible and increasingly commoditised. That alone will not solve autonomous penetration testing, because reasoning is not execution. Hypotheses are not vulnerabilities, and everything that turns one into the other lives outside the model.