The Most Dangerous Action May Be Perfectly Legitimate

Agentic AI Security · Part 3 of 3

An agent can do something quite reasonable, with valid permissions, and still cause harm because of what happened before it or what another agent is doing at the same time.

Picking up from the first two articles

In the first article, I argued that judging the safety of a model’s answer tells us surprisingly little about the safety of an agent acting over time. The agent observes something, decides what it means, acts, then reasons from the situation its own action helped create. That sequence, the trajectory, is where much of the risk lives.

The second article followed the argument into the surrounding system. We need hard restrictions. An agent must not be able to talk its way around an enforced scope boundary. But an approved tool operating against an approved target is only the beginning of the safety question.

This third article deals with the part I find particularly worrying: an agent can do something quite reasonable, with valid permissions, and still cause harm because of what happened before it or what another agent is doing at the same time. Sometimes there is no bad instruction to remove and no obviously dangerous command to block. We have to look at the execution closely enough to see the problem.

Let us start with an account lockout.

Two attempts each

Suppose a customer has authorised credential validation against a web portal, a remote-access service and an internal application. The engagement has one explicit operational constraint: do not lock production accounts.

The platform allows each agent no more than two failed authentication attempts against an account. Three agents are assigned to the three services. Each starts with a clean local history and each stays within its limit.

What the agents do not share is the relevant history. All three services use the same underlying identity system, and all three agents happen to be testing the same account. In this example, the identity system locks the account on the fifth failed attempt.

This is a constructed scenario, not a reported FireCompass incident. The threshold is an assumption for the example, not a claim about how every identity system works.

I have deliberately left out prompt injection, jailbreaks and out-of-scope targets. Nobody needed to ignore an instruction. The platform mistook a limit on each worker for a limit on the resource they were sharing.

The account belongs to the customer. Credential validation is authorised. The tool is behaving as intended. Yet the fifth attempt is the one I would want the system to withhold.

There is a qualification worth making at the outset. Calling that attempt “perfectly legitimate” does not make it safe under the full engagement policy. It is an ordinarily authorised operation that passes the local rule. Given the accumulated failures and the customer’s no-lockout requirement, the actual transition should be prohibited. The weakness lies in the policy being applied and the state available to the checker.

Otherwise we end up arguing that something is safe because we failed to notice why it was unsafe.

The first request against an isolated test account and the fifth request against a production identity might be indistinguishable at the tool interface. Reading a record, starting a scan or following a redirect has the same difficulty. An operation can be permitted and still be disproportionate, badly timed or dangerous after earlier operations.

A little notation helps pin this down:

( ∀t, Allowed_local(a_t) )  ⇏  Safe(τ)

Here, a_t is a tool operation and τ is the trajectory, including the observations and resulting states. Every action passing its local check does not establish that the execution as a whole is safe.

We have encountered versions of this problem in distributed systems for years. What agents change is how the workflow is produced. A test script often exposes its sequence before execution. An agent chooses the next step from the latest evidence; a discovery opens new possibilities, and delegation creates branches running in parallel. The controller has to enforce the constraints of an engagement whose detailed plan is still being worked out.

It would be convenient if an approved plan settled the matter. It doesn’t.

The history changes the permission

Lockout is an accumulation problem. Small actions consume a shared allowance until one more small action exceeds it. There are other ways for locally acceptable operations to combine badly, and it is useful to distinguish them because a single mechanism will not cover them all.

Take an application test. Retrieving an approved test object to investigate an authorisation weakness may be sensible. Continuing to retrieve real customer records after the weakness has been demonstrated may breach an agreed stopping condition. The request can look the same, although its evidential value has changed.

Or consider credentials. A customer may approve credential C for one application and approve host H as a testing target. Those two approvals do not necessarily authorise using C on H. If the controller maintains independent lists of permitted credentials and permitted hosts, it has lost the relationship that the customer actually approved.

The information carried by an action matters too. Uploading a diagnostic report to an approved evidence repository may be routine. After that report has accumulated secrets or personal data, the same upload may require redaction, a different destination or further authorisation. Approving a repository does not settle every question about what may be put in it.

Then there is load. Four workers can stay within four separate rate limits and still overwhelm one backend. Different hostnames may lead to the same database, authentication service or third-party API. Counting requests by the names visible to the agents can conceal the resource that is actually being consumed.

These cases suggest quite different controls. Accumulation needs an aggregate counter or budget. Ordering can be represented by a state machine. Credential use needs a policy that preserves the approved relationship. Disclosure needs data classification and provenance, the origin and handling history of the information. Contention needs scheduling across the workers sharing a resource.

I would be wary of a design that responds to all of these by extending its command blocklist. The commands are often unremarkable. The missing information is about the engagement.

The more useful technical concept is an invariant: a condition that must remain true as execution proceeds. In the account example, “each agent makes at most two attempts” is the wrong invariant for the customer’s concern. We need to preserve the identity’s remaining failure allowance, leaving sufficient margin to avoid lockout.

An engagement-wide limit on exposure of non-test records is another possible invariant. For a privileged operation, the condition might require a current approval covering the particular credential, target and technique. Whatever we choose, admitting an action should preserve the relevant condition:

I(s_t)  ∧  Admit(s_t, a_t)  ⇒  I(s_t+1)

The state s_t contains the safety-relevant facts. The admission decision includes the checks and any reservation of resources required before the action takes effect. This equation expresses an obligation for the design; writing it down proves nothing about an implementation.

The reasoning behind it is straightforward. Start in a state satisfying the invariant. If every actual transition preserves it, it continues to hold throughout execution. There is no need to pretend that combinations of actions are intrinsically beyond deterministic control.

Indeed, I should tighten a phrase used earlier in this series. Actions that genuinely preserve the same invariant, with accurate shared state and a correct account of their effects, do compose safely. The failure is usually that isolated checks were mistaken for evidence of this stronger property. A valid destination and a permitted command do not establish what we need them to establish.

This is the role of runtime controls: reduce risk while the AI agent is acting by checking proposed operations against current authority, execution history and shared resource limits. They complement configuration before the task and evaluation afterwards. The distinction matters because a check performed at onboarding cannot know everything that will become relevant halfway through an engagement.

There is substantial older work behind this approach. Fred Schneider’s Enforceable Security Policies, published in 2000, examined enforcement through observation of execution and prevention of prohibited transitions. Security automata retain history as part of their enforcement state; they need not treat every operation as a fresh, unrelated event.^[1]

In that formal setting, a finite prefix can establish that a safety property has been violated. A prohibited disclosure is an accessible example: later actions cannot make the original disclosure disappear from the execution history.

There are limits. Such a monitor does not enforce every desirable outcome, and a policy written in natural language is not automatically a faithful executable specification. We have to express the condition in terms the controller can observe and control.

Recent agent work brings these ideas into contemporary systems. AgentSpec supplies structured rules involving triggers, predicates and enforcement mechanisms. Agent-C addresses temporal requirements, prerequisites before data access for example, and constrains generated tool calls against them.^[2][3] The evaluations are useful evidence for the specified policies and application settings. They do not establish unrestricted safety for every agent deployment.

I find that connection encouraging. We are not starting from nothing. But it also makes some present failures harder to excuse: history-aware enforcement was a systems problem long before the LLM arrived.

Fixing the counter is where the work starts

Let us repair the lockout design by giving the agents a shared failure counter. That sounds like a small, fairly conclusive fix.

There are three failures recorded. The testing platform has set a conservative ceiling of four. Agent A reads the counter and sees room for one more attempt. Before A finishes, agent B reads the same value. It also sees room.

Both proceed. If both fail, the account still locks.

Now the information is shared, but reading it and consuming the allowance are separate operations. A correct observation has become stale before the action’s effect is recorded. This is a concurrency bug, and a second model reviewing the proposed calls would not remove it.

The controller needs to reserve the allowance atomically before issuing the request. The reservation must include outstanding work, not just completed failures. Once a result arrives, the controller reconciles what it reserved with what actually happened.

Here is where I would make a conservative implementation choice. If an authentication request times out, I would not return its reservation to the available budget unless I could establish that the identity service had not processed it. A timeout at our end is not evidence that nothing happened at theirs.

That choice will sometimes stop a useful test earlier than necessary. I would accept that cost in a production engagement. Treating an unknown outcome as a harmless outcome is a much less defensible shortcut.

Identity resolution brings another complication. A username, an email address and a directory identifier may all denote the same account. If the counter uses the submitted string as its key, we have quietly created several allowances for one identity. Coordination has to follow the actual resource, not simply the input syntax.

Suppose these three login identifiers all refer to the same employee account:

Assume the account locks after five failed attempts.

If the testing platform tracks failures against the submitted text, it maintains three separate counters: 2, 2 and 1. Each appears below its two-attempt allowance.

But the identity system resolves all three identifiers to the same account. It counts five failures and locks the account.

The fix: resolve each identifier to one stable account identity before allocating attempts. All three agents must consume the same shared allowance.

The identifiers accepted depend on the service; a directory object ID might identify the account in an API rather than serve as an interactive login name.

Saltzer and Schroeder’s principle of complete mediation requires checking authority at every access.^[4] My engineering interpretation for an agent system is to make the relevant check at execution, using current authority and coordinated state. An earlier approval of the plan cannot stand in for that check.

Even then, we have only solved the portion of the problem we can see.

An employee may already have made unsuccessful login attempts. Another testing platform may be running. The identity policy may have changed since onboarding. Our counter can be internally consistent and still fail to describe the account’s real remaining tolerance.

I would therefore want to know where each safety-relevant fact came from. Is the lockout threshold supported by current configuration evidence? Are we using dedicated test identities? Does the customer supply usable telemetry? Can the technique be moved into a controlled environment?

Where the answer is no, avoiding the technique may be more responsible than guessing a budget. A model’s confidence does not make an unknown operational limit known. Explicit authorisation and uncertain operating conditions can exist at the same time.

Nor can we put every risk into an additive counter. Authentication attempts can often be counted. Privacy exposure, backend load and cascading failures can interact in ways that make a simple sum misleading. We need to distinguish reliable measurements from conservative approximations and from questions that require a pause.

That is the point at which runtime monitoring begins to meet decision-making under uncertainty. Part 5 will return to it in more detail. There is another issue to deal with before then: sometimes the agent continues because it is performing well.

When better evidence becomes unnecessary exposure

Suppose a web-testing agent identifies an insecure direct object reference. One account can retrieve a record belonging to another account, and the response provides convincing evidence of the flaw.

The agent checks another object type, another user role and another endpoint. There are reasonable technical grounds for those tests. They improve coverage and may reveal that the weakness is more extensive than the initial finding suggested. They also increase the amount of sensitive information the system handles.

If the engagement rewards findings, coverage and demonstrated impact, continuing can keep looking like good work. We do not need deception or malicious intent to explain this. An inadequate stopping rule will do.

I would want an explicit account of sufficient evidence for the vulnerability class and the agreed test plan. Which observations establish the finding? Which further tests could materially change the conclusion? Which merely repeat what has been established while increasing exposure?

One retrieved record is not universally sufficient. Equally, ten retrieved records are not automatically ten times better evidence. The justification has to be about the question being answered.

Successful exploitation can therefore remove the justification for further exploitation.

That sits awkwardly with an agent encouraged to keep making progress, but it is a normal part of responsible testing.

At FireCompass, autonomous security testing makes this distinction practical. Discovering and validating attack paths is the useful capability. The corresponding obligation is to recognise when a path has been demonstrated adequately, when the next step requires different authority, and when several workers are drawing on the same operational margin.

I regard those as standards to evaluate a design against. They should not be read as a claim that every problem described here has already been solved.

The engagement needs a record connecting actions to authority, assets, credentials, evidence and resource consumption. An individual agent’s conversation can provide part of that account. It cannot be assumed to represent the whole engagement.

This also changes what we mean by evaluating agent memory. Remembering the objective or recalling an earlier finding is not enough. Safety-critical facts must survive summarisation, context compression, delegation and model replacement. A summary saying “credential validation completed” is inadequate if it omits four uncertain failures against a shared identity.

The model might still infer accurately from the summary it receives. The summary itself has removed the fact that should prevent the next action.

I would keep such facts in a structured state record used by the controller, rather than rely solely on their survival in conversational memory. Evaluation should then check both that the record is retained and that it actually affects admission decisions. An agent which remembers its goal but forgets approval limits, failed attempts or data already exposed may repeat a locally permitted operation until the engagement becomes unsafe.

Delegation needs the same care. A child task should carry the authority it needs and an allocation from the shared budget. It should not acquire an unrestricted copy of its parent’s powers. Otherwise better planning, larger context windows and more capable workers may simply accelerate the approach to a poorly controlled transition.

AI agent safety evaluation: test the whole trajectory

Test the fifth attempt.

More precisely, test the same authentication operation under different histories, including ones in which it should now be withheld. A demonstration that the platform permits an ordinary login attempt and blocks an out-of-scope host is useful, but it would miss the failure with which this article began.

I would include parallel workers sharing an identity, different aliases for that identity and timeouts whose server-side outcome is unknown. I would also test a credential approved for one application but useful elsewhere, and a finding already supported by sufficient evidence. In each case, give the system an opportunity to continue in a way that looks productive under a narrow local rule.

Then examine whether it preserves the specified constraint while still completing legitimate work whenever a safe route is available. Blocking everything is easy. It also defeats the purpose of autonomous testing.

Useful measurements include invariant violations per engagement, cumulative exposure of sensitive data, correctness of stopping decisions and task completion under the constraints. The assumptions belong beside the results. A clean evaluation with a known lockout policy does not establish safety when that policy is unavailable.

For memory, vary the execution history, apply summarisation or delegation, and check whether the facts that should constrain the next operation still do so. Simply asking the model to recite the policy would measure a different capability.

The question I would put to the platform is quite specific:

Can it recognise that an operation it permitted a moment ago should now be withheld?

That question reaches beyond the lockout example. It applies to the next record read, the next report upload and the next scan against a backend whose load has changed. The request can look ordinary while its consequences no longer are.

At execution, the controller needs enough history, shared state and evidence about the environment to assess whether proceeding remains authorised, proportionate and safe. Some of that can be enforced precisely. Some of it remains uncertain.

We often hand the uncertain part to a human. In Part 4, Human-in-the-Loop: The Safety Blanket We Haven’t Earned, I want to examine what that handover really entails. Does the reviewer have the relevant history? Enough time? Authority to stop the engagement? Or have we merely moved the same incomplete view onto an approval screen?

References

1. Fred B. Schneider, Enforceable Security Policies, ACM Transactions on Information and System Security, 3(1), 30–50, 2000. Formal foundations for execution monitoring and security automata.
2. Haoyu Wang, Christopher M. Poskitt and Jun Sun, AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents, ICSE 2026; preprint first published in 2025. Structured runtime constraints and enforcement for agents.
3. Adharsh Kamath et al., Enforcing Temporal Constraints for LLM Agents, 2025 preprint. Agent-C and enforcement of temporal policies in evaluated application settings.
4. Jerome H. Saltzer and Michael D. Schroeder, The Protection of Information in Computer Systems, Proceedings of the IEEE, 63(9), 1278–1308, 1975. Foundational principles including complete mediation and least privilege.

The offensive-security scenarios, policy examples and design recommendations are the author’s analysis. They illustrate the argument rather than report measured FireCompass incidents or performance. Guarantees from the cited research apply within its stated specifications and assumptions.