The two questions that actually decide if an autonomous agent is safe

Ask most vendors why their AI agent is safe to use and you get an answer about governance. Audit logs. Approval gates. Compliance evidence. All of that answers one question: can you prove what the agent did after the fact, the question this audit trail is built to answer.

It's a real question and most of the market genuinely can't answer it, which is worth knowing. But it isn't the question a business actually asks when it's deciding whether to let an agent run unattended. That question is simpler and less technical: will it do the wrong thing, and will I find out before it matters.

We spent a research pass on exactly that question, not the compliance one, looking hard at a dozen agent frameworks and at our own architecture. We found two specific failure modes that sit underneath almost every "is this safe" worry a business actually has. One of them we've already solved, in production, without anyone planning to. The other, nobody has solved, including us. This is what we found.

The first question: does the plan still match reality

An agent given a big goal usually breaks it into smaller steps and works through them. The obvious risk is that the plan goes stale. Something changes halfway through, and a rigid agent keeps executing yesterday's plan against today's situation, confidently doing the wrong thing because nobody told it to stop and reconsider.

The fix sounds obvious once you say it: don't treat the plan as fixed. Let the goal persist and regenerate the plan itself on every cycle, so a changed situation produces a different plan instead of a stale one being followed anyway.

Here's the honesty note this piece needs early: we didn't design this. We found it already running in our own system while researching something else entirely.

Quox's objective engine generates a fresh execution plan every time it re-checks a long-running goal, not once at the start, which means a goal that's been running for hours is always working from a plan built against current reality, not an assumption made when it started. Nobody had named this as a deliberate safety property. It was just how the re-check loop happened to work.

Once we noticed, we went and checked a system built specifically around the idea of continuous evolutionary improvement, and its research loop does something structurally similar: every iteration proposes a fresh attempt rather than committing to one long-lived plan. Different reasons, same shape. It's a reassuring pattern to find independently arrived at twice.

The second question: did the smaller goal actually serve the bigger one

This one is harder, and as far as we can tell, nobody has actually solved it. Not us, not any of the twelve systems we looked at.

When an agent breaks a big goal into smaller ones, every system we studied treats that breakdown as correct by construction. The agent decides "to fix the bigger problem, I'll do these three smaller things," does them, and the smaller things get marked done. What nobody checks is whether doing the smaller things actually moved the bigger problem at all.

An agent can complete every sub-task it assigned itself and still not have touched the thing you actually asked for, and there's no mechanism anywhere in this category that would catch it.

This matters more than it sounds like it should, because it's exactly the failure mode that erodes trust slowly rather than dramatically. Nobody notices a single agent quietly solving the wrong problem well. They notice six months of it, when someone finally asks why the metric that was supposed to move never did.

The fix we think is right, and haven't built yet, is to stop treating a sub-goal as an assumed-correct decomposition and start treating it as a claim that has to hold up: this smaller thing was chosen because we believed it would cause progress on the bigger thing, so check afterward whether it actually did, not just whether the smaller thing itself got marked complete.

That's a different question from "did the task finish" and none of the systems we studied ask it. As far as we've been able to tell, nothing in the wider literature quite asks it this way either, which is either a good sign we're onto something or a sign we haven't looked hard enough yet. Being honest about which, we don't know yet.

The part people don't think to worry about: who's allowed to hit the brakes

There's a third thing worth naming, because it's the one that surprises people. The instinct once you're worried about an agent doing the wrong thing is to add a watcher: something that observes the agent and can step in. That instinct is correct.

What people don't think through is that the watcher is now also a thing that can go wrong, and if it has unrestricted power to freeze or interrupt whatever it's watching, a single miscalibrated judgement from the watcher is now a new way for legitimate work to stop, not just a new way for bad work to get caught.

We looked at this specifically and found almost nobody in the category has thought about it as a design problem. Most systems have no watcher at all. The rare exception we found, buried inside one system's engineering, is a narrow detector that halts an agent when it notices obviously repetitive behaviour, nothing broader.

Nobody has a general answer for the harder version: a watcher that can genuinely catch subtle problems also needs genuinely serious authority to act on what it finds, and serious authority in the wrong hands, or the wrong calibration, is its own liability.

The direction we're taking, not yet built, is treating a watcher's authority as something that has to be explicitly bounded per watcher, not granted wholesale. A narrow, mechanical check, the kind that catches an agent obviously going in circles, can reasonably be trusted to pause things on its own.

A broader, more judgement-based watcher, the kind that's trying to notice something subtler, should be able to flag and ask, not freeze on its own authority. Different watchers, different power, decided deliberately rather than defaulted to either extreme.

Where this actually leaves things

To be specific about what's real and what isn't, because this whole category blurs that line more than it should: the disposable-plan pattern in the first section is genuinely shipped and running today, we found it rather than built it, and we're not aware of anyone else in this space describing it as a deliberate safety property, though at least one other system independently landed on something structurally similar for different reasons.

The sub-goal verification idea in the second section and the bounded-watcher idea in the third are both research direction. Neither is built. We think they're right, and we think naming them clearly, including the fact that we haven't solved them yet, is more useful to anyone evaluating this category than pretending the compliance-audit story covers it.

Governance answers "can you prove what happened." These two questions are "will the agent do the wrong thing" and "will the thing watching for that become its own problem." Different questions, and right now, less answered.