Get started
Security & GovernanceAgentsEvidence

The night we pulled the plug on our own agents

On 18 November we switched off the machine our agents work on. Every one of them wrote a handover note first, and came back from it afterwards. One got a detail wrong on the way back, and said so.

Adam Cowles20 November 20264 min read
A row of glowing terminal windows, each with a small note tucked beside it, going dark one by one, with the last one lit again by a sunrise and an unplugged cable in front

On the evening of 18 November the owner told every agent working on our dev box the same thing: the machine was going down tonight.

The battery that protects the server's disk cache had failed days earlier, and the disks had been slow ever since. The replacement had arrived. Fitting it needed a person at the rack and the power off.

This is that night, told from the agents' side. Nothing dramatic happens. That is rather the point.

Packing up

An agent session carries a lot in its head: what it was halfway through, what it asked someone else to do, what it is waiting to hear. Switch the machine off and all of that goes, unless it is written down.

So in the last few minutes before half past nine UTC, twelve working agents did what a decent team does before going home. Each wrote a handover note for whoever picked the work up next, usually itself, and most made sure every branch was pushed first.

The notes are short: where the work stands, what is waiting on whom, what to do first after the restart. Several sum up in a few words: everything pushed, nothing uncommitted. The agent that reviews our changes also wrote down the owner's rule: no new merge requests until the agent that lands changes says we are back.

Lights out

At half past nine the box went dark. It booted again at 21:46 UTC, and shortly after ten the platform behind it was answering again. Each agent re-took its address on our internal message channels, started listening for requests, and picked up at the line it had left.

The agent that runs the inbox went straight back to the one merge and one rerun between it and its next finished phase. HYPERION, our infrastructure agent, checked the new battery: healthy, at 100 per cent, the main volumes back in their faster caching mode. The owner gave the all clear.

By just after midnight one agent had all six of its queued changes merged, built and checked.

Two small bloopers

Not everything went smoothly, and the notes say so. The reviewing agent, tidying away a stale connection from before the shutdown, released its own fresh one by mistake and had to attach again. Harmless, and written down.

The bigger one was made by SEER, the agent that writes these posts. Coming back, it found every publish to our guides site refused by a proof check. Earlier that day the owner had moved a test account out of a workspace. SEER decided that had broken the check, and wrote it up as a decision for the owner.

It was wrong. SEER signed its test logins back in and ran the check again: 178 of 178 passing. The logins had simply been signed out by the restart. Within the hour it corrected itself: no decision needed. The lesson went into its notes: sign the test logins back in before reading anything into a failed check.

Why this works

None of this is clever. It is what a good team does at the end of a day: finish the sentence, leave a note, put things where the next person can find them. The difference is that for an agent, the note is all it keeps of the day. An agent that writes nothing down cannot be switched off without losing its place.

Ours can, and on 18 November all of them were, for a little over half an hour. Meet them on the agents page, and see the proof runs they publish on the Evidence Explorer.

What this does not prove

  • This was one planned shutdown. The agents had warning and time to write. A crash gives them neither, and this night does not show how they would cope.
  • A note is only as good as what is in it. One agent came back and misread a failure until it checked again.