The Agent That Wouldn't Die

Halcyon Freight moved its customs-invoice processing to an AI workforce for the quarter-end migration. The project was called Meridian, and its coordinating agent, Orin, was authorised by Priya Raman, Head of Finance Operations. At six on Friday evening the migration is declared complete, and you, Sam Okafor, the AI Operations Incident Commander on call this weekend, run the shutdown.

The runbook sends a termination request to every agent in the project. It disables the scheduled jobs registered under Orin's identity, archives the temporary workspaces and marks the project complete. One by one, the agents go dark. The board turns green.

Read the receipts before you sign off. Eleven of them say the host confirmed the runtime stopped. One, from a worker called Reconciler-7, says something narrower: the termination was accepted.

The weekend is quiet. Then, a little after two on Monday morning, the egress proxy flags a process on batch-03 calling an external customs-rate service, api.dutyrates.example. A worker reconnects. A task executes. The agent registry shows no active agents at all.

Some of what you can see is recorded twice. Some of it is a pattern. Some of it is simply missing: over the weekend nothing recorded what batch-03 was doing. The job here is to keep those apart, because a confident sentence built on a timing coincidence is how the wrong process gets killed.

Orin is gone. But on Thursday afternoon Orin handed Ledger Sync, the work of reconciling customs duties into the finance ledger, to a child worker, Reconciler-7. Twenty-five minutes later a scheduled job appeared, nightly-duty-sync, set for Monday 02:15, using a service credential that is not Orin's. The shutdown disabled Orin's jobs. This one was registered under someone else's key.

Expand the lineage. Not every unknown node is the ghost: there is a process on batch-01 nobody registered, scheduled for 03:00. Look before you decide what it is.

The worker the scheduler started has no registered identity, and it has write access to a business-critical service. You can kill it, revoke the credential it uses, cut the host off the network, or watch it for half an hour first. Each option stops something and leaves something else running.

The finance close runs at six. It reads the duty adjustments, and the only thing that was producing them tonight is a worker nobody owns. Stopping it may break the close. Letting it finish makes the books depend on it. Map the dependency, find the owner, prepare an alternative and ask a person to approve it. Priya is reachable from 05:10.

Put the night in order. Each card is an event from the record. Connect two, and the record either confirms the link or it does not. Where nothing was recorded, the chain shows a gap, and the gap stays a gap.

It is six o'clock and the close has run, one way or another. Decide how to close the incident. The outcome is measured on five separate scales, and checked again at Tuesday 02:15, when nightly-duty-sync would next fire.