Where the human still decides

At 20:25 the Dispatcher sent me a test plan with 32 steps. By then the three features had been in production for hours, every gate was green and nothing had been rolled back. Whether they were what I had asked for was still an open question. This part is about the checks that stay with a person, and about the day's third failure, which happened in exactly such a step.

A test plan the agent wrote for me

Automated gates cannot tell whether a feature does what the business wanted, so at the end of the second run the Dispatcher scheduled a human check for me. It wrote an acceptance test plan: 32 steps in 9 sections, each with what to do, the expected result and an empty box for my feedback. It prepared two test packages, a good one with a table, an image and a PDF, and a broken one the panel should reject. It ordered the steps so the assistant test runs on the article published a few steps earlier, and the last section cleans up, so no test content stays in the assistant's memory.

It took me about 40 minutes. I typed my results into the boxes, pasted screenshots where something looked wrong, saved the file and told the Dispatcher it was ready.

What I found

22 of the 32 steps passed without comment. One could not run yet, because the fix it needed was still on its way. The other nine were the interesting part:

• Two defects. The English and Polish article lists showed different things: the English one listed a test article, the Polish one said nothing was published yet. And the table of contents on the instructions page did not look like navigation and had scroll bars it should not have had.
• Five new requests. Open the article in a new tab after Publish. Send the contact email in my branding. Let me add facts to the assistant's knowledge, not only edit them. Let me restore a deleted fact. Let me select several articles and unpublish or delete them at once.
• One gap in the instructions. The publishing skill did not explain how to attach PDFs or how to publish an article in two languages.
• One mistake in the test plan itself. The assistant did not answer a question about the test article. The Dispatcher traced it: an earlier step had told me to delete a fact, and the next step asked the assistant about that same fact. The assistant behaved correctly and refused to claim a fact I had deleted. The agent that wrote the plan had tested its own feature against the wrong expectation.

The Dispatcher read the results, wrote a diagnosis for every comment and started the fixes as a new package within minutes.

Watch out: green checks, wrong feature. The gates are only as good as what they check. Here CI checks the code, the content rules and the build, and the live checks confirm the site responds. Nothing automated checks whether the feature does what the business wanted. That review is still done by a person, and it has to be planned. Let the agent prepare it, but read its plan as critically as its code.

Watch out: every demo grows the wish list. The moment people see a working feature, they see what else they want. My own test produced five new requests. Scope creep is an old problem; what changed is how fast it happens. Each request looks like two hours of agent work, so it feels free. Two hours here, two hours there, and the next ten hours are gone. Hold back on purpose. Write every request down, separate defects from wishes, and decide which wishes go into this release before anyone starts building.

The night shift

I went to bed. The Dispatcher kept going until 01:01 without me. It shipped the fixes from my test, built the navigation for the series format, and looked into two tests that failed now and then. One of them was hiding a real server bug: an upload over the size limit closed the connection instead of returning a clear error. I have learned to treat tests that fail now and then as bugs.

Before uploading anything real, it ran the whole publishing path on a test series: upload, preview, publishing all parts at once, navigation in both languages, withdrawal, and every address back to "not found". Then it uploaded the real series as a draft straight from a Claude Docs document and wrote me a new test plan for the morning, with the new labels it had chosen marked for my approval.

Nothing went public overnight. The last check is mine: I run the plan, and nothing goes live before I say yes.

Where the agent stops

Some steps should never be the agent's alone. During the second run the Dispatcher built a skill that publishes articles from my writing tool to the site through an API. That skill needed a machine credential for the admin panel, and there the agent stopped. It opened the right page in my browser, laid out every click and asked for my consent before changing anything in the account. I generated and stored the secret myself and put it on my machine with commands the agent prepared; the agent never saw it. Before it touched the server's configuration, it asked again. Passwords, payments and messages sent in my name stay with me. The agent can do everything else.

Failure 3: invisible characters in a credential

The step I approved went wrong anyway. With my consent, the agent added the credential's ID to the server configuration. On the way through the Windows shell the value picked up two invisible characters: a byte-order mark at the start and a line ending at the end. The service restarted, the live checks were green, and nobody noticed.

The next release found it. Fifty minutes later a small fix stopped at the configuration check before going live, and production stayed on the previous release, untouched. The agent found the cause, asked for my consent again and cleaned the one line. It also logged its own mistake: after writing the value, it had counted the lines in the file instead of checking the bytes.

I took two lessons from one line of configuration. The check that matters looks at what the change touched, as well as whether the site answers. And a step that needs a person's consent is not automatically safe; it is often the least tested step in the whole pipeline.

Watch out: the manual step nobody verifies. Pipelines get tested every day, while the steps around secrets, accounts and server settings run once and are trusted. Script them, check the result byte for byte, and make the next release verify the configuration it depends on.

If I ran your team

I would ask the agent to write the acceptance test plan for every feature, and then have the product owner run it, not the agent. I would also give every manual step around secrets and settings a script and a check, even if it runs only once a year.

Downloads

• Sample: acceptance test plan with my results (PDF, 601 KB). The plan the Dispatcher wrote for my manual check: 32 steps and the expected result for each. My results and screenshots are in the grey column; an empty box means the step passed. Internal paths removed.