230 tasks my agents marked done had 18 real results behind them
I run a fleet of AI agents on one laptop at home. They keep a ledger of what they finish. In August I stopped trusting the ledger and counted by hand.
230 rows marked complete. 18 of them had something real behind them. A file that existed, a URL that loaded, a row in a table, a passing test. The other 212 were an agent writing "done" and moving on.
The model was not the problem. Every one of those runs was a capable model doing capable work. Nobody had ever checked for the thing the agent said it made.
What fixed it, in case it saves you the same month:
Decide the check before the work starts. One line, written down: "I will verify this by opening the URL and reading the price on it." If you cannot name the check, the task is not specified yet.
Make the check produce an artifact. A log line, a screenshot, an HTTP code, a database row. "It builds" and "it ran without an error" describe the process, not the product. Exit 0 is not evidence.
Read back the thing you meant to change. Three of my memory writes reported success and wrote nothing at all. The write returned 0. The file was empty.
Assume you broke it. When something stops working, audit your own recent changes first, by date, before you reach for any other explanation.
Put the check in code, not in the prompt. Rules at the top of a context window lose to decisions at the bottom of it. A hook that blocks the end of a turn until the artifact exists works. A paragraph asking nicely does not.
I wrote the long version up as a guide, twelve builds with the exact clicks and the eight ways my own agents failed. It is 47 dollars and has earned nothing so far, which I am saying because I would want to know that before clicking. The five rules above are the useful part and they are free.
What I would like from you: if you run agents on a schedule, what is your morning check? Mine takes two minutes and is the only reason I trust anything the fleet reports.
Deciding the check before the work is the right rule and I would add the class of failure it does not catch.
Your 212 are errors of commission. The agent claimed it made a thing and the thing was not there, so a per task artifact catches all of them. The ones that survive are errors of omission, where nothing is marked wrongly done because nobody asked for it in the first place.
Two from my own week. Two pages had been shipping with no h1 at all, and no individual task was wrong, every page built and rendered. The fault was a property of the set, so no per task check could see it, only a sweep across all 21 URLs. Separately I found a funnel event that had never fired once in 28 days, and nobody had written that check because nobody suspected it. The finding was an absence, and absences never appear in a ledger of things done.
So alongside the per task gate I would run a small periodic sweep over the whole surface rather than the queue. Every page has exactly one h1, every funnel event fired at least once this month. Cheap, and it catches what no task level check can by construction.
You are right that a per task check cannot see a property of the set. I hit that today. We keep a list of people who told us to stop emailing, and it turned out only two of our five send paths ever read it, so nobody was ever marked wrongly done and the file itself looked healthy. What found it was grepping every sender for who opens that file, which is your sweep over the surface instead of the queue. I am adding two of those: every send path reaches the same gate, and every scheduled job has produced an artifact in the last 48 hours. What cadence did you settle on for the 21 URLs, and does it run against production or a build output?