AI-augmented engineering

Ten months with an agent pipeline: what the gates caught that the prompts did not

October 20265 min read

I reverted a bug fix and ran its test. The test passed.

That was the moment this project changed. Until then I had been treating a green build as information. An agent had written the fix, written a test for it, and the test went green. I went looking and found three of these in one day. One test pinned the date to today so the condition could never happen. One used an invented date outside the window the code checked. One asserted that the source contained the word "fetchmany", and it did, in a comment.

Each test looked right. Each asserted nothing.

Some context. Last December I set myself a question: how far can a requirement travel without me typing the code? Ten months later the answer is three projects, around four thousand commits, over a million lines of code and tests across Java, TypeScript, Python and Dart, built in evenings and weekends. I did not type most of it. But the lesson is not about the agents. It is about what sits around them.

The shape of it

Every repository has a working agreement file at the root. Its first line says it is not advice, it is the bar. Then a lifecycle: analyse, ticket, implement, test, document, review, close. Two phases stop and wait for me. Nothing is coded before I approve the ticket. Nothing is closed before I approve the review. Commit messages are checked by a hook: they must say why, name the ticket, and say what was tested.

That is the whole design. It is not clever. The clever part, if there is one, is what fails the build. Here are four things the gates caught.

1. The test that could not fail

After the revert incident, the working agreement got a rule. Before committing, break the thing the test protects and confirm the test goes red. Then restore it. And bad inputs must come from the data, the row that actually broke, never one invented to look broken. An invented fixture agrees with whatever you already believe.

That rule is not a script. It is the one thing in the pipeline that cannot be. It is also the one I check most.

2. The ratchet that counted its own prose

The Python project has about forty data-audit checks. I wanted proof that every check had a test showing it fires on bad input, so there is a number: the minimum count of checks shown to fire. The build fails below it. It only goes up.

In October I found three things wrong with it. The floor said 21 while 30 were covered, so nine new checks could have arrived untested. The counter scanned test files for check names, so a docstring saying "this file does not cover A and B" credited A and B. And the file holding the note "two remain: X and Y" was in the scan, so that sentence had been crediting X and Y for weeks. Excluding it dropped the count from 38 to 34.

When a number goes up, ask what moved it.

I wrote that gate. The agents fed it happily. No prompt would have found the problem. A habit did.

3. The money that was almost a double

In the Java and Dart project, money and quantities are integers. A CI script fails the build on double.parse or .toDouble() near those types. It exists because an agent parsed a kilogram field with double.tryParse and multiplied by a thousand. 0.35 through that line is 349.99999 and the truncation lands on the number someone gets paid against.

It is exactly the line a human writes. I would have written it. The first test case in the gate's own test file is that line, verbatim, so the gate cannot be weakened without something going red. A rule like this belongs in a script, where it is binary, not in a prompt, where it competes with a thousand other instructions and loses.

4. The suite that would have failed in December

A billing test in the TypeScript project started failing with a timestamp off by the current time of day. I spent an hour hunting a timezone bug. It was a fixture with a date that had been in the future when written and had quietly become the past.

The obvious fix is a guard that flags future dates in tests. I built it. It found 36 sites and 35 were fine, because this codebase injects clocks everywhere, so a future date is usually just an input. The cure for a false positive is a suppression comment, and 35 of those is how a check gets exempted into meaninglessness. I deleted the guard. Instead, every night CI runs the unit suite with the clock wound forward 90 days. Out of about twelve thousand tests it found one real case. One was enough.

What the prompts could not do

I tried to fix every one of these with a better prompt first. It does not hold. An instruction in a prompt is advice, and a model follows advice most of the time. Most of the time is not a standard I would accept from a colleague. What held was mechanical. Hooks. Scripts. A number that only goes up. A nightly job nobody can argue with.

The part I did not expect is that the gates catch me too. The prose-counting ratchet was my bug. The mutation floors, once they produced numbers, told me a module I assumed was fine scored 68 percent, and that its tests exercised the route but not the arithmetic.

If I started again

I would start with the gates, not the prompts. Every gate I have, I could have written on day one, and each would have saved me a bad week. I would distrust green sooner. And I would test any sentence that says "this cannot be tested". Four checks sat uncovered for weeks behind exactly that sentence. Every one of them was wrong.

What I will not automate is the two approvals, ticket and review. That is where I find out what I actually asked for. Ten months in, no gate replaces it.

All posts