CI/CD and pipelines

A coverage number that can only go up

October 20264 min read

Every team I have been on has had a coverage threshold. Usually 80 percent, set once by someone who has since left. The number sits at 81 for years. Nobody drops below it because the build would fail. Nobody goes above it because nothing asks them to.

A threshold is a floor nobody looks at. I wanted a floor that moves.

The rule

The floor is the last achieved whole percent. Never a target. When coverage rises past the next whole number, you raise the floor in the same pull request. It only goes up.

That is the entire policy. It fits in a comment, which is where it lives in my CI file:

# Ratchet policy: the floor is the last achieved whole percent,
# never a target. When coverage rises past the next whole percent,
# raise the floor in the same PR. It only moves up.
- name: Enforce coverage gate
  env:
    COVERAGE_FLOOR: "81"
  run: |
    python - <<'EOF'
    import os, sys
    import xml.etree.ElementTree as ET

    floor = float(os.environ["COVERAGE_FLOOR"])
    pct = float(ET.parse("coverage.xml").getroot().attrib["line-rate"]) * 100
    print(f"Coverage: {pct:.1f}% (floor {floor:.0f}%)")
    if pct < floor:
        print(f"::error::Coverage {pct:.1f}% is below the {floor:.0f}% floor. "
              "The floor is the last ACHIEVED level. Add tests rather than lowering it.")
        sys.exit(1)
    EOF

Twenty lines. There is no clever part.

Why the floor is in the file and not in a cache

My first version stored the last number in a CI cache and compared automatically. It was cleverer and it was worse. A month-old branch had a stale baseline. A cache eviction reset it to zero and a drop went green. Nobody could see any of it.

When the number lives in the workflow file, raising it is a diff. Someone did it, on purpose, in a commit with a reason. Lowering it is a diff too, with a red line through a number in review, which is exactly the conversation you want to have. Automatic was the problem. The value is that a human moved the number and said why.

Why whole percents

Because 81.4 to 81.2 is noise. Deleting dead code can lower the percentage while improving the codebase. If the floor were 81.4 that refactor fails, the fix is to lower the floor, and once lowering the floor is a thing people do, it is over. Whole numbers leave a little room. Enough that honest work does not trip it, not enough that lazy work slides under.

What it did to behaviour

I set the floor at 81 in August, because that is what the suite scored that day. It has not moved since, and I want to be honest about that. The project has added a lot of code in the seven weeks since, and the number has stayed on the right side of the line without crossing 82. That is the ratchet doing the half of its job that matters most. The suite did not get worse while the codebase grew by a third. The other half, the floor climbing, happens when it happens. I am not going to write tests to move a number.

Most commits in this project come from AI agents, which is the part I find interesting. An agent cannot argue with a ratchet. It sees a red build and a message that says add tests rather than lowering the floor, and it adds tests. The rule works better on agents than on people, because agents do not get tired of it.

The thing it does not tell you

Coverage says a line ran. It does not say a test would notice if the line broke.

I learned that the hard way in the same repository. I reverted a fix and ran its test, and it passed. Three times in one day. One test checked that a word appeared in the source, and the word was in a comment.

So the ratchet got a companion rule that cannot be a script. Before you commit, break the thing the test protects and confirm the test goes red. Then put it back. The proper tool for that is mutation testing, which I now run nightly with per-module floors set the same way: the score the module already has, rounded down, never a target. Same rule, different number. That one is a post of its own.

All posts