Actual Budget's CI shipped a bug that saved 10.23 as 1,023. I gave GitLab Duo the release instead.

CI tests what the author thought to test

In July 2025 Actual Budget shipped version 25.7.0. Type 10.23 into the amount of a new schedule and it saved 1,023. The change had passed every test in the project's CI, and it sat in people's budgets for two days before 25.7.1 fixed it. That kind of bug is not rare. When I pulled the list on October 19, Actual's maintainers had labelled 122 issues as regressions, and each one got past CI too, because CI tests what the author of a change thought to test. Nothing checks that a release still does what production already does.

No model decides whether to ship. A browser does.

A merge to main is tested, scanned and packaged, OpenTofu configures Cloud Run, and the release goes to staging. Then GitLab Duo's Release Guard flow reads the commits between production and the release, names what users rely on that this change could break, and writes Playwright checks for it. It runs every check against production first and keeps only the ones that pass there: a check that fails on production is wrong about the app, not evidence about the release. Duo never sees the new release running.

Then the pipeline decides, without asking the model anything. The kept checks run twice more on production, then on staging. If one that held on production fails on staging, the release stays on staging and the Regression Report flow opens one issue: the commit it blames, the check that failed, what staging showed, and what a fix has to keep. When everything holds, the release goes to production with a GitLab Release, Duo's release notes and a note on every issue it closes, and the checks that vouched for it run against production every six hours.

The refusal

I put the change behind 25.7.0 back in as a merge request. Duo Code Review, using this fork's amounts-and-money review instruction, flagged it ("typing 12. returns 12, so the . is dropped") and I pushed it past, the way reviews get pushed past. Duo read the release and wrote a check that types 12.50 into a new schedule. Production showed 12.50. Staging showed 1,250. The release stayed on staging, Duo opened an issue naming the commit, and production was never touched.

Duo checks Duo

Then I wanted to know whether Duo could also fix what it caught. I put the change back in again and gave each issue the guard opened to Duo Developer. It took seven merge requests. Duo Code Review approved all seven before they merged, and the guard still held five of them. Nine of the failures it caught after that approval were wrong amounts: 5.5+2.5 saved as 30.50, 1250.50 saved as 125.050,00, 1,00 typed over a selected amount applied as 100.00. None of them reached production. Production now shows thousands separators while you type, a feature Actual's maintainers reverted twice for this same kind of bug.

Issue to production, with nobody in between

In that loop I still merged each fix and assigned each issue myself. So I moved both into the pipeline, and gave it a bug real users reported upstream that is still open in 26.10.0: skip a bill's next date, rename the schedule, and the bill comes back as due (#9167). I reproduced it on production, filed it, and assigned it to Duo Developer at 19:58. Nobody touched it after that. Duo Developer opened a fix nine minutes later. Duo Code Review sent it back once, because the new test could pass without the fix, and the pipeline handed that back to Duo Developer. The second version was approved at 20:26 and the pipeline merged it. Duo wrote five checks for editing schedules, all five held on production and on staging, and the fix was in production at 21:40.

How I know it works

I did not write the bugs. I took eleven regressions Actual's maintainers labelled, each of which passed CI, shipped, and was fixed in a later release, with published images for every version. Each one was replayed from a branch holding only the release before and the release that broke it, so the fix never existed while Duo wrote its checks. A regression counts as stopped only if a check passed before, failed on the broken release, and passed on the fix. A smoke test with no Duo stopped 1 of 11. The guard that writes checks without running them stopped 3. The guard that tries its checks on production first, the one running live, stopped 2. Every catch was run five more times on each of its three releases, and a script fails the build if the README and the data ever disagree.

What is not claimed

The replay is graded by the maintainers' own labels and published images, not by me, but it is eleven regressions, and the guard lets most of them through. The live version stops 2 of 11, one fewer than the version that does not calibrate, because calibrating keeps more checks and holds more intentional changes (14 non-catch blocks against 6). It pins what production does today, including production's own bugs. It misses regressions in places the diff does not obviously touch, such as an empty-state page. It only holds what its checks cover: one release shipped a wrong amount (deleting two digits of 1,250 applied 1.20) because no check typed it, and a later Duo fix shipped the correction. It cannot show that a fix works, only that it broke nothing production already does; for #9167, Duo's unit test, Code Review and my own check on production after release covered the fix. Releases that change behaviour on purpose wait for a person to press promote-held-release, and one guard run takes about half an hour of Duo time.