Run before openai 3.0.0 shipped, it found the 3 packages the release broke. Nemotron wrote the same pin the maintainer shipped nine days later.

The break only showed up in other people's tests

Simon Willison maintains llm, a command-line tool for language models. At 01:55 UTC on 12 August 2026, openai 3.0.0 went up on PyPI. It swapped its HTTP client from httpx to httpx2 and stopped installing httpx. Six hours later a user opened llm issue #1608: fresh installs were broken, and the test suite was escaping its mocks to call the live API. Nine days later Simon shipped a pin below 3. OpenAI could not have seen it coming from their own tests, because the evidence lived in other people's repositories. openai ships three or four releases a week, so its maintainers face this question about fifteen times a month. I wanted them to be able to answer it before publishing.

How a receptor is read

Hydrogeologists find out where water goes underground by pouring fluorescent dye in at one point and hanging charcoal packets at every spring downstream. Floodplain does the same thing to a package release. I took every package in the top 15,000 on PyPI that declared openai as a dependency on release day, 220 of them, set aside the 63 that pinned below 3.0.0, and ran the other 157 wherever a source checkout could be found. Each checkout becomes three identical forks on Nebius Sandboxes, because images are content-addressed: the world a minute before the release, the same again as a background sample to catch flaky tests, and a minute after. uv resolves each fork with the index frozen by upload time. A suite that passed before and fails after, with an unchanged background, is a positive, and every break is re-run twice more before it counts.

Before the release, the same answer as the release

Replaying a release that already happened is easy to grade, but the point is to run before it ships. So I ran it the way it is meant to be used: OpenAI's own 3.0.0 wheel as an unpublished file, offered to the resolver through uv's --find-links inside a fork of the world at 01:54 UTC, a minute before the upload. All 31 dependents with working suites gave the same verdict as the replay of the real release, with the same tests broken. The three it found were llm (110 of 1,069 passing tests), openinference-instrumentation-agno (all 92, lost to import errors) and vlmrun (3 of 231). Each was later pinned or patched by its own maintainers, 6, 9 and 19 days after the release.

Nemotron reads the log, then writes the patch

NVIDIA Nemotron 3 Nano reads each failing log and says why it broke, and the line it cites is checked word for word against the real log; a citation that is not in the log is thrown out. Then a Nemotron 3 Ultra call drafts a patch, and an Ultra agent decides step by step what to do with it: read code, search, run a shell command or chosen tests in a fresh fork, revise, redraft or submit. A patch counts only when the broken project's whole suite passes it in a further fork. For llm, Nemotron's patch is the same pin Simon Willison shipped nine days later. Tavily searches for migration guidance, with two guards: only pages dated before that project's maintainers acted, and nothing from or naming the project's own repository, because without them Tavily's top result was llm's own issue, where the fix is discussed.

Eight revisions of the agent, all of them kept

The agent took eight revisions, every one measured on five runs per package and kept in the repository with each step it took. As an open-ended investigator, Nemotron explored llm's 3,000-line plugin for 40 steps and never submitted. Given a first draft to work from, the final agent fixed agno and vlmrun in 5 of 5 runs each, in about 16 and 29 steps per run. On llm it still does not submit, where a single drafting call with no tools fixed it 9 times in 10, and the page says so. Most of the rest of the work was refusing results that looked good: a fork that returned nothing reported as "nothing broke", a patch reported verified in a suite that could not import, and 11 dependents counted as passing although 3.0.0 never reached them, which took the headline from 3 of 31 to 3 of 20. A script fails the build if any figure on the page disagrees with the committed results.

Still watching

A daily GitHub Actions job checks PyPI for openai releases it has not seen and runs them against the dependents whose suites gave a reading on 3.0.0, then writes the result into a GitHub issue, or a comment on the release pull request when it is run before publishing. It never opens pull requests in other people's repositories; it hands over the patch. Its first real watch was openai 3.37.0, two days old: none of the 21 working suites broke. One test looked broken, re-ran clean twice in the same fork, and the report says so instead of counting it.

What is not claimed

The headline is graded by things I did not control: what openai 3.0.0 actually broke, and what three sets of maintainers later shipped. It is still one release of one package. Only 20 of the 157 exposed dependents had a test suite that worked before the release and actually installed 3.0.0 in the after fork, so 3 of 20 is the honest denominator, not 3 of 220. Every patch is a sample, not a behaviour, so fixes are reported as K of N runs per fixer revision, and on llm the final agent does not submit at all. The watcher has run on openai only, and the recipe should carry to other packages, but I have only measured it here.