An agent that will not put words in Canon's mouth: 21 of 22 quotes found word for word in the manufacturer's own documents

Page 100

When I was working in film, my Canon T5i would sometimes have trouble focusing in auto mode, and I would give up and switch to manual and do it myself. The explanation was in the box the whole time. Page 100 of Canon's own manual for that camera says that in the Basic Zone modes the camera will normally focus the closest subject automatically, and therefore it may not always focus your target subject. It is 388 pages long, and page 100 is the one I needed. That is the shape of the whole problem. These questions have published answers, written by the manufacturer, and the answers are hard to find and easy to misremember. Sigma's MC-11 table tells you which of its lenses keep continuous autofocus on a Sony body, and the answer for that adapter is none of them. Canon's instruction sheet for the 18-55 STM names the exact four bodies it does quiet Movie Servo AF on. Nobody reads either document before buying the lens.

The Knowledge Base put quotation marks around thirteen sentences that do not exist

I started with one Sanity Context endpoint and attached both sources to it, a Knowledge Base built from six manufacturer PDFs and a typed dataset. The first time I ran the agent against the Knowledge Base alone, it answered well and it put quotation marks around 13 sentences. Not one of those 13 appears in any of the six documents. That is not a bug in the Knowledge Base, it is the Knowledge Base doing its job: it rewrote 388 pages of Canon into ten navigable entries, and rewriting is the point of it. The mistake was mine, for asking a summariser for the manufacturer's exact words. So I split the two apart. The dataset endpoint serves records in GROQ mode and verdicts come from there. The Knowledge Base endpoint serves the explanations, and every point it contributes is labelled on the page as a summary rather than the manufacturer's words. Saving the dataset had been replacing the Knowledge Base rather than adding to it, which is what forced the split into two endpoints, and both tool sets carry a prefix so the agent can tell one initial context from the other.

A quote counts only if it is a whole sentence of the record

The rule the whole app turns on runs after the model is finished, not inside the prompt. Every quote the agent returns is looked up, by code, in the Sanity record the agent said it came from. It counts only if it is a whole sentence, or a run of whole sentences, of that record's stored quote. A near match does not count, and a tidied-up version of the sentence does not count. When the model has reworded something, the page says so and prints the record's actual sentence underneath, so you see both what the agent wrote and what the manufacturer wrote. Each autofocus state in the answer is also compared against the fields of the records the agent cited, and any state no cited record supports is labelled. The reason this has to be code rather than instruction is the 13 invented quotations: a model that has been told to quote exactly will still produce something that reads exactly like a quotation, and no amount of prompt wording turns that into a verified one.

Three runs, and the verdicts barely move

Twenty questions, chosen because their answers sit in Sigma's, Canon's and Metabones' own tables and manuals, run three ways on the same model. The Knowledge Base alone gets 37 of 43 verdicts right and 1 of 10 quoted passages verbatim. The typed dataset alone gets 39 of 43 and 28 of 31. Both together, which is what the deployed app runs, get 38 of 43 and 21 of 22. Read the two columns against each other, because the interesting thing is that the verdicts barely move. If all you want is a yes or a no, the Knowledge Base is nearly as good. The entire difference between the rows is whether the words inside the quotation marks belong to the manufacturer. I did not plan that as an ablation and it turned into the most useful thing in the repo: it says what the typed records actually contribute, in the one unit that matters, rather than asserting that they were necessary. The eval also found two of my own mistakes. I had first stored Sigma's DMF column inside a free-text note, which the agent could not query, so it answered not covered on every DMF question until it became a typed field. And the smaller model kept missing records because "Sony a7 IV" did not match the alias "a7 IV", then invented an id to cover the gap. Bodies and lenses now carry the names people actually type, and the agent is told never to construct an id.

The schema had to be allowed to say nothing

Three schema decisions did most of the work. Autofocus is split into viewfinder photo, live view and video, because the same body focuses with genuinely different hardware in each of those and a single autofocus field would have forced one answer to stand for three. A compatibility record is only ever as wide as its source: Canon's note on the 18-55 STM names four bodies, so that record references exactly four bodies, and when Sigma's table names no bodies at all the record references none, which the agent reads as not limited to particular bodies. And every field a source might simply not address can answer that no source states it. That last one matters more than it sounds. A schema that only offers yes or no forces the model to guess, and a guess in a compatibility answer is indistinguishable from a fact until somebody buys the adapter.

The best answer it gives is no answer

Put a Nikon F lens on a Canon RF body through a third-party adapter and the app returns no verdict. Nobody has published an answer for that combination, so there is nothing to quote and nothing to stand behind. This is the case I most wanted to get right, because the plausible wrong answer is easy: an adapter that physically mounts will let you take a photo, so a model with no record in front of it will happily reason its way to autofocus probably working. It does not follow, and the refusal is the feature. The same instinct is why the gate reports an unverified quote instead of dropping it. A quote that quietly disappears looks like a clean answer; a quote shown as unverified next to the manufacturer's real sentence tells you exactly how far to trust it.

Two scripts that fail the build

Every record's quote has to survive a check against the document it claims to come from. One script downloads the manufacturer PDFs, extracts their text, and fails if any quote is not on the stated page of its stated PDF, or not in the collected page text for the web sources. It passes on 137 quotes with zero failures, and it is the reason I can say the records are verbatim rather than hope so. A second script reads the numbers out of the README and the writeup and fails if any of them disagrees with the data. I keep building this one because a number written on a Tuesday and a pipeline re-run on a Wednesday is exactly how a writeup ends up lying about its own results, and I would rather the build tell me than a reader.

What is not claimed

The gate proves that a quote belongs to the record the agent cited. It does not prove the agent cited the right record for your exact combination, and that is the honest limit of the design: I can verify the words, not the relevance. The dataset is small, covering the T5i era of Canon, Sigma's MC-11, Metabones' EF-E Mark V and Canon's EF-EOS R and EF-EOS M adapters, so plenty of real combinations return no verdict. Five flange distances come from Wikipedia rather than a manufacturer, because no manufacturer page states them, and those records say so on their face instead of borrowing the authority of the ones that do. The four example questions are saved answers rather than live runs, re-checked against the current records every time they are served, and the page says which is which. And the headline number is 21 of 22 on twenty questions, which is a small sample by design: I only used questions whose answers I could point to in a document, because a bigger sample I graded myself would have been worth less than a small one Canon graded.