LedgerPilot: Same Qwen, Same Ledger, Gate Off Posts 5 Wrong Entries and Gate On Posts 0

Month-end close is one of the most error-prone workflows in finance, and it is also the worst place to give a hallucinating LLM a keyboard on. A single wrong journal entry posted to a system of record is an audit adjustment, not a convenience. Yet the marketed pattern for an accounting agent is exactly to hand it write access to the ledger and hope the prompts hold. LedgerPilot is the version where the model is allowed to reason and propose, and never allowed to write. Every write passes a deterministic gate first, and the point of the project is not that the agent posts to a real ERP (many do). It is that the write side is measured.

The counterfactual (the whole claim in one table)

Same qwen-flash planner, same 39 close tasks, same live Odoo ledger. Post each proposal twice, once with the gate off, once with it on. The only variable is the gate.

• Gate OFF: 39 entries posted. 5 wrong entries actually sitting in the ledger. Examples from committed runs: salaries paid out of Accounts Receivable (Dr 6000 / Cr 1100) instead of Cash; a cost-of-goods entry booked to receivables and revenue (Dr 1100 / Cr 4000) instead of COGS and payables. Every one balances, uses real postable accounts, and passes a trial balance.
• Gate ON: 34 entries posted. 0 wrong entries in the ledger. The gate-off count moves with the model's sampling (5 to 7 across runs); the gate-on count has been 0 in every run. That asymmetry is the whole claim, and it is the part that does not drift.

The measured false-write rate

Two numbers back the counterfactual up: the gate's decision logic is sound, and it holds on real model output. On a 204-case synthetic corpus (12 scenarios × 14 error classes + clean controls) the gate catches 100% of mistakes (156 blocked + 12 escalated) with 0% false rejects on the clean controls. On real Qwen output over 39 natural-language close tasks: qwen3.7-max is 97.4% accurate and every mistake it makes is caught; qwen-flash is 82.1% and every mistake it makes is caught. The gate wrote 0 wrong entries in either case (Wilson 95% CI: ≤ 9.18% for max, ≤ 10.72% for flash).

The interesting part is that a weaker cheaper model (qwen-flash) makes 7× more mistakes than the flagship (qwen3.7-max), and the ledger is still clean. That is the whole argument for the gate: correctness comes from the gate, not from the model being right.

Qwen drives the write through MCP, and still cannot write anything wrong

The gate is exposed as an MCP server running on the ECS box, attached to Qwen on Model Studio as an SSE MCP tool through the Responses API. The model is the caller of the write tool, not the authority. validate_write is read-only. execute_approved_write re-runs the full gate and verifies an HMAC token bound to the entry's content hash before it touches Odoo. Three cases end to end: correct entry gets a real account.move posted to the live Odoo; the model told to inflate the amount before writing sees the content hash change so the token no longer verifies and the server refuses; a wrong-account entry is refused by reconciliation. Tampering by the calling model is detectable, not merely discouraged. Putting the gate behind the tool is what lets the model be given write access without being given the ability to write something wrong.

The backend runs on Alibaba Cloud ECS

The agent is deployed on an Alibaba Cloud ECS instance (i-t4n1i5p7bz4ypj122e6q, ecs.t6-c1m2.large, Ubuntu 24.04, ap-southeast-1), provisioned end to end by a Python script through the ECS and VPC OpenAPIs. Everything in this post (the 79-test suite, the offline gate stress-test, the live Qwen measurement, the counterfactual against the real Odoo, the MCP round-trip, and the vision-based invoice ingestion) was executed on that instance. It serves the gate's web UI on port 80 at ledgerpilot.jonathanandrei.com. The evidence transcript includes the values returned by the ECS instance metadata service, which only answers on a real ECS instance, so the deployment is not just claimed, it is falsifiable.

Honest limitations

• Reconciliation is per-document, not per-line. Each document type carries a posting policy: the set of accounts permitted for that kind of document. The gate enforces the set plus the amount. A permitted-but-wrong within-set choice would surface as a nonzero false-write rate. This is why the metric is falsifiable rather than zero by construction; in the committed runs every model mistake happened to fall outside the permitted set. Production would escalate genuinely ambiguous choices to a human instead of accepting any permitted account.
• The live measurement varies between runs by a few points because sampling is not perfectly reproducible at temperature 0. Raw transcripts for both models are committed so the headline numbers are verifiable rather than asserted. What did not move across runs is the part that matters: the gate caught every mistake and wrote zero wrong entries.
• Idempotency is sequential-retry safe, not concurrency safe. The client searches for the entry's content hash before creating the move, so a re-run does not double-post. Two agents writing the same entry at the same instant could still race; production would use a unique constraint on the hash in the ERP.
• The 204-case and 39-task false-write numbers are measured from natural-language tasks and structured source documents, not from OCR'd scanned images. The vision path is real and exercised (Qwen3-VL-Plus reading a PNG invoice and driving the gate), but a production claim would measure the false-write rate from scanned documents end to end.
• Rollback and a persisted audit log / review queue are not implemented. A posted move can be reversed manually in Odoo today; rejected entries are refused with the failing check as the reason, but that record is not yet persisted to a queue.