Viva: 91 Full-Mark C Submissions Out of 626 Hide a Defect the Autograder Never Checked

A passing test suite proves a program worked on the inputs the instructor happened to check. It does not prove the student understands what the program does. Viva closes that gap. GPT-5.6 reads the assignment and proposes an input the test suite never tried, Viva runs the student program and the instructor's reference on it, and it only asks the student a question when the two programs disagree. The teacher gets a reteach brief, never a cheating accusation.

The concrete example

The assignment is print the maximum of three integers. The instructor's three tests are `1 2 3`, `6 2 1`, `-1 3 1`. Not one of them ties, so nothing ever catches the tie. Give the student's program the input `5 5 3` and it prints `3`: both strict `>` branches are false when a and b are equal, so the else prints the third value. The student passed every test and got full marks; they never learned that ties break the pattern.

Why a chat session cannot do this

Paste the code into a chatbot and it will guess whether the code is wrong, and sometimes hallucinate a bug that is not there. Viva never guesses. GPT-5.6 decides only what to test; execution decides whether this submission fails it. A finding is two programs printing different things on the same input, with both outputs shown. That split is the whole design.

• GPT-5.6 (discovery, Responses API, effort high): reads the assignment spec and the reference solution and proposes edge-case inputs a beginner would plausibly get wrong (ties, zero, negatives, single elements, boundaries). This is the part that generalizes to any exercise and that a hardcoded checker cannot do.
• Execution (verification): compiles the student program and the reference in a WSL C runner, runs both on each proposed input, and keeps only divergences that reproduce. A cosmetic float-formatting difference (5.0 vs 5.000000) is detected and excluded; only answer-changing divergences count.
• GPT-5.6 (defense, effort medium): grades the student's typed explanation against that one executed fact, and pushes back if they restate the output instead of tracing the code. Never mentions cheating.

The teacher gets a reteach brief, not a cheating accusation

The teacher view organizes the 91 findings by misconception: ties, off-by-one on boundaries, negative-input handling, and a few others. Each misconception carries the count of students affected and one canonical probe. This is a reteach signal (the class systematically missed the tie case), not a per-student accusation.

The student defense flow

The student is asked exactly one question about their own program: on the probe that showed the divergence, what does the code do and why does it print that? GPT-5.6 grades the answer against that one executed fact, constrained to the probe's evidence id. If the student restates the output, the grader follows up asking them to trace the branch that fired.

The grader's response is validated for the expected evidence id before it reaches the interface. If the model tries to cite a probe it was not shown, or invents a new input, the response is dropped and regenerated. It cannot conjure a new fact about the student's code, only reason over the single executed one, so the follow-up is always grounded in the same divergence the teacher saw.

Built in Codex, end to end

Viva was built in Codex using GPT-5.6 (gpt-5.6-sol, gpt-5.6 aliases to it). GPT-5.6 is the engine in two places: the discovery step (src/prober/generator.ts, Responses API, reasoning effort high) and the defense grader (src/viva/grader.ts, effort medium), which is constrained to the one executed probe and validated for the expected evidence id before its response reaches the interface. Codex accelerated the WSL C runner, the batched execution harness, the evidence schema, the divergence categorizer, the benchmark, and the deployed teacher and student flows.

Honest limitations

• The offline benchmark covers three integer exercises where every proposed input is in-domain and every concept label is correct. It is not a defect rate for the whole corpus. GPT-5.6 discovery extends coverage to the exercises the curated fallback deliberately skips (divisor, average, digit-sum, time-conversion, string exercises).
• The teacher brief is a product demo generated from public, anonymized corpus results. Production needs LMS integration, accessibility review, consent, and a prospective study with educators and students.
• Viva reports demonstrated understanding. It is not a cheating detector and never makes an authorship claim.