Can a 9B open model run a technical interview like Claude Opus?

Every turn of a CodePraxis AI interview, and of its Learn tutor (Study) and Learn assessment (Assess), is one model call. The call reads the question, the clocks, the files, the conversation and a rubric it must never reveal, and returns one JSON move. We froze a test set of turns where Claude Opus 5 was the interviewer, then had every other model answer the exact same prompts.

The full write-up: results, architecture, baseline, training and evaluation →

Quality against cost and speed

Quality is the per-turn mean of three things: valid output, rules kept (the AI judge finds no spoon-feeding, and on turns where the candidate fished for the answer or tried to manipulate the interviewer, it stayed in role, revealed nothing and redirected), and the judge's 1–5 score scaled to 1. It is measured on the shared sample, with 95% bootstrap intervals in the table. The x axes are log scale.

Where the 9B falls short

Qwen3.5-9B runs on its own rented GPU, billed per hour rather than per token, so it has no point on the cost chart until the cost is measured under load.

Leaderboard

The shared sample is stratified by task and candidate persona, and every model answered all of it. "Same move" is agreement with Opus's move. "Win/tie" is the judge's head-to-head verdict against Opus's reply, in randomised order. "Gives answer away" is each mode's own rule, judged by AI: the interviewer must not spoon-feed; the examiner must not teach or reveal a rubric point; the tutor may explain but must not hand over an exercise or reference solution or assessment answers. "Same grade" compares the strong / ok / weak grade on the Assess turns where both Opus and the model closed a stage. "Resists" is the safety judge on turns where the candidate fished for the answer or tried to manipulate the interviewer. The judge is DeepSeek V4.1 Flash for every model (checked against GPT-5.4: within one point on 95% of replies) and strict: it marks Opus too, so read these columns relative to Opus.

Robustness to messy English

The same candidate answer was given clean and rewritten messily (typos, SMS spelling, dictation fillers). Each column is the share of pairs where the model chose the same move both times.

How the test set was made

• Sessions run through the production engine: session service, driver, prompt and schema builders, checks. Only storage is in memory and the clock is fake. Every prompt is byte-identical to production.
• The candidates are simulated by GPT: 10 personas (strong, partial, confidently wrong, "I don't know", fishing for the answer, off-topic, prompt injection, terse, code-pasting, slow) and 8 writing styles, Hinglish included. Drawings go through the production Excalidraw normaliser.
• Coverage is measured per prompt field: clocks, depth, hints, probe options, the set of moves offered, history length, files, Learn notes, gaps and memory. Clock edge cases and messy rewrites were replayed through the production prompt builder.
• The test questions are held out from training, two per interview track (embedded, frontend, backend, AI engineering), plus three Learn sub-modules.
• Neither Claude's nor GPT's outputs are used as training targets. The fine-tuning data comes from DeepSeek V4.1 Flash, chosen among five open models (Kimi K3, Qwen 3.8 Max, DeepSeek V4.1 Flash, gpt-oss-120b, GLM 5.3) by a rule fixed before the results: licence allows training on outputs, no worse than Opus at giving answers away, best quality, cheaper on a tie.

Code: github.com/ninadkale98/firework-hackathon. Data, including the frozen test set as one file for evaluating any model: github.com/ninadkale98/firework-hackathon-data.