MIS 752 · Lab 7 · richardyoung
Catch the AI Lying: an audited pass/fail harness for one small free model
Ten gold clinical question/answer pairs, one small free model, three rulers (exact match, a lenient contains check, and an LLM-as-judge run twice), and the measured reliability of the ruler itself. Built in Google Colab on OpenRouter's free tier: 30 requests, $0, no PHI. These numbers are this run's and will vary.
The 30-second read
THE GOLD STANDARD
items : 10 (4 numeric, 3 code-lookup, 1 short-phrase, 1 yes-no, 1 one-word)
written before any model output was looked at -- that ordering is what makes it a
standard instead of a rationalisation
file : data/gold_qa.csv
THE MODEL UNDER TEST
model : liquid/lfm-2.5-2.6b:free at temperature 0.0
judge : nvidia/nemotron-3-ultra-550b-a55b:free, two passes at 0.7 and 1.1
requests used : 30 this session, of ~50 free per day
API cost : $0.000000 (every model here is on
OpenRouter's free tier, so the cost column is $0 by construction --
which is exactly why cost is in this table and not on a chart axis)
ACCURACY ON THE SAME 10 ITEMS, THREE RULERS
exact match : 6/10 = 60% deterministic, free, brittle
contains gold : 6/10 = 60% lenient, false PASSES
judge, pass A : 5/10 = 50% reads meaning
judge, pass B : 5/10 = 50% same questions, new dice
IS THE RULER ITSELF RELIABLE? (the part almost nobody ships)
judge vs itself : agreement 0.7, kappa 0.5238 = moderate
exact vs judge : agreement 0.7, kappa 0.4828 = moderate
mean |score gap| : 0.17 points, over the 6 of 10 items scored by both passes
unparseable : 2/10 judge replies ignored the output format
resolution : one flipped verdict moves accuracy by 10 points at n=10
DISAGREEMENT DIRECTION
judge passed what exact match failed : 1
judge failed what exact match passed : 0
STEP 5 -- YOUR WEEK 4 ANSWERS, RE-SCORED
no Lab 4 answers.csv found, so nothing was re-scored. Steps 1-4 are the complete lab; this step is the payoff if you kept the file.
Your judge contradicted itself
One dot per judge pass per gold item, joined by a connector, sorted so the biggest self-disagreement is on top. The number at the right of each row is pass B minus pass A. Row labels are prefixed EM+ / EM- for whether exact match passed that item, so three facts sit on one row without relying on colour alone. A judge that cannot repeat itself is not a measuring instrument, and this is the cheapest way to find out.
Exact match is the brittle one
One dot per gold item on the judge’s own 0–10 score, sorted ascending. Marker colour and shape both carry the exact-match verdict, so the chart still reads in greyscale. The dotted line is the score at which a judge verdict normally flips to PASS. Dots far from that line on the wrong side are items where the judge’s verdict and its own score do not agree — a second, free thing to audit.
The pass/fail table
The week’s deliverable: every gold item, what the model said, and what each ruler decided. Read the scorers_agree and judge_repeated columns first — the False rows are where the information is.
id type gold answer exact_verdict contains_verdict judge_a score_a judge_b score_b judge_repeated scorers_agree
icd10_t2dm code_lookup E11.9 E11.9 PASS PASS PASS 10.0 PASS 10.0 True True
icd10_ami code_lookup I21.9 I20.0 FAIL FAIL FAIL 0.0 FAIL 0.0 True True
snomed_t2dm code_lookup 44054006 40298007 FAIL FAIL FAIL 0.0 UNPARSEABLE NaN False True
acetaminophen_max_g numeric 4 4 PASS PASS UNPARSEABLE NaN PASS 10.0 False False
a1c_8_eag_mgdl numeric 183 183 PASS PASS PASS 10.0 PASS 10.0 True True
hypo_level1_mgdl numeric 70 70 PASS PASS PASS 10.0 PASS 10.0 True True
hipaa_safe_harbor numeric 18 20 FAIL FAIL FAIL 0.0 FAIL 0.0 True True
warfarin_reversal_agent short_phrase 4-factor prothrombin complex concentrate Prothrombin complex concentrate (PCC) FAIL FAIL PASS 10.0 PASS 9.0 True False
sjw_warfarin_yesno yes_no NO NO PASS PASS PASS 10.0 UNPARSEABLE NaN False True
atorva80_intensity one_word high High-intensity PASS PASS UNPARSEABLE NaN UNPARSEABLE NaN True False
Auditing the instrument
comparison n agreement disagreements unparseable kappa kappa reads as mean |score gap|
exact match vs judge (pass A) 10 0.7 3 2 0.4828 moderate n/a (two different rulers)
judge pass A vs judge pass B 10 0.7 3 5 0.5238 moderate 0.17 points, over the 6 of 10 items scored by both passes
judge_a PASS FAIL UNPARSEABLE
exact_verdict
FAIL 1 3 0
PASS 4 0 2
Confusion matrix: rows are the exact-match verdict, columns the judge pass A verdict. Off-diagonal cells are the disagreements; UNPARSEABLE is a column, not a dropped row, so an instrument failure can never flatter the accuracy.
Your Week 4 answers, re-scored
Step 5 found no Lab 4 answers.csv, so there is nothing to compare yet: no Lab 4 answers.csv found, so nothing was re-scored. Steps 1-4 are the complete lab; this step is the payoff if you kept the file.
How this was measured
Ten gold question/answer pairs were written before any model output was looked at and saved as data/gold_qa.csv. One model (liquid/lfm-2.5-2.6b:free) answered all ten at temperature 0.0 through OpenRouter, with a fixed system prompt that is identical for every item, because the prompt is part of the instrument.
Three rulers scored the same answers. Exact match normalizes case, Unicode look-alikes, Markdown emphasis, answer labels, surrounding quotes and trailing punctuation, then requires the whole string to equal the gold answer or one of its accepted variants. Contains-gold is a deliberately lenient diagnostic that produces false passes. The LLM judge (nvidia/nemotron-3-ultra-550b-a55b:free) reads the question, the gold answer and the candidate, and returns VERDICT / SCORE / REASON in three lines; the parser clamps the score to 0-10 and returns UNPARSEABLE rather than guessing when the format was not respected.
The judge was run twice, at temperature 0.7 and again at 1.1, and the two passes are compared with raw agreement and Cohen's kappa (chance-corrected, read on the Landis and Koch scale). Kappa bands are a convention, not a law, and at n=10 a single flipped item moves accuracy by 10 points, so treat every number here as a pilot measurement with its denominator attached.
Every model called is on OpenRouter's free tier at $0 per token, verified against the live catalog at run time; prices are fetched, never hard-coded. Answers and verdicts are cached to disk, so re-running a cell never spends quota twice. The harness counts requests against a daily budget and refuses to exceed it.
This is a ten-item teaching pilot, not a clinical validation. The gold items are self-contained vignettes with no PHI and no real patients, and nothing here is medical advice or a cleared clinical device.