Auditing AI I cannot read

Jorge Peres Filho, MD · Passos, Minas Gerais, Brazil · November 2026

I am a physician, not a programmer. In 2026 I built a private study platform for Brazilian residency exams (PSU-MG and Santa Casa BH) by directing Claude Code. Claude wrote the code. I wrote the medical requirements, decided what had to be tested, and reviewed the output. Because I cannot read the code, the whole system is built around one rule: the AI never grades its own work.

~700video corrections of exam questions

535flashcards

2 layersgenerator model + independent auditor

Open the demo platform

23 PSU-MG questions (2023 to 2026, five areas): question, official answer, the correction written by Claude, and the video correction. How the audit pipeline works.

Content is in Portuguese (Brazilian exam). The full platform is private because it contains licensed third-party material; a live demo is available on request.

How the pipeline works

A model writes the explanation of each exam question and a script for a short narrated video.
A separate auditor model reviews it against the official answer key and clinical sources. Disagreements are flagged, not silently resolved.
Answer-key disputes follow a fixed rule: keep the official key unless there is clear external evidence (guideline, manual, official notice).
Every approved script gets a fingerprint (hash). If anything changes it later, the change is detected and the approved version is restored.
Every automated test is paired with a negative control: a deliberately broken input that must make the test fail. A test that cannot fail is not a test.
When an error is found, the same pattern is searched and fixed everywhere, and a check is added so it cannot come back.

Real errors the system caught

9 October 2026 · clinical audit of 11 AI-written mini-lessons

The audit found confident factual errors in fluent text, for example: a script said acute hemorrhage does not change HbA1c (acute blood loss lowers it); a script on adolescent confidentiality omitted the age threshold below which sexual activity is presumed abuse and must be reported; a script stated a specific year for a guideline change that no source supported. All were corrected, and uncertain points were escalated to me instead of being decided by the model.

12 October 2026 · silent edits by reviewer models

Automated "clinical reviewer" models rewrote three video scripts that I had already approved by listening to the audio. The fingerprint check caught it; the approved versions were restored, and checking fingerprints after every batch became a fixed rule.

About August 2026 · fixation on a wrong reference

While I was studying a question about a patient with terminal cancer, a model latched onto a reference it had found and kept defending it, missing what mattered clinically. I only became sure it was wrong after checking the guideline myself. This is the failure mode that worries me most for clinical use.

What I want to study

Before AI decision support reaches primary care in small towns, I want to measure how it fails: when it is wrong with conviction, when it fixates on a wrong reference, and whether an independent blind auditor really catches the error or just agrees with the first model. Residency exams, with thousands of questions, official keys, appeals and annulled items, are a good laboratory to measure this instead of guessing.

Public example of the same approach: an admission-chance simulator (Monte Carlo, 84 automated tests with negative controls).

The full study platform is private because it contains licensed third-party material. The public demo linked above shows only a small sample: public PSU-MG exam questions with the correction written by Claude, nothing else. Content is in Portuguese (Brazilian exam). A live demo of the full platform is available on request. Code written by Claude Code under my direction; text first drafted by me and refined with Claude.