Dr. Daniel Karni
All projects

Sheet 07 · Behind the scenes

Testing a Local Model for Made-Up Facts

Built Tested by me. As of 30.09.2026.

In plain words

In tests, it pulls structure out of a dictation. Before use, it is asked for facts the dictation never contained, and a model that supplies one fails.

Title block

Status
Built Tested by me.
Engine
The model runs on a laptop.
Human check
Not in clinical use: this is the test a model must pass first.
Data touched · Reviewed
Data touched
Test material.
Reviewed
30.09.2026 · Rev A

Runs on hardware Dr. Karni controls. As of 30.09.2026. Designed by me. Built with AI coding assistants.

Step 1 Extraction
A sample dictation becomes a card with tooth, procedure and follow-up.DictationSample procedureSample toothSample intervalExtractionToothSample toothProcedureSample procedureFollow-upSample interval
Step 2 The made-up-fact test
The output adds an allergy the dictation never named, so the model fails.DictationSample procedureSample toothSample intervalToothSample toothProcedureSample procedureFollow-upSample intervalAllergysample-substancenot in the dictationfails
Step 3 The limits
Extraction runs on the laptop; letters to patients and names from a photo are not given to a local model.Reads on this laptopNot given to a local modelExtractionLetters to patientsNames from a photo
Drawn, not captured. Fictional data.
Text version

Step 1: a sample dictation about a sample procedure on a sample tooth becomes a card with Tooth, Procedure and Follow-up. Step 2: the dictation names no allergy, but an output says "Allergy: sample-substance", so it fails. Step 3: what the laptop model reads (extraction) and what it is not given (letters to patients, names from a photo).

What it does

A second AI engine on a laptop, limited to reading and extracting: in the tests, pulling the tooth, procedure and follow-up interval out of a dictation.

Before a model is used on any clinical note, it is asked for details the dictation never contained, such as a blood pressure, an allergy or a previous visit. A model that supplies one fails and is not used on clinical text. This is a gate, not a clinical validation.

Two tasks did not pass: letters to patients, and names read from a photographed sample screen. Neither is given to a local model today.

How it works

  1. A model pulls the tooth, procedure and interval out of a dictation.
  2. It is asked for facts the dictation never contained.
  3. Each question is asked several times.
  4. A run fails only on evidence of a made-up fact, not on wording.
  5. One made-up fact, and the model is not used on clinical text.

What it will not do

  • Write letters to patients. That test failed.
  • Read names from a photographed screen. A test on a sample screen failed.
  • Count as a clinical validation. It is a gate.
For colleagues

Accuracy on an easy task says nothing about honesty on a hard one, so the gate asks for facts that are not there. A run fails only on positive evidence of an invented fact, not on wording.

Contact