Baseline Imaging

A non-commercial research showcase

What AI can do with medical images, and where it falls short.

Written for radiologists and clinicians. We ran small chest X-ray experiments to show how imaging AI behaves, including how it can quietly get worse. Nothing here is a clinical tool.

See what we tested Give feedback
A small model gets worse as images get noisier

Mean AUC across 14 chest X-ray findings. 0.50 is chance; 1.00 is perfect.

0.71 0.69 0.62 0.55 0.54 Clean Faint Light Moderate Heavy

Dashed line: chance (0.50). Noise was added to 5,000 test images at four strengths. Values are approximate and rounded.

What AI can do with medical images today

Four things imaging AI is being used for

Which of these exist as cleared products, and where, varies by tool and jurisdiction. This is a plain-language map, not a product guide.

01

Flag possible findings

Software marks scans that may show a finding, such as a nodule or a pneumothorax, for a radiologist to review.

02

Prioritise the worklist

Scans with a likely urgent finding move up the reading queue so they are seen sooner.

03

Show where it looked

Heatmaps highlight the image regions that most influenced a model’s output. They prompt a closer look; they do not prove the model looked at the right thing.

04

Check quality over time

After go-live, a model can be watched for changes in the images it receives or the results it gives, for example after a scanner or protocol change.

What we tested

A small model, three questions

We trained a small research model on a public chest X-ray dataset (14 findings, 28×28-pixel images). That is far lower resolution than clinical images, chosen so the work runs on a laptop. Training runs varied, so figures are approximate.

AUC 0.71

How often did it get it right?

On clean images the model’s mean AUC was about 0.71. It varied by finding: roughly 0.58 for nodules, 0.65 for infiltration and 0.84 for edema. Small, low-resolution models are weakest on small findings.

0.71 → 0.54

What happened when images got worse?

To mimic differences between scanners or sites, we added noise, blur or lower contrast to test images. Heavy noise took mean AUC to about 0.54, close to chance. Blur and lower contrast each cost about 0.04.

0.42

Did the model warn us?

No. The model’s own uncertainty did not rise on the degraded images. Used as an alarm, it scored 0.42, where 0.50 is a coin flip. A model can get worse while looking just as sure of itself.

How to read AI performance

One number hides a lot

Four things worth asking about whenever a vendor quotes an AUC.

  • AUC is a ranking score. It describes how often the model scores an abnormal scan above a normal one. It does not give the sensitivity or specificity at the threshold a department would actually use.
  • An average hides variation. Our mean of 0.71 covered findings from about 0.58 to 0.84. Ask for results per finding.
  • Prevalence changes what an alert means. For a rare finding, even a good score can produce many false alarms for each true one.
  • Another site is another test. Scanners, protocols and patient mix differ. Ask for results on images like yours, and how performance is checked after go-live.

What didn’t work

Two ideas that fell short

We report these because negative results are useful, and because clinicians should hear about them, not only the successes.

+0.003

Treating rare findings as “unusual” images

We tested whether flagging unusual-looking scans could catch cases a standard model misses. On its own it was worse than the standard model on all 14 findings. Blended in, it added about 0.003 AUC, chosen on a separate validation set. That is too small to matter clinically.

1.00 vs 0.63

Sophisticated checks lost to a simple one

At light noise, a basic measure of fine image detail told clean from noisy scans almost perfectly. Methods built on the model’s learned features reached only 0.59 to 0.63. For spotting degraded images, a simple check may be the right answer.

Spotting noisy scans (light noise)

How well each method separates clean from noisy images. 0.50 is chance; 1.00 is perfect.

Fine-detail energy Embedding distance Outlier detector Average brightness Brightness spread simple image check model-feature based model-feature based simple image check simple image check 1.00 0.63 0.59 0.50 0.49 chance

Darker bars: simple image checks. Lighter bars: methods using the model’s learned features. Values rounded.

Limits

What this is not

We would rather say this up front than have you find it later.

  • Not a medical device. Nothing here has regulatory clearance, and nothing here should inform a clinical decision.
  • Low-resolution public images. 28×28 pixels, one dataset, one small model. Real clinical images carry far more detail.
  • Imperfect labels. The dataset’s labels were extracted from radiology reports by software and were not all verified by radiologists.
  • Synthetic problems only. We degraded the images ourselves. Real differences between scanners and sites are subtler and may behave differently.
  • Results moved between runs. Training the same model again gave somewhat different numbers, so treat every figure as approximate.

Tell us what you see

Does this match what you see in practice?

We are not selling anything. If you read or run imaging AI, we would value 20 minutes of your view on two questions: where do these tools surprise you, and what would you want checked after go-live?

About this project: built by Chandru, an AI/ML engineer (Master’s in AI/ML) with a background in banking and in fraud and anomaly detection. Not a clinician, which is why your view matters.

Contact

chandrasekaran@baselineimaging.org Connect on LinkedIn