A non-commercial research showcase
Written for radiologists and clinicians. We ran small chest X-ray experiments to show how imaging AI behaves, including how it can quietly get worse. Nothing here is a clinical tool.
Mean AUC across 14 chest X-ray findings. 0.50 is chance; 1.00 is perfect.
Dashed line: chance (0.50). Noise was added to 5,000 test images at four strengths. Values are approximate and rounded.
What AI can do with medical images today
Which of these exist as cleared products, and where, varies by tool and jurisdiction. This is a plain-language map, not a product guide.
01
Software marks scans that may show a finding, such as a nodule or a pneumothorax, for a radiologist to review.
02
Scans with a likely urgent finding move up the reading queue so they are seen sooner.
03
Heatmaps highlight the image regions that most influenced a model’s output. They prompt a closer look; they do not prove the model looked at the right thing.
04
After go-live, a model can be watched for changes in the images it receives or the results it gives, for example after a scanner or protocol change.
What we tested
We trained a small research model on a public chest X-ray dataset (14 findings, 28×28-pixel images). That is far lower resolution than clinical images, chosen so the work runs on a laptop. Training runs varied, so figures are approximate.
On clean images the model’s mean AUC was about 0.71. It varied by finding: roughly 0.58 for nodules, 0.65 for infiltration and 0.84 for edema. Small, low-resolution models are weakest on small findings.
To mimic differences between scanners or sites, we added noise, blur or lower contrast to test images. Heavy noise took mean AUC to about 0.54, close to chance. Blur and lower contrast each cost about 0.04.
No. The model’s own uncertainty did not rise on the degraded images. Used as an alarm, it scored 0.42, where 0.50 is a coin flip. A model can get worse while looking just as sure of itself.
How to read AI performance
Four things worth asking about whenever a vendor quotes an AUC.
What didn’t work
We report these because negative results are useful, and because clinicians should hear about them, not only the successes.
We tested whether flagging unusual-looking scans could catch cases a standard model misses. On its own it was worse than the standard model on all 14 findings. Blended in, it added about 0.003 AUC, chosen on a separate validation set. That is too small to matter clinically.
At light noise, a basic measure of fine image detail told clean from noisy scans almost perfectly. Methods built on the model’s learned features reached only 0.59 to 0.63. For spotting degraded images, a simple check may be the right answer.
How well each method separates clean from noisy images. 0.50 is chance; 1.00 is perfect.
Darker bars: simple image checks. Lighter bars: methods using the model’s learned features. Values rounded.
Limits
We would rather say this up front than have you find it later.
Tell us what you see
We are not selling anything. If you read or run imaging AI, we would value 20 minutes of your view on two questions: where do these tools surprise you, and what would you want checked after go-live?
About this project: built by Chandru, an AI/ML engineer (Master’s in AI/ML) with a background in banking and in fraud and anomaly detection. Not a clinician, which is why your view matters.