3 of 1,357: AI devices tested on outcomes

Three. Out of 1,357 AI medical devices the FDA had cleared through December 5, 2025, exactly three had ever been tested against a real patient outcome, meaning something like survival, a hospital readmission, or a validated symptom score, rather than a technical accuracy number (Abulibdeh et al., PLOS Digital Health, published August 19, 2026). If you have been putting off an AI purchase because you could not tell which vendor claim to trust, that figure says your instinct was right. It also is not the part that should worry you most, because the tool you are actually evaluating this month probably was not on the FDA's list at all.

What 1,357 devices and three outcomes actually means

A team of researchers reviewed every AI or machine learning device the FDA had cleared as of December 5, 2025: 1,357 in total. Of those, 34 (2.5%) had a registered prospective clinical trial on file. Twelve (0.9%) had posted results from that trial. Twelve (0.9%) made it into a peer-reviewed publication. Only three (0.2%) were evaluated against a patient-centered outcome at all, and the study's authors defined that term narrowly and specifically: mortality, major morbidity such as a stroke or heart attack, hospitalization, readmission, a validated quality-of-life score, functional status, or symptom burden. A device that measures how accurately it reads a scan, without asking whether reading the scan that way changed what happened to the patient, does not count.

Most of what got cleared is radiology software, 1,059 devices, 78% of the total, reading scans for things like fractures, nodules, or bleeds. Even inside that dominant category, fewer than 1% of devices had a prospective trial behind them. Anesthesiology tells the same story at smaller scale: 22 cleared devices, zero with a registered trial. FDA clearance tells you a device performs a task the way its maker described. It does not tell you the device made a single patient better off, and for 99.8% of the devices on the list, nobody has checked.

So what for you: if your evaluation of an AI tool stops at "the FDA cleared it," you have confirmed the tool does what it says on the label, not that using it improves care, saves time, or is worth the monthly invoice.

The tool you are about to buy is probably not even on that list

Here is the distinction most vendor pitches skip past. The 1,357 devices in the PLOS Digital Health study are things the FDA classifies as medical devices: software that makes or supports a diagnostic or treatment decision. An ambient AI scribe that types up your notes, a scheduling tool that predicts no-shows, or a billing assistant that codes a claim does not typically meet that definition. As one clinical AI legal review put it plainly, "most scribes aren't regulated by the US Food and Drug Administration (FDA) yet, as they don't offer medical advice" (Heidi Health, "AI Medical Scribe: Legal Implications and Regulatory Considerations," June 2026). The FDA's own Clinical Decision Support software guidance, last updated January 29, 2026, draws the regulatory line at whether a tool processes data and outputs a recommendation the clinician cannot independently check against the source information. Documentation-only tools generally fall on the unregulated side of that line, and for nearly every independent physical therapy, dental, psychology, OT, SLT, or optometry practice, documentation, scheduling, and billing tools are the AI actually being evaluated this year, not diagnostic imaging software.

That means the scribe, scheduler, or billing assistant most likely to land in your inbox as a demo request this month sits outside the FDA count entirely, not because it cleared a lighter version of the outcome bar the 1,357 devices faced, but because no bar applies to it at all. Nobody at the FDA has looked at whether it works. We have separately covered the accuracy-testing gap in AI scribes, which is a related but different question: whether the transcription itself is correct. This is the layer above that: whether using the tool changes anything for the patient or the practice at all.

So what for you: when a scribe or scheduling vendor uses the word "validated," ask validated against what, by whom, and where the result was published. FDA clearance is not the answer, because in most cases there is not one to have.

What a genuine outcome claim actually looks like

A real outcome claim names four things: the specific outcome measured (not "accuracy" or "satisfaction," but something concrete like time saved per note, error rate against a human-reviewed chart, or a documented change in no-show rate), the sample size, whether the study was run independently or funded by the vendor, and where it was published or registered. Phrases like "clinically proven," "AI-powered accuracy of 99%," or "trusted by thousands of clinicians" answer none of the four. They describe confidence, not evidence. If a sales call cannot produce those four details on request, the claim is marketing rather than evidence, and should be weighed accordingly.

That is not an argument for walking away from every tool without a trial behind it. Plenty of useful software has never been through formal research, largely because documentation and scheduling tools were never expected to need it. It is an argument for putting the burden of proof on the vendor rather than borrowing credibility from an FDA seal that, for most practice AI tools, was never applied for and would not apply if it were. Section 11's own house rule for this site makes the same point from the other direction: never repeat a vendor's "clinically validated" claim without naming the specific study behind it. That rule exists because most of the time, when this site has gone looking for the study, there was not one to find, which is what the outcome-monitoring piece we published last week found when we checked vendor outcome claims for OT and SLT practices specifically.

So what for you: build one question into every AI vendor call, before pricing comes up. "What outcome did you measure, and who checked your numbers." A vendor who answers cleanly has done real work. A vendor who reroutes to a testimonial has told you what you need to know.

The cost of skipping the question

The financial case for asking is not abstract. Standalone AI scribe pricing for a solo or small physical therapy practice runs from around $15 a month per clinician when bundled into an EMR like Jane App, up to $150 to $180 a month per user for a standalone tool like Heidi Health's higher tiers (pricing previously checked on this site, July 2026). Annualized across a small team, that is anywhere from a few hundred dollars to several thousand committed to a tool nobody outside the vendor's own team has measured against a real outcome.

If the tool genuinely reclaims the roughly 12 hours a week independent practices report losing to admin, that spend earns itself back many times over. If it does not, because the error rate runs higher than advertised or the workflow fit is poor, you have paid full price for an unmeasured guess, and you likely will not find out for months, by which time contract terms, staff retraining, and switching costs have added to the bill. That is the pattern worth guarding against most: not a tool that fails outright, which gets noticed and cancelled quickly, but one that underperforms quietly enough to keep renewing on inertia.

So what for you: treat the first 90 days of any new AI tool as its own trial. Track the one outcome that matters most to your practice, whether that is notes closed same day, no-show rate, or hours actually reclaimed, before you renew a second year on faith.

The distinction worth carrying into your next vendor call

AI still belongs in an independent practice's admin stack. The case here is for buying it the way the rest of healthcare is slowly being forced to buy everything else: on evidence, not on the word "validated" printed in a slide deck. The PLOS Digital Health finding is a useful gut check precisely because it shows how low the bar sits even where a regulator is genuinely involved, at 0.2%. Where no regulator is involved, which covers most of what an independent practice is actually buying this year, the bar is whatever question you decide to ask before you sign.

An AI Opportunity and Growth Assessment starts with exactly that question for your practice: what would actually need to be true for a given tool to earn its subscription, based on evidence rather than a pitch deck. You can also book a free 20-minute call to talk through what that would look like for your practice.

Related: Vendor outcome claims for OT and SLT AI tools  ·  The accuracy-testing gap in AI scribes  ·  5,944 denials: Medicare's AI pilot exposed

The core finding (1,357 FDA-cleared AI/ML devices reviewed through December 5, 2025, with 34 registered prospective trials, 12 posted results, 12 peer-reviewed publications, and 3 evaluated against a patient-centered outcome, plus the radiology 78%/anesthesiology figures) is from Abulibdeh, Cajas Ordonez, Celi, Gorijavolu, Izath, and Markussen Lunde, "1,357 AI medical devices cleared, 3 actually tested on patient outcomes," PLOS Digital Health, Volume 5, Issue 8, published August 19, 2026, DOI 10.1371/journal.pdig.0001597. The scribe regulatory characterization is from Heidi Health, "AI Medical Scribe: Legal Implications and Regulatory Considerations," published June 26, 2026, and the FDA's Clinical Decision Support software guidance, last updated January 29, 2026. AI scribe pricing figures ($15 to $180 a month per clinician) were previously verified on this site in July 2026 from vendor pricing pages. This article is for informational purposes only and does not constitute clinical, legal, or financial advice.