Opinion · Evidence
Health software should be tested like a treatment
By Element 45 · 21 August 2026

Apps and AI tools now shape decisions about care. Most reach patients with far less evidence than a medicine would need.
A new medicine reaches patients after years of trials. A new piece of health software can reach them in a product update. Some of that difference is reasonable, because an appointment reminder is not a drug. But a growing share of health software does things that look a lot like treatment: it triages symptoms, suggests diagnoses, coaches behaviour, and decides which patients a clinician sees first.
We think software that changes what happens to a patient should be held to a standard closer to a treatment than to an app. Most of it is not.
Engagement is not an outcome
The evidence offered for digital health products is usually about use: downloads, active users, minutes in the app, satisfaction scores. These tell you people opened it. They do not tell you anyone got better. A product can be popular and useless, and a product people find tedious can still lower their blood pressure.
The outcome that matters is the one the patient cares about. Fewer admissions. Better control of a condition. A shorter wait for the right clinician. If a product cannot name the outcome it is meant to change, that is the first thing a buyer should notice.
If a product claims to improve health, the claim deserves the same question we ask of any treatment: compared with what, and how do you know?
The gaps in the rules
In Australia the Therapeutic Goods Administration regulates software that meets the definition of a medical device. A great deal of health software sits outside that definition, or inside one of its exclusions, and reaches the market with no independent look at whether it works. Being legal to sell is a low bar. It is not evidence.

Software changes, so keep testing
The usual objection is that software changes too quickly for trials. A model updated every month is not the product that was studied last year. That is true, and it is an argument for continuous evaluation, not for none. A service can compare outcomes for people who used a feature against those who did not, release changes to some sites before others, and publish what it finds. Hospitals audit their surgical results every year without stopping surgery.
What to ask a vendor
What outcome does this change, and by how much? Compared with what? In which patients, and were they like ours? Who ran the study, and where can we read it? What happened in the places where it did not work?
A vendor with good answers will be glad to be asked. One without them will talk about engagement.
This is how we work ourselves. A new service is run small and measured before we build a company around it, and if the measure does not move, we stop.