AI in health care
The hard part of AI in health care comes after the model works
Michelle Mello, a Stanford professor of law and health policy, used to think patients should be told about every AI tool in their care. Then she asked them, and they wanted to know who was keeping them safe. Most AI now in hospitals handles paperwork. The harder job is deciding what to switch on and checking it afterward, and many hospitals cannot afford to do that well.
- Hospitals have adopted AI first for paperwork. In Mello’s account, scribes and billing tools spread first, mainly in well-funded systems. Tools that help with diagnosis are much less common.
- Patients asked for safety more than for forms. Mello’s patient panel wanted to know what the hospital would do to protect them, and she changed her mind about disclosure as a result.
- A human in the loop can be fooled too. In a randomized study, a biased AI made clinicians 11.3 percentage points less accurate, and explaining the AI did not clearly help.
- Many hospitals do not test their tools. Of US hospitals using predictive models in 2023, 61% checked accuracy on their own data and 44% checked for bias.
- Agents raise the stakes. When software acts by itself, a mistake can reach a patient before anyone can stop it. Mello wants AI governance to be a condition of taking part in federal health programs.
Michelle Mello helps review AI tools for Stanford Health Care. As an ethicist, she said in a September 2026 interview, she started out believing that “patients should be told about everything all the time.” Then, over the last couple of years, she talked with many patients, including a panel trained in AI that meets her team every month. In a hospital like Stanford, she points out, “a couple dozen different tools” might touch your care.
Twenty-four tools, twenty-four forms. Patients told Mello this felt like a waiver of responsibility.
Patients say things like this, in Mello’s retelling. “I’m sick, I have piles of paperwork pushed at me. The last thing that is helpful in this situation is another piece of paper asking me to sign on the dotted line. It feels like a waiver of responsibility.” The question they cared about was different. “What is the organization going to do to keep me safe?” Mello now saves formal consent for the highest-stakes choices, such as whether to have surgery guided by an AI robot.
That answer moves the work from the patient to the hospital, and it takes the conversation beyond how good the model is. The hard part of AI in health care has moved from building the model to governing it, and the hospitals that need that work most can least afford it.
In a hospital, AI governance is the process for deciding which tools to use, checking them before they go live and watching them afterward. It includes people who can say no. A model’s test score is one input to that process.
The AI in hospitals is mostly doing paperwork
Headlines are full of AI doctors. Before you read Mello’s description of what hospitals are actually using, try sorting these.
An AI scribe that listens to a clinic visit and writes it up.
In wide use. Mello names scribes first, among well-funded organizations.
A tool that handles insurance approvals and prepares bills.
In wide use. The other big category, and one that can pay for itself.
A risk predictor that warns which patients may get worse.
Less common as a tool people chose. Mello says these are not as widely implemented, though many hospitals run some model that came with their records system.
A tool that helps a doctor decide on a diagnosis.
Less common. Mello says these “might really improve quality of care,” and they are not as widely implemented.
A software agent that orders treatment by itself.
Not yet. Mostly on billboards for now, and Mello describes “a heightened concern” about checking these tools.
“I think we’re very much in the early days,” Mello says. The tools spreading fastest are scribes, which listen to a clinic visit and summarize it for the doctor, and administrative tools that handle insurance approvals and bills. The results are mixed even there. In one study, about half of patients whose doctor used a scribe reported more eye contact. “Why is it only half?” she asks. The tool takes notes so the doctor can stop typing.
The picture depends on what you count. A 2023 survey analysed by Paige Nong and her colleagues found that 65% of US hospitals used some predictive model, a program that estimates something like a patient’s risk. Of those hospitals, 79% got models from the company that sold them their electronic health records. So models are already common in hospitals.
Stanford put a lock on the switch
Stanford Health Care expected a flood of AI proposals and built a process early. A review board that people can ignore has little power, so the hospital tied the review to the moment a tool goes live. Mello calls this “a great lever.” Anyone with a patient-facing or otherwise risky tool has to go through the process, because “we will not let our digital services unit turn it on until you can certify that you’ve gone through this process.”
You want to switch on a new patient-facing AI tool. Try the switch first.
Four questions answered, so the digital services team turns it on. Then the measuring starts.
The aim is to “proactively spot problems before the organization falls in love with the technology,” Mello says. The review also forces a decision about what to measure once the tool is running, before anyone has a stake in the answer.
Many hospitals never test the tool on their own patients
A model can do well in the company’s tests and badly at your hospital, because your patients and your record-keeping differ. A widely used model for sepsis, a life-threatening reaction to infection, missed 67% of cases at Michigan Medicine. The way to find out is local evaluation, which means testing the tool on your own data.
Of the hospitals using models, 61% tested accuracy on their own data. That is about 40 of every 100 hospitals.
Hospitals that built their own models and hospitals with healthy finances were more likely to test locally. Mello calls the work “expensive and tricky.” AI sellers argue that their tools pay for themselves, and she says that is probably true for tools that handle billing and false for many others. Asked about rural hospitals facing cuts to Medicaid, the US public insurance program for people with low incomes, she says the best answer so far is sharing. Large systems publish what they learn, so “not everybody has to start at zero.”
A human in the loop can be fooled too
The usual safety answer is to keep a person in charge. Patients on Mello’s panel asked for this, with a condition. They wanted a clinician reviewing the AI who had been given the time and information to do it. Sarah Jabbour and her colleagues tested what happens when that person sees a bad model. They randomly assigned 457 clinicians to read patient cases with AI advice, and some of that advice came from a model built with a systematic bias.
A hospital adds an AI tool with a hidden bias, and shows clinicians an explanation of its reasoning. What happens to their accuracy?
The biased AI cost clinicians 11.3 points, and the explanation won back a difference small enough to be chance.
A good model helped a little. A biased one hurt a lot, and showing an explanation of the model’s reasoning did not clearly undo the harm. These were written cases and an image-based kind of explanation, so the numbers will differ on a real ward. The direction is a warning about any plan that relies on a person catching errors. Oversight also takes time.
A nurse has a spare hour in a shift to check AI alerts properly. How far does that hour go?
That needs 90 minutes, and the shift has 60. Something has to give, and usually it is the checking.
Agents change what can go wrong
Everything above assumes a person sees the output before it reaches the patient. Mello draws the line there. Human-mediated tools make a suggestion, and someone accepts or rejects it. Agentic tools act by themselves, and they “might get things wrong in ways that actually can physically hurt people very quickly and in a way that’s very hard to intercept.” She calls health care “a little bit distinct” from fields like human resources or investing, because the injury is physical and happens quickly.
Press to send a wrong order, then try to stop it.
This changes what evaluation has to measure. Checking whether the answer is correct stops being enough once software takes actions. Reviewers also need to know what the software is allowed to do. They need to know when it must stop and ask, and whether anyone can step in before harm is done.
The person who signs is the person who pays
Clinicians ask Mello one question more than any other. Am I liable if AI is involved in a mistake that hurts a patient? “That answer is not straightforward,” she says, “but usually the answer is probably, yeah, you are.”
Licensing agreements typically “immunize them from liability and disclaim ordinary consumer warranties.” The risk lands on the people using the tool.
The reason is in the contracts. Developers “typically include provisions in their licensing agreements that immunize them from liability and disclaim ordinary consumer warranties.” In plain terms, the contract refuses the usual product guarantees. That leaves the physician or nurse, and perhaps the hospital. So a person with little insight into the tool may carry the legal risk for it, which helps explain why many clinicians are wary. She adds that clinicians are often not given much information about how to be good overseers.
A gold star is optional
Standards exist, and hospitals can ignore them. NIST, a federal standards agency, runs a voluntary program where AI developers can earn what Mello calls “a gold star.” The Joint Commission, which inspects hospitals for Medicare, the US public insurance program for older people, offers a voluntary certificate for AI governance, built with the Coalition for Health AI. “Anything that is voluntary and costly will have uneven uptake,” she says.
Under voluntary rules, governance goes to the hospitals that can pay for it.
Asked what she would tell Congress, Mello had one answer. Require hospitals “to have a governance process for AI as a condition of participating in federal health programs.” Hospitals already have to meet many such rules to take part in Medicare. “We cannot justify an ecosystem in which hospitals can operate AI tools with very little vetting and no oversight and have no regulatory consequences.”
Much of this essay comes from one interview with one expert, who helps run Stanford’s process. A required process can also turn into paperwork, and it would add costs for the hospitals with the least money. The Nong survey shows who reports testing, and it cannot say how well they test.
Mello says you cannot drive for more than 30 seconds on Highway 101 without seeing a billboard promising that agentic AI will change the world. At Stanford Health Care, a patient-facing tool still stays off until someone has certified the review.
Further reading
- Michelle Mello on Health Policy Forum: In Conversation (Stanford Health Policy, September 2026), with the full transcript and the video.
- Nong et al., Current use and evaluation of artificial intelligence and predictive models in US hospitals (Health Affairs, 2025).
- Jabbour et al., Measuring the impact of AI in the diagnosis of hospitalized patients (JAMA, 2023).
- Mello and Guha, Understanding liability risk from using health care artificial intelligence tools (New England Journal of Medicine, 2024).