States Move to License AI Doctors as the FDA Steps Back

Six months later, the pilot has become a national flashpoint. Physicians, patient safety advocates and regulators are raising questions the technology cannot answer on its own. Who is responsible when AI misses a contraindication? When should the software refuse a request? And where does a state-regulated clinical tool end and a federally regulated medical device begin?

The safety concerns are specific. Dr. Alan Smith, a family physician who chairs Utah’s medical board, told the Associated Press that Doctronic’s list of 190 refillable medications includes blood thinners, which can become dangerous if patients develop conditions that cause internal bleeding.

“Many times when I see people after six months I find that their medical history or situation has changed,” Smith said. “Just because something was prescribed before does not mean it’s appropriate now.”

The American Medical Association has echoed the concern, warning that “prescription renewals aren’t routine checkboxes.” Within weeks of its launch, security researchers were able to push the Doctronic system to triple a patient’s opioid dosage and generate vaccine misinformation, The Verge reported in April.

AI’s Clinical Role Collides With a Century of Licensing Law

Doctronic executives would not say whether they have sought permission from the U.S. Food and Drug Administration (FDA) for the system, the AP reported. “Our goal here is really just to meet patients where they need healthcare,” Co-Founder Dr. Adam Oskowitz told the AP. “We try not to get too deep into the weeds on the regulatory side.”

The FDA, which typically sets national guidelines for medical technology, plans to take a hands-off approach to regulating AI prescription tools for now, according to the AP. An FDA spokesperson said the agency has not authorized any AI chatbots but is “committed to encouraging medical innovation and helping bring promising new technologies to patients, while keeping safety at the center of every decision.”

Meanwhile, lawmakers in Iowa and Idaho have advanced bills, including Idaho House Bill 945 and Iowa HSB 766, to create licensing systems for autonomous AI healthcare providers. These efforts require audits and aim to regulate systems authorized to diagnose or prescribe without human supervision.

Dr. Eric Bressman of the University of Pennsylvania framed the structural concern: “We have crossed a threshold in terms of giving something that is not human a medical license,”whether or not we want to call it that,” he told the AP.

A Second Utah Pilot Narrows the Scope. The Questions Remain.

Utah approved a second AI prescription pilot in April, this time through Y Combinator-backed Legion Health, focused on psychiatric medication renewals in a state where up to 500,000 residents lack access to adequate behavioral healthcare, according to the Utah Office of AI Policy, PYMNTS reported. Legion’s guardrails are tighter than Doctronic’s. The AI cannot issue new prescriptions, adjust doses or handle controlled substances. Patients must be stable, on an existing treatment plan with a licensed psychiatrist and must not have had a psychiatric hospitalization in the past year.

The demand side is not in question. More than 40 million people worldwide use ChatGPT daily for health-related queries, with about 70% of those interactions happening outside clinic hours, PYMNTS reported.

Daniel Aaron, a law professor at the University of Utah, captured the risk of moving faster than the evidence. “Companies may benefit in the short term by expanding their business models and kind of having the technology go beyond the evidence,” Aaron told the AP. “But in the long term, I think they risk compromising public trust and fueling backlash.”

For all PYMNTS AI and digital transformation coverage, subscribe to the daily AI and Digital Transformation Newsletters.