AI chatbots designed for medical purposes may be subject to medical device regulations. Regulatory bodies such as the United States Food and Drug Administration (FDA) or the European Medicines Agency (EMA) in Europe may be required to evaluate and approve these technologies based on their intended use and potential risk to patients.
Although LLM-based generative chat tools like ChatGPT and Google’s MedPaLM have tremendous medical potential, there are inherent risks associated with their unregulated use in healthcare. A new article addresses one of the most pressing international issues of our time: how to regulate Large Language Models (LLMs) in general and in health specifically.
“Large Language Models are neural network language models that have exceptional conversational abilities. They respond in human-like ways and engage in interactive conversations. However, they frequently produce highly convincing statements that are demonstrably incorrect or provide inappropriate responses. There is no way to know the quality, level of evidence, or consistency of clinical information or supporting evidence for any response today. “These chatbots are unsafe tools for medical advice and new frameworks that ensure patient safety are required,” said Prof. Stephen Gilbert, Professor for Medical Device Regulatory Science at the Else Kröner Fresenius Center for Digital Health at TU Dresden.
Current LLM-chatbots do not meet key AI in healthcare principles such as bias control, explainability, systems of oversight, validation, and transparency. Chatbots must be designed for greater accuracy, with safety and clinical efficacy demonstrated and approved by regulators, in order to earn their place in the medical arsenal.
Prof. Gilbert
Challenges in the regulatory approval of large language models
Most people conduct online research on their symptoms before seeking medical advice. Search engines play an important role in the decision-making process. The impending integration of LLM-chatbots into search engines may boost users’ trust in the answers provided by a chatbot that mimics conversation. When prompted with medical questions, LLMs have been shown to provide extremely dangerous information.
The underlying approach of LLM lacks a model of medical “ground truth,” which is inherently risky. Chat-interfaced LLMs have already delivered harmful medical responses and been used unethically in ‘experiments’ on patients without their consent. Almost every medical LLM use case necessitates regulatory oversight in the EU and the US. In the United States, their inability to be explained disqualifies them from being classified as “non-devices.” LLMs with explainability, low bias, predictability, correctness, and verifiable outputs do not exist at the moment, and they are not immune to current (or future) governance approaches.
The authors of this paper describe the limited scenarios in which LLMs could find application under current frameworks, how developers can seek to create LLM-based tools that could be approved as medical devices, and how they can explore the development of new frameworks that protect patient safety. “Current LLM-chatbots do not meet key AI in healthcare principles such as bias control, explainability, systems of oversight, validation, and transparency. Chatbots must be designed for greater accuracy, with safety and clinical efficacy demonstrated and approved by regulators, in order to earn their place in the medical arsenal,” Prof. Gilbert concludes.
AI chatbots in medical settings raise ethical concerns as well. Maintaining patient privacy and confidentiality, for example, is critical because these chatbots may handle sensitive health information.
Many AI chatbots are intended to supplement rather than replace healthcare professionals. Human intervention is required to ensure that the chatbot’s recommendations are consistent with established medical guidelines and best practices. AI chatbots should be programmed to learn from new information and improve over time. Regular updates and monitoring are required to address any potential issues that may arise while using them.
