A new study suggests that ChatGPT is better at diagnosing respiratory diseases than trainee doctors.
The study, presented at the European Respiratory Society in Austria, also found that Google’s Bard performed better than humans in some cases, while Microsoft’s Bing performed as well as the trainee doctors.
The research indicates that large language models like ChatGPT could help medical staff assess patients more efficiently, potentially easing the pressure on healthcare systems like the NHS.
In the study, ten trainee doctors with less than four months of experience in pediatrics were given one hour to use the internet, but not AI chatbots, to solve scenarios related to child respiratory medicine. These scenarios were created by experts and did not have obvious diagnoses. The answers from the trainee doctors were then compared to those generated by ChatGPT, Bard, and Bing.
ChatGPT version 3.5 scored the highest and was noted for giving more human-like responses than other chatbots. Dr. Manjith Narayanan, a consultant in pediatric pulmonology at the Royal Hospital for Children and Young People in Edinburgh, who presented the study, said, “These tools have several potential applications in medicine. My motivation to carry out this research was to assess how well large language modules (LLMs) are able to assist clinicians in real life.”
The researchers did not find any clear instances of “hallucinations”—a term used when AI chatbots make up information—among the chatbots in this study. However, Dr. Narayanan warned that it is important to prevent this from happening in the future. “Bing and Bard did give some answers that were deemed to be irrelevant to the questions asked, but so too did the trainee doctors,” he noted.
A survey released in July by the Health Foundation, a healthcare think tank, found that more than half of the UK public and three-quarters of NHS staff support the use of AI for patient care. However, both groups expressed concerns that AI cannot show “real empathy” or “kindness,” which they see as the biggest disadvantage of the technology.
Hilary Pinnock, a professor of primary care respiratory medicine at the University of Edinburgh, commented on the findings, saying, “It is encouraging, but maybe also a bit scary, to see how a widely available AI tool like ChatGPT can provide solutions to complex cases of respiratory illness in children. It certainly points the way to a brave new world of AI-supported care.”
Dr. Narayanan plans to continue his research by testing chatbots against more experienced doctors and exploring newer, more advanced language models.
