Researchers have found that ChatGPT, Open AI’s artificial Intelligence chatbot, can pass all three sections of the United States medical licensing exam. ChatGPT was found to have moderate accuracy and passed the exams comfortably, according to a new research experiment.
Axios reported that the research is still being peer-reviewed. This involves professionals analyzing their colleagues’ work to verify its accuracy and significance.
Researchers wrote that ChatGPT passed all three exams at or close to the passing threshold without any training or reinforcement.
The majority of authors work for Ansible Health, a startup based in Mountain View, California, that’s been exploring ways of using AI to enhance healthcare outcomes.
They said the tool demonstrated high levels of concordance, insight, and clarity in its explanations, and the results suggest large language models might have the potential to aid with medical education and possibly clinical decision-making.
The researchers did exclude some not-so-well answers because ChatGPT appeared to be programmed to prevent giving medical advice.
Morgan Cheatham, a student at Brown University in medicine, said that the answers were so general it was difficult to determine if they were correct or wrong. She is also a vice president at Bessemer Venture Partners and focuses on healthcare.
According to the study, the first exam step can be taken by medical students after two years of education. The second step is taken by a fourth-year medical student who has completed up to two years of clinical rotations. Step three can be taken by postgraduate students.
ChatGPT’s ability to create high-quality essays and understand complex topics has impressed academics. Some also pointed out that ChatGPT is vulnerable to misinformation and lacks the depth of understanding.
Anthropic, an AI safety and research company, developed another AI named Claude. According to Virginia’s George Mason University academic, Claude has also passed a university-level economics and law exam.
The study, “Performance Of ChatGPT On USMLE: Potential For AI-Assisted Medical Education Using Large Language Models,” is published on medRxiv. This online archive contains medical, clinical, and health science papers that have not been peer-reviewed.
Related Stories: