According to the findings of a recent experiment, OpenAI’s ChatGPT has passed all three components of the United States Medical Licensing Examination (USMLE) in a single sitting.
The USMLE is a high-stakes, comprehensive, three-step standardized testing program that covers all topics in physicians’ knowledge, such as basic science, clinical reasoning, medication management, and bioethics. It usually takes a medical student approximately four years and over two years of clinical rounds to pass.
ChatGPT scored at or near the passing level for all three assessments, according to the researchers, without any extra training or reinforcement. Furthermore, ChatGPT’s explanations revealed a high level of coherence and understanding. These findings imply that large language models (LLMs) have the potential to assist in medical education and, possibly, clinical decision-making.
This isn’t the first time ChatGPT has excelled at a test. Professors at the University of Pennsylvania’s Wharton School of Business determined a few days ago that the chatbot could easily pass tests on a common MBA core subject, Operations Management. However, the chatbot has limitations, such as failing to answer more complex process analysis questions.
Recently, researchers assessed GPT-3.5 using topics from the United States Bar Exam. They expect that GPT-4 and similar models will pass the exam relatively soon. Furthermore, the researchers discovered that hyper-parameter tuning and quick engineering improved GPT-3.5’s zero-shot performance.
