Google’s AI-driven medical chatbot has successfully cleared a stringent US medical licensing examination, although its responses still trail behind those of human physicians, according to a peer-reviewed study published on Wednesday.
The launch of ChatGPT last year, developed by OpenAI which is supported by Google’s competitor Microsoft, ignited a competitive fervor amongst tech titans in the rapidly evolving realm of AI.
Despite ongoing debates about the prospective opportunities and perils of AI, the healthcare sector is one domain where the technology has already made noteworthy advances. For instance, certain AI algorithms have demonstrated their capacity to interpret specific medical scans as effectively as human professionals.
Google initially revealed its AI solution, named Med-PaLM, for responding to medical queries in a preprint study in December. Unlike ChatGPT, Med-PaLM is not publicly available.
I am very excited that the MedPaLM paper is now published in @Nature https://t.co/BDqshkSIDf
It is a great way to invite the broader scientific community to LLMs. I feel like LLMs are more adopted in the general public than in the scientific community. There are just so many…
— Hyung Won Chung (@hwchung27) July 12, 2023
According to the American tech conglomerate, Med-PaLM is the first large language model – an AI method educated on extensive volumes of human-generated text – to successfully clear the US Medical Licensing Examination (USMLE).
A passing score for the USMLE, an examination undertaken by medical students and physicians in training in the United States, hovers around 60%.
A study in February reported that ChatGPT had managed to secure passing or near-passing scores.
In a peer-reviewed study disclosed in the Nature Journal on Wednesday, Google researchers stated that Med-PaLM scored 67.6% on multiple-choice questions styled after the USMLE.
In an effort to pinpoint and reduce “hallucinations” — a term used when AI models generate incorrect data — Google claimed to have established a new evaluation benchmark.
Karan Singhal, a Google researcher and the principal author of the recent study, reported that the team utilized the benchmark to evaluate a more recent version of their model, yielding “super exciting” outcomes.
According to a preprint study released in May, which hasn’t undergone peer review, Med-PaLM 2 has scored 86.5% on the USMLE exam, marking an improvement of nearly 20% over its predecessor.
