GabyA team at the University of California, San Diego compared written responses from doctors and from ChatGPT to health questions on the Reddit forum AskDocs. They wanted to see whose advice was better — medical professionals’ or the AI’s.
The researchers concluded that ChatGPT provided higher-quality responses and showed more understanding and empathy than qualified doctors.
Although ChatGPT often draws criticism for its flaws, the new study casts it in a positive light: the AI’s responses outperformed doctors’ on the same questions in 79 percent of cases.
What the researchers found
About 452,000 internet users post messages about their medical issues on the Reddit forum AskDocs, where verified medical professionals offer help, the Daily Mail reported.
The researchers randomly selected 195 questions from the forum for qualified doctors to answer, then asked ChatGPT the same questions and collected its answers.
When reviewing the responses, a panel of three doctors didn’t know which answers came from physicians and which came from the chatbot.
Jessica Kelly, a co-author of the study, later summarized that ChatGPT’s messages contained more detailed and accurate information than the doctors’ responses.
An example: doctor vs AI
Question (user on the forum): “What is the risk of becoming blind after bleach gets into the eyes, causing irritation and dryness?”
Answer — doctor: “It seems you will recover. You should rinse your eyes every time a chemical substance or foreign object gets into them. You can also contact Poison Control at 1-800-222-1222.”
Answer — ChatGPT: “I’m sorry to hear that bleach got in your eyes. It’s important to rinse your eyes as soon as possible after chemical splashes to remove as much of it as you can and prevent further irritation or damage. If you haven’t done so already, you should rinse your eyes with clean water or saline solution as soon as possible. You can do this by keeping your eyes open and directing a gentle stream of water from the inner corner of your eye to the outer corner for at least 15 minutes. If you don’t have access to water, you can use saline solution or artificial tears to rinse your eyes. If you experience severe pain, redness, or changes in vision, or if you cannot rinse your eyes properly, you should seek medical attention immediately. However, if your eye just feels irritated and dry, you can treat it at home using artificial tears or eye ointment to keep it moist and comfortable. It’s unlikely that you will go blind from bleach getting in your eyes, but it’s important to take care of them and seek medical help if necessary to prevent further irritation or damage.”
All three experts preferred ChatGPT’s responses. They gave the AI an average consultation-quality score of 4 and an empathy score of 4.67. In contrast, the doctors scored 3.33 for quality and 2.33 for empathy.
James Davenport, a professor of information technology at the University of Bath (UK), reacted more cautiously. He noted that ChatGPT’s answers were on average four times longer than the doctors’. Davenport said the research team claimed to have blinded the selection, but argued that length and style could still reveal authorship. He also pointed out that “teams of clinicians often rely on prepared responses,” while “a stochastic parrot like ChatGPT has a much broader range of answers than even the largest library of prepared responses.”
Recently, ChatGPT stirred the medical community after it was shown capable of passing the gold-standard exam required for U.S. medical practice. That raises questions about whether it could one day replace human doctors.
