Wednesday, December 4 | 2:20 p.m.-2:30 p.m. | W6-SSNMMI06-6 | S405
In response to common questions from lutetium-177 (Lu-177) prostate-specific membrane antigen (PSMA)-617 patients, Gemini provided easy-to-understand answers and ChatGPT-4 demonstrated high accuracy, yet neither AI chatbot provided completely accurate answers to all questions, according to a study in this session on prostate cancer therapy.
Presenter Gokce Belge Bilgin, MD, of the Mayo Clinic in Rochester, MN, and colleagues evaluated the accuracy, conciseness, and readability of responses from ChatGPT-4 and Gemini to 12 commonly asked questions by patients on Lu-177 PSMA-617 therapy (according to two experts). The AI-generated responses were blindly rated by eight experts, while potential safety concerns associated with the answers were also examined.
According to the findings, ChatGPT-4 provided more accurate answers than Gemini (2.95 vs. 2.73 on a 4-point scale), while Gemini's responses had better readability than ChatGPT-4 (2.79 vs. 2.94 on a 3-point scale) Both ChatGPT-4 and Gemini achieved comparable conciseness scores (3.14 vs. 3.11 on a 4-point scale).
Additionally, the experts categorized the AI-generated responses as incorrect or partially correct at a rate of 16/96 for ChatGPT-4 and 28/96 for Gemini. Gemini's answers contained significantly more misleading information than those of ChatGPT-4 (p = 0.039), the group found.
As AI chatbots are becoming increasingly popular, many patients are expected to turn to them as a rapid source of health information, which raises important questions about their reliability and effectiveness, the group noted. As things stand, the chatbots need further improvements to be considered for Lu-177 PSMA-617 patients seeking medical information, according to the researchers.
Check out this mid-afternoon session on Wednesday for all the details.
