Editor’s note: Aakanksha Pitliya, MD, Postgraduate Year 3 Internal Medicine Resident, Mercy Catholic Medical Center, one of the authors of the following abstract, answered questions about this abstract for ACR Convergence Today. Tanya Ratnani, MBBS, Postgraduate Year 2 Internal Medicine Resident at Guthrie Robert Packer Hospital, will present the research in Sunday’s Plenary I.
Abstract Presenter: Tanya Ratnani, MBBS, Postgraduate Year 2 Internal Medicine Resident at Guthrie Robert Packer Hospital
Abstract Title: Artificial Intelligence in Rheumatology: A Comparative Evaluation of Guideline Concordance and Hallucination Risk Across Platforms
Abstract Session: Plenary I (0867-0871), 10–11:30 a.m., Sunday, November 8
What is your abstract about?
Dr. Pitliya: Our study evaluated five artificial intelligence (AI) platforms—ChatGPT, Gemini, Claude, Perplexity, and OpenEvidence—using 60 standardized clinical scenarios based on ACR and EULAR recommendations across six rheumatologic conditions. We assessed how closely their answers followed guidelines, their educational quality, and their tendency to generate “hallucinations,” including fabricated citations and unsupported clinical recommendations. OpenEvidence and ChatGPT demonstrated the highest guideline concordance in our evaluation, at 91.2% and 89.4%, respectively, but clinically meaningful errors occurred across all five platforms, underscoring the importance of clinician oversight.
Why did you decide to investigate this topic?
Dr. Pitliya: We wanted to explore a practical question: When an AI-generated answer sounds clear and convincing, how closely does it actually align with the evidence? Rheumatology provided an important setting for this investigation because its guidelines are complex, and decisions about treatment selection, monitoring, and complications require careful interpretation. Our goal was to examine both the educational promise of these tools and the risks that may be less obvious to someone reading a polished response.
What are you working on next related to this research?
Dr. Pitliya: A next step I would like to pursue is examining whether grounding AI responses in verified guideline sources can reduce hallucinations and improve guideline concordance. I am also interested in identifying which types of clinical questions are most prone to consequential errors and evaluating how performance changes as these platforms evolve. These directions could help inform practical approaches to verifying AI-generated information in clinical education and decision support.
What excites you most about your work?
Dr. Pitliya: What excites me most is the opportunity to move beyond whether AI can produce an answer to understanding what makes that answer useful, reliable, and worthy of trust. Our findings highlight educational potential while also showing why a convincing response should not be mistaken for a clinically sound recommendation. I hope this work encourages clinicians and trainees to approach AI with both curiosity and critical judgment.
THERE’S STILL TIME TO JOIN US
If you haven’t registered for ACR Convergence 2026, you can still register on-site at the Orange County Convention Center in Orlando, Florida, through November 11. All registered participants receive on-demand access to scientific sessions through September 30, 2027.
