Machine learning, a key element in artificial intelligence (AI) models, is already an integral part of rheumatology research and clinical practice. It is up to rheumatologists to learn how to best make use of these models, when and how to adopt different models, and when to say no.
“These models are here, and they are being integrated into clinical care practice pipelines,” said Jamie Collins, PhD, Associate Professor of Orthopedics in the Orthopedic and Arthritis Center for Outcomes Research at Brigham and Women’s Hospital. “None of these models is perfect, all the more reason that we need to be understanding how we evaluate performance, how we assess whether these models are something we want to use or not. On the research side, some of the metrics we might report may not match with what makes a clinically useful model. We need to understand what that disconnect is and make sure we fix it.”
Dr. Collins will discuss approaches to evaluating and interpreting machine learning results and how to better understand biases, limitations, and applications in clinical practice during Making Sense of Machine Learning: How to Read, Interpret, and Apply New Research on Tuesday, October 28, from 1–2 p.m. in Room W184A of McCormick Place. Liubov Arbeeva, MSc, Biostatistician at the University of North Carolina, Chapel Hill, Thurston Arthritis Research Center, will focus on the latest recommendations for reporting machine learning model specifications and results.
AI and machine learning have been featured topics at every ACR Convergence in recent years. 2025 is no exception, with sessions devoted to AI in medical education and imaging. The full scientific program is available online.
“There have been a lot of sessions that have covered the nuts and bolts of how to actually undertake an analysis with these models and research using these models,” Dr. Collins explained. “This session will ask how we can tell if these models and results are legitimate. How do we step back and make decisions about whether these models might be applicable to clinical practice, and what are some of the pitfalls we need to be aware of?”
The reality is that AI models sometimes “hallucinate,” Dr. Collins noted. Models can make things up and present untruths as factual evidence. It is up to researchers and clinicians to understand the limitations of machine learning as well as the potential benefits.
“We are going to leave clinicians with questions, questions we should be asking when we are given one of these models, whether it’s reading a scientific paper on some new machine learning or AI model, or something that is already incorporated into clinical workflows,” she explained. “How do you even begin to make assessments of whether what you’re seeing is legitimate? There are questions we should be asking ourselves about whether or how we trust these results, and how, or if, they can be used in practice.”
But these questions shouldn’t be done in silos where computer scientists and clinicians are separated.
“We need to be talking with each other,” Dr. Collins said. “This session can facilitate some of that discussion about how we all work together to develop the best, most accurate, unbiased models and get them out in a way that can actually help to improve patient care.”
On-demand access to recorded presentations will be available to registered participants of ACR Convergence following the annual meeting through October 31, 2026.
Don’t Miss a Session
If you weren’t able to make it to a live session during ACR Convergence 2025 — or you want to revisit a session from the annual meeting — make plans to watch the replay. All registered participants receive on-demand access to scientific sessions after the meeting through October 31, 2026.