SC Connecticut News

Clinical guidelines are essential to evidence-based medicine but often pose challenges due to their complexity and length, making it difficult for busy physicians to quickly access pertinent information. Researchers in the Department of Internal Medicine are exploring methods to integrate these guidelines into existing clinical tools and workflows, with a particular focus on large language models (LLMs) that can generate responses to medical queries.

"While we were developing an LLM tool to help clinicians answer questions about hepatology and gastrointestinal conditions, we realized that LLM companies often automatically convert clinical guidelines from a PDF to a text document," said Dennis Shung, MD, PhD, assistant professor of medicine (digestive diseases). "But when you automatically convert these guidelines, you lose important data that is essential for clinical reasoning."

Shung's team observed significant data loss in tables, graphics, and flowcharts during this conversion process. The accuracy of LLMs was approximately 80% for text-only tables but plummeted to 16% with graphics and nearly zero with flowcharts.

"This is especially concerning because sometimes the most important information in a guideline is in a flowchart," noted Mauro Giuffrѐ, MD, postdoctoral associate (digestive disease). "For LLM tools to be helpful to clinicians, they must be capable of understanding all the data – not just pieces of it."

The team advocates for the creation of LLM-friendly clinical guidelines by medical societies. Such guidelines would enhance the development of accurate and comprehensive LLM tools.

Shung and Giuffrѐ recently published their findings in a paper titled “Optimizing Large Language Models for Medical Guidelines Interpretation: A Framework Based on Study on Hepatitis C Virus Guidelines Using Retrieval Augmented Generation” in njp Digital Medicine. Simone Kresevic led the software engineering effort alongside collaborators Milos Ajcevic and Agostino Accardo from the University of Trieste's biomedical engineering department and Lory S. Crocè from its gastroenterology and hepatology department.

"We started with no formatting at all – just a straight PDF to text," said Giuffrѐ. "Then we added more labels and specificity to give the LLM more information about each figure or table." This approach significantly improved the model's accuracy in reasoning over the data.

Shung believes these findings could extend beyond hepatitis C guidelines to other medical specialties. "LLMs are only as good as the information they are trained on," he stated. "Using LLM-friendly versions of clinical guidelines could help us develop point-of-care tools that provide highly accurate and relevant information for clinicians."

Ultimately, Shung and Giuffrѐ hope medical societies will adopt LLM-friendly formats for their guidelines. "Medical societies want their members to practice evidence-based medicine," Shung emphasized. "By creating LLM-friendly clinical guidelines, they can help us create tools that are up-to-date, complete, and use reputable sources."

The Yale School of Medicine’s Section of Digestive Diseases has been influential in gastrointestinal and liver disorder research for decades. More information is available at Digestive Diseases.

---