How 'cranky comments' can help spot CDS alert errors

It's an all too common occurrence for clinical decision support alerts to misfire or otherwise malfunction. And that's a common frustration for clinicians. New research in the Journal of the American Medical Informatics Association shows how data scientists at Partners HealthCare are making use of provider feedback to spotlight and fix CDS errors.

WHY IT MATTERS
Rule-based alerts in the electronic health record could help notify physicians of risks such as adverse drug interactions or nudge them to add relevant information to the EHR.

But such alerts can also malfunction, break, or might be triggered for the wrong reasons or patient populations. That could cause alert fatigue or pose safety risks.

The variety and types of these misfires can vary widely - as can clinicians' responses to inaccurate CDS alerts.

"In prior work, we developed a taxonomy of CDS alert malfunctions based on 68 cases of alert malfunctions collected from healthcare institutions in the United States," wrote Partners HealthCare researchers in JAMIA.

They also researched the ways in which those errors were found.

"Although the most common discovery method in the sample was user reports via traditional channels such as the help desk, safety reports, and contacting CDS maintenance personnel, five malfunctions were discovered by reading free-text override reasons written by providers when overriding alerts," they said.

"These malfunctions were discovered at our own institution, by chance, when a student researcher was reviewing alert overrides and noticed that some comments seemed to express strong frustration and disagreement with the alerts," researchers explained.

The researchers pointed to several instances where clinicians made note of the inaccuracies - often using colorful and indignant language:

• "BPA misfiring: no potassium on file, but there was a K done!"
• "Inappropriate warning as K is 4.3"
• "He is on beta blocker!"
• "You are stupid"
• "This is an inappropriate rec"
• "cyclosporine is eye drops!"

The Partners researchers noted that such comments "were helpful for determining why the rules were broken, and helped us fix the rules."

It's a resource that more hospitals should take advantage of to help them pinpoint where inaccuracies are occurring in their EHRs, they said.

Clinicians "sometimes provide information in override comments or other text fields in the EHR that seem to be communicating information to someone else but in fact are not sent to the implied recipient and are never acted upon.

"Override reasons are an example of this kind of text field," they added. "Although intended for audit purposes, override reasons are typically not intended for communication of clinical information or issues with the CDS itself. Nonetheless, users sometimes provide meaningful information in their override reasons."

THE LARGER TREND
CDS software can be a valuable tool to help providers make more informed medication, imaging and lab orders.

In addition to improving quality and safety, it can also save money, as we showed recently in a case study looking at Saint Luke's Health System.

But it needs to be configured properly and embraced by clinicians to prove its worth. Paying attention to physician feedback is one way to ensure it's working optimally.

ON THE RECORD
"Override comments are a rich data source for finding alerts that are broken or could be improved," said Partners researchers. "If possible, we recommend monitoring all override comments on a regular basis."