Conference on Cognitive Computational Neuroscience

Our reinforcement-learning (RL) based decision support system for sepsis (the AI Clinician) has entered prospective evaluation in 4 UK intensive care units (ICUs) and as part of this deployment critical questions arise on how to best render the action recommendations explainable and trustworthy to clinicians who may or may not choose to execute them. We therefore conducted an experimental human-AI interaction study with ICU doctors within which they were presented with 16 patient scenarios and asked how much of two supportive medications (intravenous fluid and vasopressor) they would prescribe. We used a multi- factorial experimental design with 4 arms: (B1) baseline, (B2) peer human clinician information, (B3) AI suggestion, (B4) explainable AI suggestion (using feature importance). Our results suggest that ICU clinicians are influenceable by dose recommendations. Knowing what peers had done had no significant overall impact on clinical decisions while knowing that the recommendation came from AI did make a measurable impact. However, whether the recommendation came with an explanation or not did not make a substantial difference.