You are here

Enabling Multiple Experts Achieve (greater) Consensus on (Classification) Tasks

CISA Seminar Series
Derek Sleeman, Department of Computing Science, University of Aberdeen

This project focuses on developing techniques which enable an expert to detect inconsistencies in two (or more) perspectives on the same (classification) task. Further, the INSIGHT system has been developed to support domain experts exploring, and removing inconsistencies in their conceptualization of a task. I will discuss a study with Intensive Care clinicians whose high-level task was to classify, on a five-point scale (A-E), the (hourly) reports produced by an Intensive Care Unit's patient management system. The two perspectives provided to INSIGHT were an annotated set of patient records where the expert had selected the appropriate category and the classification of the same instances by the expert's classification rules. Three experts undertook this task and each, using INSIGHT, achieved a high correlation (~97%) between his 2 perspectives (annotations & rule-set). The experts then produced a common rule-set and further refined their sets of annotations against it; this resulted in inter-expert agreements of ~97%. The presentation concludes by outlining some of the follow-up studies planned with both INSIGHT and this general approach.

Date and time: 
Monday, 27 April, 2009 - 14:00
60 minutes