The evaluator effect in usability studies: Problem detection and severity judgments

Niels Ebbe Jacobsen, Morten Hertzum, Bonnie E. John

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningpeer review

56 Citationer (Scopus)

Abstract

Usability studies are commonly used in industry and applied in research as a yardstick for other usability evaluation methods. Though usability studies have been studied extensively, one potential threat to their reliability has been left virtually untouched: the evaluator effect. In this study, four evaluators individually analyzed four videotaped usability test sessions. Only 20% of the 93 detected problems were detected by all evaluators, and 46% were detected by only a single evaluator. From the total set of 93 problems the evaluators individually selected the ten problems they considered most severe. None of the selected severe problems appeared on all four evaluators’ top-10 lists, and 4 of the 11 problems that were considered severe by more than one evaluator were only detected by one or two evaluators. Thus, both detection of usability problems and selection of the most severe problems are subject to considerable individual variability.
OriginalsprogEngelsk
TitelProceedings of the HFES 42nd Annual Meeting
Antal sider5
UdgivelsesstedSanta Monica, CA
ForlagHuman Factors and Ergonomics Society
Publikationsdato1998
Sider1336-1340
StatusUdgivet - 1998

Citationsformater