An optimized outlier detection algorithm for jury-based grading of engineering design projects

Mary Kathryn Thompson, Christina Espensen, Line Harder Clemmensen

Publikation: Bidrag til tidsskriftTidsskriftartikelForskningpeer review

Abstract

This work characterizes and optimizes an outlier detection algorithm to identify potentially invalid scores produced by jury members while grading engineering design projects. The paper describes the original algorithm and the associated adjudication process in detail. The impact of the various conditions in the algorithm on the false positive and false negative rates is explored. Aresponse surface design is performed to optimize the algorithm using a data set from Fall 2010. Finally, the results are tested against a data set from Fall 2011. It is shown that all elements of the original algorithm (the base rule and the three additional conditions) play a role in the algorithm's performance and should be included in the algorithm. Because there is significant interaction between the base rule and the additional conditions, many acceptable combinations that balance the FPR and FNR can be found, but no true optimum seems to exist. The performance of the best optimizations and the original algorithm are similar. Therefore, it should be possible to choose new coefficient values for jury populations in other cultures and contexts logically and empirically without a full optimization as long as the algorithm assumptions are valid and the limitations for its use are well understood.

OriginalsprogEngelsk
TidsskriftInternational Journal of Engineering Education
Vol/bind32
Udgave nummer1
Sider (fra-til)172-184
Antal sider13
ISSN0949-149X
StatusUdgivet - 2016
Udgivet eksterntJa

Bibliografisk note

Publisher Copyright:
© 2016 TEMPUS Publications.

Citationsformater