An optimized outlier detection algorithm for jury-based grading of engineering design projects

This work characterizes and optimizes an outlier detection algorithm to identify potentially invalid scores produced by jury members while grading engineering design projects. The paper describes the original algorithm and the associated adjudication process in detail. The impact of the various conditions in the algorithm on the false positive and false negative rates is explored. Aresponse surface design is performed to optimize the algorithm using a data set from Fall 2010. Finally, the results are tested against a data set from Fall 2011. It is shown that all elements of the original algorithm (the base rule and the three additional conditions) play a role in the algorithm's performance and should be included in the algorithm. Because there is significant interaction between the base rule and the additional conditions, many acceptable combinations that balance the FPR and FNR can be found, but no true optimum seems to exist. The performance of the best optimizations and the original algorithm are similar. Therefore, it should be possible to choose new coefficient values for jury populations in other cultures and contexts logically and empirically without a full optimization as long as the algorithm assumptions are valid and the limitations for its use are well understood.