Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning

Abstract

Incorrectly labelled test cases can adversely affect the training process of human-in-the-loop oracle learning techniques. This paper introduces ISONOISE, a technique designed to identify such mislabelled test cases introduced during human-in-the-loop oracle learning. This technique can be applied to programs taking numeric inputs. Given a compromised automatic test oracle and its training test suite, ISONOISE first isolates the test cases suspected of being mislabelled. This task is performed based on the level of disagreement of a test case with respect to the others. An intermediate automatic test oracle is trained based on the slightly disagreeing test cases. Based on the predictions of this intermediate oracle, the test cases suspected of being mislabelled are systematically presented for relabelling. When mislabelled test cases are found, the intermediate test oracle is updated. This process repeats until no mislabelled test case is found in relabelling. ISONOISE was evaluated within the human-in-the-loop oracle learning method used in LEARN2FIX. Experimental results demonstrate that ISONOISE can identify mislabelled test cases introduced by the human in LEARN2FIX with over 67% accuracy, while requiring only a small number of relabelling queries. These findings highlight the potential of ISONOISE to enhance the reliability of human-in-the-loop oracle learning.