Intelligent test oracles based on SVM for non-terminating systems

A test oracle is a mechanism that decides whether an SUT (software under test) fails or passes a test case. Modern software IPs (intellectual properties) usually have a long life cycle and are subject to ever-changing requirements and operating environments. Especially, programs like operating systems, embedded systems, servers, etc. may never terminate and their test oracles need to monitor execution traces of unbounded lengths in order to issue correct test verdicts. We investigate how to use machine learning techniques to automatically construct test oracles for such non-terminating programs without reliance on explicit specifications. Firstly, we present a library, called InTOL (Intelligent Test Oracle Library), for the convenient and flexible collection of test traces. We can flexibly use either user guidance or program assertions to collect verdicts to test traces. Such verdicts are used as supervisory signals to the supervised learning algorithm (SLA) for a test oracle. Secondly, we present several sets of feature variables for the temporal relation among events in test traces of unbounded lengths. Then we present procedures that convert test traces into feature vectors, train an SLA with the feature vectors and their verdicts, and use the trained SLA as a test oracle. The approach is plausible since program traces are usually much easier to collect than formal specifications to construct. We report the implementation of InTOL on top of SVM (support vector machine). We experiment with two open-source benchmark SUTs from the internet to check the performance of our techniques. Our experiment data shows that high-accuracy test verdicts can be achieved with our test oracles for the benchmark SUTs.