Test Oracles

"Specification-based Test Oracles for Reactive Systems," by D.J. Richardson, S.L. Aha, and O. O'Malley in Proceedings of the Fourteenth International Conference on Software Engineering, May 1992

Abstract

The testing process is typically systematic in test data selection and test execution. For the most part, however, the effective use of test oracles has been neglected, even though they are a critical component of the testing process. Test oracles prescribe acceptable behavior for test execution. In the absence of judging test results with oracles, testing does not achieve its goal of revealing failures or assuring correct behavior in a practical manner; manual result checking is neither reliable nor cost-effective.

We argue that test oracles should be derived from specifications and in conjunction with testing criteria, represented in a common form, and their use made integral to the testing process. For complex, reactive systems, oracles must reflect the multiparadigm nature of the required behavior. Such systems are often specified using multiple languages, each selected for its utility in specifying a particular computational paradigm. Thus, we are developing an approach for deriving and using oracles based on multiparadigm and multilingual specifications to enable the verification of test results for reactive systems as well as less complex systems.

from Debra J. Richardson
djr@ics.uci.edu
Department of Information and Computer Science, 
University of California, Irvine CA 92717-3425