One of the most important goals of software testing is to determine whether a system behaves the way the stakeholders want it to behave. From the stakeholders' point of view, this behavior is ideally described by a set of high-level requirements expressed so that they can be understood by all concerned, with minimum effort, and without the need of learning and working with a formal approach that is distant from stakeholders' backgrounds and interests. Studies of industry practice have shown that scenarios successfully satisfy these demands on high-level requirements. It is equally important to be able to test these requirements in a way that is transparent to the stakeholders. The stakeholders should be able to look at the test results and see that the system behaves in the way they expect based on the requirements. This need, however, is not widely being met in practice in industry today.

We are working on TR&cup ST (Testable Requirements &cup Specification-based Testing) to address this need. TR&cup ST makes requirements scenarios (or use cases) testable by augmenting a requirements notation, ScenarioML, with annotations used to derive test scenarios and construct a knowledge-based system (KBS). The KBS drives the system under test through the test scenarios and verifies conformance to the system's requirements. TR&cup ST makes powerful use of testable requirements by: (1) identifying implementation faults that prevent the system from conforming to its requirements, (2) detecting non-testable requirements early, (3) potentially automating much of the testing process, and (4) facilitating greater stakeholder involvement in requirements formulation and test evaluation.

In TR&cup ST we drive the system under test through particular paths of each requirements scenario. Each such path is called a test scenario. The KBS sends the system under test down a particular path by providing events in the system's environment that satisfy appropriate conditions on the selected path. We define requirements coverage on a set of scenarios by adapting the code-based coverage criterion Modified Condition/Decision Coverage (MC/DC). When applied to a scenario, MC/DC requires that (1) every point of entry and exit in the scenario has been used at least once, (2) every condition in a decision in the scenario has taken on all possible outcomes at least once, and (3) each condition has been shown to independently affect the decision's outcome. We use this coverage metric because it shows not only that a system can behave as each scenario branch describes, but also that each branch can be taken as a result of all the conditions related to it. We use th structure of the requirements, the coverage criterion, domain equivalence partitioning, and boundary value analysis to generate test cases from the requirements scenarios.

The expected benefits of TR&cup ST are:

1. The derived test suite covers the requirements.
2. The test design helps to validate the requirements
3. The tests are successful in detecting requirements violations in the implementation
4. Test results are traceable to the requirements
5. Much of the process can be automated.
6. Stakeholder participation is facilitated during requirements formulation and evaluation and test evaluation.