Informatics 43 Spring 2009
Course Project
Phase 4: Testing

Due date and time: Friday, June 5, 9:00pm


No matter how careful we are when we write code, mistakes are essentially unavoidable; there are just too many interlocking details that we need to get right. A good design (e.g., separation of concerns, classes with single responsibilities) can partially mitigate this issue, unit testing can certainly help, but we ultimately still need to test our system in whole, with all components completed and integrated together.

In your project this quarter, you've now completed at least some portion of the implementation and submitted it, so it's time to focus your efforts on testing, ensuring that the system has been built as specified in the Official Requirements Specification, using a combination of white-box and black-box testing techniques. This is a big job, much larger than you could reasonably be asked to complete in the time we have left this quarter, so you'll only be asked to test certain areas of functionality.

Also, because not everyone has completed the implementation — and because those of you who aren't finished haven't all completed on the same subset of it — you'll all test a separate, completed, "official" implementation instead of your own. (Of course, you're welcome to run your chosen test cases against your own implementation, if you'd like, but that won't be part of this phase of the project.)

Testing is at least partly driven by planning; deciding what test cases should be executed in order to gain confidence that a system is working as specified is often at least as much work as executing the tests you chose. As we discussed in lecture, this kind of planning can take many forms, with the goal being to find some systematic way of knowing that you've "covered all your bases." This phase of the project will provide you with practice in using some of these techniques.

In particular, you'll be working on two forms of testing:

The Official Implementation of the Triple P Enrollment System

The Official Implementation is a console-based implementation of the requirements that comprise Implementation Phase 1 of the Official Requirements Specification, along with a handful of the other requirements.

You can download the Official Implementation, in both compiled and source code forms, at the following link:

I should point out a few things about the Official Implementation:

Test matrices

For this part of Phase 4, you'll be required to develop a set of test matrices, using the technique we discussed in lecture for choosing test cases:

As an example, suppose you're testing a web site like Gmail, where users can write email messages, send them, search them, and so on. Suppose, in particular, that you're testing the login functionality. Some of the bases you might choose include:

Each basis divides test cases into subdomains, where all of the test cases in each subdomain share some interesting property. For example, the whether the entered password matches against users' passwords might yield three subdomains:

  1. The password entered matches the password for the user specified
  2. The password entered matches the password for a user other than the one specified
  3. The password entered does not match the password for any user in the system

You then choose at least one test case in each subdomain, along with test cases that span multiple subdomains (if any). In the above example, you might choose these test cases:

Note that, for each test case, the expectation is included; it's not a test case if we don't know what behavior we expect.

We can display the bases, subdomains, and test cases in the form of a test matrix, as we saw in lecture. An example of a test matrix, detailing one basis and associated test cases in the example situation above, is available at the following link:

Note the presence of an Outcome column in the example test matrix; as you execute tests, fill in the actual testing outcome that you observed (e.g., "performed as expected" or "login was successful even though it shouldn't have been").

Part 1: Black box testing (50 points)

For Part 1 of this project phase, you are required to plan and perform black box testing on part of the Official Implementation of TPES. In particular, you will separately develop and execute tests for two of the requirements from the Official Requirements Specification:

For each of these requirements, determine at least three bases, making sure that each of them is a reasonable criterion that can be used to divide the domain of possible test cases into interesting subdomains. For each basis, list a reasonable number of subdomains (usually at least three) and one or two test cases for each subdomain, along with at least one test case that covers every possible combination of subdomains. (Note that not all subdomain combinations will be possible; sometimes, none of them will be possible.)

Record your test cases in a collection of test matrices, in the same basic form as the example linked in the previous section. Run the test cases against the Official Implementation of TPES and make a note of the outcome of each: note whether the test was succesful, along with a note about any incorrect behavior that was seen if the test failed.

Part 2: White box testing (50 points)

For Part 2 of this project phase, you are required to plan and perform white box testing on part of the Official Implementation of TPES. In our case, we're using the term "white box testing" to refer to whole-program testing driven by our knowledge of the structure of the source code; this form of testing is distinct from the unit testing you did in the previous phase, which focused on testing individual classes and methods completely out of the context of the rest of the program. You'll execute your white box tests by running the program; the difference is that you'll use the code to choose your test cases.

Line numbering and node coverage

Recall from lecture that we can build a control flow graph for methods in a Java program. The first step is to number each line of code, not including the signature or any line containing only a curly brace. Statements that are split into multiple lines — which I do so that my code can be read without the need for horizontal scrolling — should only be counted as a single line. For example, the following is an example of the numbering of lines in the Classroom.addCourseMeeting( ) method from the Official Implementation:

    public void addCourseMeeting(CourseMeeting courseToAdd)
    throws CourseMeetingOverlapsException
1       if (courseMeetingToAdd.getClassroom() != this)
            throw new IllegalArgumentException(
2               "courseMeetingToAdd refers to the wrong classroom");
3       for (CourseMeeting courseMeeting : courseMeetings)
4           if (courseMeeting.getMeetingInterval().overlaps(courseMeetingToAdd.getMeetingInterval())
5               throw new CourseMeetingOverlapsException(courseMeeting, courseMeetingToAdd);
6       courseMeetings.add(courseMeetingToAdd);

We won't build control flow graphs in this phase of the project — though you're free to draw them if you'd like — but we will use this line-numbering scheme to help track whether we've achieved the most code coverage we can, where every line of code that can be reached has been reached by at least one test case.

What you need to do in this part

First, number the lines of code in the following two methods in the Official Implementation. (You don't have to turn this part in, but you do need to number the lines so that you can refer to them in your test matrices.)

Next, determine whether there are any lines of code that are unreachable in the program as it is currently written. This will require you to look around other parts of the program that use these methods. (There are tools that can help; see below.) Make a note of any unreachable lines of code; you won't need to find tests that cover them.

Finally, you're ready to choose test cases and execute them, using the following procedure for each of the two methods that you've been asked to test:

Put together a test matrix as you did in Part 1 of this phase, using the same basic format shown in the example.

Navigating an unfamiliar code base

It's been said that the worst code in the world is "someone else's code." When you're working with code that you didn't write, it can be difficult to find your way around; even if you can follow the logic of short blocks of code, you won't necessarily have a feel for its overall structure.

Since you haven't seen the code in the Official Implementation before, it may be a bit difficult, at first, to find out whether a particular line of code can be reached via the user interface. For example, some error checks in the inf43.spring2009.tpes.model package are duplicated in the user interface code; in these cases, the exception will actually never be thrown by the model, so it won't be possible to use our form of white box testing to verify that they get thrown.

(Sidebar: You might be wondering why such checks are being done in more than one place. Why should the model check things that the user interface checks? Why should the user interface check things that the model checks? The reason is that they're each doing the checking for a different purpose. The user interface checks for things that will allow it to be construct a contextual error message. The model checks for things that will cause it to fail. The model operates, rightly, under the assumption that it could be surrounded by any user interface; given that, even if today's user interface does a particular check, tomorrow's user interface might not. So we're better off making the model as bullet-proof as we can, so that we'll be more resilient to change.)

Eclipse has built-in tools to lighten the burden of navigating someone else's code base. One useful one is the "References" tool, which asks Eclipse to search an entire project or even an entire workspace, looking for references to a particular class, method, variable, etc. Suppose, for example, that you'd like to find out where in the program the Course.addCourseMeeting( ) method is called. Here's how you can find out all the places where the method is called:

Using tools like reference searching will make it easier to find your way around this obviously unfamiliar code base that I've given you.


For each part of this project phase, submit your test matrices in a single document in either Microsoft Word (.doc or .docx), PDF (.pdf), or Rich Text Format (.rtf). Note that you may find it useful to orient the pages in landscape mode (i.e., wider than they are tall) rather than the more customary portrait mode (i.e., taller than they are wide).

Follow this link for a discussion of how to submit files via Checkmate, an ICS-built online assignment submission system. Be aware that I'll be holding you to all of the rules specified in that document, including the one that says that you're responsible for submitting the version of the specification that you want graded. We won't regrade your work simply because you submitted the wrong version accidentally.