ICS 224: Software Testing and Analysis

Debra J. Richardson
Spring 2000








Course Description

Catalog Copy

Studies techniques for developing confidence in software from traditional testing schemes to integrated, multi-technique analytic approaches. Considers strengths and weaknesses and explores opportunities for synergistic technique application. Emphasis is on approaches integrated into the software process.

This course will examine software testing and analysis techniques. Students will study the goals of testing and analysis, in particular evidence of software quality or identification of software defects, as well as the basic approaches to demonstrating and assessing quality. In addition, various tools are available that support the variety of testing and analysis techniques.

At the end of the course students should be conversant with the primary current approaches to software testing and analysis. Students should be familiar with the "canonical" examples typically used in research papers. Students should also be aware of the major open problems in demonstrating quality via testing and/or analysis, such that research within this domain would be a natural follow-on to this course.

Course requirements are reading relevant papers on formal methods, participation in class, satisfactory performance on pop quizzes covering the reading material, and a quarter project, which includes two short presentations. There are several options (discussed below) for the quarter project, the primary requirement being that it focus on software testing and/or analysis.
 

Prerequisites

ICS 221 is recommended but not required.
 

Schedule and assigned readings

(subject to change; additional readings may be assigned)
 
Date Topic Readings
April 4 Course Overview  
April 6 Introduction to Analysis and Testing [ABC82] [YT89] [GHM87]
April 11 Qualities and Fault Models [Goe85] [Lev86]
April 13  Process [MDL87]
April 18 Inspections  [Fag76]
April 20 no class
April 25 Testing Introduction, Testability 
and Test Oracles		 [Wey86] [Ric94]
[Wey82] [ORD96]
April 27 Project Briefs   
May 2  Coverage and Regression Testing [CPRZ89] [RW97]
May 4 Integration and System Testing [OB88] [ER96]
May 9 Static Analysis [FO76] [Jac95] [DC96]
May 11  Model Checking [Jac96] [AG93] [Hol97]
May 16 Object-Oriented and 
Distributed System Testing		  
May 18 Student Topics  student readings
May 23 Student Topics  student readings
May 25 Student Topics  student readings
May 30 Student Topics  student readings
June 1 ICSE '00  no class
June 6 ICSE '00  no class
June 8 ICSE '00  no class
June16 Final Projects Due  

Lectures
available on line.

Readings
available in the graduate student cabinet on the fourth floor of CS.
 
[ABC82]  Adrion, Branstadt and Cherniavsky, "Validation, Verification and Testing of Computer Software", ACM Computing Surveys 14(2): 159-192, June 1982.
[AG93]  J.M. Atlee and J.D. Gannon. "State-Based Model Checking of Event-Driven System Requirements", IEEE Transactions on Software Engineering; 19(1):24-40, January 1993.
[CPRZ89]  L.A. Clarke, A. Podgurski, D.J. Richardson and S.J. Zeil. "A Formal Evaluation of Data Flow Path Selection Criteria", IEEE Transactions on Software Engineering; 15(11):1318-1332, November 1989.
[DC96] M. Dwyer and L.A. Clarke. "A Flexible Architecture for Building Data Flow Analyzers", Proceedings of the International Conference on Software Engineering, March 1996.
[ER96] N. Eickelmann and D.J. Richardson, Evaluating Software Testability based on Software Architecure, Proceedings of theSecond International Software Architecture Workshop (ISAW-2), October 1996. 
[Fag76]  M.E. Fagan. "Design and Code Inspections to Reduce Errors in Program Development", IBM Systems Journal; 15(3):182-211, 1976.
[FO76] L. Fosdick and L.J. Osterweil. "Data Flow Analysis in Software Reliability", ACM Computing Surveys, September 1976.
[GHM87]  J.D. Gannon, R.G. Hamlet, and H.D. Mills. "Theory of Modules". IEEE Transactions on Software Engineering; SE-13(7):820-829, 1987.
[Goe85]  A.L. Goel. "Software Reliability Models: Assumptions, Limitations, and Applicability", IEEE Transactions on Software Engineering; SE-11(12):1411-1423, 1985.
[Hol97] G.J. Holzmann. "The Model Checker SPIN," IEEE Transactions on Software Engineering; SE-23(5), May 1997.
[Jac95] D. Jackson. "Aspect: Detecting Bugs with Abstract Dependencies", ACM Transactions on Software Engineering and Methodology 4(2):109-145, April 1995.
[Jac96] D. Jackson. "Elements of Style: Analyzing a Software Design Feature", IEEE Transactions on Software Engineering; 22(7):484-495, July 1996.
[Lev86]  N.G. Leveson. "Software Safety: What, Why, and How", ACM Computing Surveys; 18(2):125-163, June 1986.
[MDL87]  H.D. Mills, M. Dyer and R.C. Linger. "Cleanroom Software Engineering", IEEE Software; 4(5):19-25, September 1987.
[ORD96] T.O. O'Malley, D. J. Richardson, and L.K. Dillon, Efficient Specification-Based Test Oracles, Second California Software Symposium (CSS'96), April 1996. 
[OB88] T.J. Ostrand and M.J. Balcer. "The Category-Partition Method for Specifying and Generating Functional Tests," Communications of the ACM; 31(6):676-686, June 1988.
[Ric94]  D.J. Richardson. "TAOS: Testing with Analysis and Oracle Support". in Proceedings of the 1994 International Symposium on Software Testing and Anlaysis, pp. 138-153, Seattle, WA, August 1994.
[RW97] D.S. Rosenblum and E.J. Weyuker. "Using Coverage Information to Predict the Cost-Effectiveness of Regression Testing Strategies", IEEE Transactions on Software Engineering; 23(3):146-156, March 1997.
[YT89]  M. Young and R.N. Taylor. "Rethinking the Taxonomy of Fault Detection Techniques", in Proceedings of the 11th International Conference on Software Engineering, pp. 53-62, Pittsburgh, PA, May 1989.
[Wey82] E.J. Weyuker. "On Testing Non-Testable Programs," The Computer Journal; 25(4):465-470, 1982.
[Wey86]  E.J. Weyuker. "Axiomatizing Software Test Data Adequacy". IEEE Transactions on Software Engineering; SE-12(12):1128-1138, December 1986.

Suggested topics for student presentations

Assessment

5% Class Participation

15% Pop Quizzes on reading material

10% Project Description (10 minutes)

50% Term Project, chosen from

Graded on effort and on final report (and demonstration, if applicable) at end of class

Two options:

A project done with another person would naturally be larger in scope than one done individually.
Both options require each student to do a half-hour lecture (see next)

20% Presentation of a topic in analysis and testing relevant to chosen project (30 minutes),
including presentation of background readings and description of project application of topic

NOTE: There will be no incompletes granted, except in truly rare and unavoidable circumstances. All course work must be completed by the end of finals week.