Introduction

1 Software Uncertainty Modeling

Software is a large and complex artifact, produced primarily by humans for humans. Software development and operational use are therefore dominated by human action, human judgment and decision making, and inevitably human error. The outcome is, therefore, often uncertain and unpredictable and, along with uncertainty in natural phenomena, leads to unavoidable software uncertainties. Most if not all software failures are experienced by or otherwise affect human users. Similarly, most software defects can be traced to human origins, be it an error introduced directly by a developer or a defect that is only indirectly human–related. An IEEE software engineering standard, for example, identifies two key sources of software uncertainty as “poor knowledge or understanding of the problem domain” and “insufficient, untimely orientation to user needs and to overall quality” [IEE94].

The observation that “software engineering is fraught with uncertainty” is often used to explain the known symptoms of the software crisis, including high costs and unexpected slips in development schedule as well as unpredictable failures of operational software. The abundance of uncertainty also justifies extensive measures taken by researchers and practitioners attempting to identify and alleviate software risks and uncertainties.

While software risk management has long been an active field of research producing significant contributions (cf. [Boe89, Boe91]), the subject of software uncertainties has largely been overlooked, with the notable exception of software reliability models [Lit79] and models for validation of software dependability [LS93]. This omission is particularly surprising in light of extensive research in modeling and management of uncertainty carried out in Artificial Intelligence (AI) domains (cf. [Pea88, Nea90, HMW95]).

Thus, despite the pervasiveness of human involvement and related software uncertainties, surprisingly few attempts have been made to model those uncertainties explicitly. This dissertation aims to remedy this situation, initially by presenting the Maxim of Uncertainty in Software Engineering (MUSE) and as a corollary, urging that software uncertainties be modeled directly, using an appropriate uncertainty modeling technique.

Ideally, an uncertainty modeling technique should include probabilistic notions of uncertainty and confidence, provide for multiple sources of evidence and influence, and support dynamic updating of uncertainty values during software development. In this dissertation, we claim that Bayesian Belief Networks meet these desiderata and are therefore suitable for software uncertainty modeling.

There are many perceived benefits of modeling software uncertainty, including, among others:

1. Uncertainty permeates software development, yet is generally absent in most software models. A model providing explicit representation of software uncertainties would therefore be more accurate (and possibly more useful) than one where uncertainties are disregarded.

2. The search for uncertainty plays a pivotal role in an overall software risk management strategy. Following Gemmer [Gem97], “It is impossible to manage risk without ferreting out elements of uncertainty. Uncertainty is present in all decisions we face.” A key risk minimization activity is to “systematically search for uncertainty wherever it may be.”

3. Finally, software uncertainty modeling may help guide and ultimately improve software process decisions. Based on changes and fluctuations in software uncertainty levels, for example, a project manager may decide to “conclude testing,” “instigate regression testing,” or else “shuffle testing priorities.”

We generally believe that software uncertainty modeling holds promise for improved human understanding and decision making in a wide range of development activities. In this dissertation, however, we focus on specific benefits of uncertainty modeling for specific development situations. To this end, we describe several premises and propose a specific research hypothesis to be validated against a real software project. The premises and hypothesis for this dissertation are described next.

2 Premises and Hypothesis

To arrive at the approach taken in this dissertation, one must follow the statements in this section. Most statements, such as “software complexity hinders its understanding” or “users get disoriented in large hypertext spaces,” are either widely accepted or substantiated later. These statements therefore constitute the premises for this dissertation. One statement, however, is less accepted and requires further investigation: “Explicit modeling of software uncertainties improves human judgment and decision making during software development.” This statement therefore constitutes the hypothesis for this dissertation and will be further elaborated and validated.

Our initial observation is that software is extremely complex, i.e., that “software systems are perhaps the most intricate of man’s handiworks” [Bro95]. Software complexity is not only ubiquitous, it is also intrinsic: “The complexity of software is an essential property, not an accidental one” [Bro87]. “By essential we mean that we may master this complexity, but we can never make it go away” [Boo94]. Complexity hinders software understanding, yet understanding must be afforded if development is to be successful.

To manage complexity, software development concerns are often separated in time and space. Thus, software is typically developed in distinct phases by different teams of developers. Large-scale software is often longlived and delivered in multiple version, thereby comprising a family of related programs [Par79].

The product of software development is typically a large and complex collection of software elements of diverse types. Software artifact collections often contain not only millions of source code lines, but also versioned releases, volumes of user manuals and documentation, requirements and design specifications, test cases and test results, and so forth. Moreover, these software objects are typically integrated and correlated within themselves and with each other in surprisingly intricate and subtle ways [SZ92]. This poses substantial impediments to software understanding, particularly with respect to traceability, visualization, and navigation issues, thereby leading to the following research question.

Research Question:

Given the intricacy and complexity of software artifact relationships, what can be done to improve traceability, visualization, and navigation of large artifact collections?

As a first step, it is proposed that software systems be viewed as large hypertexts, since hypertext offers a coherent and consistent metaphor for viewing both inter– as well as intra–artifact relationships (cf. [ZO95, SZ92]). Here, it suffices to define hypertext informally as providing node-and-link views and navigation of software artifact collections. Hypertext is defined more formally and completely in Chapter ??. We therefore contend that a hypertext metaphor of software artifacts and relations should improve developer’s ability to trace or track related elements in large and complex software spaces.

Despite the above, hypertext is no “silver bullet” for relieving software artifact complexities and uncertainties. Specifically, hypertext introduces several concerns regarding its efficacy for software engineering [ZO95]. Here, we are particularly concerned with the well-known navigation problem of user confusion and disorientation in large hypertext spaces as well as the fact that hypertext systems generally are not designed with software engineering in mind. This means that hypertext does not necessarily address the unique needs of software developers and users nor does it model or make use of the unique characteristics of software systems. Additional means should, therefore, be provided to further facilitate understanding of software artifact collections. The following statements concern this facilitation.

We claim that software engineering is fraught with uncertainties. Software uncertainties contribute significantly to the overall complexity and unpredictability of software development. Like complexity, uncertainty is inherent to the engineering of software systems. This observation is summarized succinctly in [ZRK96] as the Maxim of Uncertainty in Software Engineering:

Uncertainty is inherent and inevitable in software processes and products.

Examples of software uncertainties abound – many are provided in this dissertation. Starting in Chapter Four, we focus on uncertainties associated with developers’ confidence levels in software artifacts, including, among others, requirements specifications, design elements, code modules, and testing information. These confidence levels fluctuate frequently during development. Their modeling should, therefore, include a scheme for confidence revision and updating.

We now arrive at the high-level statement of the hypothesis for this dissertation.

High-level Research Hypothesis:

Explicitly modeling uncertainty improves human judgment and decision making during software development.

This hypothesis is too vague to be investigated and validated directly. Instead, we require a more specific formulation, including a specific technique for software uncertainty modeling. To model software uncertainties, we turn to artificial intelligence research in modeling and management of uncertainty. We select a specific uncertainty modeling technique called Bayesian belief networks [Pea88]. Detailed reasons for this choice are provided in Chapter Four.

Here, it suffices to say that Bayesian networks offer a clear conceptual model of causality among related elements and include algorithms for belief revision and updating. This leads to our specific hypothesis below. The hypothesis speaks in terms of software artifacts, which include, among others, requirements elements, design nodes, code modules, and test information, relationships among those artifacts, established as part of the development process, and confidence levels, which reflect developer confidences in certain qualities and properties of software artifacts.

Specific Research Hypothesis:

Bayesian-network models of software artifact uncertainties improves understanding of associated confidence levels, compared to simply following linked information, ultimately leading to better human decision making.

In the long run, we believe that proper adoption and use of uncertainty modeling techniques will improve human judgment and decision making during software development. For the purposes of this dissertation, we focused on the specific hypothesis above and took steps toward its validation, as follows:

1. An existing software system was selected as case study. The system of choice, called CEquencer, is currently under development at Beckman Instruments in Fullerton, CA. CEquencer is designed to control and communicate with various hardware devices that in turn are used by biologists, chemists, and other scientists to separate laboratory specimens into molecular constituents in order to determine their DNA sequences. CEquencer software artifacts are described in Chapter Five.

2. CEquencer requirements were investigated throughly, and a hypertext model as well as a Bayesian-network model of those requirements were constructed.

3. A key subsystem of CEquencer, called FSM Run, was investigated thoroughly, resulting in a hypertext model of software artifacts, including all FSM Run code modules as well as related requirements.

4. Software uncertainty information was collected for FSM Run artifacts. The information was elicited from Beckman experts and developers and organized in a Bayesian network.

5. Our main hypothesis was evaluated against FSM Run code modules and related requirements, as identified in (2) above. In a questionnaire session with the Beckman team, developers were asked to trace artifacts and associated confidence levels with and without the Bayesiannetwork representation of confidence values.

6. Results were gathered for the case study and analyzed for statistical significance. Results of the questionnaire session are summarized in Chapter Six.

3 Dissertation Contributions

We consider the key contributions of this dissertation to be:

1. Defining succinctly the Maxim of Uncertainty in Software Engineering, identifying the abundance of uncertainties and their relevance to software process decisions and project risk management. Also, as corollary, we suggest strongly that the software life cycle include search, identification, modeling and management of software uncertainties.

2. Defining an approach to explicitly model uncertainty in relevant software situations. This approach is anchored in a Bayesian interpretation of relationships and dependencies among software artifacts. It is suggested and demonstrated that Bayesian belief networks, originally described by Pearl [Pea88], may be used for this purpose.

3. Implementing a prototype Java applet that allows software systems to be defined as graphs of related artifacts as well as be annotated with uncertainty information. These software belief networks can then be subject to Bayesian updating.

4. Conducting a case study to substantiate the maxim above and to evaluate the applicability of Bayesian belief networks to a real software project. To this end, we selected as case study the CEquencer system described earlier. Though confined, our case study indicates that explicit modeling of software uncertainties improves developers’ ability to identify and track changes to levels of confidence in software artifacts and relations.

4 Dissertation Organization

The rest of this dissertation is organized as follows: In Chapter Two we present background information in the form of a brief historical perspective of thirty years of related software engineering research. Specifically, Chapter Two covers areas of software visualization, hypertext and hypermedia, object-oriented methods, and AI inference diagrams. Chapter Three discusses software uncertainty in detail, including sources of uncertainty and uncertainty in development phases. In Chapter Four we present an uncertainty modeling technique called Bayesian Belief Networks and justify its applicability to software engineering situations. Several examples of Bayesian networks are included. Chapter Five describes the architecture and components of the CEquencer system, selected as case study for this dissertation. In Chapter Six we present results and impressions from our study of uncertainty modeling at Beckman. Chapter Seven describes three specific applications of Bayesian networks in practice as well as related work in uncertainty modeling. Finally, Chapter Eight presents conclusions and future work.

In conclusion of the introduction, we wish to point out that uncertainty abounds not only in software development but also in most engineering and scientific pursuits as well as many everyday situations (For uncertainty in everyday situations see, for example, Stefik [Ste95], pp. 460.). Detailed exploration of uncertainty in general is, therefore, beyond the scope of this dissertation. Nevertheless, we hope that the maxims, techniques, tools, and results described here will help identify research opportunities as well as provide a solid foundation for future work in software uncertainty modeling. We encourage readers to consider occurrences and consequences of uncertainty in their own domains of interest and expertise. Our future plans include additional modeling of software uncertainties at Beckman as well as exploring other domains where uncertainty modeling may be useful. We are particularly interested in “Bayesian Internets,” i.e., the modeling of information retrieval uncertainties on the Internet and WWW.

References

[Boe89] Barry W. Boehm, editor. Software risk management. IEEE Computer Society Press, Washington, D.C., 1989.

[Boe91] Barry W. Boehm. Software risk management: principles and practices. IEEE Software, 8(1):32–41, January 1991.

[Boo94] Grady Booch. Object Oriented Analysis and Design with Applications. The Benjamin/Cummings series in object-oriented software engineering. The Benjamin/Cummings Publishing Company, Redwood City, California, second edition, 1994.

[Bro87] Frederick P. Brooks. No silver bullet: Essence and accidents of software engineering. IEEE Computer, 20(4):10–19, April 1987.

[Bro95] Frederick P. Brooks. The Mythical Man-Month: Anniversary Edition. Addison-Wesley, Reading, MA, 1995.

[Gem97] Art Gemmer. Risk management: Moving beyond process. IEEE Computer, 30(5):33–43, May 1997.

[HMW95] David E. Heckerman, Abe Mamdani, and Michael P. Wellman. Real-world applications of bayesian networks. Communications of the ACM, 38(3), March 1995. Special Issue on Uncertainty in AI.

[IEE94] Computer Society IEEE. IEEE guide for the use of IEEE standard dictionary of measures to produce reliable software. In IEEE Standards Collection Software Engineering. IEEE Computer Society Press, New York, New York, 1994.

[Lit79] Bev Littlewood. How to measure software reliability and how not to. IEEE Transactions on Reliability, R-28(2):103–110, June 1979.

[LS93] Bev Littlewood and Lorenzo Strigini. Validation of ultrahigh dependability for software-based systems. Communications of the ACM, 36(11):69–80, November 1993.

[Nea90] Richard E. Neapolitan. Probabilistic reasoning in expert systems: theory and algorithms. Wiley, New York, New York, 1990.

[Par79] David Lorge Parnas. Designing software for ease of extension and contraction. IEEE Transactions on Software Engineering, SE-5(2):128–138, March 1979.

[Pea88] Judea Pearl. Probabilistic reasoning in intelligent systems: Networks of plausible inference. Morgan Kaufmann Publishers, San Mateo, CA, 1988.

[Ste95] Mark Stefik. Introduction to Knowledge Systems. Morgan Kaufmann, San Francisco, CA, 1995.

[SZ92] Dani Steinberg and Hadar Ziv. Software Visualization and Yosemite National Park. In Proceedings of the Twenty-Fifth Annual Hawaii International Conference on System Sciences, January 1992.

[ZO95] Hadar Ziv and Leon J. Osterweil. Research issues in the intersection of hypertext and software development environments. In Richard N. Taylor and Joëlle Coutaz, editors, Software Engineering and Human-Computer Interaction, volume 896 of Lecture Notes in Computer Science, pages 268–279. Springer-Verlag, Berlin Heidelberg, 1995.

[ZRK96] Hadar Ziv, Debra J. Richardson, and René Klösch. The uncertainty principle in software engineering. Technical Report UCI-TR-96-33, University of California, Irvine, August 1996.