Conclusions and Future Work

1 Summary of Contributions

This dissertation offers the following contributions:

• Defining the Maxim of Uncertainty in Software Engineering. This maxim recognizes the abundance of software uncertainties and their relations and impacts on process decisions and risk management. Also, as corollary, it is suggested that software development include the search, identification, modeling and management of software uncertainties.

• Defining an approach to explicitly model uncertainty using software belief networks. This approach is anchored in a Bayesian interpretation of relationships and dependencies among software artifacts. It includes notions of human judgment, confidence, belief, and evidence, often ignored by software modeling techniques.

• Implementing the SoftBayes Java applet. This applet allows software systems to be defined as graphs of related artifacts and to be annotated with uncertainty information. Once defined, software belief networks are subject to Bayesian updating, using a core engine such as provided by JavaBayes.

• Conducting a case study to substantiate the maxim above and to evaluate the applicability of Bayesian belief networks to a real software project. Results of the CEquencer study indicate that explicit modeling of software uncertainties improves developers’ ability to identify and track changes to levels of confidence in software artifacts and relations.

• Finally, several confidence factors were identified as influencing expert’s beliefs in software requirements. Recording those confidence factors is deemed beneficial to future similar projects where requirements uncertainty is captured explicitly.

Several impediments and limitations of our approach were observed, including the initial cost of obtaining prior belief values, the need to ensure that software belief networks retain causality and variable independence, and the assumption that software developers, domain experts, and related project information are available and accessible.

Our experience in using Bayesian networks for software uncertainty modeling, specifically for the CEquencer system, indicates that:

• The conceptual view of both software systems and Bayesian networks as interrelated “webs” of nodes and links seems to offer a convenient metaphor that also maps well into subsequent design and implementation. Specifically, early depictions of CEquencer artifact webs were developed as Java applets and placed on the World Wide Web (see [Ziv97]); these applets were then viewed and reviewed by Beckman developers for accuracy and relevancy.

• The CEquencer system, like most other software, proved to be fraught with uncertainties, thereby confirming our suspicions for at least one real-life system. Specifically, CEquencer software embodies many, often subtle, problem domain uncertainties, including uncertainties stemming from laws of physics and chemistry in the software’s operational environment as well as from vaguely defined boundaries between software versus hardware components.

• Our notion of software uncertainties was well received by Beckman developers, offering convenient means in which to describe their confidences and beliefs regarding CEquencer software. The CEquencer requirements analyst has recently incorporated confidence levels into her requirements capture process.

• The questionnaire session at Beckman revealed that, for the given set of questions, a significant difference between the means of the subjects’ scores was identified. This suggests that subjects’ ability to answer the questions increased with the addition of confidence level information. This confirms a statistically significant relationship between the availability of confidence level information and developers’ ability to trace and track software artifacts and related confidences.

2 Future Work

2.1 Comprehensive Case Study

As discussed in chapter ??, the case study at Beckman was confined to a single group of developers, a single Bayesian model of confidence-level uncertainty, and a single scenario of change with associated questions. We are very interested in expanding the scope of our field studies, both within Beckman as well as with other software development organizations.

A more comprehensive study than the one reported in this dissertation would include one or more of the following enhancements:

• Measure the effects of using the Bayesian models over time, especially its impacts on the quality of human judgment and decision making.

• Construct additional Bayesian models for additional artifact networks or to model other situations or properties. We are particularly interested in constructing Bayesian models at different levels of granularity, comparing the efficacy of resulting models, and, eventually, combining Bayesian models with different granularities.

• Afford automated support for definition and revision of software belief networks. Of special interest are UI design issues in constructing beliefnetwork software as well as integration of Bayesian capabilities with existing tools, e.g., REBUS, DOORS, and RequisitePro.

• Finally, this dissertation focused on incorporation of subjective information into software models. We are interested in combining subjective and objective data in future models. Specifically, we wish to include project data such as code quality and developer productivity, failure rate and severity, etc., in our models as factors that influence confidence and, ultimately, human judgment and decision making.

2.2 Monitoring the testing process

We intend to study the relationship between the Bayesian approach and software testing. This relationship is important not only since testing should have significant implications for belief and confidence levels, but also since a Bayesian model should have ramifications for human judgment and decision making during testing. For example, one important question in software testing is “How much testing is enough?”. This question may be addressed by explicit modeling of uncertainty, if sufficient testing is defined in terms of levels of confidence in select system entities, for example, its code modules. As testing progresses, confidence levels increase as long as test execution is successful. Testing is guided and monitored by continuous update and comparison of confidence levels against predefined thresholds. Testers are notified and may take appropriate action whenever thresholds are exceeded.

This approach may be especially useful in safety-critical systems, where confidence requirements and constraints are often specified numerically. With regard to CEquencer, we intend to enhance our Bayesian model to include testing information - as it becomes available at Beckman - and monitor the effects of the Bayesian approach on testing decisions.

2.3 Other software qualities

In this dissertation, we focused on implementability and later validity as interesting software qualities for which causal relationships can be established and belief values can be determined. We believe, however, that uncertainty should also be modeled for additional software qualities, including, among others, correctness, reliability, efficiency, safety, and maintainability. Of particular relevance is the work carried out at CSR, discussed in chapter ??, where uncertainty is modeled for software testability and dependability assessment tasks. As mentioned earlier, qualities associated with entities must be consistent with causal relationships such that the resulting network is meaningful.

2.4 Other uncertainty modeling techniques

In this dissertation, we have used Bayesian networks to model uncertainty in software development. Alternative approaches to uncertainty modeling and logical inference exist, however, that must be weighed and compared against Bayesian methods. Those include Certainty–Factor approaches, Dempster– Shafer approaches, fuzzy logic, and default, monotonic, and nonmonotonic logic [Ste95]. Specifically, relative merits and pitfalls of these techniques should be evaluated against the Bayesian approach in the context of software engineering situations.

2.5 Modeling uncertainty in software processes

In this dissertation, we focused on software product uncertainties, such as modeled for CEquencer software, though process uncertainties were treated briefly (e.g., path selection criteria in Table ??). Despite the emphasis on artifact uncertainties, we firmly believe that process uncertainties should also be modeled. We contend that software-process modeling formalisms must be augmented to include uncertainty values; that an environment for supporting definition and execution of process models should include capabilities for representation and interpretation of belief values and should allow for Bayesian updating of those values; and that Bayesian updating procedures must be carried out during process execution, such that belief values and confidence levels are continuously updated as new evidence arrives.

The provision and update of belief values may be greatly enhanced in software process frameworks that include process measurement capabilities. Such capabilities constitute a rich source of information regarding the current state of various elements and support the collection of statistical and empirical data that may significantly improve the accuracy of prior belief value estimation.

We expect that by modeling software process uncertainties, one may achieve a more realistic representation of the process, enable automated belief revision by means of Bayesian updating, and support prediction and guidance of future development activities.

3 Final Thoughts

This dissertation offers a path that, if followed, helps address software development problems for which uncertainty plays an important role, specifically with respect to human judgment and decision making. We strongly believe that software uncertainty modeling - using Bayesian belief networks - will help managers to better plan, schedule, estimate, and evaluate software projects. It will also help developers to better trace and navigate interrelated artifacts in large software spaces, understand and discover system qualities and properties, and, ultimately, achieve better judgment and make better decisions.

References

[Ste95] Mark Stefik. Introduction to Knowledge Systems. Morgan Kaufmann, San Francisco, CA, 1995.

[Ziv97] Hadar Ziv. Java applets for Beckman CEquencer software, March 1997. http://www.ics.uci.edu/ ziv/java/bayesian example.html.