Kendrick Hang

The key concepts of software maintenance both parallel and extend concepts from software development. Similar to software development, software maintenance is concerned with requirements, design, coding, testing, delivery, and the ongoing project management concerns: resource allocation, configuration management, and quality assurance. However, all of these concerns are within a context of a system that is already in operation. There is an element of building in place: making changes without adversely affecting what is already there. Clearly, building in place requires careful planning and execution around a well-defined and logical process.

Software maintenance revolves around the maintenance request (MR). Unlike the change request (CR) in software development, which results in modifications to the major project artifacts, the MR is the driving project artifact. An initiation of an MR is what starts the process of evaluation, change, and approval: determining what the requirements of the change are, designing and implementing the change, and testing and approving the change to be accepted into a new version of the system. In software maintenance, the project artifacts are created to satisfy the MR. Since the MR and its resolution process is so important in the maintenance phase of the system lifecycle, the process should be documented clearly.

Although the initiation of MRs drive all successive activities in the maintenance process, every activity in the software maintenance phase, including the handling of MRs is listed in the software maintenance plan. Just as the software development plan describes the rationale behind the project management in a development project, the maintenance plan details how activities should be managed in a maintenance effort. The MRs address the technical issues of a change, but the maintenance plan addresses all of the other real-world organizational issues: what is and is not going to be maintained, who will be performing maintenance, how maintenance will be conducted, where the maintenance will be performed. In addition, it details the resource allocation: budget, cost, schedule, and staffing. In essence, the maintenance plan describes how the manager and team will conduct and control the maintenance effort.

A key concept that pervades the maintenance effort is configuration management. Configuration management has three major aspects: configuration identification, change control, and status accounting and auditing. Change control is the element of configuration management that allows the system to evolve in a stable manner. Meanwhile, if something happens during evolution and one has to revert back to a previous (stable) version, the historical aspect of change control then becomes more important to help move the system back to a controlled and stable state. Configuration management is the bridge between software development and software maintenance. Software maintenance involves understanding the current system well enough to make changes to the system, whether they be corrections or enhancements and having proper and consistent configuration management during development and into maintenance is invaluable.

The transition from development into maintenance is described in a transition plan. The transition plan addresses how the system will be transferred from the development team to the maintenance team. In addition to the obvious fact that the maintenance team needs to be brought up to speed on the current state of the system, the transition plan describes how the project artifacts, source code, and software development tools will be transferred to the maintenance team. Since so much in the development phase affects what the maintenance team will inherit in in the maintenance phase, one should consider maintenance from the early stages of development where a significant portion of a system’s maintainability is determined.

The software lifecycle should be viewed as a whole. Although they are often treated as separate efforts, software development and software maintenance all lead back to the same system: a system that will always be evolving over time, regardless if it is under development or in maintenance. Unless the system is being retired, change is inevitable and the system will continue to evolve and one needs to be aware of how to manage that change through process, planning, execution, and evaluation.

I wrote this lit review-style survey after doing a research dig on software safety to get a sense of what the current research and practice is for safety-critical software systems. This paper really does not do justice to the significant amount of material that is out there, but I think it at least skims the surface for someone who might just be trying to find a place to start reading and learning more. At the same time, I think the structure of this paper (each section subsequently shorter in length) mirrors the literature that is out there. There’s a lot of folks writing about how important safety is, a few less people writing about safety requirements, even less on designing software for safety, and not much at all on testing safety-critical systems. I originally wrote this paper in July 2007 and I am now publishing it on the web now for the software community to read and refer to as needed.

Abstract
Although software is increasingly becoming a part of safety-critical systems that control aircraft, automobiles, heavy machinery, and medical devices, few software engineers are aware of the issues relating to safety-critical systems and the techniques that are currently available, along with their limitations. To effectively design a safety-critical software system one must consider safety throughout the software development lifecycle, and not as an afterthought. Safety requirements must be explicitly specified, an active approach is used to address safety in design and implementation, and the safety constraints of the software must be verified and tested properly before the software is deployed in a safety-critical situation. Instead of being an afterthought to software engineering, software safety is more of an extension of the software development methods of which software engineers are already familiar, with added emphasis on specific techniques to address safety-specific concerns.

Software Safety: A Survey of Concepts and Techniques
When software professionals think of software safety, the infamous Therac-25 incident comes to mind. The Therac-25 was a medical device designed to administer radiation therapy. However, flaws in the Therac-25 software resulted in some patients being seriously harmed by radiation overdoses and burns. Leveson and Turner’s (1993) case study is required reading in many undergraduate computer science programs, but beyond knowing that software safety is an issue, few software engineers know how to make software safer. Creating safer software involves using methods and techniques during all phases of the software development life cycle. From determining safety-related requirements to developing test cases for safety conditions, the notion of safety must embody the approach to software development for the resulting software to be considered safe.

Although emphasis is placed on making software safer, safety must be considered at the system level. Moray and Huey (1988) define the layers of a complex socio-technical system as:

1. the technical/engineering system,
2. the workers that operate the system,
3. the organizational/management infrastructure that governs the workers, and
4. the environmental context in which the organization operates.

Vicente (1999) states, “There is now a great deal of evidence to show that organizational factors play a very important role in system performance and safety.” This implies that the design of the technical system, including the software, should take factors external to itself into consideration when addressing safety issues. Since people use the software system, human factors need to be considered for safety to anticipate the potential actions a user might take in the software if certain scenarios arise. In addition, users may follow guidelines from their management and the software may need to address those guidelines to maintain safe conditions in the overall system.

Software safety is important in situations that have potential to cause harm to humans, but situations that pose hazards are equally important. It is natural to think that a scenario where software failure results in bodily harm or death as a safety-critical system, particularly since the Therac-25 is the most well-known case study on software safety. Knight (2002) defines safety-critical systems as “those systems whose failure could result in loss of life, significant property damage, or damage to the environment.” However, many accidents are not caused by a single error, but rather multiple errors that compound to create an unsafe condition in a system. Wickens (2000) classifies errors in terms of active failures and latent conditions. An active failure occurs when a specific action causes a error, while a latent condition is a scenario where an action introduces or contributes an error condition into the system, but the error itself does not occur until a later state when other conditions fall into place at the right time. Similar types of latent errors occur in software: the software may continue to function “normally” but an error may surface later or in another form, either within the software, or outside the software within the realm of the socio-technical system. Leveson (1995) notes that safety-critical systems are rarely defined in terms of the potential catastrophic consequence of an error, but rather the hazard conditions that can be introduced by software, which in turn may contribute to a safety-critical condition in the system.

Safety Requirements
In creating software for safety-critical system, the first step is to consider and determine the safety requirements. In their software safety standard, NASA (1997) stresses the importance of requirements by stating, “The successful development of safety requirements for the software requirements specification is essential to developing safe software and allows for safety to be built into the software early in the life cycle while it is relatively inexpensive.” In the requirements phase of a project, a hazard analysis (U.S. Department of Defense, 1984) should be performed to identify a list of potential hazards in the system, assess the hazards, and then translate the hazards into software requirements, specifically in the form of design constraints. For example, in a software-controlled building security system, the doors not being able to open during a fire might be a hazard. The resulting requirement (or constraint) generated from that hazard might be that the doors can only be locked when no fire alarm in the building has been pulled. In addition to performing a hazard analysis, an investigation of the conditions that cause a hazard to occur, through fault tree analysis, for example, may help to elicit additional safety-related requirements.

Most software engineers are familiar with using natural language (English) to describe requirements in a specification document, but more formal mechanisms can better express safety-related requirements. Formal mechanisms allow for better analysis and reasoning about the modes and states the software can be in, as well as the values that the software may encounter. A finite state machine is an example of a formal mechanism that describes a safety feature of a system or how the system reacts to hazard conditions. In the building security example, a finite state machine can describe the possible transitions between a normal operating state and an emergency (fire, evacutation) operating state, along with the resulting effects. That particular finite state machine can then be examined to see if any transitions were not considered or if any of the resulting effects can pose a hazard. Another means of stating safety requirements formally is through formal specification languages, such that formal analysis techniques can be used to later verify that the safety properties are satisfied (Lutz, 2000). The formal specification languages express safety requirements in mathematical terms such that the interpretation is definite, especially compared to natural language, which can be interpreted differently in varying contexts. Leveson (2004) developed a CASE tool, SpecTRM, to help software developers more formally state system and software requirements in a combination of graphics and text, and then allows users to run automated routines to check the models for inconsistencies and errors that may lead to hazard conditions.

Once the safety requirements have been analyzed and stated in the software requirements specification (SRS), it is necessary to validate the requirements for completeness. The IEEE (1998) lists the following as one of the criteria for completeness: “Definition of the responses of the software to all realizable classes of input data in all realizable classes of situations.” In addition to the broad statement, Leveson (1995) lists 60 specific criteria to examine for requirements completeness. A sample of these criteria includes: startup and shutdown, mode transitions, inputs and outputs, value and timing, load and capacity, failure states and transitions, and latency. Just as most software errors trace their source back to requirements, most software safety issues trace back to non-existent or insufficient safety requirements. Therefore, effective validation of the requirements specification, including the safety-related requirements, will result in fewer errors and hazards being introduced into the system and propagated into subsequent phases in the software development lifecycle.

Safety in Design and Implementation
The software requirements, including the safety requirements among them, are the basis in conceptualizing the software design and producing the software implementation. That is, every requirement should be traceable to an element in the design and to lines of code in the implementation. Several design elements exist to implement some of the safety requirements, including inhibits, traps, interlocks, and assertions to control, reduce, or eliminate hazard conditions. The underlying strategy behind these design elements is to actively check for failure or hazard conditions and handle them appropriately before they propagate to another part of the software. Various levels of checking can be designed into the software to stop the propagation of errors. Leveson (2002) details a hierarchy of software checking. The levels include:

• Hardware checks, where the software actively detects hardware failures and instruction errors.
• Code-level checks, where the software actively checks for invalid values, ranges, and states.
• Audit checks, where a separate process checks data and timing in the main process.
• Supervisory checks, where a separate system (human or computer) checks the main system. Lutz (2000) refers to this as runtime monitoring.

In addition to the safety-related design elements, the design of the non-safety-critical portions of the program needs to be considered as it may in turn affect safety. Parnas, van Schouwen, and Kwan (1990) state that in safety-critical systems, “the system must be structured in accordance with information hiding to make it easier to understand, review, and repair”—a hint that designing for safety starts with good general design practices. Integrating these design elements and strategies, one can explicitly design and code features to not only prevent hazards, but to detect and manage them as well, thus providing an end result of a safer software system.

Testing for Safety
Even if the software is designed with a safety strategy in mind and implemented with safety-related design elements, hazard conditions in the resulting software may still exist because of errors in the design and implementation. For this reason, verification of the code for correctness and validating the implementation against safety-related requirements is an important step in the process of developing safer software. Two primary strategies are static and dynamic analysis, where static analysis involves using tools or people to analyze or inspect the code, while dynamic analysis involves running tests and examining the results to ensure none of the safety requirements (constraints) are violated. Parnas, van Schouwen, and Kwan (1990) suggest that effective testing of safety-critical systems requires a basis of mathematical documentation to test against. That is, using the finite state machines to define the test cases. Although many testing strategies exist, many of which improve the effectiveness of checking safety requirements, a parable of testing still holds true: testing will not prove the non-existence of errors in a system, it will only show the presence of errors in a system. This implies that even if safety is taken into consideration throughout the entire software development lifecycle, there is no such thing as safe software, there is only safer software.

Conclusions
In analyzing the currently available techniques to develop safety-critical software systems, one notices that the principles of designing safer software are not independent from the fundamental principles of software engineering. Software safety is not a separate field and does not involve a different approach to developing safer software, but rather rooted in good software engineering. For instance, good software engineering principles call for investing sufficient time and effort into requirements analysis because that is where most errors in a system originate. Likewise, most safety issues in software originate because of insufficient requirements and safety constraints that need to be designed into the software. Software safety involves extending software engineering practices by supplementing it across the entire software development life cycle with analysis techniques, design strategies, and verification methods to emphasize safety by preventing hazards.

References
Dunn, W. R. (2003). Designing Safety-Critical Computer Systems. Computer, 36(11), 40-46.

Herrmann, D. S. (1999). Software safety and reliability: techniques, approaches, and standards of key industrial sectors. Washington, DC: IEEE Computer Society.

Heimdahl, M. (2007). Safety and Software Intensive Systems: Challenges Old and New. Proceedings of the Conference on The Future of Software Engineering, 137-152.

IEEE. (1998). “IEEE Std. 830: IEEE Recommended Practice for Software Requirements Specifications.” Software Engineering Standards Collection. New York, NY: The Institute of Electrical and Electronics Engineers.

Knight, J. C. (2002). Safety-critical systems: challenges and directions. Proceedings of the 24th International Conference on Software Engineering, 547-550.

Leveson, N. G. (2002). System Safety Engineering: Back to the Future. Unpublished manuscript, Massachusetts Institute of Technology. Retrieved on June 24, 2007, from http://sunnyday.mit.edu/book2.pdf.

Leveson, N. G. (2004). A Systems-Theoretic Approach to Safety in Software-Intensive Systems. IEEE Transactions on Dependable and Secure Computing, 1(1), 66-86.

Leveson, N. G. (1995). Safeware: System Safety and Computers. Boston, MA: Addison-Wesley Professional.

Leveson, N. G., & Turner, C. S. (1993). An Investigation of the Therac-25 Accidents. Computer, 26(7), 18-41.

Lutz, R. R. (2000). Software engineering for safety: a roadmap. Proceedings of the Conference on The Future of Software Engineering, 213-216.

Moray, N., & Huey, B. (1998). Human factors research and nuclear safety. Washington, DC: National Research Council, National Academy of Sciences.

McDermid, J. A., & Pumfrey, D. J. (2001). Software Safety: Why is there no Consensus? Proceedings of the 19th International System Safety Conference.

NASA. (1997). NASA-STD-8719.13A Software Safety Standard. Washington, DC. Retrieved on July 15, 2007, from http://satc.gsfc.nasa.gov/assure/nss8719_13.html

Parnas, D. L., van Schouwen, A. J., & Kwan, S. P. (1990). Evaluation of safety-critical software. Communications of the ACM, 33(6).

Storey, N. R. (1996). Safety Critical Computer Systems. Boston, MA: Addison-Wesley Longman.

U.S. Department of Defense. (1984). MIL-STD-882B System Safety Program Requirements (AMSC Number F3329 FSC SAFT). Washington, DC. Retrieved on June 24, 2007, from http://sunnyday.mit.edu/safety-club/882b.htm.

Vicente, K. J. (1999). Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work. Lawrence Earlbaum: Mahwah, NJ.

Wickens, C. D., & Hollands, J. G. (2000). Engineering Psychology and Human Performance, 3rd ed. Prentice Hall: Upper Saddle River, NJ.

I originally wrote this article in October 2007 for a course in Software Maintenance, now making it available on the web. The intended audience is software engineers and software project managers who are faced with a task of estimating software size and cost.

Planning for a software development (or maintenance) project involves estimating the effort in terms of cost and time to establish a budget and a schedule. The estimated effort is directly proportional to the size of the project—larger projects require more time and money. Therefore, a size estimation (scope, or amount of work) must be made to serve as the basis for the effort estimation (cost and time).

In software projects, lines of code (LOC) immediately comes to mind as a measure of size. However, estimating in terms of lines of code can be problematic as there is no consistent definition of what a line of code is. Does whitespace count as a line of code? Is each line in an if-statement a line of code, or does the entire if statement count as one line of code? In addition, lines of code are dependent on the programming language being used: a 100-line program written in Perl may take 500 lines in C (Kernighan and Pike). Finally, if concise code is a design goal, a count of the lines of code may not reflect the necessary effort required to develop a concise design to satisfy a requirement. Instead of using lines of code as a metric, function points provide an alternative unit of size that addresses some of the shortcomings of using lines of code.

Function points are a measure of the functions—as in functionality—of a system, independent of how the software requirements are specified (natural language or formal notations), how the software is designed (structured or object-oriented), and how the software is implemented (programming language). The determination of the number of function points in a system depends on five parameters described by Albrecht (1983): external input types, external output types, logical internal file types, external interface file types, and external inquiry file types. In more modern terminology used by Futrell, Shafer, and Shafer (2002), these parameters can also be referred to as: inputs, outputs, data structures (files), interfaces, and inquiries. The following list provides brief definitions for each of the parameters.

• An external input is any object (data or control) that provided to the system, regardless of whether the source is a user or another system.
• An external output is any object that the system provides to an external recipient be it a user or another system. Reports produced for a user, messages displayed to a user, and messages sent to another system are examples of outputs.
• A data structure or internal file type is a structure maintained within the system to manage some logical grouping of information. A data structure may be some object in memory that maintains the data to be manipulated by the system.
• An interface is any data that crosses the system boundary: a file that resides outside the application or data structures received from or sent to other systems.
• An inquiry is an input-output pair where some request is made of the system and that request triggers a response by the system. One can think of an inquiry as an event triggered by a user (command) or another system (request) that results in an immediate response.

Upon identifying the parameters in the system, each parameter is classified as simple, average, or complex. For example, if the system has ten outputs, each of the ten outputs are individually classified as simple, average, or complex. The classifications are then used to determine the weight of each parameter. After counting and classifying the parameters in the system under consideration, a weighted sum of the parameters is taken to obtain the raw function point count.

The raw function point count serves as the basis for the final function point count, but it first must be adjusted for environmental factors that can affect the size of the system, up to 35 percent (Jalote, 1997), due to increased complexity. There are 11 environmental factors that are taken into consideration: data communications, distributed processing, performance objectives, operation configuration load, transaction rate, online data entry, end user efficiency, online update, complex processing logic, reusability, installation ease, operational ease, multiple sites, and desire to facilitate change (Jalote, 1997). Each of these environmental factors are evaluated and determine the complexity adjustment factor (CAF). The number of raw function points multiplied by the CAF results in the final function point count, also referred to as the number of adjusted function points or delivered function points. Futrell, Shafer, and Shafer (2002) provide a summary of the steps described above to determine the number of (adjusted) function points in a system.

1. Count the number of functions (parameters) in each category.
2. Apply complexity weighting factors.
3. Apply environmental factors.
4. Calculate the complexity adjustment factor.
5. Compute the adjusted function points.

As an optional last step, one can convert adjusted function points to lines of code. There are known conversion factors that help to estimate the number of lines of code in a particular programming language to implement one adjusted function point. Although it might seem counterintuitive to convert function points into lines of code after noting all of the deficiencies of size estimation by lines of code, this last step is often performed to provide additional insight into the software size estimate. An estimate in lines of code provide developers a sense of the size of the system as it is often easer to visualize scale in lines of code, rather than in function points. In addition, cost estimation models like the widely-known Constructive Cost Model (COCOMO) use lines of code as the estimation unit for its size input. Note however that there is distinct difference between estimating a system’s size directly in lines of code and estimating a system’s size by function points and then converting function points into lines of code. Estimating size in lines of code via function point analysis is less subjective than estimating size ad-hoc in lines of code.

Estimating software size through function point analysis provides a methodical and consistent means of gathering input for software project planning, particularly in determining an initial budget and schedule. Function point counts allows for tracking progress as the system is developed in terms of number of function points satisfied. Even during the maintenance phase of the software lifecycle, it can be an invaluable tool in understanding the scale of the system to be maintained and evaluating the impact of changes on the system to effectively plan for maintenance efforts.

Note to readers: Function point analysis is one, but not the only method of estimating software size. This article is meant as an introduction to the function point analysis, which is commonly mentioned in the literature and one of the more prevalent size estimation methods, used in conjunction with COCOMO to produce a cost estimate.

References
Albrecht, A. J., & Gaffney, J. E. (1983). Software function, source lines of code, and development effort prediction: A software science validation. IEEE Transactions on Software Engineering, 9(6), 639-648.

Futrell, R. T., Shafer, D. F., & Shafer, L. I. (2002). Quality Software Project Management. Upper Saddle River, NJ: Prentice Hall PTR.

Jalote, P. (1997). An Integrated Approach to Software Engineering (2nd ed.). New York, NY: Springer-Verlag.

Kernighan, B. W., & Pike, R. (1999). The Practice of Programming. Reading, MA: Addison Wesley.

I originally wrote this literature review on Software Inspections, with an emphasis on empirical evidence, in December 2007 for a course on Software Verification and Validation. I wanted to publish it on the web because a good portion of the literature that I reviewed is inaccessible to software practitioners and the general public. (One has to pay exorbitant fees to get access to the full-text journal articles and some of the articles that were previously published on the web have since disappeared.)

Abstract
The software engineering community, researchers and practitioners, accept software inspections as a basic and essential tool in conducting software verification and validation activities. In terms of untestable work products such as requirements specifications and design documents, inspections may be one of the few available tools to conduct an effective verification and validation of these work products. However, many in the community want to know how effective inspections actually are, in empirical terms, and how to make them both more effective and efficient in detecting the maximum number of errors with a minimal amount of effort or cost.

In this review of the literature on inspection technologies, the Fagan Inspection is used as a baseline for study and evaluation as it is the most well known inspection technique. Since Fagan (1976), others have noted problems and limitations of the Fagan Inspection technique and proposed their own inspection techniques. Two major variants include Active Reviews by Parnas and Weiss (1985) and Phased Inspections by Knight and Myers (1991). More recent inspection techniques focus on the changing nature of the software industry, in terms of how software teams work and the types of software being developed. Specifically, recent variations on inspections address the impact of globalization (distributed teams, outsourcing, offshoring) and the advent of the World Wide Web (distributed technologies, decentralized architectures). All variants attempt to address flaws in the Fagan Inspection technique with the hopes of making inspections more effective and efficient. Although publications describing the varying inspection techniques claim effectiveness through a case study of a single instance or application, few are able to show the improvement empirically.

Although there is a lack of empirical data that covers broad applications to prove the effectiveness of inspections and to calculate the return on investment of inspection activities, the varying inspection research and techniques still contribute to the larger verification and validation toolkit. Software engineers and software project managers are currently not able to use empirical data to make and justify their decisions on the varying inspection techniques, but they are able to pick and choose from a rich variety of options to suit their application. However, researchers are continuing to improve upon inspection techniques and study how to reliably measure and compare techniques to provide empirical data for decision making. There is no consensus on a “best” inspection technique, but rather the consensus among the research community is that there is still a lot of further study to be done to better understand inspections.

Software Inspections
Experienced software engineers and software project managers recognize that the earlier errors are detected in the software lifecycle, the less costly it will be to resolve the error. As a result, software engineers and project managers plan verification and validation activities into the development lifecycle to detect errors as early as possible in the lifecycle. One of the most well known verification and validation activities is the software inspection, a staple of many verification and validation toolkits and plans. The Software Engineering Institute (2001) defines software inspection as a disciplined engineering practice for detecting and correcting defects in software artifacts, and preventing their leakage into field operations. Where testing can only verify executable work products, software artifacts that can be inspected include requirements specifications, software designs, source code, test plans, and test procedures. The goal of this paper is to present software engineers and software project managers with a survey of available inspection techniques with an empirical evaluation of their effectiveness to allow for software engineers and project managers to not only identify inspection as a necessary activity in a verification and validation plan, but to choose the techniques that are most appropriate for their projects.

Fagan Inspection
The Fagan Inspection is the most commonly cited software inspection technique in software engineering literature. In a sense, the Fagan Inspection is a standard software inspection technique as its essence is included in the IEEE Standard for Software Reviews (1997). Developed by M. E. Fagan in 1972 at IBM, the Fagan Inspection consists of six steps: (1) Planning, (2) Overview, (3) Preparation, (4) Inspection, (5) Rework, and (6) Follow-Up. Compared with walkthroughs, which are informal and consist only of two steps, Preparation and Walkthrough, Fagan Inspections are a more formal alternative requiring an inspection team, with each member playing a role: moderator, designer, coder, or tester.

Fagan’s original work included an empirical evaluation of his documented inspection technique in finding errors. Fagan (1976) showed that inspections caught 82 percent of the errors in the application under case study. Although the findings proved successful for that particular case, Fagan later showed in follow-up work that the effectiveness of an inspection depends on how the inspection was conducted. Fagan (1986) noted that well-conducted inspections resulted in the discovery of 82 and 93 percent of all defects in two specific case studies. However, in cases where inspection moderators were not trained or inspection was conducted on a limited portion of the development lifecycle, defect discovery rates were slightly over 50 percent. Fagan (1986) considered these reduced rates to be a success given the shortcomings of the inspections performed with limitations, but the high rate of variation between moderately successful and highly successful inspections served as an indication to others that inspection techniques could be improved to provide more consistency in finding defects.

Active Reviews
Parnas and Weiss (1985) developed Active Reviews, an alternative inspection technique that addresses the limitations of Fagan Inspections. In describing the motivation behind Active Reviews, Parnas and Weiss (1985) detail the problems of traditional (Fagan) inspections, including: reviewers being overloaded with information and not having sufficient time to work with it, reviewers not being familiar with all of the requirements and constraints imposed on the artifact under inspection, and reviewers attempting to examine all of the artifact without focusing on the parts they might be most familiar with. In an attempt to address these shortcomings, the Active Review technique involves: inspectors working on a single technical area, inspectors working alone, authors of the artifact supplying questionnaires to the inspectors to check comprehension, all of which results in individual discussions between each inspector and the author to arrive at agreed-upon feedback that results in product rework.

Although Parnas and Weiss (1985) mention in their conclusions that they were convinced that Active Reviews achieved their review objectives and that the problems with Fagan Inspections had been addressed, there was no experimental study conducted. That is, there was no formal experiment that compared Active Reviews against a control or baseline method of Fagan Inspections. Despite no empirical evidence existing that shows the effectiveness of Active Reviews, a search of the literature today (in the ACM Digital Library, CiteSeer, or IEEE Computer Society Library) reveals many dozens of papers citing Parnas and Weiss (1985). In essence, the software engineering community generally accepts Active Reviews as an effective practice based on a logical and sensible rationale, even though no empirical evidence shows it to be more effective than Fagan Inspections.

Phased Inspections
Following on Fagan’s work, Knight and Myers (1991) developed an improved inspection technique, Phased Inspections. Knight and Myers recognized several shortcomings in the Fagan Inspection technique, many of them similar to what Parnas and Weiss (1985) found. Knight and Myers realized that there is generally too much material to check in a single inspection activity. Instead of having one inspection meeting as in the Fagan Inspection, the Phased Inspection technique involves conducting an inspection as series of tightly focused steps (phases) instead of in one inspection meeting.

Each inspection phase may involve either a single or multiple inspectors and each phase is concerned with inspecting only one specific quality goal. The inspector responsible for checking for a particular goal assumes that other quality goals either have been checked by other people in a previous inspection phase or that they will be checked in an upcoming inspection phase. This allows for inspectors, individual or group, to remain focused on reviewing one specific aspect of a work product instead of having to try to examine the entire product. For example, a Phased Inspection of source code may consist of four inspection phases: a source code format inspection, a documentation (code comment) inspection, a module-internal logic inspection, and a module-external interface inspection. In a Fagan Inspection, all of the concerns would be addressed in one meeting and source code formatting issues may end up distracting attention away from module-internal issues. The premise behind Phased Inspections is to provide more focus in each of the inspection phases to result in a more effective overall inspection effort.

Although Knight and Myers (1991) conclude that Phased Inspections are an improvement over the well-accepted Fagan Inspection, Knight and Myers only mention that preliminary data supports their conclusion. The preliminary data mentioned was the result of an experiment conducted for a masters thesis, but the data does not appear in a published peer-reviewed journal. As with Active Reviews, Phased Inspections have some level of acceptance in the software engineering community based on the number of citations in the literature on inspections, but no empirical evidence supports inspections, of any form being predictably effective.

Recent Trends in Inspections
Recognizing that major improvements could be made to the traditional Fagan Inspection technique, more recent literature presents both techniques specific for the domain of the system and techniques arising as a result of the evolving nature of software projects. In the last decade, the software industry has shifted from developing all applications in-house to either outsourcing part or most of their software development projects. In addition, software development teams within an organization may not be co-located, but rather be distributed across the country or around the world. The nature of the software applications themselves has evolved in the past decade as well: instead of having standalone applications that are completely self-contained, many applications now consist of multiple components and use services external to the application. For example, applications developed using rapid development frameworks and code generators (as in Microsoft Visual Studio), applications that use open-source components, applications that are developed for the World Wide Web. Each of these examples present cases where conducting a traditional inspection only makes sense if the inspection is tailored particularly for the application.

Stellman and Greene (2006) detail a variant of the Fagan Inspection technique for software development in situations that involve outsourcing as most inspection techniques described in the literature assume that all software is developed in house. Similarly, Tom Gilb, an author of a textbook on classical (Fagan) inspection techniques (Gilb, Graham, & Finzi, 1993), now has new material that describes inspection techniques for Agile software projects (Gilb & Gilb, 2007). Additional research in Agile Software Methods (Phongpaibul & Boehm, 2006) shows that a separate inspection activity may not be necessary, but may rather be counter to the objectives of the Agile Methodology. Some instances of the Agile Methodology call for Pair Programming (Cockburn & Williams, 2000), where programmers work together as a pair while they are coding. The rationale is that one person will be writing code, and the other will be both helping and checking as the code is being written. In essence, the second person of the pair acts as an inspector (as in Fagan Inspections) while the code is being written and asking questions (as in Active Reviews). Since these authors claim that Pair Programming is in effect combining the coding activity with the inspection activity, a separate inspection activity or process is not needed. Although this claim may be true for code, traditional inspections may still play a role in the Agile Methodology as only programming is done in pairs. That is, there are still other artifacts and work products developed in the Agile Methodology (artifacts that correspond to traditional requirements specifications, design documentation, and test plans) that still need to be verified and validated.

The Search for Empirical Evidence
Where the results of Fagan’s original study are often cited, where inspections can detect approximately 50 to 90 percent of software errors, few studies since then have been able to state conclusions based on empirical evidence, rather than on the results of a case study. This is likely due to the fact that most studies, including Fagan’s original study, are conducted on real-world production applications and that establishing the components necessary for a valid experimental design. For instance, in a production project, few organizations would have the resources allocated or the motivation to conduct a parallel inspection effort just to make comparisons. Likewise, an academic study of a theoretical application would not reflect the scale and issues involved in developing production software. In either case, results obtained from a case study, including Fagan’s original study, reflect the situation under study, including the software development process and the people employed on the project.

Given these limitations, there have been some recent studies that attempt to provide empirical data to help evaluate the effectiveness of inspections. Porter, Siy, Marcus, and Votta (1998) focus on creating an model not to evaluate inspections directly, but to rather identify the sources of variation in inspection activities. Identifying the sources of variation allows for software engineers to have a greater understanding of what specifically determines the effectiveness of a software inspection effort to both maximize the defect detection rate and to make the defect detection rate more consistent and predictable. Porter et al (1998) found that the inputs to the inspection activity created the most variation in an inspection effort, compared to the process by which the inspection was conducted and the structure of the inspection team.

In addition to evaluating inspections from a software engineering point of view, Biffl and Halling (2003) conducted a cost-benefit analysis on software inspections to provide more perspective from a software project management point of view. Although many software engineers would agree that inspections are an essential and beneficial tool in the verification and validation process, inspections are expensive activities in a project due to the number of people involved in an inspection team and the amount of time spent preparing and conducting the inspection activities. A cost benefit analysis views inspection activities as an investment in a software project. That is, Biffl and Halling (2003) sought to understand how to allocate sufficient resources to perform inspections that result in beneficial returns through effective and early defect detection, while not overspending effort and resources that result in diminishing returns. Although the authors did not explicitly state it in their conclusions, it is evident that the effectiveness-versus-cost relationship of software inspections follows the Pareto principle, better known as the 80-20 rule. This observation is supported by the Biffl and Halling’s (2003) conclusion that investing a moderate amount into a software inspection effort results in significant detection of defects while investing beyond that amount results in limited gains for the effort spent.

Conclusions
As versatile as software inspections are throughout the development lifecycle, the effectiveness of inspections in detecting errors is not necessarily guaranteed by their execution alone. The effectiveness of a software inspection depends on how the inspection is conducted. The majority of the literature on software inspections shows ways to improve how software inspections are performed, using the Fagan Inspection as the baseline. However, few facts about inspections have been proven empirically through solid experimental design.

Researchers and practitioners often cite Fagan’s original defect detection rates of 50-90 percent detection using inspections, or similarly state that inspections will find the majority of defects in work products. However, few have shown improvements in Fagan’s original results: either by increasing the percentage of errors detected or by narrowing the variance between moderately and highly effective inspections. Instead, improvements have been focused around matching inspection techniques to the current state of the software industry in terms of how software teams work together (processes) and the types of software they are working on (products). This may be discouraging for some as without solid empirical evidence, one wonders how one can make decisions on inspections in verification and validation activities.

There is no silver bullet when it comes to software inspections, and as of today, there is no “best” inspection technique and no easy formula to calculate the return on investment when it comes to conducting inspections. However, what software engineers and software project managers have at their disposal are a library of software inspection techniques they can add to their toolkit and many rules of thumb on when to conduct inspections and how much effort should be put into them. As with many other issues in software engineering, the software engineer and project manager should view inspections as a customizable tool they can use to achieve the objectives of detecting errors and detecting them early in the lifecycle to help reach the overall goals of delivering a high quality product at the lowest possible cost by first assessing the situation, knowing the inspection tools they have available, and choosing (or synthesizing) the right ones for the situation.

References
Biffl, S., & Halling, M. (2003). Investigating the Defect Detection Effectiveness and Cost Benefit of Nominal Inspection Teams. IEEE Transactions on Software Engineering, 29(5), 385-397.

CeBASE: NSF Center for Empirically Based Software Engineering. (2003). eWorkshop on Software Inspections and Pair Programming. Retrieved on December 1, 2007, from http://www.cebase.org/www/home/index.htm.

Cockburn, A., & Williams, L. (2000). The Costs and Benefits of Pair Programming. Proceedings of the First International Conference on Extreme Programming and Flexible Processes in Software Engineering.

Fagan, M. E. (1976). Design and Code Inspections to Reduce Errors in Program Development. IBM Systems Journal, 15(3).

Fagan, M. E. (1986). Advances in Software Inspections. IEEE Transactions on Software Engineering, 12(7).

Glib, T., Graham, D., & Finzi, S. (1993). Software Inspection. Boston, MA: Addison-Wesley.

Glib, T., & Gilb, K. (2007). Agile Inspection. Retreived on December 1, 2007, from

http://www.spipartners.nl/data/train_course/gilbinspect_agile_en.php.

Knight, J. C., & Myers, E. A. (1991). Phased Inspections and Their Implementation. ACM SIGSOFT Software Engineering Notes, 16(3), 29-35.

IEEE. (1998). IEEE Std 1028-1997: Standard for Software Reviews. Los Alamitos, CA: IEEE Computer Society Press.

Parnas, D. L., & Weiss, D. M. (1985). Active Design Reviews: Principles and Practices. Proceedings of the 8th International Conference on Software Engineering.

Phongpaibul, M., & Boehm, B. (2006). An Empirical Comparison Between Pair Development and Software Inspection in Thailand. Proceedings of the 2006 ACM/IEEE International Symposium on Empirical Software Engienering.

Porter, A., Siy, H., Mockus, A., & Votta, L. Understanding the Sources of Variation in Software Inspections. ACM Transactions on Software Engineering and Methodology, 7(1), 41-79.

Software Engineering Institute. (2001). Software Inspections. Retreived on December 1, 2007, from http://www.sei.cmu.edu/str/descriptions/inspections_body.html.

Stellman, A., & Greene, J. (2006). Applied Software Project Management. Sebastopol, CA: O’Reilly Media.