Design for Six Sigma (DfSS) in Software 133 4.2 Software FMEA Failure mode and effects analysis (FMEA) is one of the well-known analysis methods having an established position in the traditional reliability analysis. The purpose of FMEA is to identify “UPFRONT” possible failure modes of the system components, evaluate their influences on system behaviour and propose proper countermeasures to suppress these effects. A failure mode and effects analysis (FMEA) can be described as a systematic way of identifying failure modes of a system, item or function, and evaluating the effects of the failure modes on the higher level. A bottom-up technique such as FMEA is an effective way to identify component failures or system mal-functions, and to “design rightly” the system under consideration (Pentti & Atte, 2002). The standard guidelines provided by the FMEA cannot be directly used and would have to be tailored for applying it to software. Typically the best definition for “Severity” would be the one that the software teams use for their problem report classifications. Similarly for “Occurrence” and “Detection” it is better that the teams use their own tailored guideline based on a simplistic criteria of “Very high” to “Very Low”. By nature, software failure modes generally are unknown—“software modules do not fail in the literal sense as hardware failure, they only display incorrect behaviour”—and depend on dynamic behaviour of the application. The aim of the FMEA is to then uncover those situations. The following are certain alerts/pitfalls/learning’s to be aware of when doing software FMEA:- 1) Use case explosion – Software due to its very nature has many permutations /combinations of inputs and outputs which could be prone to failures. Hence FMEA would soon run into thousands of use-case combinations of failure-modes. Hence it is advisable to focus on failure modes associated with CTQs, Critical components/modules/functionalities etc 2) Capturing “Requirements not meeting” as failure modes e.g. set not recording as a failure mode for a DVD recorder etc. Recording is a basic requirement itself of a recorder so listing it as failure mode at a global level would not help. Instead the failure mode should delve deeper into the features 3) Not having the appropriate subject matter experts in the analyses. Failure modes largely dependent on competence, hence knowledge of domain (not software engineering but rather the usage of product in actual environment) is crucial 4) Attempting to perform FMEA on 100% of the design or code instead of sampling the design/code most likely to cause a serious failure 5) Excluding hardware from the analysis or isolating the software from the rest of the system as many of the failures result from the combination and not software alone 6) Typically for software, the severity “SEV” would remain unchanged and it is mainly the occurrence and detection that can be improved. For e.g. a hang/crash in a normal user operation is a severity “A” failure mode translating to a value of 8 for SEV. By taking various actions, its occurrence can be reduced/ eliminated or detectability can be improved. However even after taking actions, the severity would remain unchanged 7) The occurrence “OCC” value can be tricky sometimes for software. In a product development environment, normally a test will be done on few devices say 5 to 10 and issues do not surface out. When long duration tests are conducted in the factory on a larger sample say 100 devices then the product starts failing. So OCC value could be different based on the sample taken and has to be accordingly adapted when validating the results 8) From software development life-cycle perspective, the DET value can take on different values for the same detection levels. For e.g. a control mechanism may have a high chance of detecting a failure mode making the DET value 4 as per the guideline. However based on whether that detection can happen in design itself or testing may vary the value. The team might give a higher vale for DET for something that can be detected only in testing as against that which can be detected in design. 4.3 Use of Statistics in software Often this is one of most important challenge when it comes to using concepts like DfSS for software. Many software requirements fall into the Yes/No, Pass/Fail category so limit setting is fuzzy. Most of them would become critical factors (CFs) and not CTQs in the “continuous data” sense  Predicting DPMO (defects per million opportunities) may be misleading (out of limits). This is because the specifications limits in cases like responsiveness are soft targets. Just because it takes 0.5 seconds more than Upper Specification Limit to start-up does not necessarily classify it as a defective product. In Six sigma terms anything beyond Upper spec limit and less than Lower spec limit becomes a defect  Random failures due to only software are rare due to which concept like Mean-Time- Between-Failures (MTBF) for software alone is questionable, however it makes sense at overall product level  No concept of samples – the same piece of code is corrected and used, so advanced statistical concepts have to be applied with discretion However this does not mean that statistical concepts cannot be applied at all.  The starting point is to challenge each specification to ensure if some numbers can be associated with it. Even abstract elements such as “Usability” can be measured as seen in section 3.5.2  For many of the software CTQs, the Upper limits and lower limits may not be hard targets, nevertheless it is a good to use them as such and relax it during the course of the development  The change in Z-scores over the releases would be more meaningful rather than absolute Z-scores  All Statistical concepts can be applied for the “Continuous CTQs”  Many of the Design of experiments in software would happen with discrete Xs due to nature of software. So often the purpose of doing these is not with the intent of generating a transfer function but more with a need to understand which “Xs” impact the Y the most – the cause and effect. So the Main effects plot and Interaction plots have high utility in such scenarios  The hypothesis tests such as t-Tests, F-Tests, ANOVA are useful in the Verify and Monitor phase to determine if indeed there have been statistical significant changes over the life cycle or from one product generation to next etc. Quality Management and Six Sigma134  Statistical Capability analysis to understand the variation on many of the CTQs in simulated environments as well as actual hardware can be a good starting point to design in robustness in the software system. 5. References Ajit Ashok Shenvi (2008). Design for Six Sigma : Software Product Quality, Proceedings of the 1st India Software Engineering Conference, pp. 97-106, ISBN:978-1-59593-917-3, Hyderabad, India, February 19 - 22, 2008. ISEC '08. ACM, New York, NY, DOI= http://doi.acm.org/10.1145/1342211.1342231 Haapanen Pentti & Helminen Atte, Stuk-yto-tr 190/August 2002. Failure modes and effects analysis of software based-automation systems Jeannine M. Siviy and Eileen C. Forrester. (2004). Accelerating CMMi adoption using Six Sigma,Carnegie Mellon Software Engineering Institute Jeannine M. Siviy (SEI), Dave Halowell (Six Sigma advantage). 2005. Bridging the gap between CMMi & Six Sigma Training. Carnegie Mellon Sw Engineering Institute Jiantao Pan. 1999. Software Reliability. Carnegie Mellon http://www.ece.cmu.edu/~koopman/des_s99/sw_reliability/ Minitab tool – Statistical tool. http://www.minitab.com Philips DFSS training material for Philips. 2005. SigMax Solutions LLC, USA Statistical Process Control for Software: Fill the Gap 135 Statistical Process Control for Software: Fill the Gap Maria Teresa Baldassarre, Nicola Boffoli and Danilo Caivano X Statistical Process Control for Software: Fill the Gap Maria Teresa Baldassarre, Nicola Boffoli and Danilo Caivano University of Bari Italy 1. Introduction The characteristic of software processes, unlike manufacturing ones, is that they have a very high human-centered component and are primarily based on cognitive activities. As so, each time a software process is executed, inputs and outputs may vary, as well as the process performances. This phenomena is better identified in literature with the terminology of “Process Diversity” (IEEE, 2000). Given the characteristics of a software process, its intrinsic diversity implies the difficulty to predict, monitor and improve it, unlike what happens in other contexts. In spite of the previous observations, Software Process Improvement (SPI) is a very important activity that cannot be neglected. To face these problems, the software engineering community stresses the use of measurement based approaches such as QIP/GQM (Basili et al., 1994) and time series analysis: the first approach is usually used to determine what improvement is needed; the time series analysis is adopted to monitor process performances. As so, it supports decision making in terms of when the process should be improved, and provides a manner to verify the effectiveness of the improvement itself. A technique for time series analysis, well-established in literature, which has given insightful results in the manufacturing contexts, although not yet in software process ones is known as Statistical Process Control (SPC) (Shewhart, 1980; Shewhart, 1986). The technique was originally developed by Shewhart in the 1920s and then used in many other contexts. The basic idea it relies on consists in the use of so called “control charts” together with their indicators, called run tests, to: establish operational limits for acceptable process variation; monitor and evaluate process performances evolution in time. In general, process performance variations are mainly due to two types of causes classified as follows:  Common cause variations: the result of normal interactions of people, machines, environment, techniques used and so on.  Assignable cause variations: arise from events that are not part of the process and make it unstable. In this sense, the statistically based approach, SPC, helps determine if a process is stable or not by discriminating between common cause variation and assignable cause variation. We can classify a process as “stable” or “under control” if only common causes occur. More precisely, in SPC data points representing measures of process performances are collected. 8 Quality Management and Six Sigma136 These values are then compared to the values of central tendency, upper and lower limit of admissible performance variations. While SPC is a well established technique in manufacturing contexts, there are only few works in literature (Card, 1994; Florac et al., 2000; Weller, 2000(a); Weller, 2000(b); Florence, 2001; Sargut & Demirors, 2006; Weller, & Card. 2008; Raczynski & Curtis, 2008) that present successful outcomes of SPC adoption to software. In each case, not only are there few cases of successful applications but they don’t clearly illustrate the meaning of control charts and related indicators in the context of software process application. Given the above considerations, the aim of this work is to generalize and put together the experiences collected by the authors in previous studies on the use of Statistical Process Control in the software context (Baldassarre et al, 2004; Baldassarre et al, 2005; Caivano 2005; Boffoli, 2006; Baldassarre et al, 2008; Baldassarre et al, 2009) and present the resulting stepwise approach that: starting from stability tests, known in literature, selects the most suitable ones for software processes (tests set), reinterprets them from a software process perspective (tests interpretation) and suggest a recalculation strategy for tuning the SPC control limits. The paper is organized as follows: section 2 briefly presents SPC concepts and its peculiarities; section 3 discusses the main differences and lacks of SPC for software and presents the approach proposed by the authors; finally, in section 4 conclusions are drawn. 2. Statistical Process Control: Pills Statistical Process Control (SPC) (Shewhart, 1980; Shewhart, 1986) is a technique for time series analysis. It was developed by Shewhart in the 1920s and then used in many contexts. It uses several “control charts” together with their indicators to establish operational limits for acceptable process variation. By using few data points, it is able to dynamically determine an upper and lower control limit of acceptable process performance variability. Such peculiarity makes SPC a suitable instrument to detect process performance variations. Process performance variations are mainly due to: common cause variations (the result of normal interactions of people, machines, environment, techniques used and so on); assignable cause variations (arise from events that are not part of the process and make it unstable). A process can be described by measurable characteristics that vary in time due to common or assignable cause variations. If the variation in process performances is only due to common causes, the process is said to be stable and its behavior is predictable within a certain error range; otherwise an assignable cause (external to the process) is assumed to be present and the process is considered unstable. A control chart usually adopts an indicator of the process performances central tendency (CL), an upper control limit (UCL = CL+3sigma) and a lower control limit (LCL = CL-3sigma). Process performances are tracked overtime on a control chart, and if one or more of the values fall outside these limits, or exhibit a “non random” behavior, an assignable cause is assumed to be present. Fig. 1. Example of SPC charts (X charts) “Sigma” is calculated by using a set of factors tabulated by statisticians (for more details refer to (Wheeler & Chambers, 1992)) and it is based on statistical reasoning, simulations carried out and upon the heuristic experience that: “it works”. A good theoretical model for a control chart is the normal distribution shown in figure 2 where: the percentage values reported express the percentage of observations that fall in the corresponding area;  is the theoretical mean;  is the theoretical standard deviation. In the [-3, +3] interval, fall 99.73% (i.e. 2.14 + 13.59 + 34.13 + 34.13 + 13.59 + 2.14) of the total observations. Thus only the 0,27 % of the observations is admissible to fall outside the [-3, +3] interval. Fig. 2. Normal distribution, the bell curve If we consider sigma in place of , the meaning and rational behind a control chart results clear. For completeness it is necessary to say that the normal distribution is only a good theoretical model but, simulations carried out have shown that independently from the data distribution, the following rules of thumb work:  Rule1: from 60% to 75% of the observations fall in the [CL-sigma, CL+1sigma]  Rule2: from 90% to 98% of the observations fall in the [CL-2sigma, CL+2sigma]  Rule3: from 99% to 100% of the observations fall in the [CL-3sigma, CL+3sigma] Statistical Process Control for Software: Fill the Gap 137 These values are then compared to the values of central tendency, upper and lower limit of admissible performance variations. While SPC is a well established technique in manufacturing contexts, there are only few works in literature (Card, 1994; Florac et al., 2000; Weller, 2000(a); Weller, 2000(b); Florence, 2001; Sargut & Demirors, 2006; Weller, & Card. 2008; Raczynski & Curtis, 2008) that present successful outcomes of SPC adoption to software. In each case, not only are there few cases of successful applications but they don’t clearly illustrate the meaning of control charts and related indicators in the context of software process application. Given the above considerations, the aim of this work is to generalize and put together the experiences collected by the authors in previous studies on the use of Statistical Process Control in the software context (Baldassarre et al, 2004; Baldassarre et al, 2005; Caivano 2005; Boffoli, 2006; Baldassarre et al, 2008; Baldassarre et al, 2009) and present the resulting stepwise approach that: starting from stability tests, known in literature, selects the most suitable ones for software processes (tests set), reinterprets them from a software process perspective (tests interpretation) and suggest a recalculation strategy for tuning the SPC control limits. The paper is organized as follows: section 2 briefly presents SPC concepts and its peculiarities; section 3 discusses the main differences and lacks of SPC for software and presents the approach proposed by the authors; finally, in section 4 conclusions are drawn. 2. Statistical Process Control: Pills Statistical Process Control (SPC) (Shewhart, 1980; Shewhart, 1986) is a technique for time series analysis. It was developed by Shewhart in the 1920s and then used in many contexts. It uses several “control charts” together with their indicators to establish operational limits for acceptable process variation. By using few data points, it is able to dynamically determine an upper and lower control limit of acceptable process performance variability. Such peculiarity makes SPC a suitable instrument to detect process performance variations. Process performance variations are mainly due to: common cause variations (the result of normal interactions of people, machines, environment, techniques used and so on); assignable cause variations (arise from events that are not part of the process and make it unstable). A process can be described by measurable characteristics that vary in time due to common or assignable cause variations. If the variation in process performances is only due to common causes, the process is said to be stable and its behavior is predictable within a certain error range; otherwise an assignable cause (external to the process) is assumed to be present and the process is considered unstable. A control chart usually adopts an indicator of the process performances central tendency (CL), an upper control limit (UCL = CL+3sigma) and a lower control limit (LCL = CL-3sigma). Process performances are tracked overtime on a control chart, and if one or more of the values fall outside these limits, or exhibit a “non random” behavior, an assignable cause is assumed to be present. Fig. 1. Example of SPC charts (X charts) “Sigma” is calculated by using a set of factors tabulated by statisticians (for more details refer to (Wheeler & Chambers, 1992)) and it is based on statistical reasoning, simulations carried out and upon the heuristic experience that: “it works”. A good theoretical model for a control chart is the normal distribution shown in figure 2 where: the percentage values reported express the percentage of observations that fall in the corresponding area;  is the theoretical mean;  is the theoretical standard deviation. In the [-3, +3] interval, fall 99.73% (i.e. 2.14 + 13.59 + 34.13 + 34.13 + 13.59 + 2.14) of the total observations. Thus only the 0,27 % of the observations is admissible to fall outside the [-3, +3] interval. Fig. 2. Normal distribution, the bell curve If we consider sigma in place of , the meaning and rational behind a control chart results clear. For completeness it is necessary to say that the normal distribution is only a good theoretical model but, simulations carried out have shown that independently from the data distribution, the following rules of thumb work:  Rule1: from 60% to 75% of the observations fall in the [CL-sigma, CL+1sigma]  Rule2: from 90% to 98% of the observations fall in the [CL-2sigma, CL+2sigma]  Rule3: from 99% to 100% of the observations fall in the [CL-3sigma, CL+3sigma] Quality Management and Six Sigma138 The control limits carried out using SPC are based on a process observation and they are expression of it. They are not the result of expert judgment and, furthermore, they can be clearly obtained. In general, control charts are used as follows: samples are taken from the process, statistics (for example, average and range) are calculated and plotted on charts, and the results are interpreted with respect to process limits or, as they are known in SPC terminology, control limits. Control limits are the limits within which the process operates under normal conditions. They tell us how far we can expect sample values to stray from the average given the inherent variability of the process or, to use the SPC terms, the magnitude of common-cause variation. Data points beyond the control limits or other unusual patterns indicate a special-cause variation. 3. SPC for Software Software processes and manufacturing ones present deep differences that the use of SPC in software cannot exempt from considering. Moreover, according to the discussions in (Jalote, 2002(a); Eickelmann & Anant, 2003) we can consider three main differences between manufacturing and software processes that have to be kept in mind in order to assure a more appropriate use of SPC in software context in terms of control charts, run test indicators, anomalies interpretation and control limits calculation. Measurement of Software Processes. In manufacturing, the observed and actual number of defects is not significantly different. In software development, these two numbers routinely vary significantly. Possible causes for extreme variation in software measurement include the following:  People are the software production process.  Software measurement might introduce more variation than the process itself.  Size metrics do not count discrete and identical units. Such extreme variations in software processes need different indicators for the anomalies detection and more specific interpretations. Product Control and Product Rework. The primary focus of using SPC control charts in manufacturing is to bring the process back in control by removing assignable causes and minimize as much as possible the future production losses. In the manufacturing process when an anomaly occurs the products usually do not conform to the expected standards and therefore, must be discarded. On the other hand, in the software process the product can be “reworked”. For example, when using control charts for an inspection process, if a point falls outside the control limits, besides the process improvement actions like improving the checklist, inevitably, product improvement actions like re-reviews, scheduling extra testing also occurs. With software processes, besides improving the process, an important objective of using control charts is to also control the product. In (Gardiner & Montgomery, 1987), which is perhaps the first paper on the use of SPC in software, Gardiner and Montgomery suggest "rework" as one of the three actions that management should carry out if a point falls outside the control limits. The use described in (Ebenau, 1994) clearly shows this aspect of product control. The survey of high maturity organizations also indicates that project managers also use control charts for project-level control (Jalote, 2002(b)). Due to this product-control, project managers are more likely to want test indicators and interpretations that highlight potential warning signals, rather than risk to miss such signals, even if it means more false alarms. Shutdown and Startup is “Cheaper”. The cost parameters that affect the selection of control limits are likely to be quite different in software processes. For example, if a manufacturing process has to be stopped (perhaps because a point falls outside the control limits), the cost of doing so can be quite high. In software, on the other hand, the cost of stopping a process is minimal as elaborate "shutdown" and "startup" activities are not needed. Similarly, the cost of evaluating a point that falls outside the control limits is likely to be very different in software processes as compared to manufacturing ones. For these reasons the control limits could be recalculated more often than in manufacturing processes. Due to these differences, it is reasonable to assume that, to get the best results, control charts, the use of the indicators and their interpretation, as well as the tuning of process limits, will need to be adapted to take into account the characteristics of software processes. Finally, in spite of the rather simple concepts underlying statistical process control, it is rarely straightforward to implement (Card, 1994). The main lacks for software processes are listed below: Focus on individual or small events. The indicators generally used in SPC highlight assignable causes related to the individual events. However the high variability of a software process and its predominant human factor make such indicators ineffective because they usually discover occasional variations due to passing phenomena that should be managed as false positives (false alarms). Therefore the SPC indicators, in software processes, should detect the assignable variations and then also interpret them if occasional variations (as false positives) or occurred changes in the process (in the manufacturing processes the passing phenomena are very rare). For such reasons the control charts should be constructed with a view toward detecting process trends rather than identifying individual nonconforming events (Figure 3). Fig. 3. SPC variations tree Statistical Process Control for Software: Fill the Gap 139 The control limits carried out using SPC are based on a process observation and they are expression of it. They are not the result of expert judgment and, furthermore, they can be clearly obtained. In general, control charts are used as follows: samples are taken from the process, statistics (for example, average and range) are calculated and plotted on charts, and the results are interpreted with respect to process limits or, as they are known in SPC terminology, control limits. Control limits are the limits within which the process operates under normal conditions. They tell us how far we can expect sample values to stray from the average given the inherent variability of the process or, to use the SPC terms, the magnitude of common-cause variation. Data points beyond the control limits or other unusual patterns indicate a special-cause variation. 3. SPC for Software Software processes and manufacturing ones present deep differences that the use of SPC in software cannot exempt from considering. Moreover, according to the discussions in (Jalote, 2002(a); Eickelmann & Anant, 2003) we can consider three main differences between manufacturing and software processes that have to be kept in mind in order to assure a more appropriate use of SPC in software context in terms of control charts, run test indicators, anomalies interpretation and control limits calculation. Measurement of Software Processes. In manufacturing, the observed and actual number of defects is not significantly different. In software development, these two numbers routinely vary significantly. Possible causes for extreme variation in software measurement include the following:  People are the software production process.  Software measurement might introduce more variation than the process itself.  Size metrics do not count discrete and identical units. Such extreme variations in software processes need different indicators for the anomalies detection and more specific interpretations. Product Control and Product Rework. The primary focus of using SPC control charts in manufacturing is to bring the process back in control by removing assignable causes and minimize as much as possible the future production losses. In the manufacturing process when an anomaly occurs the products usually do not conform to the expected standards and therefore, must be discarded. On the other hand, in the software process the product can be “reworked”. For example, when using control charts for an inspection process, if a point falls outside the control limits, besides the process improvement actions like improving the checklist, inevitably, product improvement actions like re-reviews, scheduling extra testing also occurs. With software processes, besides improving the process, an important objective of using control charts is to also control the product. In (Gardiner & Montgomery, 1987), which is perhaps the first paper on the use of SPC in software, Gardiner and Montgomery suggest "rework" as one of the three actions that management should carry out if a point falls outside the control limits. The use described in (Ebenau, 1994) clearly shows this aspect of product control. The survey of high maturity organizations also indicates that project managers also use control charts for project-level control (Jalote, 2002(b)). Due to this product-control, project managers are more likely to want test indicators and interpretations that highlight potential warning signals, rather than risk to miss such signals, even if it means more false alarms. Shutdown and Startup is “Cheaper”. The cost parameters that affect the selection of control limits are likely to be quite different in software processes. For example, if a manufacturing process has to be stopped (perhaps because a point falls outside the control limits), the cost of doing so can be quite high. In software, on the other hand, the cost of stopping a process is minimal as elaborate "shutdown" and "startup" activities are not needed. Similarly, the cost of evaluating a point that falls outside the control limits is likely to be very different in software processes as compared to manufacturing ones. For these reasons the control limits could be recalculated more often than in manufacturing processes. Due to these differences, it is reasonable to assume that, to get the best results, control charts, the use of the indicators and their interpretation, as well as the tuning of process limits, will need to be adapted to take into account the characteristics of software processes. Finally, in spite of the rather simple concepts underlying statistical process control, it is rarely straightforward to implement (Card, 1994). The main lacks for software processes are listed below: Focus on individual or small events. The indicators generally used in SPC highlight assignable causes related to the individual events. However the high variability of a software process and its predominant human factor make such indicators ineffective because they usually discover occasional variations due to passing phenomena that should be managed as false positives (false alarms). Therefore the SPC indicators, in software processes, should detect the assignable variations and then also interpret them if occasional variations (as false positives) or occurred changes in the process (in the manufacturing processes the passing phenomena are very rare). For such reasons the control charts should be constructed with a view toward detecting process trends rather than identifying individual nonconforming events (Figure 3). Fig. 3. SPC variations tree Quality Management and Six Sigma140 Failure to investigate and act. Statistical process control only signals that a problem may exist. If you don’t follow through with a detailed investigation, like an audit, and follow-up corrective action, there is no benefit in using it. In these sense a larger set of anomalies indicators and a more precise anomalies interpretation is necessary. Incorrect computation of control limits. Several formulas exist for computing control limits and analyzing distributions in different situations. But although they are straightforward, without proper background, it is easy to make mistakes. Such mistakes might concern:  the correct calculation of control limits  the appropriate timing for the recalculation of control limits (“tuning” activities) In order to mitigate such differences and face these issues, in the past the authors have proposed and experimented an SPC framework for software processes (Baldassarre et al., 2007). Such framework, based on the software process peculiarities, proposes the most appropriate control charts, a set of indicators (run-test set) and related interpretations (run- test interpretation) in order to effectively monitor process variability. When such indicators are used, SPC is able to discover software process variations and discriminate between them. For these reasons such indicators:  are able to detect process trends rather than identify individual nonconforming events (i.e. occasional variations that in software processes would be considered like the false alarms)  enable to discover assignable variations and address some quality information about “what happens” in the process. Thereby such framework supports the manager during the causes-investigation activities. Furthermore, our framework faces problems related to incorrect computation of control limits and proposes “when” and “how” to recalculate the SPC control limits (the “tuning” activities) that supports manager in:  Choosing the control charts and measurement object to use in SPC analysis  Selecting the appropriate data-points, building the Reference Set and calculating the control limits needed for monitoring process variations  Monitoring the process variations and detecting run-tests failures  Evaluating the assignable events occurred and then undertaking the appropriate actions (for example recalculating the control limits) Figure 4 summarizes the steps for applying the framework: first, process characterization is carried out, i.e. a process characteristic to monitor is observed over time, and related data points are collected; the appropriate control chart is selected and upper and lower control limits are calculated (Step 1); secondly anomaly detection occurs, i.e. each new data point observed is plotted on the chart, keeping control limits and central line the same; the set of run tests (RT1…RT8) is executed and anomalies are detected each time a test fails (Step 2); at this point, causes investigation is carried out, i.e. the cause of the anomaly pointed out is investigated in order to provide an interpretation (Step 3). Finally, according to the process changes occurred and identified in the previous step, appropriate tuning actions are applied to tune the sensibility of the monitoring activity and adapt it to the new process performances (Step 4). Fig. 4. SPC based Process Monitoring guidelines 3.1 Process Characterization A reference set must be determined in order to characterize a process, i.e. a set of observations that represent the process performances and do not suffer from exceptional causes. In short, the reference set provides a reference point to compare the future performances with. After determining the reference set, each following observation must be traced on the control chart obtained and then the set of tests included in the test set must be carried out in order to identify if eventual exceptional causes come up. More precisely, the following two steps are executed: • Identify the measurement object • Identify the reference set Identify the measurement object. The process to evaluate is identified along with the measurement characteristics that describe the performances of interest. The most appropriate control charts for the phenomena being observed are selected. There are charts for variables data (measurement data such as length, width, thickness, and moisture content) and charts for attributes data (“counts” data such as number of defective units in a sample). Statistical Process Control for Software: Fill the Gap 141 Failure to investigate and act. Statistical process control only signals that a problem may exist. If you don’t follow through with a detailed investigation, like an audit, and follow-up corrective action, there is no benefit in using it. In these sense a larger set of anomalies indicators and a more precise anomalies interpretation is necessary. Incorrect computation of control limits. Several formulas exist for computing control limits and analyzing distributions in different situations. But although they are straightforward, without proper background, it is easy to make mistakes. Such mistakes might concern:  the correct calculation of control limits  the appropriate timing for the recalculation of control limits (“tuning” activities) In order to mitigate such differences and face these issues, in the past the authors have proposed and experimented an SPC framework for software processes (Baldassarre et al., 2007). Such framework, based on the software process peculiarities, proposes the most appropriate control charts, a set of indicators (run-test set) and related interpretations (run- test interpretation) in order to effectively monitor process variability. When such indicators are used, SPC is able to discover software process variations and discriminate between them. For these reasons such indicators:  are able to detect process trends rather than identify individual nonconforming events (i.e. occasional variations that in software processes would be considered like the false alarms)  enable to discover assignable variations and address some quality information about “what happens” in the process. Thereby such framework supports the manager during the causes-investigation activities. Furthermore, our framework faces problems related to incorrect computation of control limits and proposes “when” and “how” to recalculate the SPC control limits (the “tuning” activities) that supports manager in:  Choosing the control charts and measurement object to use in SPC analysis  Selecting the appropriate data-points, building the Reference Set and calculating the control limits needed for monitoring process variations  Monitoring the process variations and detecting run-tests failures  Evaluating the assignable events occurred and then undertaking the appropriate actions (for example recalculating the control limits) Figure 4 summarizes the steps for applying the framework: first, process characterization is carried out, i.e. a process characteristic to monitor is observed over time, and related data points are collected; the appropriate control chart is selected and upper and lower control limits are calculated (Step 1); secondly anomaly detection occurs, i.e. each new data point observed is plotted on the chart, keeping control limits and central line the same; the set of run tests (RT1…RT8) is executed and anomalies are detected each time a test fails (Step 2); at this point, causes investigation is carried out, i.e. the cause of the anomaly pointed out is investigated in order to provide an interpretation (Step 3). Finally, according to the process changes occurred and identified in the previous step, appropriate tuning actions are applied to tune the sensibility of the monitoring activity and adapt it to the new process performances (Step 4). Fig. 4. SPC based Process Monitoring guidelines 3.1 Process Characterization A reference set must be determined in order to characterize a process, i.e. a set of observations that represent the process performances and do not suffer from exceptional causes. In short, the reference set provides a reference point to compare the future performances with. After determining the reference set, each following observation must be traced on the control chart obtained and then the set of tests included in the test set must be carried out in order to identify if eventual exceptional causes come up. More precisely, the following two steps are executed: • Identify the measurement object • Identify the reference set Identify the measurement object. The process to evaluate is identified along with the measurement characteristics that describe the performances of interest. The most appropriate control charts for the phenomena being observed are selected. There are charts for variables data (measurement data such as length, width, thickness, and moisture content) and charts for attributes data (“counts” data such as number of defective units in a sample). Quality Management and Six Sigma142 Fig. 5. Decision Tree for Control Chart Selection In software processes, where data points are not so frequent, generally, each data point is individually plotted and evaluated. Hence, charts that work on single observation points (like the XmR or the U charts) are more suitable for software (Gadiner & Montgomery, 1987; Weller, 2000(a); Zultner, 1999) and are the most commonly used charts, as reported in the survey (Radice, 2000). On the other hand, in manufacturing, the Xbar-R charts, which employ a sampling based technique, is most commonly used. Consequently, modeling and analysis for selecting control limits optimal performance has also focused on Xbar-R charts. Identify the Reference Set. Identifying the “reference set” is a mandatory activity for correctly monitoring and evaluating the evolution of process performances in time. It consists in a set of observations of the measurement characteristics of interest. The set expresses the “normal” process behaviour, i.e. the process performances supposing that the variations are determined only by common causes. As so, first, process performances in time must be measured and, CL and control limits must be calculated. The observations collected are then traced on the control charts and the tests included in the test set are carried out. If no anomalies are detected, the process can be considered stable during the observation period. The observations collected along with the CL and control limits values become the reference set. If one of the tests points out anomalies, then the process is not stable. As so, it must be further investigated. The exceptional causes, if present, need to be eliminated from the process and, the CL and control limits must be recalculated. This is repeated until a period of observed data points indicate a stable process, i.e. until a new reference set can be determined. In an X chart: each point represents a single value of the measurable process characteristic under observation; CLX is calculated as the average of the all available values; UCLX and LCLX are set at 3sigmaX around the CLX; sigmaX is the estimated standard deviation of the observed sample of values calculated by using a set of factors tabulated by statisticians (for more details refer to (Wheeler & Chambers, 1992; Park, 2007)). In a mR chart: each point represents a moving range (i.e. the absolute difference between a successive pair of observations); CLmR, is the average of the moving ranges; UCLmR = CLmR+3sigmamR and LCLmR=0; sigmamR is the estimated standard deviation of the moving ranges sample. For example, given a set of 15 observations X = {213.875, 243.600, 237.176, 230.700, 209.826, 226.375, 167.765, 242.333, 233.250, 183.400, 201.882, 182.133, 235.000, 216.800, 134.545}, the following values are determined: ii mi xx m mR       1 1 1 1 1 = 33.11 3sigma X = 2,660 * mR = 88.07 CL X = X = 210.58 UCL X = X + 2,660 * mR = 298.64 LCL X = X - 2,660 * mR = 122.52 CL mR = mR =33,11 UCL mR = 3,268* mR =108,2 LCL mR = 0 Fig. 6. Example of Individual and moving ranges charts (XmR charts) 3.2 Anomalies Detection In software processes, one should look for systematic patterns of points instead of single point exceptions, because such patterns emphasize that the process performance has shifted or is shifting. This surely leads to more insightful remarks and observations. There is a set of tests for such patterns referred to as “run rules” or “run tests” (see (AT&T, 1956; Nelson, 1984; Nelson, Grant & Leavenworth, 1980; Shirland, 1993)) that aren’t well known (or used) in the software engineering community. Run-Test Description RT1: Three Sigma 1 point beyond a control limit (±3sigma) RT2: Two Sigma 2 out of 3 points in a row beyond (±2sigma) RT3: One Sigma 4 out of 5 points in a row beyond (±1sigma) RT4: Run above/belo w CL 7 consecutive points above or below the centreline RT5: Mixing/Overcontrol 8 points in a row on both sides of the centreline avoiding ±1sigma area RT6: Stratification 15 points in a row within ±1sigma area RT7: Oscillatory Trend 14 alternating up and down points in a row RT8: Linear Trend 6 points in a row steadily increasing or decreasing Table 1. Run-Test Set Details [...]... common and frequent in software processes It is the result of an 1 48 Quality Management and Six Sigma induced process improvement, such as the introduction of a new technology, or a spontaneous one, such as the maturation effect This test, give insightful remarks when it fails on R chart and it is interpreted jointly between X and R charts For example:  If R chart shows a decreasing trend as in Figure 8( d),... as the area between 2 and 3 times sigma above and below the center line; Zone B is defined as the area between 1 and 2 times sigma, and Zone C is defined as the area between the center line and 1 times sigma For the execution of the zone based tests, the distribution of the values in the charts need to be assumed as symmetrical around the mean This is not the case for mR charts and thus, in general,...Statistical Process Control for Software: Fill the Gap mR 143 1   xi 1  xi = 33.11 m1 i 1 m1 3sigmaX = 2,660 * CLX = X 88 .07 = 210. 58 UCLX = 2 98. 64 LCLX = mR = X X + 2,660 * - 2,660 * mR = mR = 122.52 CLmR = mR =33,11 UCLmR = 3,2 68* mR =1 08, 2 LCLmR = 0 Fig 6 Example of Individual and moving ranges charts (XmR charts) 3.2 Anomalies Detection In software processes, one should look for systematic... the selected SPC indicators (Table 1) in logical classes: sigma (RT1, RT2, RT3), limit (RT4, RT5, RT6) and trend (RT7, RT8), for details refer to (Baldassarre, 2004) 3.3.1 Sigma Tests They provide an “early” alarm indicator that must stimulate searching for possible assignable causes and, if the case, identify and eliminate them One and Two sigma tests point out a potential anomalous “trend” that “may”... remarks and observations There is a set of tests for such patterns referred to as “run rules” or “run tests” (see (AT&T, 1956; Nelson, 1 984 ; Nelson, Grant & Leavenworth, 1 980 ; Shirland, 1993)) that aren’t well known (or used) in the software engineering community Run-Test Description RT1: Three Sigma 1 point beyond a control limit (± 3sigma) RT2: Two Sigma 2 out of 3 points in a row beyond (± 2sigma) ... beyond (± 2sigma) RT3: One Sigma 4 out of 5 points in a row beyond (± 1sigma) RT4: Run above/below 7 consecutive points above or below the centreline CL RT5: Mixing/Overcontrol 8 points in a row on both sides of the centreline avoiding ± 1sigma area RT6: Stratification 15 points in a row within ± 1sigma area RT7: Oscillatory Trend 14 alternating up and down points in a row RT8: Linear Trend 6 points in... consecutive sample means are independent, and (3) that the distribution of means follows the normal distribution Statistical Process Control for Software: Fill the Gap 145 Fig 7 Run-tests set 3.2.1 Sigma Tests These tests point out the possible presence of an assignable cause The three sigma test can be applied to both, X and R charts The One and Two sigma tests are Zone Tests and thus they should not be applied... object No Action Table 3 Relationship between Process Changes and the necessary SPC Tuning Actions According to such relations and to “Process Changes” described through the “run-test interpretation” we have defined the following function: ψ: {Process Changes}  {Tuning Actions} “what happens” “what to do” 150 Quality Management and Six Sigma Ψ is defined so that it assigns the appropriate tuning actions... three sigma test is the most (and often the “only” one) used test in software engineering literature 3.2.2 Limit Tests All the tests included in this class use chart Zones and thus they are applicable to the X charts only 1 Run above or below the Centerline Test: This test watches for 7, 8 or 9 consecutive observations above or below the centerline The presence of such a run indicates 146 Quality Management. .. ± 1sigma area RT7: Oscillatory Trend 14 alternating up and down points in a row RT8: Linear Trend 6 points in a row steadily increasing or decreasing Table 1 Run-Test Set Details 144 Quality Management and Six Sigma As sigma, the run rules are based on "statistical" reasoning For example, the probability of any observation in an X control chart falling above the CL is at a glance equal to 0.51 Thus, . [CL -sigma, CL+ 1sigma]  Rule2: from 90% to 98% of the observations fall in the [CL- 2sigma, CL+ 2sigma]  Rule3: from 99% to 100% of the observations fall in the [CL- 3sigma, CL+ 3sigma] Quality Management. 242.333, 233.250, 183 .400, 201 .88 2, 182 .133, 235.000, 216 .80 0, 134.545}, the following values are determined: ii mi xx m mR       1 1 1 1 1 = 33.11 3sigma X = 2,660 * mR = 88 .07 CL X . 242.333, 233.250, 183 .400, 201 .88 2, 182 .133, 235.000, 216 .80 0, 134.545}, the following values are determined: ii mi xx m mR       1 1 1 1 1 = 33.11 3sigma X = 2,660 * mR = 88 .07 CL X