TY - JOUR
T1 - Evaluating testing methods by delivered reliability
AU - Frankl, Phyllis G.
AU - Hamlet, Richard G.
AU - Littlewood, Bev
N1 - Funding Information:
This work was carried out, in part, during visits by Richard G. Hamlet and Phyllis G. Frankl to the Centre for Software Reliability, with support from EPSRC visiting fellowship Grant Nos. GR/K68134 and GR/L00445. Bev Littlewood and Lorenzo Strigini were funded, in part, by the European Commission via the ESPRIT Long Term Research Project 20072 DeVa. Phyllis G. Frankl was funded, in part, by the National Science Foundation under Grant No. CCR-9206910.
PY - 1998
Y1 - 1998
N2 - There are two main goals in testing software: 1 ) to achieve adequate quality (debug testing); the objective is to probe the software for defects so that these can be removed and 2) to assess existing quality (operational testing); the objective is to gain confidence that the software is reliable. The names are arbitrary, and most testing techniques address both goals to some degree. However, debug methods tend to ignore random selection of test data from an operational profile, while for operational methods this selection is all-important. Debug methods are thought, without any real proof, to be good at uncovering defects so that these can be repaired, but having done so they do not provide a technically defensible assessment of the reliability that results. On the other hand, operational methods provide accurate assessment, but may not be as useful for achieving reliability. This paper examines the relationship between the two testing goals, using a probabilistic analysis. We define simple models of programs and their testing, and try to answer theoretically the question of how to attain program reliability: Is it better to test by probing for defects as in debug testing, or to assess reliability directly as in operational testing, uncovering defects by accident, so to speak? There is no simple answer, of course. Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The "better" method delivers higher reliability after all test failures have been eliminated. This comparison extends previous work, where the measure was the probability of detecting a failure. Revealing special cases are exhibited in which each kind of testing is superior. Preliminary analysis of the distribution of the delivered reliability indicates that even simple models have unusual statistical properties, suggesting caution in interpreting theoretical comparisons. A
AB - There are two main goals in testing software: 1 ) to achieve adequate quality (debug testing); the objective is to probe the software for defects so that these can be removed and 2) to assess existing quality (operational testing); the objective is to gain confidence that the software is reliable. The names are arbitrary, and most testing techniques address both goals to some degree. However, debug methods tend to ignore random selection of test data from an operational profile, while for operational methods this selection is all-important. Debug methods are thought, without any real proof, to be good at uncovering defects so that these can be repaired, but having done so they do not provide a technically defensible assessment of the reliability that results. On the other hand, operational methods provide accurate assessment, but may not be as useful for achieving reliability. This paper examines the relationship between the two testing goals, using a probabilistic analysis. We define simple models of programs and their testing, and try to answer theoretically the question of how to attain program reliability: Is it better to test by probing for defects as in debug testing, or to assess reliability directly as in operational testing, uncovering defects by accident, so to speak? There is no simple answer, of course. Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The "better" method delivers higher reliability after all test failures have been eliminated. This comparison extends previous work, where the measure was the probability of detecting a failure. Revealing special cases are exhibited in which each kind of testing is superior. Preliminary analysis of the distribution of the delivered reliability indicates that even simple models have unusual statistical properties, suggesting caution in interpreting theoretical comparisons. A
KW - Debugging
KW - Reliability
KW - Software testing
KW - Statistical testing theory
UR - http://www.scopus.com/inward/record.url?scp=0001134764&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=0001134764&partnerID=8YFLogxK
U2 - 10.1109/32.707695
DO - 10.1109/32.707695
M3 - Article
AN - SCOPUS:0001134764
SN - 0098-5589
VL - 24
SP - 586
EP - 601
JO - IEEE Transactions on Software Engineering
JF - IEEE Transactions on Software Engineering
IS - 8
ER -