Specification Requires Testing… Really?

Abstract

Chris and Fred discuss scenarios where it is (for whatever reason) impossible to demonstrate reliability through testing. Some organizations think that if you can’t demonstrate through testing … then it can’t be a requirement that appears in a specification. So … does this mean that the customer can’t get something that is reliable if we can’t test for it? … can customers even ask for it? Of course they can. The customer knows what they want. Everyone needs to understand that you can verify performance through activities that don’t involve testing (… like analysis). This is how it works for nuclear power plants … so what makes your organization special? Listen to this podcast if this intrigues you.

Key Points

Join Chris and Fred as they discuss scenarios where organizations either struggle or cannot demonstrate reliability through testing. We engineers tend to like unequivocal or ‘certain’ information. The problem is … this doesn’t exist in the real world. Testing often has the ‘visage’ of objectivity, but will always involve uncertainty. Testing can have more uncertainty than analysis that incorporates ‘soft’ information such as expert judgment, physics of failure models and so on. This is because there is so much information out there if you choose to use it. So why do we think that testing results in ‘perfect’ information that no other information source can approach?

Topics include:

The specification summarizes what the customer’s wants. It is not a product of Quality Assurance (QA) or verification functions. The end.
There are plenty of examples where ‘highly reliable’ things cannot be tested for reliability. Nuclear power plants cannot be tested until failure. Neither can spacecraft, aircraft and many other things that need to be very reliable. How can they be allowed to operate?
Things you can’t test for still inform design. If your manager or leader is a robot, then perhaps testing is the only way you can influence design. But this doesn’t apply if we remember that engineers and designers are human beings.
Testing is a form of verification. Verification does not have to be testing. All you need is ‘evidence.’ Physics of failure models have been around for a long time. Evidence may be ‘soft’ and could be based on things like expert judgment. This is OK, if you take into consideration uncertainty.
… and you don’t need to have verification to influence design. If you constantly update the reliability analysis of your system (which doesn’t need testing), you can work out which components are driving failure. And this allows you to influence design.
Analysis is everything. You constantly analyze during design. By the time it comes to verification … the design is supposed to be done. So it is too late anyway.
Why do we have to act like we have forgotten everything we learned when it comes to testing? By this we mean that if you have information sources like the previous model’s operational data (for example), why wouldn’t you use this? You are wasting information, and by extension money if you don’t maximize what you have.

Enjoy an episode of Speaking of Reliability. Where you can join friends as they discuss reliability topics. Join us as we discuss topics ranging from design for reliability techniques, to field data analysis approaches.