Demonstration Testing

Abstract

Chris and Fred discuss the whole idea of ‘demonstration testing’ … and how it may be ‘chronically’ overused or at least misunderstood. Heard of ‘demonstration testing’ and need to learn more?

Key Points

Join Chris and Fred as they discuss

Topics include:

What is ‘demonstration’ testing? Typically … ‘demonstration testing’ involves testing a system in ‘real world’ scenarios (no acceleration), without any assumed knowledge, using no models based on previous information. This might sound very ‘objective’ and ‘scientific’ … but it is very expensive and time-consuming. Think about something that is ‘really’ reliable. Say something has an MTBF of 10 years. How long does testing need to be to get enough failure points to be confident about the real MTBF?
Who likes ‘demonstration’ testing? Typically governmental, military and some commercial organizations.
But do we really ‘demonstrate’ reliability? Not really. For example, we typically do reliability demonstration testing to work out if we are going to purchase a large number of these systems. Which means we conduct reliability demonstration testing on a pre-production prototype. And what can really affect reliability? Manufacturing. So what is the point of testing a prototype that is not built using the ‘actual’ manufacturing process? Look at the United States Department of Defense (DoD) acquisition of the armored Joint Light Tactical Vehicle (JLTV). It passed reliability demonstration testing really well … it actually ‘demonstrated’ a reliability that was three times higher than the requirement. But now it is being introduced into service … it has a lot of reliability problems. Another example? Ever heard of the Hewlett-Packard (HP) digital camera? No. The prototype pass a reliability demonstration test – but the contract allowed the manufacturer to substitute any component it wanted to at will. And it rarely worked longer than a month.
So ‘demonstrating reliability’ is not really the same as making something reliable. Demonstrating reliability is often an arduous activity where we need to pass a test. And this means we aren’t testing to learn.

Enjoy an episode of Speaking of Reliability. Where you can join friends as they discuss reliability topics. Join us as we discuss topics ranging from design for reliability techniques to field data analysis approaches.