Using Statistical Confidence to Protect your Family

A helpful analogy  in communicating the concept of statistical reliability confidence is the “new airplane” example. Let’s say I am developing  an entirely new technology for airplanes. The airplane has an engine that has never been used before for air travel; a fusion engine. I tell the world that this new airplane with a fusion engine will have a reliability of 99.99999999%, the highest any airplane has ever had. It’s not possible to fully demonstrate this reliability until every single unit of this airplane has been produced, used to full life, and the full fleet is retired.  As long as one is still flying it can add or subtract from the reported reliability number.  So, how do we make decisions at product launch regarding the design’s reliability?  No products have yet to be produced or used by customers, so how can we trust the design? We use statistics to make projections of how a future field population of planes may perform based on our test data during development.  These projections are made in statistical confidence statements.  We may say that to approve launch of the airplane into production we need to have demonstrated a statistical confidence of 85% that the full airplane fleet will perform to its design reliability goal of 99.99999%. The 85% is measuring how well we have measured the design in addition to measuring the design itself. In reliability, this confidence is a mathematical calculation based on data.  But we can use the emotional definition of confidence as an analogy. In fact, that is why this mathematical method was labeled “confidence.” If I test my prototype of this plane and it performs perfectly, would you let your family be the first passengers on the first fusion plane off the production line?  No? Why not, the prototype performed perfectly in the test?  The reason is that there is very little data supporting that a larger population would perform in the same manner as that one prototype.  If we did the math using the statistical distributions, we would probably end up calculating that we have demonstrated a 2% confidence that a 50-unit fleet of this fusion airplane design may achieve its 99.99999% reliability once in the field. As we continue our testing program and measure the reliability of the design, that confidence will grow. Now, our confidence will grow in either of two ways.  If the product performs well, our confidence in a high reliability will grow. Alternatively, if the product performs poorly, we will grow confident in a low reliability. Either way, we are more confident that we can predict how the field population will perform, and make a decision if it is ready to be handed to the customer. The last piece is that we want to make these confidence calculations throughout the program.  It doesn’t do much good to only calculate the confidence in the design’s reliability at launch.  At that point, it is too late to do anything if the design is insufficient.  We establish interim confidence goals throughout the program.  They are determined based on a known likely trajectory of confidence growth during development that will result in the correct confidence at launch. -Adam