Statistical vs. Practical Significance
When we’re looking at results (like measures of a characteristic), we need to take care not to get too hung-up on what the statistics is trying to tell us. Yes, statistical tools are a good way for us to make decisions and the results can act as proof for us. But, there’s a practical, engineering side to results, too. We need to evaluate the statistical significance along with the practical significance.
We review an example and how to document it.
We use tools to help us make decisions and predict outcomes with some confidence. Some of the techniques are a way for us to apply a mathematical model, like statistics. Some are graphical organizers. And others, still, help a team of people organize and coordinate their ideas. They’re all tools for people to use to make decisions. Ultimately, we use the results of many inputs in order to make our best decisions.
It is important to distinguish between statistical significance and practical significance. Statistical significance simply means that we reject the null hypothesis. The ability of the test to detect differences that lead to rejection of the null hypothesis depends on the sample size. For example, for a particularly large sample, the test may reject the null hypothesis that two process means are equivalent. However, in practice the difference between the two means may be relatively small to the point of having no real engineering significance. Similarly, if the sample size is small, a difference that is large in engineering terms may not lead to rejection of the null hypothesis. The analyst should not just blindly apply the tests, but should combine engineering judgement with statistical analysis. (NIST)
Citations
“1.3.5 Quantitative Techniques, Practical Versus Statistical Significance”. NIST/SEMATECH e-Handbook of Statistical Methods. NIST. www.itl.nist.gov/div898/handbook//eda/section3/eda35.htm. Accessed Oct. 16, 2021.
Montgomery, Douglas C. Design and Analysis of Experiments, 5th Ed. John Wiley & Sons, Inc. 2001, pp 19.
Episode Transcript
When we’re looking at results (like measures of a characteristic), we need to take care not to get too hung-up on what the statistics is trying to tell us. Yes, statistical tools are a good way for us to make decisions and the results can act as proof for us. But, there’s a practical, engineering side to results, too. We need to evaluate the statistical significance along with the practical significance. I’ll tell you more of what I mean after this brief introduction.
Hello and welcome to quality during design the place to use quality thinking to create products others love, for less. My name is Dianna. I’m a senior level quality professional and engineer with over 20 years of experience in manufacturing and design. Listen in and then join the conversation at QualityDuringDesign.com.
I had some great mentors when I first started doing Quality Engineering-type analysis. One of the first lessons I learned was the difference between statistical significance and practical significance.
I was evaluating test data. I think I was comparing two things: we were modifying a manufacturing technique, and I wanted to find out how that change affected the product. I wanted to learn if the performance of a quality characteristic had changed. Did the change have any effect? And, if it did, was it worse than before? If it was no worse than before, then we’d go ahead and make that change.
I designed the test and asked parts to be made. Then, I did a comparison using statistical hypothesis testing. My results showed that there was a statistically significant difference between the two manufacturing methods, that the performance had gotten worse. These results indicated to me that this change had a bad effect on our product and that we shouldn’t make that change. When I reviewed the results with my mentor, he said, “Even though the statistical comparison concludes that the new method is worse, from a practical standpoint it’s negligible. I think this change is okay to implement.”
Hmmm…we then had a good discussion about statistical vs practical significance.
Here’s what the NIST Engineering Statistics Handbook has to say about this:
It is important to distinguish between statistical significance and practical significance. Statistical significance simply means that we reject the null hypothesis…The analyst should not just blindly apply the tests, but should combine engineering judgement with statistical analysis.
“1.3.5 Quantitative Techniques.” NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/. Accessed Oct. 17, 2021.
My thoughts about it are this:
We use tools to help us make decisions and predict outcomes with some confidence. Some of the techniques are a way for us to apply a mathematical model, like statistics. Some are graphical organizers. And others, still, help a team of people organize and coordinate their ideas. They’re all tools for people to use to make decisions. Ultimately, we use the results of many inputs in order to make our best decisions.
When I did my statistical comparison test, I got results that showed that the change in manufacturing had a negative effect to the product performance. The parts made with technique A were different from the parts made with technique B. The product is likely to perform worse with technique B, more than just a matter of chance. But, when we looked at the data compared to the larger performance requirement, we decided that the negative change to the performance was miniscule. We looked at the engineering aspects of the part, its requirements, its purpose in the whole system, and all aspects of the data we collected (like the failure modes and the range and location of the data) and we considered the reasons why we were making the change to begin with. When we put everything into context, we felt we had enough information all together to conclude that we’re going to go ahead with the change.
How did I capture this decision? I wrote it in a report. I showed the results of the statistical test as they were, showing a statistical significance. I also put the results in context with the bigger picture. Then, concluded that we were going to go ahead with the change. There’s no need to ever hide results. In fact, omitting the results of the analysis would’ve been a bad practice. We show all the results and explain our decision. If it doesn’t make sense to others, then maybe we’re not making a good decision.
What’s today’s insight to action? Let’s use all of our data and analyses and our engineering know-how to make decisions. Realize that statistics is a tool,(albeit a powerful one), but it’s just one tool to use when we’re making decisions. A result can show us that there’s a statistical significance, but maybe it’s not a practical one.
It is important to distinguish between statistical significance and practical significance. Statistical significance simply means that we reject the null hypothesis. The ability of the test to detect differences that lead to rejection of the null hypothesis depends on the sample size. For example, for a particularly large sample, the test may reject the null hypothesis that two process means are equivalent. However, in practice the difference between the two means may be relatively small to the point of having no real engineering significance. Similarly, if the sample size is small, a difference that is large in engineering terms may not lead to rejection of the null hypothesis. The analyst should not just blindly apply the tests, but should combine engineering judgement with statistical analysis.
Please visit this podcast blog and others at qualityduringdesign.com. Subscribe to the weekly newsletter to keep in touch. If you like this podcast or have a suggestion for an upcoming episode, let me know. You can find me at qualityduringdesign.com, on LinkedIn, or you could leave me a voicemail at 484-341-0238. This has been a production of Denney Enterprises. Thanks for listening!
Leave a Reply