Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Way of the Quality Warrior
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • AI & Predictive Maintenance
      • Asset Management in the Mining Industry
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
      • The People Side of Maintenance
      • The Reliability Mindset
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • R for Engineering
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • Reliability Engineering for Heavy Industry
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by Fred Schenkelberg 5 Comments

When to Stop Testing

When to Stop Testing

Stop testing when the testing provides no value.

If no one is going to review the results or use the information to make a decision, those are good signs that the testing provides no value. Of course, this may be difficult to recognize.

Some time ago while working with a product development team, one of the tasks assigned was to create an ongoing reliability test plan. This was just prior to the final milestone before starting production. During development, we learned quite a bit about the product design, supply chain, and manufacturing process. Each of which included a few salient risks to reliable performance.

 

Investigating previous ongoing test plans

Being new to the group and knowing the project was a small evolution for an existing product, I suspected there was a previous plan already in place. Now, the motivation was not to replicate the previous plan. My intent was

  •  To understand what data previous testing produced
  • How that data was being collected and presented
  • And, how the information helped the team make decisions.

The new plan should be consistent and useful where it worked well and maybe an improvement where it didn’t work so well.

I also asked a couple of people that had previous created ongoing reliability tests how they approached the task. Each one said they basically replicated the design qualification test plan and conducted the full suite of testing each month.

That is a lot of testing, it’s expensive and just think of all the data that has been collected over the past few years. I really wish I had known about that treasure trove of data during the development process as we could have avoided a few tests, improved others and focused on the largest risks based on that data.

Finding the Data

The first person that told me about the existence of previous test plans and that the testing was accomplished monthly. She said the data was probably on the team’s share drive. Since everything was on the shared drive we spent a few minutes looking, with no luck.

Then off to other engineers and managers on the team that would be most likely to know the whereabouts of the said data. Everyone I talked to over a month of searching was aware of the testing and that the data was somewhere. It was turning into a quest and not sure if it was becoming a search for the holy grail or not.

To make a long story very short, with the help a country manager (the manufacturing and testing were being done in China) and the financial engineer(?) or person, we found the data after two months of searching. The person collecting and organizing the data did a wonderful job and the data was complete and well presented, including the raw data.

I asked when the last time anyone asked for the test data, and he said I was the first in the five years he had been maintaining the database.

Using Test Data

The requirement to create and run testing that evaluated the product’s performance and durability was written into the product lifecycle and development guidelines. At some point in the past the testing was considered worth the expense to create test plans and pay for samples, testing, and data collection.

And, somewhere along the way, the value of that data diminished to have little to no value. No one in the development team nor the manufacturing team took the time to review or even monitor the data.

Of course, I grabbed the data, did a few simple plots, discovered in the historical record indicators of most of the excursions of higher than expected field failures (most would have been prevented or minimized if someone had looked at the test data and made a decision to do something about it.) Then revised the test plan I had created by eliminating many of the tests as they did not show any failures or adverse variation over multiple generations of design. And, increased the samples and frequency of a few based on larger than expected variation in the data.

More importantly, stopped about two-thirds of the existing testing sequences. First, as no was really needed to look at the results, second, the process and supply chain demonstrated years of stability and capability. Those tests didn’t show any indication of risk of failure.

That left a manageable set of meaningful tests tailored to each product’s set of unique risks. Those become useful to monitor and take action to improve the existing and future products.

When to Stop Testing

Ideally, stop before setting the testing in motion. Not just ongoing reliability tests, any test. Anytime we take a product out of the process of development or manufacture to conduct an evaluation, it takes time and resources to do so. Therefore, the testing and the results if collected to support a decision by someone that knows they are going to use the data to make a decision, then drive on.

Sure there are all kinds of tests and some are inexpensive, exploratory or quick. The impact of no meaningful information is minor. When the testing is expensive, time-consuming, etc then the value of the results had the best warrant the investment.

Do not design and conduct a test unless there is some purpose. If it is done because we always do it, that is clear signal to stop and ask a few questions.

Second, if the testing is established as a routine and ongoing test, when it no longer serves a meaningful (valuable) purpose. Stop the test.

 

Summary

Testing appears to be like government agencies. Once established they exist as any living being with a will to survive and will find ways to continue despite having outlived any useful purpose. Tests are not entities in and of themselves (not sure about government committees, so no comment there). They are tools we use as engineers and managers to understand characteristics of our products or processes.

Look at your existing testing and sort out for each test

  • What is the purpose?
  • What do the results mean?
  • What question or decision does the data support?
  • What is the value of this test?

If the answers to these questions indicate the test either will not or does not have sufficient value given the investment to create the data, then stop the test.

Filed Under: Articles, Musings on Reliability and Maintenance Topics, on Product Reliability Tagged With: testing

About Fred Schenkelberg

I am the reliability expert at FMS Reliability, a reliability engineering and management consulting firm I founded in 2004. I left Hewlett Packard (HP)’s Reliability Team, where I helped create a culture of reliability across the corporation, to assist other organizations.

« Why Doesn’t Product Testing Catch Everything?
Reading a Standard Normal Table »

Comments

  1. Wei says

    June 11, 2015 at 6:15 AM

    It really helps!
    I am thinking about reducing some foolish ORT tests

    Reply
  2. John Healy says

    June 19, 2015 at 4:46 PM

    Fred,

    There is a whole theory on when to stop testing. If there were a certain amount of randomness in the discovery of faults, it would be easy to come up with mathematical algorithms to determine when to stop testing. Unfortunately, software testers run every script that they are aware of looking for software faults. What ends up happening is that all easy problems are fixed and the number of software problems found in subsequent tests dwindles to zero. That does not mean we have done the right amount of testing.

    John

    Reply
    • Fred says

      June 19, 2015 at 5:02 PM

      thanks John, I suppose if we knew what was the total number of faults we wished to resolve, we could test to find them. I still maintain the best test is the one not conducted. If it is not useful, then do not test. cheers, Fred

      Reply
  3. Askhat Turlybayev says

    June 25, 2015 at 10:21 AM

    Fred,

    Good article. Tests can also be eliminated if data can be grandfathered from historical or field data; expensive phisical testing can be substituted by numberical analysis and/or simulations; component level tests can be eliminated or postponed until system level testings take place.

    Askhat Turlybayev

    Reply
    • Fred says

      June 25, 2015 at 10:22 AM

      Thanks Askhat – all great suggestions. cheers, Fred

      Reply

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Article by Fred Schenkelberg
in the Musings series

Join Accendo

Receive information and updates about articles and many other resources offered by Accendo Reliability by becoming a member.

It’s free and only takes a minute.

Join Today

Recent Articles

  • test
  • test
  • test
  • Your Most Important Business Equation
  • Your Suppliers Can Be a Risk to Your Project

© 2025 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy