Your Reliability Engineering Professional Development Site
I suspect reliability of the products and services in your world plan an important role in your day to day existence. For me, maybe I just pay attention to reliability, yet today in particular I tried to notice when things were just working as expected.
Rather then consider everything that touched my life today, I’ll narrow this down to just five. [Read more…]
In a recent reliability seminar I learned that the younger engineers did not have to take a statistics course, nor was it part of other courses, in their undergraduate engineering education. They didn’t dislike their stats class as so many before them have, they just didn’t have the pleasure.
Generally I ask how many ‘enjoyed’ their stats class. That generally gets a chuckle and opens an introduction to the statistics that we need to use for reliability engineering. I’ll have to change my line as more engineers just do not have any background with statistics.
I suspect this is good new for Las Vegas and other gambling based economies.
On average there are a few folks that get statistics. No me. There are those that intuitively understand probability and statistics, and demonstrate a mastery of the theory and application. No me.
I like many others that successfully use statistical tools, think carefully, consider the options, check assumptions, recheck the approach, ask for help and still check and recheck the work. Statistics is a tool and allows us to make better decisions. With practice you can get better at selecting the right tool and master the application of a range of tools.
Sure, it’s not easy, yet as many have found, mastering the use of statistics allows they to move forward faster.
Politicians, marketers, and others have a message to support and citing an interesting statistic helps. It doesn’t matter that the information is out of context nor clear. When someone claims 89% of those polled like brand x, what does that mean? Did they ask a random sample? Did they stop asking when they got the result they wanted? What was the poll section process and specific questions? What was the context?
The number may have been a simple count of positive responses vs all those questioned. The math results in a statistic, a percentage. It implies the sample represented the entire population. It may or may not, that is not clear.
We hear and read this type of statistic all too often. We discount even the well crafted and supported statistic. We associate distrust with statistics in general given the widespread poor or misleading use.
To me that means, we just need to be sure we are clear, honest and complete with our use of statistics. State the relevant information so others can fully understand. Statistics isn’t just the resulting percentage, it’s the context, too.
Even working to apply a statistical tool appropriately, there is a finite chance that the laws of random selection will provide a faulty result. If we test 10 items, there is a chance that our conclusion will show a 50% failure rate even though the actually population failure rate is less then 1%. Not likely to happen, yet it could.
We often do not have the luxury of the law of large numbers with our observations.
So, given the reality that we need to make a decision and that using a sample has risk, does that justify not using the sample’s results? No. The alternative of using no data doesn’t seem appealing to me, nor should it to you.
So, what can we do, we:
As with wood working there are many ways to cut a board, with statistics there are many tools. Learn the ones that help you characterize and understand the data you have before you. Master the tools one at a time and use them safely and with confidence.
When I buy a product, say a laptop, I am making an educated guess that Apple has done the due diligence to create a laptop that will work as long as I expect it to last. The trouble is I don’t know how long I want it to last thus creating some uncertainty for the folks at Apple. How long should a product last to meet customer expectations when customers are not sure themselves? [Read more…]
I will not say ‘never’, which is probably what you expect. There are a rare set of circumstances which may benefit with the use of MTBF as a metric. Of course, this does not include being deceitful or misleading with marketing materials. There may actually be an occasion where the MTBF metric works well.
As you know, MTBF is often estimated by tallying up the total hours of operation of a set of devices or systems and dividing by the number of failures. If no failures occur we assume one failure to avoid dividing by zero (messy business dividing by zero and to be avoided). MTBF is essentially the average time to failure.
The metric we select should be measurable and of a measure we have an interest. We would like to detect changes, measure progress, and possibly make business decisions with our metrics. If we are interested in the expected value of the time to failure for our devices, then MTBF might just be useful.
When making a device we often hear of executives, engineers and customers talk about how long they expect the product to last. An office device may have an expected life of 5 years, a solar power system – 30 years, and so on. If by duration we all agree that we expect 5 years of service on average, then using the average as the metric makes sense.
Before starting the use of MTBF, just make sure that a 5 year life implies half or two thirds of the devices will fail by the stated duration of 5 years. Yes, if the time to failure distribution is actually described by the exponential distribution (and a few other distributions) it means that two third of the units are expected to failure by the MTBF value. Thus if we set the goal to 5 years MTBF we imply half or more of the units will fail by 5 years.
Having a goal helps the design and development team make decisions and eventually conduct testing to prove the design meets the reliability objectives. Setting the goal a the expected value allows the fewest number of samples for testing. Testing for 99% reliability over 5 years is much tougher. We may require many samples to determine a meaningful estimate of the leading tail (i.e. first 1% or 5% of failures) of the time to failure distribution.
If the time failure pattern fits an exponential distribution, then testing becomes simplified. We can test one unit for a long time, or many units a short time, and arrive at the same answer. The test planning can maximize our resources to efficiently prove our design meets the objective. When the chance of failure each hour is the same, every device-hour of testing provide an equal amount of information.
Unlike products that wear out or degrade with time, when the design and device exhibit an exponential distribution we do not need any aging studies. We can just apply use or accelerated stress and measure the hours of operation and count the failures. Also any early failures are obviously quality issues and most likely do not count toward failures that represent actual field failures. Or do they?
When the industry, organization, vendors, and engineering staff already use MTBF to discuss reliability, then management would be wise to establish a metric using MTBF. Makes sense, right? The formula to calculate MTBF is very simple. Even the name implies the meaning (no pun intended). MTBF is the mean time between (or before) failure. It’s an average, which calculators, spreadsheets, smart phones, and possibly even your watch can calculate.
While the spread of the data is often of importance when making comparisons, estimating a sample set of data’s confidence bounds, or estimating the number of failures over the warranty period, if we assume the data actually fits an exponential distribution, we find the mean equals the standard deviation. Great! One less calculation. We have what we need to move forward.
Nearly every reliability or quality textbook or guideline includes extensive discussions about MTBF the exponential distribution and a wide range of reliability related calculations. Our common understanding generally is supported by the plentiful references.
Ask a few folks around you when considering using MTBF. What do they define MTBF as representing? If you receive a consistent answer, you may just have a common understanding. If the understanding is also aligned with the underlying math and assumptions, even better.
In summary all you need is:
I submit we are rarely interested in the time till the bulk of devices fail, rather interested in the time to first failures or some small percentage fail
I suggest that very few devices or system actually fail with a constant hazard rate. If your product does, prove it without grand waves of assumptions.
I have found that engineers, scientists, vendors, customers, and manager regularly misunderstand MTBF and how to properly use an MTBF value.
So back to the opening statement, it is possible though not likely you will find an occasion to effectively use MTBF as a metric. Instead use reliability: the probability of successful operation over a stated period with stated conditions and definition of success. 98% of office printers will function for 5 years without failure in a office…. Pretty clear. Sure we can fully define the function(s) and environment, and we need to do that anyway.
In English there is a lot of confusion on what reliability, availability and other ‘ilities mean in a technical way. Reliability as used in advertising and common discussions often means dependable or trustworthy. If talking about a product or system it may mean it will work as expected. [Read more…]
On social media the other day ran across a comment from someone that took my breath away. They were looking forward to starting a new reliability, no, MTBF report. They were tasked with creating a measure of reliability for use across the company and they choose MTBF.
Sigh.
I certainly do not blame the person. They have read about MTBF in many textbooks. Studied reliability using MTBF and related measures, plus found technical papers using the same. They may have seen industry reports and standards also.
MTBF is prevalent and no wonder someone tasked with setting a metric would select MTBF. It’s easy to calculate. Just one number and bigger is better.
MTBF is roundly criticized across any reliability related forum or discussion group. There is progress in books, papers and standards. And, it’s not reaching those new to reliability engineering.
This note will be short and have one request. Please tell those just getting started in reliability engineering to please not consider using MTBF. To not request MTBF from vendors. And, to actually do some thinking before selecting MTBF as their organizations metric.
Better yet, challenge those using MTBF to explain in a coherent and rational manner why they are doing so. Ask them to validate their assumed constant failure rate or similar assumptions. Working together we can start a ripple that may help build the wave of knowledge to improve the state of reliability engineering.
In an ideal world the design of a product or system will have perfect knowledge of all the risks and failure mechanisms. The design then is built perfectly without any errors or unexpected variation and will simply function as expected for the customer.
Wouldn’t that be nice.
The assumption that we have perfect knowledge is the kicker though, along with perfect manufacturing and materials. We often do not know enough about:
We do know that we do not know everything we need to create a perfect product, thus we conduct experiments.
We test. [Read more…]
Let’s say we 6 months of life data on 100 units. We’re charged with looking at the data and determine the impact of fixing the problems that caused the earliest failures.
The initial look of the data includes 9 failures and 91 suspensions. Other then the nine all units operated for 180 days. The MTBF is about 24k days. Having heard about Weibull plotting and using the beta value as a guide initially find the blue line in the plot. The beta value is less than one so we start looking for supply chain, manufacturing or installation caused failures, as we suspect early life failures dominant the time to failure pattern.
Given clues and evidence that some of the products failed early we investigate and find evidence of damage to units during installation. In fact it appears the first four failures were due to installation damage. The fix will cost some money, so the director of engineer asks for an estimate of the effect of the change on the reliability of the system.
The organization uses MTBF as does the customer. The existing MTBF of 24k days exceeds the customers requirement of 10K days, yet avoiding early problems may be worth the customer good will. The motivation is driven by continuous improvement and not out of necessity or customer complaints.
One way to estimate the effect of a removal of a failure mechanism is to examine the data without counting the removed failure mechanism. So, if the change to the installation practice in the best case completely prevents the initial four failures observed we are left with just the 5 other failures that occurred over the 6 months.
Removing the four initial failures and calculating MTBF we estimate MTBF will change to about 300 days.
Hum?
We removed failures and the MTBF got worse?
The classic calculation for MTBF is the total time divided by the number of failures. Taking a closer look at time to failure behavior of the two different failure mechanisms may reveal what is happening. The early failures have a decreasing failure rate (Weibull beta parameter less than 1) over the first two months of operation. Later, in the last couple of months of operation, 5 failures occur and they appear to have an increasing rate of failure (Weibull beta parameter greater than 1).
By removing the four early failures the Weibull distribution fit changes from the blue line to the black line (steeper slope).
Recall that the MTBF value represents the point in time when about 63% of units have failed. With only 9 total failures out of 100 units we have only about 10% of units failed so the MTBF calculation is a projects to the future when most of have failed, it does not providing information about failures at 6 months or less directly.
In this case when the four early failures are removed the slope changed from about 0.7 to about 5, it rotated counter clockwise on the CDF plot.
If only using MTBF the results of removing four failures from the data made the measured MTBF much worse and would have prevented us from improving the product. By fitting the data to a Weibull distribution we learned to investigate early life failures, plus once that failure mechanism was removed revealed a potentially serious wear out failure mechanism.
This is an artificial example, of course, yet it illustrates the degree which an organization is blind to what is actually occurring by using only MTBF. Treat the data well and use multiple methods to understand the time to failure pattern.
Finished it. 130 pages long and packed with advice on why and how to switch from MTBF to reliability.
Based in large part on comments, feedback, discussions and input from you, my peers in the NoMTBF tribe. Thanks for the encouragement and support.
First off, not sure why anyone would want to do this, yet one of the issues I’ve heard concerning abandoning the use of MTBF is client ask for MTBF. If they will not accept reliability probabilities at specific durations, and insist on using MTBF, you probably should provide a value to them.
Let’s say you have a Weibull distribution model that described the time to failure distribution of your product. You’ve done the testing, modeling, and many field data analysis and know for the requestor’s application this is the best estimate of reliability performance. You can, quite easily calculate the MTBF value.
As you know, if theβ parameter is equal to one then the characteristic life, η, is equal to MTBF. If β is less than or greater than one, then use the following formula to determine the mean value, MTBF, for the distribution.
$latex \displaystyle&s=4 \mu =\eta \Gamma \left( 1+\frac{1}{\beta } \right)$
You’ll need the Gamma function and the Weibull parameters. The further β is from one, the bigger the difference between η and MTBF.
You can find a little more information and background at the article Calculate the Mean and Variance on the accendoreliability.com site under the CRE Preparation article series.
After a short convulsion of disbelieve I became shocked. This was a guide for a government agency advising design teams and supporting reliability professional. It suggested using 217 to create the estimate of field reliability performance during the development phase.
Have we made not progress in 30 years?
Let’s say you are reviewing a purchase contract and find a request for a reliability estimate based on Mil Hdbk 217F (the latest revision that is also been obsolete for many years), what would you do? Would you contact the author and request an update to the document? Would you pull to 217 and create the estimate? Would you work to create and estimate the reliability of a product using the best available current methods? Then convert that work to an MTBF and adjust the 217 inputs to create a similar result. Or would you ignore the 217 requirement and provide a reliability case instead?
When a customer demands a parts count prediction as a condition of the purchase, is that useful for either the development team or the customer?
No.
So, given the contract is signed and we are in the execution phase, what are your options?
Ask the customer to adjust the agreement to include a meaningful estimate.
Ignore the 217 requirement and provide a complete reliability case detailing the reliability performance of the product.
Find a new position that will not include MTBF parts count prediction.
I hope you would call out the misstep in the contract and help all parties get the information concerning reliability that they can actually use to make meaningful decisions.
MTBF is the inverse of a failure rate, it is not reliability. Physics of failure (PoF) is a fundamental understanding and modeling of failure mechanisms. It’s the chemistry or physical activity that leads a functional product to fail. PoF is also not reliability.
Both MTBF and PoF have the capability to estimate or describe the time to failure behavior for a product. MTBF requires the knowledge of the underlying distribution of the data. PoF requires the use stresses and duration to allow a calculation of the expected probability of success over time.
MTBF start with a point estimate. PoF starts with the relationship of stress on the deterioration or damage to the material. One starts with time to failure data and consolidates into a single value, the other starts with determining the failure mechanism model.
Given the ability to model at the failure mechanism level even for a complex system, is there a need to summarize the time to failure information into a single value?
No.
MTBF was convenient when we had limited computing power and little understanding of failure mechanisms. Today, we can use the time to failure distributions directly. We can accommodate different stresses, different use pattern and thousands of potential failure mechanisms on a laptop computer.
MTBF has no purpose anymore. MTBF describes something we have and should have little interest in knowing.
Sure, PoF modeling takes time and resources to create. Sure, we may need complex mathematical models to adequately describe a failure mechanism. And, we may need to use simulation tools to estimate time to failure across a range of use and environmental conditions. Yet, it provide an estimate of reliability that is not possible using MTBF at any point in the process. PoF provides a means to support design and production decisions, to accommodate the changing nature of failure rates given specific experiences.
When will we stop using MTBF? I think the answer to both is about the same time. It is going to happen when we, reliability minded professionals, decide to use the best available methods to create information that support the many decisions we have to make. PoF will become dominant soon. It provides superior information and superior decision, thus superior products. The market will eventually decide, and everyone will have to follow. Or, we can decide now to provide our customers reliable products.
We can help PoF become dominant by not waiting for it to become dominant.
A troublesome question arrived via email the other day. The author wanted to know if I knew how and could help them adjust the parameters of a parts count prediction such that they arrived at the customer’s required MTBF value.
I was blunt with my response. [Read more…]
Let’s say you have a reliability goal for your product of 95% survive 2 years in an outdoor portable environment with the primary function of providing two way communication. There is an engineering reference specification detailing the product functions and requirements for performance. There is a complete document of environmental and use conditions . And you have similar detailed goals for the 1st month of use the expected useful life of 5 years. [Read more…]
Every now and then I receive an interesting question from a connection, colleague or friend. The questions that make me think or they discussion may be of value to you, I write a blog post.
In this case, there are a couple of interesting points to consider. Hopefully you are not facing a similar question. [Read more…]