The MTBF Conspiracy Theory

When my son was young he asked a lot of questions that were difficult to answer. For example:

• Why is the sky blue?
• Why do I have to go to school?
• What is a conspiracy theory?

The first two were expected, yet the third set me back a little. How do you explain conspiracy theory to a 5^th grader? The dictionary type definitions just seemed to confuse everyone. So, I made up a conspiracy theory.

I said, “Did you know, North Dakota, is not really a state?”

For those that haven’t heard of North Dakota, which on many maps is in the north central part of the US, that just reinforces the theory that it doesn’t exist.

My son, having recently memorized all fifty US states and their capital cities in school, said I was wrong and he even knew that was true as he still recalled the capital city name.

“Prove it.”, Was all I said in response.

“Well it’s on the map on the country as a state.” My reply included how maps change and are arbitrary. Anyone could have drawn the map, and how do we know it is accurate. Maybe the good folks in South Dakota paid the map maker to draw in the fictions state of North Dakota.

“It’s listed in Wikipedia!” And, my reply, was about how anyone can create a posting on the site, what is the proof it’s actually true? Have you ever seen a car with ND plates or meet someone from there?” He hadn’t.

My son knew I was only demonstrating the idea of a conspiracy theory. We had fun with it for years.

I was glad he never asked me,

“Why do people use MTBF?”

Just with the blue sky, a shrug and smile just wasn’t a good enough answer. There has to be a rational reasons people use MTBF.

After writing about perils of MTBF use for a few years, my current theory is it has to be a conspiracy.

The MTBF conspiracy theory revealed

Here’s what I think happened.

A bright engineer was tasked with estimating the reliability of a nuclear submarine’s electronics. He was given about a month to achieve this task, which is not enough time to conduct any testing. So, he gathered all the component failure rate data, tallied it up and reported the expected failure rate. {Parts count prediction}

The marketing department noticed the failure rate value and the word failure. The admission that the submarine might fail didn’t help to sell summaries, so they flipped the failure over, creating the average time between failure, or mean time between failures, MTBF.

The lower the failure rate the higher the MTBF went. Up was good. Failure is bad. {That’s how I think marketing folks think – sorry}

The engineers understood failure rates the math to create MTBF was pretty simple. So whatever, tis the same thing. Then management got involved.

The management team only wanted to read and talk about MTBF {again the word ‘failure ’  is bad thinking}. They set MTBF goals, they expected glowing reports of increasing MTBF values, and so on.

Then something really bad happened.

The US Military created a standard. And, a company used a computer to automate the standard’s estimate  of MTBF. Other’s did too. Now there was profit to be made by estimating MTBF, not reliability. So, they sold MTBF estimations. After all, that is what the management team wants, MTBF.

The military standard spawned many industry standards. The standards become parts of purchase contracts. MTBF flourished.

“What is your MTBF?” became an acceptable way to ask about reliability performance.

The murky bit of the theory involves why very few stood up to say, “Let’s not use MTBF, it is not very useful. Let’s use the probability of success over a duration (reliability) instead.” You may have said these very words or words to the same affect. And you felt the resistance.

• We always use MTBF.
• Everyone in our industry uses MTBF.
• The vendor only provides MTBF values.

My theory is we all know better, {maybe not the marketing folks – sorry} and we just do feel able to overcome the resistance to change. We know we could do much better with better metrics, yet the backlash is unrelenting.

Just as that first engineer figured out a quick way to come up with a failure rate estimate, we too face the necessity to use MTBF. We do not have the time or energy to change our company or industry to stop using MTBF. So, we just do it.

It’s easy.

I don’t know if the spread of MTBF use is organized by a secret group or not. I suspect not. Yet the ease of use and avoidance of the word failure (or anything the smells like we would have to do statistics) conspired to trap us into using MTBF.

That’s my theory. If you know of any critical bits of information to support this theory, let me know. If we expose the conspiracy for what it is, it may just fade away. We then may get back to work doing reliability engineering and creating reliable products.

Does a Certification Make You a Professional Reliability Engineer?

No, it doesn’t.

It’s just a piece of paper that conveys you mastered some body of knowledge. You most likely also committed to abide by a code of ethics. Plus you may have committed to continuing eductions to maintain the certification.

Having a certification means you know the terms, definitions, techniques and concepts concerning reliability engineering. Thanks all.

Does it mean you are a professional? No.

Being Professional

The dictionary describes professional as being associated or involved with a profession. You are professional by working or studying the profession of reliability engineering. Yet, we commonly consider a professional as being more than just a person with a job title.

A professional, in my mind exemplifies the essence of a noble, caring, capable engineer. One that works for the greater good. Someone the strives to make the world a better place. (Insert pedestal here.)

This is the nature of the engineering code of ethics that professional societies draft and encourage members to live. The following are just examples of the many similar codes that exist:

American Society for Quality Code of Ethics

http://asq.org/about-asq/who-we-are/ethics.html

National Society of Professional Engineers Code of Ethics

http://www.nspe.org/resources/ethics/code-ethics

Institute of Electrical and Electronics Engineers Code of Ethics

http://www.ieee.org/about/corporate/governance/p7-8.html

There are many others and they are all similar. Be honest, forthright and fair in your work.

You probably already adhere to these various codes of ethics. You do not have to pay membership dues to demonstrate you are ethical. It’s how you work, behave and conduct your life.

You are a professional reliability engineer by way you solve problems, continue to learn, assist others willingly, and exemplify how the reliability engineering profession makes the world a better place.

Certifications are Good, too.

There are different types of certifications and many organization offer certificates. For reliability engineering there are three professional societies that I know about that offer certifications.

American Society for Quality Certified Reliability Engineer

http://asq.org/cert/reliability-engineer

Society for Maintenance and Reliability Professional Certified Maintenance & Reliability Professional

http://www.smrp.org/i4a/pages/index.cfm?pageid=3578

Association for maintenance Professionals Certified Reliability Leader

http://www.maintenance.org/pages/crl

Some engineers have all three certifications. Some only one. Many professional engineers do not have any certification. It’s a personal decision. You can strive to work as a professional with or without securing one or more of the certifications offered by professional societies.

I should mention there are many other certifications offered in our industry. Conferences, software companies and consulting & training organizations offer certifications.  These like the ones offered by professional society are not licenses (state license or charter). The various certifications simply mean the person meet some level of experience, course work, demonstrated body of work or passed a test.

It doesn’t mean they are a professional.

If you are pursuing a certification, why? Please add a comment on what certification means to you and your career.

The Convenient Use of MTBF

Sometimes making an assumption is a good thing. You can achieve more with less. A well placed assumption saves you time, work, and worry. The right assumption may even be left unstated, it’s so good.

Have you ever assumed the failures for a system follow an exponential distribution? Did you assume tallying up the total hours and dividing by the number of failures was appropriate? Did you even check? (You don’t need to answer.) [Read more…]

Just Because the Customer Requests MTBF

Is that justification to use MTBF?

No.

It’s not. In this case the customer is probably not asking for MTBF, what they most like want to know is something meaningful about the expected reliability performance of the item in question. They want to know if what they will or did purchase will last as long as they expect. [Read more…]

The Constant Failure Rate Myth

Have you said or have you heard someone say,

• “Let’s assume it’s in the flat part of the curve”
• “Assuming constant failure rate…”
• “We can use the exponential distribution because we are in the useful life period.”

Or something similar? Did you cringe? Well, you should have. [Read more…]

MTBF is a Statistic, Not the Only One

We often face just a sample of life data with the request of estimating the reliability of the system. Or, we have a touch of test results and want to know if the product is reliability enough, yet. Or, we gather repair times to grapple with spares stocking.

We need to know the reliability. We need to know the number.

MTBF (or close cousin MTTF) is just that number. It is easy to calculate. A higher number means the system is more reliable. And, the metrics are in the units of time, often hours, which is easy to understand (and misunderstand).

In early chapters of reliability engineering books, or in introduction to reliability, we learn about the exponential distribution and the population parameter, theta. We also learn about the sample statistics which provides an unbiased estimated for the population parameter. In both cases, MTBF, or the mean time between failure, is the one value we have to master.

Other Statistics

Reliability is pretty easy using just one statistic. One calculation, one number, and we’re done.

Then a couple of things start to happen.

First, we notice that the actual time to failure behavior is not predicted, nor follows, the expected pattern when using just MTBF and the exponential distribution. The average time to fail changes as the system ages. We find that we run out of spares based on calculations using MTBF as the parts fail more and more often.

Second, we learn just a little more. We turn the page in the book or attend another webinar. We hear about another distribution commonly used in reliability engineering. The Weibull distribution. But, wait, hold on there. The Weibull has two and sometimes three parameters. I’ll need to learn about plotting, censored data, regression analysis, goodness of fit, confidence intervals, and a bunch of statistical methods.

Life was good with just one statistic.

We didn’t sign up to be reliability statisticians.

Well, too bad.

Actually, when using even just the one statistic, MTBF, we also should have been

• Checking assumptions
• Fitting the data to the exponential distribution function
• Evaluating the goodness of fit
• Calculating confidence bounds
• And, using those other statistical methods

In order to understand and use our sparse and expensive datasets, we need to use the tools found in the statistics textbooks.

Yes, the Weibull distribution has two or three parameters, thus we need to evaluate how well our statistics describe the data in a more rigorous way. And, we learn so much more. For example, we can model and predict a system with decreasing or increasing failure rates over time. We can estimate the number of required spares next year with a bit more accuracy then using just MTBF.

There are more benefits. Have you advanced past the basic introduction and embraced the use of reliability statistics? How’s it going and what challenges are you facing?

Plot the Data

Just, please, plot the data.

If you have gathered some time to failure data. You have the breakdown dates for a piece of equipment. You review your car maintenance records and notes the dates of repairs. You may have some data from field returns. You have a group of numbers and you need to make some sense of it.

Take the average

That seems like a great first step. Let’s just summarize the data in some fashion. So, let’s day I have the number of hours each fan motor ran before failure. I can tally up the hours, TT, and divide by the number of failures, r. This is the mean time to failure.

$latex \displaystyle&s=3 \theta =\frac{TT}{r}$

Or, if the data was one my car and I have the days between failures, I can also tally up the time, TT, and divide by the number of repairs, r. Same formula and we call the result, the mean time between failure.

And I have a number. Say it’s 34,860 hours MTBF. What does that mean (no pun intended) other than on average my car operated for 34k hours between failures. Sometimes more, sometimes less.

Any pattern? Is my car getting better with age, or worse?

A Histogram

In school we used to use histograms to display the data. Let’s try that. Here’s an example plot.

 

 In this case the plot is of service and repair times (most likely similar to the times the garage has my car for a oil change and tune up). Right away we see more than just a number. The values range from about 50 up to about 350 with most of the data on the lower side. Just a couple of service times take over 250 minutes.

Using just an average doesn’t provide very much information compared to a histogram.

Mean Cumulative Function Plot

Over time count the number of failures. If the repair time is short compared to operating time, than this simple plot may reveal interesting patterns that a histogram cannot.

Here is a piece of equipment and each dot represented a call for service. The x-axis is time and the vertical axis is the count of service calls. While it’s not clear what happened shortly after about 3,000 hours, it may be worth learning more about what was going on then.

 

Even after the first there or four point after 3,000 hours would have signaled something different is happening here.
MCF plots show when something is getting worse (more frequent repairs) by curving upward, or getting better, (longer spans between repairs) by flattening out. Again, a lot more information than with just a number.

Plot the Fitted Distribution

Let’s say we really want to assume the data is from an exponential distribution. We can happily calculate the MTBF value and continue with the day. Or, we can plot the data and the fitted exponential distribution.

Let’s say we have about five failure times based on customer returns out of the 100 units placed into service. We can calculate the MTBF value including the time the remaining 95 units operated, which is about 172,572 hours MTBF. And, we can plot the data, too.

Here’s an example. What do you notice, even with a fuzzy plot image?

 

The line intersects the point where the F(t) is 0.63 or about the 63^rd percentile of the distribution, and the time is at the point we calculated as the MTBF value (off to the right of the plot area).
Like me, you may notice the line doesn’t seem to describe the data very well. It seems to have a different pattern than that described by the exponential distribution. Let’s add a fit of a Weibull distribution that also was fit to the data, including the units that have not failed.

 

The Weibull fit at least appears to represent the pattern of the failures. The slope is much steeper than the exponential fit. The Weibull tells a different story. A story that represents the story within the data.

Again, just plot the data. Let the data show you what it has to say. What does your data say today?

The Fear of
Reliability Statistics

When reading a report and there is a large complex formula, maybe a derivation, do you just skip over it? Does a phrase, 95% confidence of 98% reliability over 2 years, not help your understanding of the result?

Hypothesis testing, confidence intervals, point estimates, parameters, independent identically distributed, random sample, orthogonal array, …

Did you just shiver a bit?  [Read more…]

5 Ways Reliability Was Important to Me Today

I suspect reliability of the products and services in your world plan an important role in your day to day existence. For me, maybe I just pay attention to reliability, yet today in particular I tried to notice when things were just working as expected.

Rather then consider everything that touched my life today, I’ll narrow this down to just five. [Read more…]

Statistics and the Bad Reputation

In a recent reliability seminar I learned that the younger engineers did not have to take a statistics course, nor was it part of other courses, in their undergraduate engineering education. They didn’t dislike their stats class as so many before them have, they just didn’t have the pleasure.

Generally I ask how many ‘enjoyed’ their stats class. That generally gets a chuckle and opens an introduction to the statistics that we need to use for reliability engineering. I’ll have to change my line as more engineers just do not have any background with statistics.

I suspect this is good new for Las Vegas and other gambling based economies.

Statistics are hard

On average there are a few folks that get statistics. No me. There are those that intuitively understand probability and statistics, and demonstrate a mastery of the theory and application. No me.

I like many others that successfully use statistical tools, think carefully, consider the options, check assumptions, recheck the approach, ask for help and still check and recheck the work. Statistics is a tool and allows us to make better decisions. With practice you can get better at selecting the right tool and master the application of a range of tools.

Sure, it’s not easy, yet as many have found, mastering the use of statistics allows they to move forward faster.

Statistics are abused

Politicians, marketers, and others have a message to support and citing an interesting statistic helps. It doesn’t matter that the information is out of context nor clear. When someone claims 89% of those polled like brand x, what does that mean? Did they ask a random sample? Did they stop asking when they got the result they wanted? What was the poll section process and specific questions? What was the context?

The number may have been a simple count of positive responses vs all those questioned. The math results in a statistic, a percentage. It implies the sample represented the entire population. It may or may not, that is not clear.

We hear and read this type of statistic all too often. We discount even the well crafted and supported statistic. We associate distrust with statistics in general given the widespread poor or misleading use.

To me that means, we just need to be sure we are clear, honest and complete with our use of statistics. State the relevant information so others can fully understand. Statistics isn’t just the resulting percentage, it’s the context, too.

Statistics can be wrong

Even working to apply a statistical tool appropriately, there is a finite chance that the laws of random selection will provide a faulty result. If we test 10 items, there is a chance that our conclusion will show a 50% failure rate even though the actually population failure rate is less then 1%. Not likely to happen, yet it could.

We often do not have the luxury of the law of large numbers with our observations.

So, given the reality that we need to make a decision and that using a sample has risk, does that justify not using the sample’s results? No. The alternative of using no data doesn’t seem appealing to me, nor should it to you.

So, what can we do, we:

• Do the best we can with the data we have.
• Do exercise due care to minimize and quantify measurement error.
• Strive to select samples randomly.
• Apply the best analysis available, and,
• Extract as much information from the experiment and analysis as possible.

As with wood working there are many ways to cut a board, with statistics there are many tools. Learn the ones that help you characterize and understand the data you have before you. Master the tools one at a time and use them safely and with confidence.

How Many Assumptions Are Too Many Concerning Reliability?

When I buy a product, say a laptop, I am making an educated guess that Apple has done the due diligence to create a laptop that will work as long as I expect it to last. The trouble is I don’t know how long I want it to last thus creating some uncertainty for the folks at Apple. How long should a product last to meet customer expectations when customers are not sure themselves? [Read more…]

When to Use MTBF as a Metric?

I will not say ‘never’, which is probably what you expect. There are a rare set of circumstances which may benefit with the use of MTBF as a metric. Of course, this does not include being deceitful or misleading with marketing materials. There may actually be an occasion where the MTBF metric works well.

As you know, MTBF is often estimated by tallying up the total hours of operation of a set of devices or systems and dividing by the number of failures. If no failures occur we assume one failure to avoid dividing by zero (messy business dividing by zero and to be avoided). MTBF is essentially the average time to failure.

Expected Value as Metric

The metric we select should be measurable and of a measure we have an interest. We would like to detect changes, measure progress, and possibly make business decisions with our metrics. If we are interested in the expected value of the time to failure for our devices, then MTBF might just be useful.

When making a device we often hear of executives, engineers and customers talk about how long they expect the product to last. An office device may have an expected life of 5 years, a solar power system – 30 years, and so on. If by duration we all agree that we expect 5 years of service on average, then using the average as the metric makes sense.

Before starting the use of MTBF, just make sure that a 5 year life implies half or two thirds of the devices will fail by the stated duration of 5 years. Yes, if the time to failure distribution is actually described by the exponential distribution (and a few other distributions) it means that two third of the units are expected to failure by the MTBF value. Thus if we set the goal to 5 years MTBF we imply half or more of the units will fail by 5 years.

Product Testing Advantages

Having a goal helps the design and development team make decisions and eventually conduct testing to prove the design meets the reliability objectives. Setting the goal a the expected value allows the fewest number of samples for testing. Testing for 99% reliability over 5 years is much tougher. We may require many samples to determine a meaningful estimate of the leading tail (i.e. first 1% or 5% of failures) of the time to failure distribution.

If the time failure pattern fits an exponential distribution, then testing becomes simplified. We can test one unit for a long time, or many units a short time, and arrive at the same answer. The test planning can maximize our resources to efficiently prove our design meets the objective. When the chance of failure each hour is the same, every device-hour of testing provide an equal amount of information.

Unlike products that wear out or degrade with time, when the design and device exhibit an exponential distribution we do not need any aging studies. We can just apply use or accelerated stress and measure the hours of operation and count the failures. Also any early failures are obviously quality issues and most likely do not count toward failures that represent actual field failures. Or do they?

Metrics Should Have a Common Understanding

When the industry, organization, vendors, and engineering staff already use MTBF to discuss reliability, then management would be wise to establish a metric using MTBF. Makes sense, right? The formula to calculate MTBF is very simple. Even the name implies the meaning (no pun intended). MTBF is the mean time between (or before) failure. It’s an average, which calculators, spreadsheets, smart phones, and possibly even your watch can calculate.

While the spread of the data is often of importance when making comparisons, estimating a sample set of data’s confidence bounds, or estimating the number of failures over the warranty period, if we assume the data actually fits an exponential distribution, we find the mean equals the standard deviation. Great! One less calculation. We have what we need to move forward.

Nearly every reliability or quality textbook or guideline includes extensive discussions about MTBF the exponential distribution and a wide range of reliability related calculations. Our common understanding generally is supported by the plentiful references.

Ask a few folks around you when considering using MTBF. What do they define MTBF as representing? If you receive a consistent answer, you may just have a common understanding. If the understanding is also aligned with the underlying math and assumptions, even better.

When to Use MTBF Checklist

In summary all you need is:

• A business interest in the time till half or more of product fail
• A design with a fixed chance to failure each hour of operation
• A well educated team that understands the proper use of an inverse failure rate measure

I submit we are rarely interested in the time till the bulk of devices fail, rather interested in the time to first failures or some small percentage fail

I suggest that very few devices or system actually fail with a constant hazard rate. If your product does, prove it without grand waves of assumptions.

I have found that engineers, scientists, vendors, customers, and manager regularly misunderstand MTBF and how to properly use an MTBF value.

So back to the opening statement, it is possible though not likely you will find an occasion to effectively use MTBF as a metric. Instead use reliability: the probability of successful operation over a stated period with stated conditions and definition of success. 98% of office printers will function for 5 years without failure in a office…. Pretty clear. Sure we can fully define the function(s) and environment, and we need to do that anyway.

Reliability and Availability

In English there is a lot of confusion on what reliability, availability and other ‘ilities mean in a technical way. Reliability as used in advertising and common discussions often means dependable or trustworthy. If talking about a product or system it may mean it will work as expected. [Read more…]

Why Doesn’t Product Testing Catch Everything?

In an ideal world the design of a product or system will have perfect knowledge of all the risks and failure mechanisms. The design then is built perfectly without any errors or unexpected variation and will simply function as expected for the customer.

Wouldn’t that be nice.

The assumption that we have perfect knowledge is the kicker though, along with perfect manufacturing and materials. We often do not know enough about:

• Customer requirements
• Operating environment
• Frequency of use
• Impact of design tradeoffs
• Material variability
• Process variability

We do know that we do not know everything we need to create a perfect product, thus we conduct experiments.

We test. [Read more…]