Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Way of the Quality Warrior
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • AI & Predictive Maintenance
      • Asset Management in the Mining Industry
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
      • The People Side of Maintenance
      • The Reliability Mindset
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • R for Engineering
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • Reliability Engineering for Heavy Industry
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by JD Solomon Leave a Comment

Qualitative Assessments: Do the Fine Points of Risk Matrices Really Matter?

Qualitative Assessments: Do the Fine Points of Risk Matrices Really Matter?

Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article summarizes a review of ten risk matrices performed by facility owners or their consultants.

Qualitative Assessment Scales

Likert scales are five-point ordinal scales where participants select a label (or numerical value) that equates to an opinion or attitude. Labels are paired (i.e., agree-disagree, strongly agree-strongly disagree) symmetrically around a neutral center. Today, we see 5-, 7-, and 9-point versions of Likert scales in everything from customer surveys to risk assessments.

Ten Risk Matrices Evaluated

Ten risk assessments were randomly evaluated that used consequences of failure (CoF) and likelihood of failure (LoF) in a two-axis matrix. In this case, random means from my files of numerous projects I have reviewed over the past two decades. All of the examples are related to some form of facilities or infrastructure.

Seven of the examples were developed under the leadership of consultants and three were developed by facility staff. One consulting company appeared twice; however, the lead consultant and the geographic regions were different.

Focus of Design and Analysis

The review focused on the design and analysis of the qualitative assessments. In most cases, the approaches and techniques of administering the assessments were not known.

Poor Application of Risk Matrices

Eight of the ten risk matrices had flawed analysis or design. In theory, these flaws produce significant impacts on the results.

Scales (and bad math)

In seven of the ten risk matrix evaluations examined, applying scalar properties to ordinal data was found to be a major source of error. The ordinal data from 5- and 9-point Likert risk surveys were treated parametrically. In five of the seven cases, the numbers (ratings) were multiplied as if the data were continuous. 

The correct analysis in these cases is for the scales to be handled non-parametrically and, if combined, treated additively. A comparative analysis of the results indicated much different risk rankings and prioritization depending on how the data was treated.

From a practical perspective, the ordinal data can be treated as interval data if the scales. This was done correctly in only three of the ten cases. The incorrect treatment of ordinal data is consistent with similar reviews I have performed over the past twenty years. The improper analysis of ordinal scales underscores a significant practitioner problem.

Flawed Internal Consistency

Eight of the risk matrices examined involved the concept of vertical consistency. By way of example, in one risk matrix involving the consequences of failure, the value associated with a human life was 10. However, a 10 was also correlated to a budget impact of $150,000 or a customer experiencing five days without utility service.   Clearly, these are not equivalent.

The results in eight of the ten cases were blindly weighted and aggregated without regard to vertical equivalency. Technically, this is a violation of the measurement principle of representativeness. From a practical perspective, it skews many lesser important systems to be treated as equivalent to those where human health and safety should be dominant.

Similar to the improper treatment of ordinal scales, flawed internal consistency is common and a significant practitioner problem.

Limited Practical Ramifications

Although eight of the ten cases had some form of poor assessment design and analysis, there were few practical ramifications. In all ten cases, I reviewed the final prioritization and discussed the results with an owner’s representative one to three years after the work was implemented.

Common Sense

In four of the eight cases, the risk matrix was technically performed wrong from a design or analysis perspective. Still, the final selections of risk treatment yielded acceptable results for the organization. It appears that common sense or maybe even luck overcame poor analytical techniques. 

Insignificant Magnitude

In four of the eight cases, the owners acknowledged that the assessment produced a risk evaluation that included or excluded some wrong items. The costs of the associated implemented items (projects) were in the range of hundreds of thousands to over ten million dollars.

However, the facility and infrastructure system values ranged from a half billion to more than one billion dollars. While acknowledging some degree of undesirability, the impacts were not insignificant or outside the “normal” process for the organization.

Extremes Remained the Same

In four of seven cases, it was possible to re-analyze the ordinal data were treated as continuous data and parametrically. Meaningful distinctions were made in the average scores (calculated to two to three decimal places) that were applied in the risk matrices. When the data was re-evaluated as ordinal and analyzed non-parametrically, the order of the prioritization resulted in significant differences in the risk matrices and associated project prioritizations.

However, items that ranked at the extremes, either high or low, were considered properly regardless of the strict correctness of the assessment design and analysis. In all cases, the issues were most significant in the middle, where an action (funding) cut-off level would have impacted projects on the “bubble.”

Overall, most organizations stated they could not significantly fund projects that were not high priorities. In those cases, the fine-tuning in the middle made no practical difference.

Acceptable, Not Optimal, Results

One conclusion that can be taken from the examination of ten cases is that risk matrices yield acceptable but not optimal results.

Some more specific conclusions include the following:

  1. A minority of risk matrices are designed, analyzed, and administered properly.
  2. When done correctly, the results of risk matrices are acceptable for initial prioritization purposes. (more analysis and decision making may be needed, depending on the context)
  3. There is a significant practitioner problem in analyzing ordinal scales and associated data.
  4. There is a significant practitioner problem in designing and analyzing multi-attribute variables, particularly related to the consequences of failure (CoF).
  5. Common sense or luck often prevails in the finalized prioritization.
  6. Organizations have some slop in the normal selection of priorities, and using risk matrices produces no worse than normal results.
  7. Many organizations can only afford to invest in high priorities, so the fine differences in the middle often do not matter.

Applying It with FINESSE

Do the fine points of risk matrices really matter? Yes, because someone is paying a trained professional to do the technical assessment correctly. The risk matrix is often not constructed as it should be.

Qualitative assessments are cost-effective and highly flexible tools. Using Likert-type questionnaires as a principal evaluation test must be accompanied by the appropriate design, analysis, and administration. Do it correctly, and communicate it with FINESSE, or do not do it at all.


Communicating with FINESSE is a not-for-profit community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.

Filed Under: Articles, Communicating with FINESSE, on Systems Thinking Tagged With: Likert scales, Qualitative assessments, Risk matrix

About JD Solomon

JD Solomon, PE, CRE, CMRP provides facilitation, business case evaluation, root cause analysis, and risk management. His roles as a senior leader in two Fortune 500 companies, as a town manager, and as chairman of a state regulatory board provide him with a first-hand perspective of how senior decision-makers think. His technical expertise in systems engineering and risk & uncertainty analysis using Monte Carlo simulation provides him practical perspectives on the strengths and limitations of advanced technical approaches.  In practice, JD works with front-line staff and executive leaders to create workable solutions for facilities, infrastructure, and business processes.

« Small Multiples, Huge Advantage
Indoor Dispersion: A Simple Model To Estimate Indoor Concentrations »

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Headshot of JD SolomonArticles by JD Solomon
in the Communicating with FINESSE article series

Join Accendo

Receive information and updates about articles and many other resources offered by Accendo Reliability by becoming a member.

It’s free and only takes a minute.

Join Today

Recent Posts

  • test
  • test
  • test
  • Your Most Important Business Equation
  • Your Suppliers Can Be a Risk to Your Project

© 2025 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy