Accendo Reliability

Your Reliability Engineering Professional Development Site

  • Home
  • About
    • Contributors
  • Reliability.fm
    • Speaking Of Reliability
    • Rooted in Reliability: The Plant Performance Podcast
    • Quality during Design
    • Way of the Quality Warrior
    • Critical Talks
    • Dare to Know
    • Maintenance Disrupted
    • Metal Conversations
    • The Leadership Connection
    • Practical Reliability Podcast
    • Reliability Matters
    • Reliability it Matters
    • Maintenance Mavericks Podcast
    • Women in Maintenance
    • Accendo Reliability Webinar Series
  • Articles
    • CRE Preparation Notes
    • on Leadership & Career
      • Advanced Engineering Culture
      • Engineering Leadership
      • Managing in the 2000s
      • Product Development and Process Improvement
    • on Maintenance Reliability
      • Aasan Asset Management
      • AI & Predictive Maintenance
      • Asset Management in the Mining Industry
      • CMMS and Reliability
      • Conscious Asset
      • EAM & CMMS
      • Everyday RCM
      • History of Maintenance Management
      • Life Cycle Asset Management
      • Maintenance and Reliability
      • Maintenance Management
      • Plant Maintenance
      • Process Plant Reliability Engineering
      • ReliabilityXperience
      • RCM Blitz®
      • Rob’s Reliability Project
      • The Intelligent Transformer Blog
      • The People Side of Maintenance
      • The Reliability Mindset
    • on Product Reliability
      • Accelerated Reliability
      • Achieving the Benefits of Reliability
      • Apex Ridge
      • Metals Engineering and Product Reliability
      • Musings on Reliability and Maintenance Topics
      • Product Validation
      • Reliability Engineering Insights
      • Reliability in Emerging Technology
    • on Risk & Safety
      • CERM® Risk Insights
      • Equipment Risk and Reliability in Downhole Applications
      • Operational Risk Process Safety
    • on Systems Thinking
      • Communicating with FINESSE
      • The RCA
    • on Tools & Techniques
      • Big Data & Analytics
      • Experimental Design for NPD
      • Innovative Thinking in Reliability and Durability
      • Inside and Beyond HALT
      • Inside FMEA
      • Integral Concepts
      • Learning from Failures
      • Progress in Field Reliability?
      • R for Engineering
      • Reliability Engineering Using Python
      • Reliability Reflections
      • Testing 1 2 3
      • The Manufacturing Academy
  • eBooks
  • Resources
    • Accendo Authors
    • FMEA Resources
    • Feed Forward Publications
    • Openings
    • Books
    • Webinars
    • Journals
    • Higher Education
    • Podcasts
  • Courses
    • 14 Ways to Acquire Reliability Engineering Knowledge
    • Reliability Analysis Methods online course
    • Measurement System Assessment
    • SPC-Process Capability Course
    • Design of Experiments
    • Foundations of RCM online course
    • Quality during Design Journey
    • Reliability Engineering Statistics
    • Quality Engineering Statistics
    • An Introduction to Reliability Engineering
    • Reliability Engineering for Heavy Industry
    • An Introduction to Quality Engineering
    • Process Capability Analysis course
    • Root Cause Analysis and the 8D Corrective Action Process course
    • Return on Investment online course
    • CRE Preparation Online Course
    • Quondam Courses
  • Webinars
    • Upcoming Live Events
  • Calendar
    • Call for Papers Listing
    • Upcoming Webinars
    • Webinar Calendar
  • Login
    • Member Home

by JD Solomon Leave a Comment

What Are Best Practices for Facilitating Qualitative Assessments?

What Are Best Practices for Facilitating Qualitative Assessments?

Opinion-based data is the foundation of qualitative assessments. Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. The article provides best practices for improving administering and facilitating qualitative assessments.

A Long History with Many Forms

The modern basis of opinion-based data’s scientific use can be traced from the western hemisphere to the late 1800s. Educators and psychologists were seeking to quantify their clinical observations of human behavior. A similar movement was underway in the fields of natural science and statistics.

Rensis Likert is credited with creating one of the first data instruments, which employs the 5-point ordinal scales currently used in most opinion-based surveys. There are five major qualitative measurement scales: Likert, ranking, Thurstone, Guttman, and semantic differential.

Proper Administration and Facilitation

According to the Institute of Defense Analysis (IDA), “It is true that surveys measure subjective experiences (i.e., thoughts) and they are not accurate measures of anything other. However, when designed, administered, and analyzed correctly, surveys provide objective, reliable, and valid measurement of these subjective experiences. There is a substantial body of research on how to collect accurate survey data (e.g., Likert, 1932; Babbitt & Nystrom, 1989).”

IDA adds, “The motivations of the respondents is affected by what the respondents were asked to do before the survey, their beliefs as to what will happen with the survey data, how interested they are in the topic of the survey, and how frequently they are surveyed. Motivation can be increased or decreased by the design of the test (e.g., completing the same survey multiple times), the design of the survey (e.g., length, question wording), and the administration of the survey (e.g., when administered).”

Administration and facilitation constraints, which come from things such as the test design, the survey delivery method (paper, electronic, verbal), and the time of day, can impact results.

Likert’s Survey Administration Recommendations

Likert provides few recommendations in terms of survey administration. However, he clearly believes that the survey should be evaluated for reliability (repeatability) and used the Spearman-Brown method as a primary tool. He used data from resurveys of a given group of respondents up to 30 days following the initial survey.

Modern References

Several modern-era sources of good practices are available. In addition to the author’s own experiences, the basis for the ones provided relates closely to the field of reliability engineering. It includes The Handbook of Human Factors Testing and Evaluation, the Institute for Defense Analysis, and the International Handbook of Survey Methodology.

  • avoid giving surveys at the beginning or end of a day or work shift; avoid administering detailed surveys at the beginning or end of a work week
  • facilitate the questions orally
  • use facilitators that are available throughout the time(s) the survey is being completed to answer questions
  • provide written instructions or frequently asked questions so that each respondent has a common basis of knowledge
  • any single session/scenario should be able to be completed by a respondent working independently in 15 minutes.

Additional Observations

Qualitative assessments are cost-effective and highly flexible tools. They measure the opinions, attitudes, knowledge, perceived behaviors, observations, beliefs, and experiences of individuals who use a system the most. Human factors should be included in any system evaluation, as well as aspects of system performance that we cannot or do not understand how to measure.

Solomon and Benson use some best practices related to risk evaluations of infrastructure systems, some of which include:

  • beginning and ending the session with “easier” subsystems to account for mental warm-up and to avoid the potential effects of fatigue
  • using a combination of paper and electronic forms
  • going through each question as a group to avoid the potential for respondents to feel as if they were in a mental race
  • providing a comment section for each question to allow respondents to give more detail and to avoid potential frustration if a question is not well understood

Facilitating with FINESSE

There are greater opportunities than at any time in our history to leverage different types and combinations of data and utilize this information to improve decision quality. All data potentially leads to knowledge, and knowledge can lead to greater understanding. 

Qualitative assessments are used in various applications, including asset management, risk management, human reliability analysis, and customer surveys. The usefulness of any qualitative assessment is a function of design, analysis, and administration. This article explores aspects and good practices of survey administration.

Are you Facilitating with FINESSE?

References

Rensis Likert, R.S. Woodruff, editor, Archives of Psychology, Columbia University, New York, Volume XXII, Nos. 146-146, 1932-1933, pp.4-43.

Handbook of Human Factors Testing and Evaluation, 2nd edition, edited by S.G. Charlton and T.G. O’Brien, publishers Lawrence Erlbaum Associates, Inc., 2002.

Institute of Defense Analysis, “ICH Q9 Briefing Pack II”, July 2006.

European Association of Methodology (EAM), International Handbook of Survey Methodology, edited by E.D. de Leeuw, J.J. Hox, and D.A. Dillman, 2008.

J.D. Solomon, Daniel Vallero, and Kathryn Benson, “Evaluating Risk: A Revisit of the Scales, Measurement Theory, and Statistical Analysis Controversy,” Proceedings of the 2017 international Reliability and Maintainability Symposium.


JD Solomon Inc provides solutions for facilitation, asset management, and program development at the nexus of facilities, infrastructure, and the environment. Founded by JD Solomon, Communicating with FINESSE is the community of technical professionals dedicated to being highly effective communicators and facilitators. Learn more about our publications, webinars, and workshops. Join the community for free.

Filed Under: Articles, Communicating with FINESSE, on Systems Thinking

About JD Solomon

JD Solomon, PE, CRE, CMRP provides facilitation, business case evaluation, root cause analysis, and risk management. His roles as a senior leader in two Fortune 500 companies, as a town manager, and as chairman of a state regulatory board provide him with a first-hand perspective of how senior decision-makers think. His technical expertise in systems engineering and risk & uncertainty analysis using Monte Carlo simulation provides him practical perspectives on the strengths and limitations of advanced technical approaches.  In practice, JD works with front-line staff and executive leaders to create workable solutions for facilities, infrastructure, and business processes.

« So, What’s Still Wrong with Maintenance
Risk Prioritization in FMEA – a Summary »

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Headshot of JD SolomonArticles by JD Solomon
in the Communicating with FINESSE article series

Join Accendo

Receive information and updates about articles and many other resources offered by Accendo Reliability by becoming a member.

It’s free and only takes a minute.

Join Today

Recent Posts

  • test
  • test
  • test
  • Your Most Important Business Equation
  • Your Suppliers Can Be a Risk to Your Project

© 2025 FMS Reliability · Privacy Policy · Terms of Service · Cookies Policy