Weibull Analysis & Risk Forecasting
• 8:30 am – 4:30 pm
• Lunch included
• In-class work problems – bring your laptop
Outline
• Weibull analysis – what is it?
• Weibull Plots
o Estimation methods
o Understanding confidence bands
• Goodness-of-Fit measures
• Interpret Results
o Infant mortality, Random/Constant, Wear-out
• Weibayes Analysis, 3-parameter Weibull
• Risk Forecasting
o Issues: Mitigation effectiveness, population growth
• In-class examples
Instructor:
Trevor Craney, ASQ Fellow, CRE, CQE
Bio:
Trevor Craney is a Sr. Tech Fellow for Statistics, Risk & Reliability at
Collins Aerospace. He has primary responsibility for safety risk analyses,
including performing analysis, training and evaluating individuals to be
authorized risk analysts. Trevor holds an MS degree in Statistics from the
University of South Carolina, an MS degree in Industrial & Systems
Engineering from the University of Florida, and an MS degree in
Reliability Engineering from the University of Maryland.
Effective Reliability Test Development with DFMEA Integration (Matt Anfang, Keyanna Qi)
1 Introduction
Reliability testing had been challenging, especially of new reliability engineers or for the company which just established new team. This presentation will give overview the types of reliability test, and how to integrated DFMEA concept into test development process.
2 Type of Reliability Tests
This will discuss the Pros and Cons for traditional reliability demonstration test, ALT and HALT. In the real world, reliability engineers are critical team contributor in many other types of design verification and validation tests, like new supplier evaluation, design optimization, field failure root cause analysis, firmware bug fix, etc.
3 Test Development Process
DFMEA has been widely used in NPD (New Design Development) projects by many companies as risk mitigation tools. The concepts of DFMEA could be integrated into test development process
1) Test justification: why we need test it?
2) Design requirements: understand design specification (limits) and/or reliability requirements
3) Critical Functions: what should be tested?
4) Major Failure Modes: what should be detected?
5) Test Conditions/Matrix: based on previous steps, understand causes of failure and failure mechanisms apply reasonable stress level,
6) Failure Detection: clearly define “failure”, and requirements of data acquisition.
4 Case Study
Two or three examples on different types of tests which following the process with review and approval process.
– Door switch life cycle test
– GUI firmware validation
– Unit level long term reliability tests
5 Summary
Integrated DFMEA into reliability test development process, it will improve effectiveness of tests by understand the design requirements, intended functions, failure modes, and causes of the failures with failure mechanisms. With well-defined test development process, it also
– Drive design specification improvement
– Shortens the test fixture development time with clear requirements.
– Easy for test data analysis and test reporting
This study outlines the reliability qualification process of drawer sub-systems of commercial refrigerated display cases via the DMAIC process. The drawer was defined from marketing via customer requirements. The critical details of the design are then identified via FMEA process. Then the design was analyzed through performance & reliability testing. With the testing, the results were fed back to respective project members & product vendors, such as via the FRACAS process, to improve the design. Lastly, with the completed design, lessons learned were explained to the Quality Assurance group so that they can continue to monitor the end-of-line tests and control charts.
This document highlights the usage of a structural hang test on commercial doors to assess robustness. This HALT/HAST is used to rapidly identify design issues in the door structure that are normally found after thousands of door cycles is traditional reliability testing, reducing months of testing to a day. By evaluating commercial doors under this type of stress, it will expose potential failure points within the system earlier in the design process.
This hang test is used to apply a load that stresses the door structure in it’s intended orientation this allows design engineers to quickly assess door structural concepts reducing product development cycle times and time to market while increasing quality and reliability of the doors our customer purchase. By performing this test, we have developed a standard that is applied across all commercial doors ensuring consistent quality and reliability in our designs.
During the development of products, the proof of product reliability is often a challenging activity. Basic planning actions have to be executed in an early design stage of the product. One possible approach is the zero failure testing, also called reliability demonstration testing (RDT).
At the planning phase, many questions have to be analysed to perform a reliability proof:
• What is an appropriate reliability target for the product?
• What could be a useful product level for testing – system test vs. component test.
• What is the probability of a successful reliability proof?
• Which number of specimen and testing duration is necessary?
• How to handle the unlike event of product failures while performing a RDT?
• Which uncertainties have to be regarded and how to consider in the evaluation phase.
In addition to a procedural flow of test planning, correlations between reliability requirements and various test parameters such as test duration and sample size are shown. The test quality is evaluated by the confidence level, whose dependencies are also determined.
The statistical basis is the Binomial distribution. The practical implementation of RDTs is often challenging, but at the same time offers a solid methodological foundation. Limits to realization can arise, for example, with high product requirements, which then cause very high sample sizes and test durations. The consideration of uncertainties in the input parameters of a test planning must also be taken into account.
After performing a RDT, the impact of uncertainties on the results are once again under consideration. The effects of uncertainties of the planning phase, e.g. uncertainties of the estimation of the Weibull shape parameter, to the demonstrated reliability and the confidence level are shown. Finally, case studies give an impression of the opportunities, challenges and limitations of RDTs. Some hints for the practical implementation close the whole topic.
In this talk, I will draw on my research experience from the past twenty years on accelerated life testing (ALT) to discuss how taking a different perspective of statistical modeling can open a new way to the solutions of several engineering problems. ALT is a classic technique used by reliability engineers for predicting product failure time distribution. The responses of these experiments are failure times, but oftentimes, failure times cannot be observed; instead, only survival times are observed. That is the so-called censored response. Instead of modeling failure time, it can be shown that the design and analysis of ALT can be simplified by modeling a discrete pseudo-variable via generalized linear models. At the same time, more complicated and realistic testing methods can be studied by this approach.
Overview of research and resulting methodology to deliver consumer focused reliability. Redefining the classical reliability paradigms to meet the needs within a global market that are continuing to transform regarding expectations of quality and reliability in products and services.
Keywords: Consumer loyalty, Robustness, Reliability, New Product Development, Design for Reliability, Risk Assessment
The off-highway sector is witnessing transformation through integration of evolving technology stack to revolutionize production systems in agriculture and construction to unlock customer economic value across the lifecycle. At the forefront of this revolution are connectivity, precise measurements and autonomous solutions, which not only promises to manage peak seasons and achieve increased crop yields but also introduces new data streams, creating opportunities for more innovative growth in reliability engineering methodologies.
This paper introduces methods that applies artificial intelligence (AI) to create a customer usage profile for agriculture equipment use across various operations and tasks, developing deep customer understanding on how customers get work done through a series of job steps to produce their output. For this purpose, diverse data sets, like customer experience data across lifecycle, telematics data, dealer business systems data and farm specific agronomic data are utilized.
Building on these insights, customer use profiles can then be applied to various reliability engineering analysis methods to provide a deeper and more precise insights into the overall reliability of the system.
Positioned at the intersection of reliability engineering and data science, this approach highlights the potential of data-driven strategies and showcases the transformative impact of integrating of AI in advancing reliability engineering.
This talk presents a Bayesian modeling methodology for estimating failure probabilities under low-probability, high-consequence events. Specifically, a hierarchical Bayesian regression model is proposed to combine an analytical model with experimentally observed capacity data to predict failure probability under hurricane wind loading. The goal is to predict the failure probability of residential building roof systems and number of damaged buildings in building portfolios under severe wind loading to determine the cost-effective wind mitigation options. The proposed approach is illustrated on a case study involving a residential building portfolio in southeast Florida.
Historically when R&M community discussed risk management in medical device industry, they only considered product risk that affected consumer safety. However in today’s changing world companies need to consider risks from cradle to grave such as project management risk, human capital retention risk, component sourcing and supply chain management risk, regulatory compliance risk, data privacy and security risk, and end of life product disposal risk. This paper/presentation outlines all the risks a company needs to manage to be successful and provides tips to manage these risks.
Risks that will be discussed:
• Project Management Risks:
o Requirements Gathering
o Resource and budget estimation
o Managing global team (especially communication)
o Project risk register (including example template)
• Human Capital Retention Risk
o Forming a high performing core project team
o Ensuring high engagement level of all team members
o Retaining people throughout life cycle of product
• Component Sourcing and Supply Chain Management risk
o Globalization – parts sourced from all over the world ( e.g. managing LiIon battery shipping requirements)
o Managing multi-tiered component suppliers (especially investigations and supplier corrective actions)
o Part shortage
o Managing ramp down of current product and ramp up of new product
• Regulatory Compliance risk
o EU MDR compliance (mandatory in EU by 2020)
o Managing labeling and compliance to individual country regulations (including wireless connectivity regulation examples)
o ISO standards compliance risk
• Data Privacy and Security risk
o Ensuring data security (HIPPA, financial data etc.)
o Managing data server location regulations
• End of Life Product Disposal risk
o RoHS, REACH, e-waste disposal
o managing sustainability index
All the risks, tips to manage risk, templates, and examples will be from the medical device industry but they can be applied in non-medical applications too.
It is well known that FMEAs are best when teams do them in working meetings versus individually. A team made up of varied backgrounds and levels of expertise can provide valuable insights.
It can be challenging to facilitate working meetings with a cross-functional team. Getting started can be difficult. Keeping a series of meetings meaningful with exploratory discussions can also be a challenge.
Facilitators can follow a discussion method to get ADEPT at these types of working meetings. ADEPT combines best practices from quality, product design sprints, and brainstorming alternatives. It is a framework anyone can use to guide their teams through working meetings and help them move the team forward toward action.
• Align to a goal
• Discover individually and share
• Examine together
• Prioritize for action
• Teamwork follow-up
Advanced packaging combines dies and surface mount components into system-in-package applications, thus packing more into highly compact spaces within a cadence set by Moore’s Law. As the demand increases for sophisticated AI devices, manufacturers require higher performance and productivity from their manufacturing equipment. Quality and reliability assurance proves challenging in this time-cost tradespace, where supply chain co-development complexity carries a variety of risks in addition to those already intrinsic to high-value low-volume products. This presentation outlines an overview of technical and organizational challenges in achieving high levels of quality and reliability for advanced packaging deposition equipment. Additionally, we provide an update on the neutral prior Bayesian reliability and quality estimation work presented previously.
Reliability engineers focus on failures. We also want to help set target reliability goals for innovative concepts, based on customer expectations.
How do we transition from our usual failure mode-thinking into a useful examination of a concept for potential reliability goals? Instead of focusing on potential customer symptoms of a problem, we focus on targeted customer benefits. We break apart our benefits, examine their potential drivers, and utilize a quality Kano Model to help us prioritize.
The US Congress created a new category of medical devices, the cyber device, defined as a medical device that contains software and has the ability to connect to the internet. The law requires that device manufacturers develop a plan to monitor postmarket cybersecurity, provide postmarket updates, generate a software bill of materials, and comply with other FDA requirements.
FDA has issued final guidance documents that create the device cybersecurity framework. FDA views cybersecurity as applied medical device risk management following ISO 14971:2019. In particular, the guidance document invokes AAMI TIR57:2016/(R)2023 Principles for Medical Device Security—Risk Management. Because the risk management standard came after the AAMI document, the alignment is not always clear.
In addition, the guidance documents explain threat evaluation, cites standards for the software bill of materials (SBOM), and provides information on postmarket updates to customers.
The aim of the presentation is to explain FDA’s cybersecurity framework for medical devices using the guidance documents. This includes the risk management elements, the software bill of materials, submission requirements, threat evaluation, and postmarket updates. The presentation takes a life-cycle approach to implementation.
Circuit board-level electronic component lifetime prediction has been a long-studied topic in the reliability engineer field, ranging from decades-old actuarial handbook predictions to more modern simulation-based approaches. These methods are most commonly used to create design-specific reliability predictions prior to production, often utilizing representative test environments or conservative “worst-case” loading conditions.
For example, a ball grid array (or BGA) is a type of electronic component commonly soldered to circuit boards. Its physical characteristics can make it prone to solder fatigue during thermal cycling. A variety of accepted methods exist for a designer to predict solder lifetime during expected environmental thermal cycling when choosing a BGA package. Similar prediction methodologies exist for other component types and loading events. These predictions are extremely valuable for estimating the expected life for a class of products and identifying design issues that must be addressed prior to fielding a product. However, because they focus on the design itself, they do not provide any insight to individually fielded products.
In recent years, a new frontier for electronics reliability engineering has emerged – real-time monitoring of consumed life for individual circuit board assemblies in the field. The goal of such an approach would be to refine the pre-production design lifetime predictions for each individual product as it is used by the end customer. The methodology is still in its infancy and could potentially be executed in a wide variety of ways on a product-to-product basis. One thing is evident however – any effective implementation of a real-time electronics wearout monitoring system would require a strong understanding of a circuit board’s critical components and their most likely failure modes in a given environment. It would also require a robust feedback system that accurately captures and reports relevant assembly-level, board-level, and component-level environmental load data.
This talk will present a high-level overview of the current challenges and considerations for real-time electronics reliability monitoring. It will examine how to use existing reliability modelling approaches to identify the critical components in a design and the key loads they experience. It will also investigate the types of input information that would be necessary for a reduced order reliability model of an electronic component.
In an era of widespread uncertainties across various sectors, an effective framework for determining risk controls is crucial for robust risk management. This presentation delves into a systematic methodology for identifying risk controls, using a simplified analysis. By focusing on a real-world case study, the presentation dissects the process of mapping potential risks to effective control measures and stresses the importance of aligning risk controls with organizational objectives. Attendees will gain insights into practical steps for translating complex risk scenarios into clear, manageable control actions in diverse operational contexts. Additionally, the presentation illustrates how artificial intelligence can be leveraged in control determination analysis.
For any product, software, or service, it’s expected that the device would be subjected to a failure at some point in time during the life cycle of the product. Such failures manifest as risks to the user and cause inconvenience or in some extreme situation’s lead to an injury to the user. Though individual risks may be mitigated as far as possible, a user may be apprehensive on opting a device considering the risk profile of a product. Such situations warrant a review on the results of the risk management process not just considering the outcomes of an individual risk but by comparing them to the benefits that the device has to offer. While the assessment may seem simple, considering the application of medical devices and the rather high risks the devices may result to, it’s critical that an established framework be used in determining the overall risk acceptance of a device and to give a conclusive benefit risk estimation. Use of such a framework is highly beneficial during product development process particularly when justifying risks which are defined to be “High” or “cautionary” but have been reduced as far as possible.
Regulatory agencies such as the FDA (Food and Drug Administration) in the United States and the EU MDR (European Union Medical Device Regulation) emphasize the need for a benefit-risk evaluation to determine the acceptability for use of a device. This evaluation process is fundamental in ensuring that the benefits of the device outweigh its risks, ultimately ensuring patient safety. Infact considering the increased importance to evaluation of benefit risk, ISO 14971:2019 version introduced a new definition for benefit that defines it as “The positive impact or desirable outcome of the use of a medical device on the health of an individual, or a positive impact on patient management or public health”.
To prevent polarization regarding the final risk acceptability for a device, it’s essential to compare the overall risk of the system against the benefit the device is expected to provide to the user. Further such an evaluation confirms the disclosures/statements which need to be included in product labeling to indicate users on expectations from use of the device.
The purpose of this presentation is to provide a comprehensive framework for establishing and determining the acceptability of the overall risk of a medical device and to outline how to define a benefit statement. This will be followed by a detailed discussion on the comparison of a qualitative versus quantitative methods for benefit-risk estimation utilizing results from the medical device risk management process. Additionally, the presentation will include a summary of existing available methods utilized by regulatory bodies to establish acceptance of benefit-risk.
The main objective of this topic is to raise awareness among engineers about the importance of benefit estimation and the methods that facilitate making an informed decision. By understanding the balance between risks and benefits, stakeholders can make more effective decisions regarding the development, regulation, and use of medical devices, ultimately improving patient outcomes.
Frustrated by the complexity of health insurance products, the customers moved away from a large healthcare payer organization. The complexity impacted the product development, sales and revenue tracking process, and made it difficult to provide the clients with what they need. Moreover, only about one-third of all features in a product were relevant to the consumers in that product group, and many features were duplicate across the product lines. In a product-driven organization, the definitions and parameters of product quality were getting lost in translation from the sales team to senior managers, senior managers to the developers, the development team to the marketing team, and so on. A strategic project was undertaken to isolate, index and prioritize each feature end-to-end, and to restructure the business operations around each feature. The strategic goal was to create a feature-based dynamic product offering system in which each consumer could select the features and build a health insurance product which serves their needs the most. In such a system, the organization is able to focus and prioritize resources towards most sought-after features and mobilize resources away from low performing features. The pricing was simplified for the customers as well. They could now pick a product basket and add all the features they need or can afford. It was a massive cultural change, training, education, and getting used to the new process for the organization. In a sense, the customers built the product as per their needs, and as a result, the customer support requests also went down.
Does your company perform Root Cause Analysis (RCA) or just failure analysis? What distinguishes these tools and when are they appropriate? Does your company have a written policy or procedure? Are these performed routinely after an incident or notable failure? Listen to one company’s process, challenges and successes in initiating an RCA program.
What do we say when our managers ask us how our FMEA process is going? Are we getting push-back on the value of FMEA and cannot answer it? Are we doing FMEA because we must for our files and not really evaluating the process itself for improvements?
We can plan to evaluate our FMEA processes with targeted tactical and strategic metrics. We borrow common methods from project management and enterprise risk management and apply them to our FMEA processes.
This investigates crack propagation of a Formica (a countertop material) used in commercial applications.
Assessment to study the of physics of failure, design methodology, and establishing a design guideline for future applications.
Fault Tree Analysis (FTA) is employed to first understand sources of high stress concentrations. Then a test method was developed to replicate crack formation and define primary contributors. Of three possible sources, it is discovered that the physics failure mode is primarily due to thermal stresses imposed by radiant heat cycling. Lastly, theoretical & experimental calculations are used to establish a design guideline for future applications and materials.
John Skowron, Steven Korda, Daniel Conrad
Advances in machine learning (ML) have led to applications in safety-critical domains, including security, defense, and healthcare. These ML models are confronted with dynamically changing and actively hostile conditions characteristic of real-world applications, requiring systems incorporating ML to be reliable and resilient. Many studies propose techniques to improve the robustness of ML algorithms. However, fewer consider quantitative techniques to assess changes in the reliability and resilience of these systems over time. To address this gap, this study demonstrates how to collect relevant data during the training and testing of ML suitable for the application of software reliability, with and without covariates, and resilience models and the subsequent interpretation of these analyses. The proposed approach promotes quantitative risk assessment of machine learning technologies, providing the ability to track and predict degradation and improvement in ML model performance and assist ML and system engineers with an objective approach to compare the relative effectiveness of alternative training and testing methods. The approach is illustrated in the context of an image recognition model, which is subjected to two generative adversarial attacks and then iteratively retrained to improve the system’s performance. Our results indicate that software reliability models incorporating covariates characterized the misclassification discovery process more accurately than models without covariates. Moreover, the resilience model based on multiple linear regression incorporating interactions between covariates tracks and predicts degradation and recovery of performance best. Thus, software reliability and resilience models offer rigorous quantitative assurance methods for ML-enabled systems and processes.
Contention & Chaos in the Courtroom:
How Reliability and Statistical Methods Prevent Product Liability Drama
Presenter: Allise Wachs, Ph.D.
Abstract: Large manufacturing companies spend billions of dollars on warranty and product liability lawsuits annually. Warranty, product liability suits, safety recalls, and associated costs could easily be minimized with the proper use of industrial statistical and reliability methods. Many product defect cases are a result of suboptimal designs or unstable manufacturing processes.
The presentation will describe product liability cases through the eyes of a reliability expert witness. In these cases, the proper use of reliability/warranty analysis, measurement systems assessment, statistical process control, designed experimentation, and data analysis would have prevented excessive warranty burden, costly lawsuits, safety recalls, and reputation losses to manufacturers.
In any manufacturing or service oriented endeavor to produce something of exceptional quality, or provide excellence in service to a growing clientele in general, this abstract proposes a new framework that combines risk management into quality design so that it is not an afterthought, but a strategic aspect built in from the very beginning of a project formulation. This methodology captures key quality principles using latest methodologies to weave in risk practices thus producing a more robust platform for anticipatory quality control.
The methodology proceeds from an extensive risk identification exercise in which all potential quality hazards along the product journey are enumerated and then risk-rated using quantitative techniques such as Failure Mode and Effects Analysis (FMEA) and Fault Tree Analysis (FTA); where a fault tree relates an occurrence of a significant quality hazard (eg, an engine failure) to a set of basic events (eg, leaking oil) and the means through which these might be causally linked (eg, faulty bearings, wiring, seals). The quality hazard can be approached with a greater degree of severity and probability.
Next, these risk assessments are integrated into QFD so that customer requirements are met and secured against an identified risk. Here, the framework positions ‘risk mitigation’ as a design activity externally aligned with the ‘customer-centered quality’ of QFD to achieve an overall quality- and reliability-oriented design ethos.
Also, the framework is demonstrated to work with two case studies from the automotive and healthcare industries. The case studies describe how this framework could have been applied within these industries. The case studies also illustrate how the framework can be applied across different industries, and can be used to deal with diverse quality challenges.
The abstract concludes that: ‘This integrated consideration will have implications on future quality management practices where proactive risk management embedded in quality design is not only to prevent risk failures but also an essential part of organizational resilience and customer confidence.’
It is in this way that the framework pioneers risk-based quality design – one that best meets the challenges of our time, in which new demands on the humanities push for excellence and dependability, and where planetary-scale complexity and uncertainty point to breakdowns.
Remaining life assessment (RLA) and reliability engineering (RE) of complex assets has always been a challenge for the process plant industries such as hydrocarbon, petrochemical, fertilizers, and green energy. Complex assets such as well stimulation service units in the hydrocarbon upstream, feed filter package, air fin coolers in hydrocarbon downstream, and other skid mounted assets across all process plant industry segments require a comprehensive reliability engineering framework that addresses both the loss of containment fluid scenarios for pressure boundary sub-assets and the loss of function scenarios for the dynamic and non-pressure boundary sub-assets. Failure mode effects and criticality analysis (FMECA) methodology has proven to be effective in the remaining life and reliability engineering assessments of complex assets across all process plant industry segments. This technical presentation will outline a FMECA driven thorough reliability engineering framework (holistic methodology) that evaluates both the mechanical integrity (loss of containment fluid) and reliability (loss of function) aspects of complex assets such as air fin coolers in the hydrocarbon downstream facilities and well stimulation service units in the hydrocarbon upstream facilities.
In this talk, we discuss the challenging issues that reliability practitioners face in conducting destructive tests for one-shot devices, especially in estimating the defect rate of products. The working condition of a one-shot device is unknown until testing the device. We consider one-shot device test data with defects induced in a realistic manufacturing process. Manufacturers need to measure the defect rate for quality control purposes, but obtaining enough defective devices for accurate estimation is challenging when the defect rate is relatively low. Frequentist and Bayesian approaches are proposed to estimate the mean-time-to-failure and defective rate. Masked data are also considered when we cannot distinguish whether a failed device is initially defective. A Monte Carlo simulation study is conducted to evaluate the impacts of the masking effect on the estimation under different settings. Some practical guidelines and recommendations are provided.
The existing definition of reliability is known as the ability of a product to consistently perform its intended functions under specified conditions for a specified period. A widely recognized situation in industry, however, is that this definition does not necessarily tell exactly what it means, and it can help with when people may need it to communicate or carry out a task with a specific reliability goal. people could say something about reliability which means different to others. There is an obvious gap between the definition and our day-to-day practices, making reliability remain nebulous in concept.
This article provides an insight into, from a fundamental standpoint, and therefore, a further understanding of this issue about why this happens and what we could do to possibly close the gap. It presents examinations of some widely known and discussed discrepancies and/or issues we see in real world engineering practices for reliability, including:
These examinations suggest potential benefits we could possibly receive in associated engineering practicing of maintenance/RCM, asset management, and decisions from this fundamental understanding of reliability.
Major sections of this article will include:
1.Introducion, existing definition of reliability and associated issues
2.Differentiating a concept from technical terms
3.Ultimate dominance of statistics in reliability
4.Reliability defined on random events
5.Reliability, a measure as well as an objective
6.A tool or an approach
7.A matter of decision making