Chapter 6

Develop Certifiable AI Models

In recent years, the first AI-based medical devices hit the market, a milestone for medical industry innovators using proprietary black box algorithms to diagnose or treat a disease.

Developing AI-based models that can be certified through a regulatory authority is a key challenge for medical device developers. Most recent devices approved by regulators like the US FDA have been seen as low risk for patients and have had pre-market approval or De Novo classification as low-risk devices.

Regulatory authorities have yet to approve a device determined to have high potential risk to patients, such as a diagnostic algorithm where a false positive could lead to a risky procedure. Extra controls will likely be needed to approve such an algorithm.

With MATLAB and Simulink, it is possible to not only develop AI models, but also to collect the data needed to ensure software quality. In addition:

• MATLAB can help you build an explainable and interpretable AI model.
• You can also follow a formalized process to validate that the model adheres to the IEC 62304 guidelines during the model development phase, which is a requirement for certifying the AI-based Software as a Medical Device (SaMD).

Interpretability and Explainability

Interpretability and explainability are closely related. Interpretability is used more often in the context of classic machine learning, while in the context of deep neural networks many use “AI explainability.”

Interpretability and explainability are the degree to which machine learning algorithms can be understood by humans. Machine learning models are often referred to as “black boxes” because their representations of knowledge are not intuitive and their decision-making processes are not transparent. As a result, it is often difficult to understand how they work.

Interpretability techniques help to reveal how machine learning models make predictions by revealing how various features contribute (or do not contribute) to predictions. Interpretability techniques can help you:

• Validate that the model is using appropriate evidence for predictions.
• Find biases in your model that were not apparent during training.

Some machine learning models, such as linear regression, decision trees, and generative additive models are inherently interpretable. However, interpretability often comes at the expense of power and accuracy.

Interpretability has been highlighted as a standard guiding principle in the Good Machine Learning Practice document for medical device development published jointly by the US FDA, Health Canada, and the United Kingdom’s Medicines & Healthcare products Regulatory Agency (MHRA) [6]. Several techniques can be used to produce a successful interpretation result for the various models by using MATLAB for machine learning and deep learning. Examples include:

• Local interpretable model-agnostic (LIME) explanations
• Partial dependence (PDP)
• Individual conditional expectation (ICE) plots
• Shapley values
• Class activation mapping (CAM)
• Gradient-weighted class activation mapping (Grad-CAM)
• Occlusion sensitivity

Interpretability and explainability are a first step towards understanding your AI models. However, as you move to later steps in the workflow, you may also want to ensure that your model has been built in a robust manner. Neural networks can be susceptible to a phenomenon known as adversarial examples, where very small changes to an input (often imperceptible to humans) can cause the input to be misclassified. You can use Deep Learning Toolbox Verification Library to verify whether a deep learning network is robust against adversarial examples and to compute the output bounds for a set of input bounds.

Model Validation and Deployment

Medical device manufacturers are responsible for validating their AI-based models in partnership with clinical researchers. The Good Machine Learning Practice document also emphasizes the importance of validating AI-based models by performing clinical studies that include the intended patient population [6].

With MATLAB, you can share an AI-based application with clinicians or hospitals for validation. MATLAB provides you options for creating a standalone application that could be shared, installed individually for the end-user, or hosted on a web server for multiple user access through a web browser.

MATLAB also supports capabilities for deploying AI applications on production servers for quickly processing concurrent requests, together with parallel servers for processing larger patient data volumes in batch mode. There are options for generating self-contained, deployable C/C++ code of the algorithms that could be easily deployed standalone or integrated with existing clinical application for performing validation studies.

Certification Workflows (IEC 62304)

MathWorks makes it easy for you to adhere to IEC 62304 guidelines for developing software as a medical device. MATLAB workflows incorporate verification and validation into the software development workflow. As a result, the software is comprehensively tested and verified before you integrate it into a medical device.

MATLAB also supports software development processes by providing capabilities for implementing detailed software development planning, software requirement analysis, software architectural design, software detailed design, software unit implementation and testing, software integration and testing, and software system testing as a part of the medical device development workflow. In addition, MATLAB and Simulink will help you assess the impact of a proposed software change by enabling simulation of the affected software items.

MATLAB will generate most of the regulatory compliance documentation required by the IEC 62304 standard for software development and software maintenance. You can read more about the IEC 62304 workflows with MATLAB and Simulink in the Developing IEC 62304–Compliant Embedded Software for Medical Devices white papers.

Khwaja Achieves IEC 62304 Certification

Challenge

Electrocardiogram (ECG) data analysis is essential for the recognition and treatment of cardiac diseases.

Results

Working with Simulink, the team reduced the development time by 40% compared to traditional approaches by adhering with the reference workflows from IEC Certification Kit to verify and validate the models and automatically generated code. They also accelerated the certification and audit with TÜV SÜD for ISO^® 13485 and IEC 62304.