In regulated environments like medical device manufacturing, it is not enough for an AI model to achieve high accuracy. You must also be able to document exactly how it works, how it was trained, and how you ensure it continues operating within controlled boundaries.
At JLI, we work a lot with AI-based vision systems for regulated industries, and in this article, we address some of the key challenges in validating AI for quality control.
Eliminating the “black box” problem
One of the main concerns surrounding AI in regulated industries is the so-called black-box problem. Many AI systems can produce highly accurate results, but without sufficient transparency into how those results were achieved.
That level of uncertainty is unacceptable in medical device manufacturing.
To validate an AI vision system, the entire development process must be documented in detail - from data collection and annotation to training, testing, and deployment. Every step must be reproducible and traceable.
This places extremely high demands on how the AI model is trained and maintained over time.
Maintaining strict separation between datasets
One of the most important requirements is having full control over the images used throughout the process.
When training a model, we divide the data into three separate pools with watertight boundaries:
-
Training data
-
Validation data
-
Test data
A typical auditor question is simple but critical: How can you prove that the same image was not used for both training and testing?
To address this, we use a sophisticated system of checksums and digital tracking that uniquely identifies every image and ensures it cannot be accidentally moved between datasets or modified unnoticed during the process.
Handling new training data correctly
AI-based vision systems are rarely static. Over time, additional training images may need to be introduced to improve the model or account for new production variations.
In regulated environments, this process must be handled carefully.
New images cannot simply be mixed into the existing datasets, as this may introduce unintended bias or compromise the integrity of the validation process. Instead, new data must go through the exact same structured handling, splitting, and documentation process as the original training material.
Without this discipline, even a highly accurate model may no longer be considered validated.
Annotation must be fully documented
Annotating training images is another area where regulatory requirements significantly increase complexity.
In a standard AI project, annotation is often treated as a practical task. In a validated environment, annotation itself becomes part of the documented process.
We must be able to document:
- Who annotated the images
- Their qualifications and background
- Which criteria they followed
- That all annotations were performed consistently
This ensures that the AI model is trained on reliable and standardized data rather than subjective individual interpretations.
Monitoring the model after deployment
An equally important requirement is ensuring that the system continues operating within the conditions it was originally trained for. In other words, the model must not make uncontrolled decisions when presented with unfamiliar input.
We handle this through model monitoring.
By extracting statistical parameters from incoming production images and comparing them with the training data, we continuously verify that the inspected products remain within the expected operating range.
If the input begins drifting away from the training distribution, the system can raise an alarm before inspection reliability is affected.
This ensures the AI system remains within its validated boundaries.
Vision systems in the medical industry require more than accuracy
Even though the real goal is achieving 99.9% accuracy, accuracy alone is not enough in regulated environments. A validated AI vision system also requires:
- Full traceability
- Controlled data handling
- Consistent annotation procedures
- Transparent model training
- Continuous monitoring after deployment
It is a demanding process, but with the right tools, infrastructure, and methodology, it is possible to deploy AI-based quality control in highly regulated environments.
And as AI becomes increasingly important in medical manufacturing, those processes become just as critical as the models themselves.
.jpeg)