Hybrid Vision: Pushing the Boundaries of Quality Control

How to solve the biggest challenges in industrial QC with innovative machine vision systems

Download the guide as PDF

Why use Hybrid Vision?

How can we produce faster? With higher quality? With less labor? More sustainably? 

The evolution of industrial production has always been driven by a need for optimization, and here automation plays a key part. This is also true when it comes to quality control and visual inspection.

Today, the need for automated quality control is becoming increasingly important. A key driver is the increasing tendency of European manufacturers towards nearshoring. This strategy focuses on ensuring robust supply chains but has the side effect of more expensive domestic labor markets. This economic landscape reinforces the need to automate manual inspection activities.

In some cases, visual inspection is easy, but how do you automate the more complex inspection tasks - the ones that don't fit easily into a formula? 

The rapid development of AI and machine learning is opening up new possibilities, but AI cannot stand alone. There is a need to unlock multiple toolboxes, which is why we at JLI vision have introduced Hybrid Vision. 

Hybrid Vision combines traditional machine vision in 2D or 3D with machine learning and unites the two with a third element: craftsmanship. The result is solutions that push the boundaries of what vision technology has been able to achieve in quality control.

In this guide, we'll explain how Hybrid Vision works and describe 5 use cases for this technology, so you can be inspired to see new ways to optimize your production.

What is Hybrid Vision?

Hybrid Vision is the combination of traditional 2D or 3D machine vision, machine learning, and craftsmanship. 

Traditional 2D machine vision has been used for decades for automated quality control in the manufacturing industry. It basically consists of one or more cameras that capture an image and a computer that processes that image and, based on an engineer-coded algorithm, determines whether the inspected part meets the specified quality requirements. 

In recent years, 3D technology has literally added an extra dimension to the possibilities of machine vision.

Traditional machine vision is ideally suited for inspecting dimensions, shapes, numbers, positioning, and a wide range of defect types that can be described and narrowed down. 

Machine learning often comes into play when the sample space becomes difficult to describe in an algorithm - for example, when inspecting aesthetic surface defects that require an experienced inspector to identify and assess. By training a neural network with images of good and bad samples, we can use machine learning to mimic the experienced inspector's assessment and achieve a more consistent and accurate inspection process.

Even though machine learning standard components have become available off-the-shelf, it's still not plug-and-play to buy a smart camera and set it up in complex industrial production environments. It usually requires a customized solution. To achieve the optimal result, you need deep insight into the use of ML and a vision technology skill set that involves thinking creatively and controlling the camera and light to capture suitable images for inspection. 

In a Hybrid Vision solution, the combination of elements could look like this if you were to inspect, for example, a wooden panel for aesthetic defects:

Hybrid Vision - Pre-processeing
Step 1

Pre-processing with traditional vision

First, we pre-process the image with traditional 2D/3D vision.

To optimize the process and avoid the neural network having to process an entire image, which is both resource-intensive and time-consuming, traditional vision technology is used to identify possible defects in a given subject - in other words, to point out areas that need further investigation.

Hybrid Vision - Processeing with ML
Step 2

Processing with ML

The selected areas are then served to a neural network trained on a large number of good and bad samples, which processes the image and gives it a score that defines whether it is within the established quality standard. 

Hybrid Vision - Post-processeing (1)
Step 3

Post-processing with traditional vision

If the score is a borderline case, you can choose to do post-processing with traditional vision technology, looking for specific, predefined defect types or, for example, measuring the size of defects.

Craftsmanship as the foundation

Image quality will always be paramount to the outcome of the inspection, which is why "craftsmanship" is the glue that binds the elements of Hybrid Vision together.

Both in terms of lighting, exposure, optics, and other imaging parameters and when it comes to the ability to train the neural network in the best possible way for the specific task.

Point is, it's the combination of all the elements that makes it possible to push the boundaries of what's possible with machine vision.

When should you choose Hybrid Vision?  

Often the need for Hybrid Vision increases with the complexity of the task. Hybrid Vision is particularly well-suited for tasks that would otherwise require a trained eye and a certain amount of experience. The best example is inspecting surfaces for aesthetic defects - in other words, whether the surface "looks good enough" to pass quality control. 

This is typically an assessment based on so many parameters that it is difficult to specify the sample space precisely enough to develop an algorithm that is sufficiently robust. That's why we turn to artificial intelligence, as we can train a machine learning network to make the same assessment as an experienced inspector. 

In the case of large surfaces that need to be inspected for defects, it will often be very resource-intensive to process the entire image with a machine learning network. This calls for a Hybrid Vision approach, where we can use traditional 2D vision to pre-process images and identify the areas of the image that contain potential defects so that only the relevant pixels are run through the neural network.

So, Hybrid Vision makes sense when traditional vision and machine learning can be combined to create a more robust solution and when deep vision expertise is needed to capture suitable images. 

This will often be in the context of industrial production environments where the physical environment challenges the ability to capture the right images. This can be both the size of the items, surface, material, etc. as well as the physical environment, e.g. speed of the conveyor belt, lighting conditions, temperatures, clean rooms, dusty production environments, etc.

When is Hybrid Vision unnecessary?

If the task is simply to "count pixels" and measure dimensions, quantities, or similar to assess whether a part is within the specified tolerances, it is rarely necessary to use machine learning. Here, traditional 2D or 3D vision can stand alone as the most efficient way to achieve the goal. 

Similarly, some tasks can be handled with standard machine learning solutions. These will typically be relatively simple tasks where it is easy to identify and annotate the errors and where the sample space in relation to errors is smaller. In standard ML solutions, no preprocessing is done to select image sections with candidates, instead, the entire image is processed. Therefore, it is typically a prerequisite that the production takes place at a relatively low speed or stops completely so that the network has time to process the image and find any errors.

What is Hybrid Vision not suitable for?

If the production is low volume or consists of small batches of different items, Hybrid Vision is typically not a suitable approach. 

This is because it takes a lot of work to train a machine learning network to be robust, and therefore it is not easy to adapt the vision solution to a new production where new factors need to be taken into account.

5 examples of Hybrid Vision use cases

Example 1

Inspection of aesthetic defects on glulam boards

Purpose

Inspection of knots and resin pockets in glulam boards.

Hybrid vision Gluelam boards-1

Challenge

When using wooden boards for quality furniture, it is very important to detect knots that can cause holes or are at risk of falling out prior to using the board in a costly production. Some resin pockets can also cause problems, as the resin becomes more fluid if the furniture is placed in a warm environment and can cause yellow stains if the wood is lacquered afterward.

In this kind of wood inspection it is very difficult to distinguish between acceptable knots and resin pockets and those that mean the panel should be discarded or repaired. It's a combination of several factors that determine whether the board is good or bad, and traditionally, it would take an experienced inspector to determine this. 

It's a resource-intensive process, and since there will be multiple inspectors working on the line, their subjective assessments will often produce inconsistent results.

Solution

The glulam boards are scanned with multiple line scan cameras as they pass the inspection system on the conveyor belt.
First, traditional machine vision is used to identify potential defects and select candidates for further processing. We use 2D inspection to identify knots and resin pockets and 3D inspection to inspect the surface for irregularities and holes.
 
The selected candidates are then processed in a machine learning network that has been trained on up to 30,000 good and bad images to learn which knots and resin pockets can pass a quality check.

This results in a very high level of accuracy of +95%, which is far better and more consistent than manual inspection. This means that fewer defects pass quality control and fewer good boards are rejected by mistake. 

This Hybrid Vision solution also post-processes the defects that the ML network identifies. Whether a knot is loose or not is important and cannot be assessed by simply looking at an image. However, by linking images of the top and bottom of the plates, it is possible to determine if the knots go all the way through, which increases the likelihood of a loose knot. 

The machine learning network cannot determine the size of the detected knots, but this information is essential for the repair of the knots. Therefore, the images of knots are post-processed with 2D vision. An algorithm calculates the size of the knots that need to be repaired. This information is then passed on to the repair equipment on the production line. 

Why Hybrid Vision is necessary

The variation in knots and resin pockets is high, and it would be very expensive or downright impossible to create an algorithm that could classify all defect types. At the same time, it would be too time-consuming to process an image of the entire glulam board with a neural network. It would require the production line to work slower, which would be counterproductive.

However, by combining the two approaches, it is possible to get the best of both worlds and achieve an efficient, automated inline inspection with a high level of accuracy.

Example 2

Inspection of cracks in bricks

Purpose

To detect small cracks in the surface of bricks 

Hybrid Vision craks in bricks

Challenge

Bricks are made of an organic material with a natural surface texture that will vary from brick to brick. However, there should be no cracks in the bricks, as in addition to the aesthetic appeal, it can become a functional problem if water penetrates the bricks.

However, it can be difficult to distinguish small cracks from other structures in the surface, and not all cracks are an issue. Only if they exceed a certain size. As there will be a number of borderline cases, the challenge here is to develop a solution that can robustly identify the bricks that have critical cracks and need to be discarded.

Solution

The developed solution consists of three parts:

First, a pre-processing of images of bricks, where candidates - i.e. possible defects - are selected. Next, the image sections with candidates are processed in a machine learning network, and finally, post-processing is done with traditional 2D machine vision to determine the size of the cracks found.

To identify cracks with a high degree of accuracy, we use a machine learning network trained on a large number of images of bricks without defects and bricks with cracks. In this way, the network has learned to recognize the cracks and gives each brick a score based on the probability that there is a crack in the brick. 

The selected candidates are then processed using traditional 2D machine vision, which measures the size of the crack and defines whether the defect is outside tolerance and should be discarded.

Why Hybrid Vision is Necessary

Hybrid Vision is all about combining the best tools for the job to create the most effective solution - in this case for the building industry. Traditional 2D/3D machine vision falls short when the task at hand is to detect aesthetic surface imperfections, which can appear in many different ways.

On the other hand, it is a very effective tool for measuring sizes. Therefore, you can let the neural network identify the cracks that need further investigation and then use traditional vision technology to measure the exact size of the defect and determine if it is within tolerance or if the stone should be discarded.

Example 3

Content verification

Purpose

Quality control of parts packaging 

Box Inspection-1

Challenge

Many products consist of different components that need to be assembled into a single package - think building kits, furniture, kitchenware, etc. 

It's crucial that the right parts are packaged together, but errors can easily occur during packaging, which is often done manually. This leaves the customer frustrated and with a product that can't be assembled. 

The contents of such packages can be quality-checked with machine vision during the process. The challenge, however, is that the contents can be placed in many different ways. A bag of screws might be half hidden under a piece of paper. A plate may be placed in the box but upside down. In other words, we are looking at a large sample space.

Solution

The task is solved by placing a number of 2D cameras above the conveyor belt at selected locations where the contents need to be checked. First, traditional vision technology is used to identify the packages that are potentially faulty, eliminating the need to use machine learning on the obviously good items. 

The images that deviate from the defined standard are then processed through a neural network trained on a large number of images of different correct packages.
Using anomaly detection technology, the neural network can then identify the packages that deviate too much from the norm to be approved.

Why Hybrid Vision is necessary

Because the range of outcomes is so large that it is too resource-intensive to use traditional vision technology to solve the task, it is necessary to bring machine learning into play.

The machine learning network can be trained to assess an item in the same way an experienced inspector would. Therefore, this technology is particularly well suited for these types of tasks where it is difficult to make exact, complete descriptions of what a good and a bad item looks like.

Example 4

Glass tube inspection

Purpose

Classification of open-ended and closed airlines in glass tubes

JL Vision-ekspert-SK-1

Challenge

With traditional machine vision, it is possible to detect defects in glass tubes, such as
airlines. 

The challenge arises when quality control requires you to differentiate between different types of defects, such as airlines with open and closed ends. Airlines with closed ends are okay, while open ends are a critical error. A standard glass inspection system cannot robustly make this distinction.

Solution

Machine vision can detect all defects, including airlines, so the first step in the process is to identify areas with defects. Images of the identified defects are then transferred to a machine learning network, trained with images captured by the standard system. The neural network processes the image and assigns each fault a probability of being an open or closed airline.

Why Hybrid Vision is necessary

Prior to this solution, automated sorting between open-ended and closed airlines in glass tubes was not possible.

But by adding a machine learning layer on top of the traditional vision system, automated inspection can be taken a step further. 
At the same time, the synergy between standard vision and machine learning means that it's possible to implement the solution quickly.

Instead of training the machine learning network on a large amount of training data, this is done within the first week of the system's implementation by training the machine learning network with actual images from the standard system. They are continuously annotated so that the network learns to recognize error types.

Example 5

Lacquered surfaces

Purpose

Inspection of defects in the surface of lacquered boards

Hybrid vision Laqured surfaces

Challenge

The challenge of inspecting painted surfaces for defects, in this example painted kitchen fronts, is the sheer number of different defects that an algorithm needs to identify and classify.

There can be over 25 different types of defects in the lacquer - small holes, scratches, orange peel, dust, etc. It would be very resource-intensive to develop an algorithm capable of finding and classifying all defects, especially due to the high variance in defects.

Solution

In this example, we have developed a solution that first pre-processes the image and identifies areas in the image that are likely to have a defect. We can use traditional machine vision for this part, as it doesn't need to do anything other than find areas that deviate from how the surface otherwise looks. 

We then process these image sections in a neural network that has been trained on a large number of good and bad images that have been annotated so that the network learns the different types of defects. The network can then assess with a high degree of probability whether an error is present and at the same time define which type of error it is.

The neural network identifies the errors by setting a probability that a particular error is present in an image section.

In some cases, the value will be close to the threshold we have defined for an error. If there are specific errors that are particularly important to avoid, we can use traditional machine vision to post-process the image sections with borderline cases.

Since we have narrowed the sample space through machine learning, we now can create an algorithm that looks for specific characteristics of a given defect. In this way, we increase the accuracy of the system.

Why Hybrid Vision is necessary

It's often a bad idea to process an entire image with machine learning, as it will take too much time and require too many resources. The system becomes much more efficient when we use traditional 2D and 3D vision to identify the areas of an image that we want to process through the machine learning network.

When the complexity of a task is sufficiently large, it is often not possible to solve it with standard machine learning solutions. Training the network to be robust enough to identify and classify a high number of different errors with high variance requires a great deal of professional expertise.

 

Save it for later? Download the guide as PDF