Pushing boundaries - the story of JLI vision

For 40 years, JLI has navigated both gradual technological developments and big leaps forward. The most significant change gained momentum around 2016, when the potential of artificial intelligence for quality control began to emerge.

That year, JLI invited a professor from Aalborg University to the annual company seminar to give a presentation on machine learning and deep learning.

"Afterwards, we all sat there thinking, 'What now? Are we going to be unemployed? Can we continue doing what we do today?' Sometimes these kinds of presentations are just good inspiration, but this one was groundbreaking on a technical level and sparked a lot of discussion afterwards. We quickly concluded that we needed to know more about it. Instead of seeing it as a threat, we wanted to find out how we could benefit from AI," says vision engineer Esben Korre, who was given responsibility for driving the work with artificial intelligence at JLI.

The first AI task

He began researching the subject and, among other things, attended an intensive DTU course that provided concrete examples of how the technology could be used. Around the same time, Google released its first version of TensorFlow, a software library for machine learning, and ViDi Systems launched a user-friendly interface for it, which JLI also tested.

In parallel with the initial online research, JLI was contacted by IKEA regarding a task that would be very difficult to solve with traditional machine vision. It involved finding knots and resin pockets on wooden boards, an inspection task with so many variables that it had previously required manual inspection with a trained eye.

JLI saw the task as a perfect opportunity to try out its newly acquired AI knowledge on a real project and bid for the task. However, IKEA rejected JLI and instead went with a Swedish start-up company that had developed an advanced solution.

"We realized that securing a project to fund our development was unlikely. It would need to stem from our own ambition to gain more knowledge in this area," says Esben Korre.

However, JLI did not want to give up on the task and made an agreement that if IKEA would supply a number of test items, JLI would continue the project on its own, and after 3-6 months, IKEA could evaluate the finished solution.

Together with ViDi, JLI developed a proof-of-concept showing that it was possible to distinguish between resin pockets and knots. After testing 30-50 images over a week, the results appeared promising.

AI and traditional vision combined

Part of Esben Korres' research was carried out through a machine learning community, Kaggle, where he explored various competitions involving vision tasks and descriptions of the winning solutions.

"Among other things, there was a winning solution that had trained an ensemble, as it is called. You train several networks on part of the total data set and give them all an idea of what, for example, a resin pocket looks like. Each network then becomes more refined in its own specialty, and you can average several networks' assessments to obtain a more robust solution," he explains.

The winning solution also demonstrated another point that Esben Korre took with him and applied to the IKEA project.

"It showed how we can combine machine learning with traditional vision, and that's exactly how we began using it. We built a classification network based on multiple candidates. We used traditional vision to find contrasts that indicate a defect, cut out the small image section, and then gave these candidates to an ensemble of networks and asked: Is this a burr or not? Is it a resin pocket or not? So it was a relatively simple network that gave some yes/no answers," explains Esben Korre.

Partway through the process, the team stumbled upon a hurdle that has since proven to be a fundamental challenge for AI-based vision systems: obtaining sufficient training material.

"We could see that we needed a larger amount of data. And we had to do it the hard way. We asked for 100 more boards and 100 more and ended up with 3-400 boards, which we used to train the network. We set up our system at the machine builder’s site, where they had constructed the production line, so we could test it effectively. After running a plate through the system, we analyzed what our model detected and what it missed. This required us to retrain the network to improve its accuracy. This process involved a significant amount of data collection, which we now refer to as annotation," says Esben Korre.

"It didn't even crash?"

Eventually, the network reached a sufficiently high level of accuracy, and IKEA was invited to Denmark to see a presentation of a system installed on a test line.

"They had said in advance that they had seen good results from the Swedish supplier, so we were under a bit of pressure, but everything went well, and everyone was happy – especially us," says Esben Korre.

The system delivered the promised detection rate, but it was something else entirely that caught the IKEA team's attention.

"They said, 'It didn't even crash!' And no, it didn't, but we hadn't given that any thought. We had our tried-and-tested platform, which this system was integrated into. But the other company had apparently had a lot of trouble getting it to work in real life," says Esben Korre.

IKEA ultimately changed its decision, terminating its collaboration with the Swedish partner and purchasing JLI's solution instead.

The system became JLI's first application with machine learning, and the fact that getting artificial intelligence to work on a production line was no trivial feat was confirmed when Esben Korre attended a machine learning seminar in Germany in 2018.

“It was still very much in the conceptual stage, and there was a call for speakers who had something operational, as this was still rare. That was at a time when we had our solution in production.”

The way paved for more systems

From there, development began to pick up speed. Among other things, JLI developed a classification network for the SK systems to classify defects in glass tubes, and in 2019, they also began exploring segmentation networks, which can, for example, identify areas with defects in images.

This paved the way for a longer dialogue with Novo Nordisk about a system for inspecting glass ampoules.

"They had been working on it themselves, but one of the problems they had was that it was a bit too much of a black box, in that you didn't know what the network was responding to. We were then able to tell them that we were researching segmentation networks, where you can specifically find out which pixels the network is responding to," says Esben Korre.

When the network was deployed on JLI's platform, it also enabled logging user activity and regulating brightness to achieve a consistent light level, among other benefits.

The dialogue ultimately led to JLI's first AI-based vision system for inspecting medical equipment. The company has cultivated this niche ever since and developed one of the first AI-based vision systems for the pharmaceutical industry, validated to meet the industry's strict requirements.

Started with a pioneering spirit

Artificial intelligence has now become an integral part of JLI's toolbox, but it began with curiosity and a conscious decision to invest in becoming smarter.

“In the beginning, it was a pioneering spirit that drove it. One thing that characterizes JLI is that we like to work with new ideas and come up with innovative solutions. And we had the advantage of having full support to explore it. We didn't have to report on how far we had gotten and what we could do at a given point in time. We just poured hours into it to see how far we could get,” says Esben Korre.

AI before traditional vision

Today, it is difficult to find a machine vision supplier that does not talk about artificial intelligence in one way or another, and for JLI, the use of AI has turned many things upside down. Initially, the idea was that AI should be used where traditional vision fell short. Today, the approach is often to start from the other end and clarify whether the task can be solved effectively with artificial intelligence before looking at traditional vision techniques.

The competitive landscape has become crowded when it comes to AI software for quality control, but JLI still has an advantage in its ability to make neural networks work in a production environment.

"What we can do with AI, many others can also do, but what we can do in terms of seamlessly integrating it into an advanced ecosystem, I don't see many others who can," says Esben Korre.