Using CV for Critical Inspections on Large Equipment

Matroid’s no-code platform teaches ordinary cameras, hand-held or fixed, to think like seasoned inspectors. Even as an operator is walking a simple GoPro around a 10-foot engine or turbine blade, Matroid turns their footage into a pixel-level checklist, catching defects that human eyes can miss, and doing it with surprisingly small data sets.

Introduction

When it comes to manufacturing mission-critical products, there is little room for error. The integrity of the whole system depends on many small components being installed correctly. Overlooking missing bolts or incorrectly installed safety wiring can lead to catastrophic failure, costly damages, and injury or death. Inspectors manually check each engine, scouring its many components for defects. This process is time-consuming and can yield inconsistent results. 

Pairing an inspector’s practiced sweep with computer vision provides the best of both worlds: human intuition guides the camera, and AI double-checks every pixel in the footage, eliminating blind spots that are a human inevitability.

One manufacturer was facing a problem: the parts are massive and are built over a span of weeks, making the inspection process extremely tedious. Even with multiple rounds of human inspection, mistakes could still slip through. There had to be a better way.

They didn’t have thousands of historical images to train on, only a handful of builds. Any solution had to thrive on limited data while still meeting a zero-fail standard.

Solution

Matroid partnered with this manufacturer, which makes large engines. Matroid rapidly built a custom computer vision solution to aid human inspectors, helping catch improper safety wiring, missing safety wiring, and loose bolts. Using a GoPro, inspectors scan all parts of the engine, upload the footage to Matroid, and receive back an inspection report showing which bolts were loose and where to find them on the engine. Reports also show which wirings were done the wrong way, and where safety wiring was omitted.

Because the camera travels with the inspector, every vantage point the human deems important becomes data for the model, creating a natural and beneficial feedback loop.

This approach works very well for the inspectors as it piggybacks off their existing flow of walking around to scan every part of the engine. It also beats having a fixed camera array all around the part which would be exponentially more expensive, and may still miss parts of the engine due to self-occlusions. Moreover, the inspectors get time back as they can get to other tasks while the footage is processed.

In practice, a single 10-minute walk-around generates thousands of annotated frames, more virtual “eyes” than a permanent rig might deliver, yet costs nothing in additional hardware.

Engineering Effort

Loose bolts are especially tricky to detect with visual inspection alone. When first installed, each bolt is given a torque stripe. Each stripe is a paint marking that crosses both the surface of the bolt and the surface to which the bolt is fastened. Then, when the engine returns for maintenance, bolts that have loosened from their original place can be discerned visually because the torque stripe becomes broken.

A standard segmentation detector that outputs masks for torque stripes, combined with a small post-processing step to check for brokenness, got the deployment most of the way to a robust solution. The solution successfully detected all torque stripes throughout the scans. However, there was an additional challenge: when viewed at an oblique angle, a torque stripe can appear broken when it is actually not. This visual discrepancy initially produced some false positives that were tricky to account for.

To overcome this challenge, Matroid needed to account for the fact that the bolts may not be viewed head-on and adjust the interpretation of the results. It was critical to maintain a recall-first approach since missed defects could not be tolerated. Matorid employed a two-stage process where the team trained a model to surface broken torque stripes with high recall. Then, they layered an additional review step to adjudicate them as true or false positives, accounting for oblique angles.

This two-tier method kept recall at the forefront while geometry-aware filters safeguarded precision, an essential balance when any missed defect could be a catalyst for devastating outcomes in real-world usage.

Another challenge faced with this solution was data scarcity. This manufacturer makes engines in extremely low volume. Having just a few full scans per month, there were significant limits to how much data could be collected. The segmentation approach had the advantage of also being more data efficient, learning the objects of interest on much less data than a localizer would require. This process worked well because segmentation masks omit the background space behind an object in a bounding box, reducing the variance that the model has to learn.

As a result, the system reached production readiness after only dozens, not thousands, of annotated examples, proving that “big data” is not a prerequisite for big reliability.

Results

To date, Matroid has caught many loose bolts, noncompliant configurations of safety wiring, and missing safety wires in production. Without Matroid, these mistakes could have led to very costly or even deadly consequences. It’s a major crisis averted via Computer Vision.

About the Author

Matthew Dworkin is a Deep Learning Field Engineer at Matroid. He studied Computer Science & Statistics at the University of California, Berkeley, and enjoys swimming, lifting weights, and watching baseball in his free time. He can be found at the pool, gym, or hanging around San Francisco with friends.