Leveraging Open Set Detectors to Create a Custom Detector

Introduction

In large-scale warehousing and distribution operations, conveyor belts are an essential infrastructure that must operate with near-zero downtime to ensure the timely delivery of products. The presence of loose or foreign items on a conveyor belt can pose a serious risk to these operations. Small parts can slip through gaps and cause jams, sharp objects can tear belts, and heavy items can pose serious safety hazards.

If left unremedied, even minor belt tears can escalate into a larger problem and cause downtime for the system. Detecting loose objects allows them to be removed before they cause problems, and detecting belt defects early enables proactive maintenance before the belt becomes unusable. However, continuous monitoring for loose objects can be difficult for extensive facilities to implement, and manual inspection of belts can be time-consuming and inconsistent.

Solution

Matroid partnered with one such high-volume fulfillment center, where maintaining the health of the conveyor system is critical to meeting their customers’ expectations. Matroid implemented a real-time computer vision system to monitor belt conditions and raise alerts when a potential issue was detected. The solution detects various types of belt defects as well as loose objects. Strategic locations were selected for the initial set of cameras, based on package volume and historical patterns of defect occurrences. One of the key technologies enabling rapid deployment was “open set detection,” which allows the creation of a machine learning model by simply providing a text label for the object of interest. These models eliminate the need for a dataset on which to train the detector, enabling the rapid deployment of an initial model.

Engineering Effort

At the beginning of the project, Matroid conducted a thorough process of staging occurrences of both belt defects and foreign objects to begin building a dataset. However, this data was both limited in volume and not truly representative of real-world scenarios, as the belt was not in motion during the staged scenarios. 

Instead of solely relying on this data, Matroid leveraged an open set detector with the labels “package,” “foreign object,” and “belt tear”. We were able to deploy this detector across all of the cameras immediately. Although open-set detectors are not fine-tuned to specific customer environments, this one could distinguish between regular packages and loose objects, and locate belt defects well enough to begin surfacing meaningful detections immediately.

Although imperfect, these initial detections were an invaluable way to automatically compile a training dataset for a traditional localizer model. After being validated by Matroid’s annotators, the Matroid Solutions team was able to leverage the open set detections and the staged data to train a highly accurate Matroid detector. By using an open set detector for data collection, we can train and iterate on detectors quickly, thereby avoiding traditional data collection techniques and manual review.

Results

Within a week of deployment, Matroid detected an early-stage belt tear, allowing for preventative action before damage occurred. The trained detector can distinguish between packaged and loose items, even in complex cases such as items wrapped in clear packaging that visually resemble foreign objects. Loose items continue to be flagged regularly so they can be removed immediately, and the occasional belt defects are also detected well. 

By combining flexible open-set detectors with more robust localizers trained on customer-specific data, Matroid was able to reliably detect potentially hazardous conditions and substantially increase the facility’s resilience against conveyor downtime.

About the Author

Rebecca Derham is a Deep Learning Field Engineer at Matroid. She studied mathematics, statistics, and machine learning at Carnegie Mellon University. She enjoys swimming, taking care of her houseplants, and spending time with her donkeys.