A Neural Network Powers This Lego-Built Brick Sorter That Can Recognise Every Piece Ever Made

A Neural Network Powers This Lego-Built Brick Sorter That Can Recognise Every Piece Ever Made
Gif: <a href="https://www.youtube.com/channel/UCIg7y12vUUslNgs2VqDXeSw">Daniel West</a>, <a href="https://youtu.be/04JkdHEX3Yk">YouTube</a>

Lego becomes more and more enjoyable as your collection grows, until the point when your morass of plastic bricks becomes so overwhelming that it’s impossible to find the piece you’re looking for. At that point, you need to develop a sorting system, or do what Daniel West did, and use a bunch of your Lego to build a machine that automatically sorts the rest of your bricks for you.

This isn’t the first time Lego has been used to build a contraption like this. But previous attempts have all required some level of pre-sorting, and are very limited when it comes to categorising the pieces and dropping them into separate bins. West’s machine was designed with a complex mechanism supporting 18 different bins and categories of pieces, but its overall capabilities allow for pieces to be sorted into thousands of categories and bins potentially—assuming someone has enough pieces to build such a device.

West estimates it took somewhere around 10,000 Lego pieces to build his sorter, including six Lego motors, plus a handful of other non-Lego components like servos, a Raspberry Pi, lighting for the camera, and a laptop that powers the convolutional neural network that identifies each piece. Getting the machine to accept buckets of pieces but eventually present only one at a time to its camera was a huge challenge which West eventually solved with a pair of vibrating plates that ensures pieces pass through individually. The machine itself took six months to build, but the neural network and software behind it took two-and-a-half years to perfect.

In order to train the neural network to recognise the images of each part as captured by the sorter’s video camera, West needed access to a database of images for all the pieces Lego has ever produced. That’s not something Lego readily shares with the public, and there’s no chance West could have tracked down and photographed all those pieces on his own. His solution was to rely on the 3D generated images of those parts that online databases such as LDraw.org and Rebrickable have created over the years.

Getting image recognition software to match a photo of a brick taken with sketchy lighting on a moving Lego conveyor belt to a pristine 3D render of that same part is all but impossible; the subtle differences would be completely confounding. But that’s where the convolutional neural network came into play, it essentially learned to match 3D renders of Lego pieces to the photos of the real thing, and at a speed that allows West’s machine to process roughly two pieces every second.

In addition to thoroughly documenting his creation and everything he’s learned in the process, West plans to write an academic paper on it. What he has no intention of doing is making instructions allowing Lego fans to build their own. He might release the software for free, but he’s leaving the sorting machine to the ingenuity of other Lego enthusiasts who will have to come up with their own.