The frontier of natural artificial intelligence is a game of Jenga in a warehouse in San Leandro, California.
This warehouse is occupied by Recordinga company that makes data tools used to train AI models. Andrew Ceja is a pilot—the company’s term for its robotic trainers—and he carefully pulls blocks of wood from a wobbly tower while wearing a headset with a camera that tracks what he sees. That in itself is common enough for collecting robot training data, but this headset includes sensors that measure its brain waves as it carefully dismantles the block tower.
Encord is one of a small but growing number of startups betting that the next real limitation in humanoid and warehouse robotics won’t be model architecture, but the sheer scarcity of real-world physical training data. Rather than simply helping robotics companies manage the data they have, Encord is building a business around producing the data they don’t.
The brain headset Ceja is wearing was made by Zander Labsa German neuroscience startup betting on measuring brain activity—to infer mental states like error, intention, and surprise—can create a more useful data set for training models. Encord’s work with Zander is currently on trial. Encord says the goal is to create an initial brain-tagged dataset, run it through customer robotics models, and assess whether it actually improves performance before deciding whether to scale it up.
Lucas Gehrke, a Zander neuroscientist overseeing the work, says the amount of brain activity used at any point during a given task offers clues for model builders trying to figure out when to deploy the highest-effort models.
This is the “bleeding” of the effort to solve robotics’ data bottleneck, according to Vineeth Velmurugan, Encord’s head of robot learning. A veteran of OpenAI’s robot lab and warehouse automation company Berkshire Gray, Velmurugan joined Encord to build the company’s internal data generation team.
Encord was founded to help companies building machine vision applications annotate data and evaluate models. As their customers—Velmurugan says they work with several leading robotics companies but is not authorized to name them—started applying end-to-end learning to robotic manipulation tasks, executives realized they would have to generate training data themselves, rather than just manage it. “The data just isn’t there,” Velmurugan said.
The bet that genetic AI can do for robots what it does for chatbots continues to run into the same wall. Self-driving car companies collect physical data themselves, but this is difficult to scale. Video training can work, but it lacks the fidelity of real-world data. Velmurugan says it will take a data set roughly five times the size of YouTube’s video corpus to break through — a scale that explains why data generation itself has become a business rather than just a research problem.
Feed your egocentric data needs
Companies making robot brains are now turning to two main sources: “self-centered” video collected by workers wearing cameras, often augmented with additional camera angles and other measurements, and data from remotely operated robots. Encord does both, pulling egocentric data from multiple factories around the world and using its San Leandro facility to experiment with new methods, such as brainwaves, or collect datasets around specific skills for refinement.
When TechCrunch visited, the pilots were using leader-follower platforms—paired robotic arms, one controlled directly by a human operator and one that mimics his movements—to generate data on tasks like pouring coffee from a pot into mugs (very slowly) and stacking poker chips. “Every humanoid company has asked us for these pieces,” says Velmurugan.
Storage shelves contained cartons of fake flowers in vases, books, plastic vegetables, cat litter trays and scoops, bags and bundles of wire, the stock in trade for training operators for domestic work.
At one of those stations, another pilot, Sofia Infante, maneuvers robotic arms to connect and disconnect ethernet cables from the back of a server—the kind of work that data center operators wish could be automated, if only robots could handle it with the precision required. Taking a spin behind the controls, I could see why this is still out of reach: Tweezers are much less dexterous than human fingers and don’t have the degrees of freedom we take for granted in our arms.
Another new way of data that Encord is developing uses a set of sensors strapped to the forearm to detect electrical signals in the muscles. Video taken of human hands manipulating objects typically doesn’t capture the entire hand, but Velmurugan hopes to create a 3D representation of where the hand is at any given moment based on the arm’s sensors, creating a more robust understanding for the models.
Encord datasets are annotated with physical descriptions of what each video contains—”right tightens bolt”—to help LLM-based models understand what’s going on. Velmurugan estimates that this kind of dense annotation is worth 100 times more than “junky ego data” for task-specific training and costs only 20 times more to produce, which is a good trade-off, on paper.
But “20x more” is still real money, and that’s the catch: scraping text from the internet, the way LLM builders built their models by pulling from Stack Overflow and the rest of the web, costs borderline labs next to nothing. Generating physical training data does not, and this is the limit of comparing physical AI as an LLM. This kind of data needs to be generated, not just collected, and that changes the economics of building these models.
Velmurugan says that progress has been made — by showing Encord to programs across the industry, he’s able to see startups and frontier labs figure out what works and what doesn’t for improving physical AI models. That advantage — which sits among several robotics companies at once — is also part of Encord’s playing field. It can identify which data techniques are gaining industry-wide traction before any customer.
This will keep the dozens of pilots at Encord’s facilities busy. Both Infante and Ceja are part of a growing workforce developing the building blocks for neural networks. they previously worked at Scale, another AI data annotation company, before joining Encord.
Ceja had worked for a waste management company where his interest in technology found him responsible for keeping a robotic waste sorter in good working order. Now, as the Jenga tower comes tumbling down, he says he enjoys the challenge of solving training tasks for robots — “It’s something new every day!”
When you purchase through links in our articles, we may earn a small commission. This does not affect our editorial independence.
