Robot AI

How Egocentric Video Scales Whole-Body Manipulation

Turning egocentric video into model-ready data for learning complete human skills.

Sep.8, 2026

What Is Whole-Body Manipulation Data

Useful physical-world skills require more than watching hands or objects in isolation. Household chores, assembly work, and long-horizon mobile interactions all depend on coordinated motion across the body, both hands, manipulated objects, and the surrounding environment. Yet egocentric video — the most natural and scalable source of human demonstrations — rarely captures the full body, while traditional motion capture demands expensive equipment, controlled spaces, and specialized operators, leaving the long tail of real-world behavior out of reach.

We have built a data production system that turns egocentric observations into Whole-Body Manipulation Data. A portable, vision-only, multi-view ground-truth system provides high-quality supervision first; DFM then reconstructs structurally plausible, temporally coherent whole-body motion from Ego observations, unifying the body, both hands, objects, contact, vision, and task semantics in one common data representation. The output is designed to flow directly into robot policy training and evaluation.

Under our current internal evaluation protocol, the system reaches approximately 3 cm mean whole-body error — about 2 cm for the upper body and 3.5 cm for the lower body. Just as importantly, the automated pipeline moves data collection out of fixed laboratories and into homes, commercial spaces, and industrial environments, laying the foundation for continuously producing Whole-Body Data at the 100,000-hour scale.

Why Ego Data Alone Is Not Enough

Ego devices stay close to the human point of view: they are lightweight, portable, preserve natural behavior, and impose few constraints on where a task happens. A person can wear one while cleaning a room, organizing objects, moving tools, or executing a long action sequence — making egocentric capture one of the most promising entry points for collecting real human skills.

The first-person view, however, poses a fundamental challenge: the camera only ever sees part of the body and the environment. Arms occlude each other; bending or squatting changes the geometry between head and torso; the lower body frequently leaves the frame entirely; fast motion and long-distance movement constantly shift camera orientation and global position. The visible evidence in a single frame is simply not enough to recover complete, stable, physically plausible whole-body behavior.

For robot learning, these missing signals cannot just be thrown away. A policy needs to know more than what the hands touch — it must understand where the person stands, how weight shifts, when contact starts and ends, and how body motion evolves through each stage of a task. Partial Ego observations, in other words, have to be transformed into complete Whole-Body Manipulation Data.

That raises a bigger question: what makes data truly learnable by a model? In our view, model-ready Whole-Body Data must meet at least four conditions — complete motion structure, aligned multimodal signals, stable long-horizon coordinates, and measurable, traceable quality. Only then does a human demonstration stop being a video to watch and become a data asset that enters robot training and evaluation directly.

Key Results at a Glance

Based on our internal evaluations, the system already delivers across three critical parts of the Whole-Body Data stack:

  • Whole-body reconstruction — The current model achieves approximately 3 cm mean whole-body error, including about 2 cm for the upper body and 3.5 cm for the lower body.
  • Ground-truth capture — Our self-developed, portable multi-camera capture system completes on-site calibration in under 30 seconds, with calibration reprojection error below 0.4 pixels.
  • Production at scale — An automated pipeline connects data ingestion, reconstruction, quality validation, valid-frame filtering, and training-format export, establishing the production foundation for Whole-Body Data at the 100,000-hour scale.

Together, these results cover the three practical requirements of model-ready Whole-Body Data: accurate reconstruction, reliable ground-truth capture in real-world environments, and a production pipeline that scales with growing data volume.

From Ground Truth to Production Scale

Our approach does not hinge on any single model or capture device. It is an end-to-end system that spans ground-truth production, model training, and data generation at scale.

1. Producing Reliable Whole-Body Ground Truth

Reliable supervision is the starting point of high-quality Whole-Body Data. We built a low-cost, markerless, vision-only ground-truth capture system that synchronizes an Ego headset with multiple portable third-person cameras watching the same behavior. Collection needs no reflective markers, no motion-capture suit, and no dedicated studio, so the system works in homes, offices, outdoor spaces, and other real environments.

Visual calibration boards establish a shared coordinate system across the Ego and third-person cameras. One calibration finishes in under 30 seconds with multi-camera reprojection error below 0.4 pixels. Cross-view pose estimation, temporal stabilization, and smoothing then yield human-mesh ground truth directly, giving the whole-body model a consistent supervision target.

2. DFM Reconstructs Complete Motion from Partial Observations

At production scale, DFM turns Ego video into continuous whole-body motion by combining the body evidence visible right now, motion cues from neighboring frames, and structural priors about the human body. Even when part of the body temporarily leaves the camera view, DFM keeps inferring the relationships among the head, torso, hands, and lower body.

The goal is not to assemble a human-looking mesh frame by frame, but to preserve consistent body structure, global position, and action semantics across an entire sequence. A stable global coordinate frame matters most: it keeps the relationships among the person, the scene, and the objects explicit throughout long-horizon tasks, so the data can represent continuous movement, contact, and manipulation.

3. More Than a Body Mesh

A robot policy needs a learnable manipulation process, not isolated human geometry. Whole-Body Data therefore goes beyond body-mesh reconstruction by aligning these signals within the same sequence:

  • Whole-body mesh and global motion
  • Two-hand tracking and mesh
  • Objects and their spatial relationships to the body
  • The onset, duration, and release of contact
  • Egocentric visual observations
  • Task and action semantics

Representing these signals in a shared spatiotemporal coordinate system and a unified data format cuts down downstream conversion and secondary fitting, so the output can more directly support robot policy training, evaluation, and failure analysis.

Built to Survive Real-World Behavior

Accuracy only counts when it stays stable during real behavior. Our internal evaluation therefore goes well beyond static poses and examines whether the system can recover human motion throughout complete action sequences.

We focus on several challenges that occur constantly in Ego-based whole-body reconstruction:

  • Bending and squatting — Headset orientation and body configuration change together, so the system must preserve a complete relationship among the head, torso, and lower body.
  • Fast motion — Motion blur and abrupt viewpoint changes make single frames uncertain, and temporal context is required to keep predictions stable.
  • Long-distance, long-horizon movement — The global coordinate frame must stay consistent for the whole task, preventing the relationships among person, environment, and objects from drifting over time.
  • Partial occlusion and out-of-view body parts — When arms cross, the person turns, or part of the body disappears, the system combines visible evidence, motion context, and human-body constraints to keep predictions plausible.

Our self-developed ground-truth production system runs automated quality checks for bone-length consistency, keypoint discontinuities, severe occlusion, out-of-frame subjects, and mesh penetration, filtering invalid frames before export — replacing large-scale manual review with dataset-wide algorithmic inspection.

An Automated Pipeline That Keeps Scaling

Capturing one high-quality demonstration solves only a small part of the data problem. The harder part is moving thousands of real behaviors continuously into a training system under one consistent quality standard.

Our production pipeline connects data ingestion, multi-stream merging, cross-view calibration, human-mesh generation, training-format export, algorithmic quality control, valid-frame selection, and dataset analytics. Work that once required manual review, conversion, and organization now runs end to end, and the path from completed capture to training-ready output shrinks from a days-long workflow to an hours-long one.

Portable hardware enables parallel collection across many real environments, while automated software lets processing capacity grow with the number of devices and the volume of data — without quality-control headcount having to scale linearly. The system also maintains dataset versions and supports batch reprocessing, so when models or quality standards improve, historical raw data can be regenerated and benefit from the new capabilities.

Together, this pipeline establishes a sustainable foundation for Whole-Body Data at the 100,000-hour scale.

Beyond Standard Motions: Real-World Skills

A fixed motion-capture studio delivers precise data, but it cannot easily represent the complexity of the real world. Human skills play out across changing spaces, lighting, backgrounds, clothing, objects, and occlusions — and only sufficiently diverse data gives robot policies a chance to learn transferable behavior instead of memorizing a single laboratory setup.

Lightweight, portable Ego devices move collection out of fixed spaces and into homes, commercial locations, and industrial environments. From organizing, cleaning, and pick-and-place tasks to longer-horizon movement and bimanual manipulation, the same data system can keep absorbing new tasks and scenes. Real demonstrations preserve natural motion and physical plausibility, while simulation built on a common human representation can further expand diversity in lighting, background, occlusion, clothing, and appearance.

What Comes Next

Whole-Body Data is still in its early days. Fast motion, severe occlusion, complex human-object contact, and longer-horizon tasks all remain important areas for improvement, and we will keep expanding real-world task coverage, strengthening robustness in difficult scenarios, and advancing the data representations and transfer methods that connect human behavior to robot policy learning.

Our goal is simple: a natural human demonstration should no longer remain just a video — it should become complete skill data that robots can understand, train on, and reuse.

See the Whole Body. Learn the Whole Skill.

Chat with me