HumanEgo: An Open-Source Egocentric Human Dataset Built for Training Embodied AI
1,800+ hours of real human skills, openly released with synchronized 3D hand motion, large-FOV depth, and layered semantic labels.
Sep.23, 2026
Why We Built HumanEgo
For embodied AI, egocentric video offers a natural supervision signal: it shows the world from the actor's own perspective, recording what a person does, which objects they touch, how their hands move, and how a task develops step by step.
But volume by itself does not make data ready for training. In our view, training-ready data must pair diverse real-world behavior with enough structure to answer three questions: how the action is carried out, where it happens in 3D space, and what the task means.
HumanEgo was designed around exactly these requirements. The dataset holds 1,800+ hours of first-person demonstrations spanning 10,000+ distinct tasks, all recorded with one unified six-camera rig. Our Data Foundation Model (DFM) then transforms the multi-view Ego footage into structured, synchronized training signals: 3D hand reconstruction for human motion, large-FOV depth for spatial understanding, and hierarchical semantic annotations for task understanding.
Every signal is aligned to the same spatiotemporal coordinate system, together forming a complete representation of the physical world — making HumanEgo an open-source, training-ready egocentric dataset for robot learning.
One Backbone: From Raw Ego Video to Structured Training Signals
At the input side, DAS-Ego records multi-view RGB sequences over 6 visual streams, capturing spatial relationships more completely than any single viewpoint. Temporal context links observations across frames, which improves motion consistency, trajectory continuity, and prediction stability.
To convert raw footage into learnable representations, we built a DFM-powered stack: a single unified perception backbone with task-specific modules on top. Related tasks can therefore share the same visual and spatial features, and the system can grow as new supervision and data arrive.
Fusing Multiple Views into One Consistent Representation
A lone Ego view degrades quickly under occlusion, fast motion, or limited visibility, so DAS-Ego captures six synchronized visual streams.
DFM models the multi-view RGB sequence jointly, letting each view compensate for the others. Complementary angles preserve evidence when a hand disappears behind an object or drifts toward the edge of a camera's field of view, while the continuous sequence supplies context — where the hand came from, how it moved, what followed — giving the reconstruction pipeline a much stronger starting point.
Temporal context further enforces motion consistency across frames, stabilizing trajectories and suppressing frame-level noise. Combined, these cues yield a stronger representation of the interaction.
The shared perception backbone behind the whole pipeline is our Data Foundation Model (DFM). It fuses synchronized views with temporal context, and task-specific heads extract structured signals from the resulting features.
Cross-view fusion is guided by camera intrinsics and extrinsics: these geometric constraints associate observations of the same physical region across cameras, so feature exchange focuses on spatially relevant information. If one view is occluded or motion-degraded, another fills in. Temporal modeling exploits frame-to-frame continuity to reduce jitter and keep trajectories stable.
For pose estimation, sparse target-aware queries gather only the features relevant to the target instead of running identical dense computation everywhere. Geometry, time, and targeted selection together let the model recover a more coherent account of each interaction.
A shared backbone also means related perception tasks reuse common visual and spatial features. As more data and supervision become available, the same stack keeps improving and can produce richer annotations for future dataset releases. In HumanEgo it powers three central forms of supervision: hand motion, scene geometry, and task semantics.
DFM for Human Motion: Multi-View 3D Hand Tracking and Full Hand Reconstruction
RGB video shows the interaction, yet it does not directly expose the hand trajectory, finger articulation, or grasp geometry that describe how an action is performed — exactly the details that matter when learning how a person manipulates an object.
HumanEgo therefore relies on DFM-powered HandTrack to produce a structured 3D representation of human hand motion:
- 21 3D keypoints per hand
- MANO parameters
- Full 3D hand meshes
Reconstructing the Full 3D Hand from Multi-View Features
Natural first-person interaction is hard on hand tracking: hands are routinely occluded by objects, blurred by fast motion, partly out of frame, or mixed with other visible hands.
HandTrack consumes the shared multi-view features from DFM, fuses complementary observations across views, and passes them to a dedicated hand reconstruction head. Sparse hand queries pick out the features most relevant to each hand, and the model predicts the MANO representation directly:
Multi-View Features → Cross-View Fusion → Sparse Hand Queries → MANO + 3D Keypoints
MANO describes the hand as a structured parametric model — pose, shape, global orientation, and translation. From these parameters HandTrack recovers both the 21-point 3D hand skeleton and the full 3D hand mesh.
Compared with keypoints alone, the mesh retains far more of the physical interaction: finger articulation, palm orientation, grasp configuration, and hand-object geometry.
Hand Tracking Ground Truth and a Continuous Improvement Loop
Reliable 3D ground truth underpins high-quality hand reconstruction. For precision evaluation we use a dedicated motion-capture rig, where hand-mounted markers are tracked by multiple cameras to supply high-precision 3D reference positions.
Mocap data is accurate but expensive to scale, so HandTrack pairs high-precision mocap data for calibration and evaluation with large-scale real-world Ego data for training and iteration. The production pipeline can additionally derive structured hand labels straight from Ego recordings, closing a continuous improvement loop:
Real-World Ego Data → Hand Reconstruction → Structured Training Data → DFM Training → Improved HandTrack
Evaluation spans geometric accuracy and difficult real-world conditions alike. Under the current setup, HandTrack reaches sub-centimeter tracking error while outputting 21 3D keypoints and MANO-based full hand reconstruction.
We also test beyond clean sequences, on the hard cases that naturally occur in Ego data: motion blur, hand-object occlusion, partial out-of-frame motion, and multi-person / multi-hand scenes. These matter because tracking quality depends on whether enough reliable visual evidence survives when interactions get messy.
DFM for Spatial Understanding: Large-FOV Ego Depth
Hand reconstruction explains how the wearer manipulates an object; depth maps out the 3D workspace surrounding that interaction.
For embodied learning this distinction is essential. A model must know not just that an object sits in front of the wearer, but where it is in space, how far away it lies, and how the surrounding surfaces are arranged.
Ego-Depth is the spatial component of the DFM-powered data stack.
Depth Beyond the Stereo Overlap
Most practical depth systems rely on stereo vision and therefore need sufficient overlap between a camera pair. Large-FOV fisheye imagery makes this harder: overlap shrinks toward the image borders, while mirrors, reflective materials, and transparent glass further erode depth reliability.
Ego-Depth instead outputs a single large-FOV fisheye depth map. Its core module, Multi-View Cross-View Attention, models relationships across camera views and merges complementary information in a shared feature space, recovering depth even where a local view offers weak geometry or limited overlap.
Millimeter-Level Ground Truth and Evaluation
We built a dedicated indoor Depth Ground Truth system from multiple high-precision infrared stereo devices, with trigger synchronization held under 5 ms, delivering millimeter-level Depth Ground Truth. For mirrors, high-reflectance surfaces, and transparent glass, the pipeline adds QR-code-assisted supervision to keep references reliable.
Ego-Depth is scored on depth accuracy, temporal stability, and valid coverage across the full field of view. In Full-FOV evaluation it spans 157° × 132° ± 5°, reaches ≤ ±4% depth accuracy at 1 m, and sustains a ≥ 99.5% fill rate.
The system has also been validated on challenging scenes, including large empty space, dynamic interaction, mirrors, and transparent glass.
DFM for Semantics: Record Naturally, Annotate Afterwards
Hand tracking and depth describe the physics of an interaction; semantic annotation describes what the person is doing and how the task unfolds over time.
Instead of handing collectors a rigid atomic-action script, HumanEgo records natural behavior first and applies structured annotation afterwards through the DFM-powered annotation pipeline, preserving natural variation in task order, motion, object placement, retries, and recovery.
Three Levels of Semantic Annotation
Every episode is organized at three levels by the annotation pipeline:
- Video-level description — Summarizes the overall activity
- Task-level annotation — Marks the main goal across a continuous time segment
- Fine-grained subtask annotation — Describes the actions and object interactions inside the task
Each task and subtask is aligned to explicit time and frame ranges, keeping the semantic description synchronized with the source video. Fine-grained labels may also capture the manipulated object, visible attributes, source and destination, spatial relationships, and whether the action completed successfully.
One HumanEgo episode, for instance, is annotated as follows:
Video description: straighten up the sofa — set the throw pillows and plush toys against the sofa back and smooth out the sofa cover.
Task: straighten the sofa (scene: living room; duration: 0–168.5 s)
Subtasks:
- 0–4.5 s — Set the red-and-black throw pillow against the sofa back
- 5–9.5 s — Set the light-purple throw pillow against the sofa back
- 10.5–18 s — Put the brown teddy bear onto the red-and-black pillow
- 28–42 s — Smooth the sofa cover over the right seat
- 140.5–149.5 s — Adjust the sofa cover along the sofa's right edge
Every subtask additionally carries an is_success field, so successful, failed, and retried attempts all remain part of the structured sequence. The full released example contains twelve temporally aligned subtasks under a single task.
Grounded in What the Camera Sees
Fine-grained annotations describe only what is actually observable in the video. When an object cannot be identified with confidence, the annotation falls back on visible properties such as color, shape, and size rather than guessing a category.
For example:
Move the small silver rectangular object from the tabletop into the storage box.
This keeps annotations visually grounded while preserving the useful details of the action, the object, and the spatial relationship.
The resulting semantic output weaves together overall task context, temporal segmentation, fine-grained actions, object attributes, spatial relationships, and success state — without constraining how the original demonstration was performed.
These signals are precisely aligned across video, action, language, and depth, and then encoded into unified tokens: a shared, training-ready representation of human interaction for robot learning.
Where We Go Next
HumanEgo captures interaction today through first-person vision, hand motion, scene geometry, and task semantics. The next step is to extend this representation from the hands to the complete human motion behind the interaction.
Many real-world skills involve far more than end-effector motion: reaching, bending, squatting, walking, shifting body position, and coordinating both hands all shape how a task is physically executed — yet an Ego camera only sees part of it.
We are therefore developing Whole-Body Manipulation Data, where DFM reconstructs continuous whole-body motion from egocentric observations using visible body evidence, temporal context, and human-body structural priors.
HumanEgo is open-source now. Download the dataset, or reach out with questions, collaboration ideas, or feedback.