2024 · Meta AI and academic partners · Advanced
Ego-Exo4D: First- and Third-Person Skilled Activity Dataset
Ego-Exo4D pairs synchronized first-person and third-person video with language, gaze, audio, pose, and 3D signals.
Direct answer
What does Ego-Exo4D: First- and Third-Person Skilled Activity Dataset contribute?
Ego-Exo4D pairs synchronized first-person and third-person video with language, gaze, audio, pose, and 3D signals.
Background
The dataset captures skilled human activities from egocentric and exocentric views, with expert commentary, narrate-and-act descriptions, atomic actions, and multiple sensor modalities.
Problem
The work addresses a central constraint in Egocentric Data: building systems that learn useful representations or actions while remaining general enough to transfer beyond a single demonstration or environment.
Core idea
Ego-Exo4D pairs synchronized first-person and third-person video with language, gaze, audio, pose, and 3D signals.
Architecture and method
The dataset captures skilled human activities from egocentric and exocentric views, with expert commentary, narrate-and-act descriptions, atomic actions, and multiple sensor modalities.
- Synchronized ego-exo video
- Expert language annotations
- Multimodal signals for skilled activity understanding
Results and impact
It is directly relevant to training VLMs and VLAs that need to understand action from both the actor's view and an observer's view.
Prerequisites
- Egocentric video
- 3D pose
- Video-language learning
Recommended reading order
Read the explanation above, review the related topic pages, then use the primary-source links below to inspect the abstract, figures, experiments, and released implementation.
Primary sources
External links are provided after the context needed to evaluate the work.
Follow-up research
Related papers and concepts
Ego4D: Unscripted First-Person Video Dataset
Ego4D is a large first-person video dataset for studying what people see, remember, manipulate, and anticipate.
Aria Gen 2 Pilot Dataset
Aria Gen 2 Pilot Dataset captures daily activities with research glasses and synchronized multimodal sensor data.
V-JEPA 2: Self-Supervised Video Models for Physical Planning
V-JEPA 2 learns predictive video representations that support visual understanding and zero-shot robot control.
RT-1: Robotics Transformer for Real-World Control at Scale
RT-1 trains one transformer policy on a large multi-task dataset of real robot demonstrations.
Common questions
Frequently asked questions
What is the main idea of Ego-Exo4D: First- and Third-Person Skilled Activity Dataset?
Ego-Exo4D pairs synchronized first-person and third-person video with language, gaze, audio, pose, and 3D signals.
Why is Ego-Exo4D: First- and Third-Person Skilled Activity Dataset important?
It is directly relevant to training VLMs and VLAs that need to understand action from both the actor's view and an observer's view.
What should I learn before reading Ego-Exo4D: First- and Third-Person Skilled Activity Dataset?
Recommended prerequisites are Egocentric video, 3D pose, Video-language learning.