2023 · Google DeepMind · Advanced
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
RT-2 co-trains vision-language models on web and robot data so semantic knowledge can influence actions.
Direct answer
What does RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control contribute?
RT-2 co-trains vision-language models on web and robot data so semantic knowledge can influence actions.
Background
Robot actions are represented as text-like tokens and included in vision-language model training. This lets the model connect concepts learned from broad visual-language data to new manipulation instructions.
Problem
The work addresses a central constraint in VLA: building systems that learn useful representations or actions while remaining general enough to transfer beyond a single demonstration or environment.
Core idea
RT-2 co-trains vision-language models on web and robot data so semantic knowledge can influence actions.
Architecture and method
Robot actions are represented as text-like tokens and included in vision-language model training. This lets the model connect concepts learned from broad visual-language data to new manipulation instructions.
- Action-as-token formulation
- Web knowledge transfer
- Emergent semantic robot skills
Results and impact
RT-2 clearly defined the modern vision-language-action model and demonstrated semantic generalization in robot control.
Prerequisites
- RT-1
- Vision-language models
- Tokenized actions
Recommended reading order
Read the explanation above, review the related topic pages, then use the primary-source links below to inspect the abstract, figures, experiments, and released implementation.
Primary sources
External links are provided after the context needed to evaluate the work.
Follow-up research
Related papers and concepts
RT-1: Robotics Transformer for Real-World Control at Scale
RT-1 trains one transformer policy on a large multi-task dataset of real robot demonstrations.
Open X-Embodiment and RT-X
Open X-Embodiment combines robot datasets across institutions and trains policies that transfer across embodiments.
OpenVLA: An Open-Source Vision-Language-Action Model
OpenVLA is an open 7B-parameter VLA trained on the Open X-Embodiment dataset.
Gemini Robotics 1.5
Gemini Robotics 1.5 turns visual observations and instructions into motor commands while supporting multi-step physical tasks.
Common questions
Frequently asked questions
What is the main idea of RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control?
RT-2 co-trains vision-language models on web and robot data so semantic knowledge can influence actions.
Why is RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control important?
RT-2 clearly defined the modern vision-language-action model and demonstrated semantic generalization in robot control.
What should I learn before reading RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control?
Recommended prerequisites are RT-1, Vision-language models, Tokenized actions.