← All guides

IAEI / RESEARCH GUIDE

VLA and world models: what problems do they solve?

Vision-language-action (VLA) models connect visual observations and instructions to actions; world models predict aspects of an environment and how it changes. Compare RT-2, original OpenVLA and DreamerV3 through six fields, and identify which component actually chooses the action.

RT-2: action tokens for robot control

Version scope: the 2023 RT-2 paper, including RT-2-PaLI-X (55B) and RT-2-PaLM-E (12B). RT-2-X is a different model variant and is outside this comparison. [1] [2]

  • Input — a current robot-camera image and a language task instruction. [2]
  • Output — action tokens. The VLA policy selects an action; decoding converts its tokens into robot commands for repeated observation and control. [2]
  • Training — co-fine-tuning a pretrained vision-language model with robot demonstrations and web vision-language tasks. [2]
  • Action interface — six end-effector displacement coordinates, gripper extension and an episode-termination command. Continuous dimensions are discretized into 256 bins. [1]
  • Evaluation — real mobile-manipulator tasks; separate Language Table experiments include simulation and real-world data. These settings must stay distinguishable. [1] [2]
  • Adaptation boundary — new semantic instructions do not create unseen motor skills. Coverage of robot demonstrations and inference latency still constrain use. [1]

Original OpenVLA: a policy adapted with demonstrations

Version scope: openvla-7b and the original paper’s arXiv v3, dated September 5, 2024. The repository separately identifies the later OFT recipe; its capabilities are not attributed to this original model. [3] [4]

  • Input — one image and a language instruction. The original model has no observation history or proprioceptive input. [3] [5]
  • Output — action tokens decoded into one seven-dimensional robot action, rather than an action chunk. [3] [4]
  • Training — action-token prediction after vision-language pretraining, using 970,000 robot trajectories from Open X-Embodiment. The action-training stage uses robot data rather than RT-2’s web-data co-fine-tuning. [5]
  • Action interface — single-arm end-effector control. Deployment must use the appropriate action unnormalization statistics and robot control integration. [3] [4]
  • Evaluation — direct WidowX and Google Robot tests; Franka tests after adaptation; the paper’s LIBERO simulation results also use fine-tuning. [3] [5]
  • Adaptation boundary — new setups generally require target-domain demonstrations. Verify camera conventions, control timing and action replay; a compatible software call alone does not establish transfer. [4]

DreamerV3: the world model trains an acting policy

Version scope: DreamerV3 in the 2025 Nature paper. The author’s code repository describes a reimplementation; record its revision separately from the paper. Earlier Dreamer generations are not interchangeable with V3. [6] [7]

  • Input — image or vector observations and previous actions, with recurrent state preserving temporal information. [6]
  • Output — the world model predicts latent states, rewards and continuation; observation reconstruction helps learn the representation. These predictions are not the action policy. [6]
  • Training — environment experience trains the world model; imagined trajectories train an actor and critic. The critic estimates returns, while the actor learns behaviour. [7]
  • Action interface — the actor selects discrete or continuous environment actions. During interaction it acts without lookahead search; imagination is used in training. [6]
  • Evaluation — games and simulated control, including Atari, Minecraft and visual/proprioceptive Control Suite tasks. These are not the physical manipulation trials above. [6]
  • Adaptation boundary — configure and train on the target environment. Shared hyperparameters do not establish that one pretrained checkpoint transfers across tasks. [7]

Who chooses the action in a combined system?

Editorial illustration: suppose a robot must put a cup on a tray. A VLA could propose a motion and a separate world model could predict consequences. A specified selector would still have to choose the command. After execution, fresh sensor feedback would update the next decision. This hypothetical combination is not an implementation reported by the projects above.

An integration needs matching coordinate frames, action units, time steps and observation conventions. It also needs a defined response when prediction and observation disagree. Calling two components “VLA” and “world model” does not resolve these interfaces or show that they work together.

Compare evidence under matching conditions

Editorial reading checklist: record the exact checkpoint and paper revision, sensor inputs, action interface, training and adaptation data, test environment and success definition. Mark missing information as “not reported”; do not fill it from a related model’s documentation.

Keep direct deployment, target-task fine-tuning and learning from environment interaction in separate columns. A robot-task success rate, a game return and prediction quality measure different things. This guide compares system roles, not performance ranks, and reports no IAEI deployment or independent experimental reproduction.

This is an explanatory guide. Examples and reading advice are editorial synthesis; cited findings remain bounded by the original studies. Sources and corrections policy

Continue reading