The research question focuses on developing a comprehensive model capable of handling 3D perception, action prediction, and goal-directed planning for egocentric dexterous manipulation tasks.
The proposed approach, P³-WAM, employs a unified world action model that combines three modes: Perceive, which reconstructs hand motion and camera trajectories from video; Predict, which generates future visual outcomes based on specified actions; and Plan, which creates action sequences and their expected visual effects given an initial image, language instruction, and optional visual goal.
The authors report that experiments demonstrate improved performance across all three capabilities when trained together compared to single-mode training. The model outperforms specialized baselines in 3D hand reconstruction and action-conditioned video prediction on four datasets, and generates manipulation videos that better follow instructions and show more coherent action progressions.