Splat2Act: From Captured Scenes to Personalized VLA Manipulation
Automatically prepared brief
Splat2Act is an automated pipeline that personalizes a pretrained vision-language-action policy to a specific scene and object set using a handheld phone scan and images of the user's objects.
The research question addresses how to adapt a generalist manipulation policy to successfully pick up objects in various user-specific environments without requiring extensive real-world demonstrations.
The proposed approach, Splat2Act, uses a handheld phone scan and images of the user's objects to create a photorealistic digital twin of the environment. It identifies receptacles, constructs pickup tasks, and generates successful demonstrations through simulation.
The authors report that when applied to 101 scanned campus environments, the pipeline yields 852 receptacles and over 68,000 pickup demonstrations. Fine-tuning the policy in two stages resulted in a personalized policy achieving 62.5% pickup success in the target scene and 63.9% success on a real robot.
About this brief
This article summarizes a public preprint abstract. Its findings have not been independently reproduced for this publication. Consult the paper for methods, evaluation scope and limitations. Automated briefs may contain errors; the original source takes precedence.