QF3 is an online off-policy RL algorithm that enhances flow policies using filtered Q-gradients, enabling fast training and transfer to hardware.
The research question addresses the need for efficient reinforcement learning methods to improve or learn flow policies from scratch, particularly for complex robotic tasks like humanoid locomotion.
The proposed approach, QF3, combines flow matching with the critic's action gradient, filtered to ensure reliability. It uses a one-step prediction of the flow's output to backpropagate gradients selectively based on proximity to replay actions.
The authors report that QF3 achieves a 10x speedup in training humanoid locomotion and motion-tracking policies compared to FPO++. They also demonstrate its ability to fine-tune pretrained manipulation policies on simulation tasks.
About this brief
This article summarizes a public preprint abstract. Its findings have not been independently reproduced for this publication. Consult the paper for methods, evaluation scope and limitations. Automated briefs may contain errors; the original source takes precedence.