The research question focuses on how robots can understand and execute complex tasks based on human intentions conveyed through various cues, including language and gaze, in naturalistic settings.

HEIR captures human speech, gaze, and egocentric video alongside robot observations and teleoperated navigation-manipulation trajectories, preserving successive episodes where each request involves multiple execution subtasks.

The authors report conducting offline experiments to assess how models improve with additional human-centric context and real-robot experiments to evaluate task completion, demonstrating HEIR's value for studying user-centric interactions.