1. Identify what the experiment actually tests
Start with the task and the train–test split. LIBERO separates several kinds of knowledge transfer, including spatial relationships, objects and goals. Its suites address different questions; a score without a named suite loses important context. [1]
Our reading recommendation is to write the claim in one sentence: this policy completed these tasks, under these conditions, after this training. Then identify exactly what was unfamiliar at evaluation. A new camera view, a new object instance and a new skill are different tests of generalization.
- Record the benchmark version, task subset and training data.
- Check whether test objects, scenes or demonstrations appeared during training.
- Distinguish variation within a known task from transfer to an unseen task.
2. Read the success rate together with the setup
RLBench exposes different observation and action settings, and explicitly specifies a robot arm for benchmark comparisons. This makes the configuration part of the result, rather than a minor implementation detail. [2]
For your comparison sheet, record the sensors, camera views, controller, action frequency and episode limit. A policy given object coordinates has different information from one using images alone. Likewise, a planner-assisted action interface changes what the learned model must solve. Do not rank results until their evaluation conditions are sufficiently comparable.
- Look for the success criterion, trial count and uncertainty reporting.
- Check whether retries, resets or human interventions count as failures.
- Read per-task results alongside the average; inspect repeated failure cases.
3. Separate data diversity from demonstrated transfer
The DROID policy documentation recommends including demonstrations from the target domain in the training mixture. When reading a DROID-based result, therefore, check whether the tested setting contributed additional training data. [3]
Our interpretation rule is to keep simulation, physical-robot evaluation and deployment claims separate. Simulation can test a defined hypothesis; evidence of physical performance requires physical trials. Neither setting alone establishes reliability in every environment. Ask which changes were tested individually and which combinations remain untested.
- List the tested changes: objects, lighting, layout, camera or robot.
- Identify target-domain adaptation and the demonstrations it used.
- For physical trials, record the tested hardware and recovery procedure.
- Treat untested environments as open questions, not implied capabilities.
4. Keep conclusions within the evidence
The robomimic study reports sensitivity to demonstration quality, algorithm choices and stopping criteria. Its findings illustrate why a headline score needs a description of how the model was trained and selected. [4]
Before citing a paper, note its version, publication status and available code or evaluation protocol. An arXiv link alone does not establish peer review; check the stated venue separately. A compelling video illustrates behavior but does not reveal the full trial distribution. These are reading recommendations, not an independent reproduction of the cited studies.
- Separate the authors’ observations from their explanations and future aims.
- Label your own interpretation, and retain the experiment’s limitations.
- Finish with three lines: what was tested, what was observed and what remains unknown.