← All guides

IAEI / RESEARCH GUIDE

How to read robot-learning papers: beyond success rates

Use comparability and transfer-evidence records to read robot-learning results: check experimental conditions, target-scene demonstrations and adaptation, with worked examples from official sources.

1. Identify what the experiment actually tests

Start with the task and the train–test split. LIBERO separates several kinds of knowledge transfer, including spatial relationships, objects and goals. Its suites address different questions; a score without a named suite loses important context. [1]

Our reading recommendation is to write the claim in one sentence: this policy completed these tasks, under these conditions, after this training. Then identify exactly what was unfamiliar at evaluation. A new camera view, a new object instance and a new skill are different tests of generalization.

  • Record the benchmark version, task subset and training data.
  • Check whether test objects, scenes or demonstrations appeared during training.
  • Distinguish variation within a known task from transfer to an unseen task.

2. Read the success rate together with the setup

RLBench exposes different observation and action settings, and explicitly specifies a robot arm for benchmark comparisons. This makes the configuration part of the result, rather than a minor implementation detail. [2]

For your comparison sheet, record the sensors, camera views, controller, action frequency and episode limit. A policy given object coordinates has different information from one using images alone. Likewise, a planner-assisted action interface changes what the learned model must solve. Do not rank results until their evaluation conditions are sufficiently comparable.

  • Look for the success criterion, trial count and uncertainty reporting.
  • Check whether retries, resets or human interventions count as failures.
  • Read per-task results alongside the average; inspect repeated failure cases.

3. A ten-field record for comparing results

For each result, fill in the ten fields below with the reported value, a paper section or configuration link, and a status: aligned, different or not reported. This is an editorial reading aid, not an official benchmark standard. The robomimic study motivates checking training data and stopping criteria alongside scores. [4]

Start by stating what you want to compare: algorithms under matched conditions, or complete systems with different resources. A shared benchmark name is insufficient. Missing information leaves comparability unconfirmed; an intentional difference should remain visible in the conclusion.

  • 1. Benchmark version — Record the release or commit, simulator and relevant dependencies, plus local patches. Save the exact evaluation entry point; a repository name or a moving branch alone does not identify the protocol.
  • 2. Task set — List task IDs, task order where relevant, train–test split, variations and initial-state distribution. Check whether both results cover the same tasks and conditions; do not substitute a suite average for a selected subset.
  • 3. Training data — Record datasets, demonstration counts, collection or filtering rules, pretraining and target-task fine-tuning. Mark any overlap with evaluation objects, scenes or demonstrations. Unreported data are not evidence of a data-free or zero-shot setting.
  • 4. Observation inputs — List camera views, image size, RGB, depth, robot state, task instructions and history. Record preprocessing and privileged simulator information. Check what the policy actually consumes, not merely what the environment can return.
  • 5. Action interface — Record the robot, gripper, command representation, controller, action scaling, control frequency and any planner. Distinguish joint commands from end-effector targets and note how many environment steps each model decision executes.
  • 6. Trial count — Record attempted episodes per task, independent training seeds, initial states and episode limits. Keep repeated rollouts of one checkpoint separate from independent training runs; record unfinished or excluded trials and their reasons.
  • 7. Success criterion — Locate the task predicate or scoring code, thresholds, stopping conditions and timeout rule. State whether success must persist and whether the score measures full completion, partial progress or another outcome.
  • 8. Human intervention — Record permitted prompts, teleoperation, object repositioning and recovery during an attempt. Distinguish scheduled resets between trials from assistance within a trial, and state how assisted attempts and retries affect the denominator.
  • 9. Model selection — Identify the checkpoint, stopping rule, selection metric and dataset used to choose it. Record whether test outcomes influenced checkpoint or hyperparameter selection; distinguish a preselected model from the best score found after repeated evaluation.
  • 10. Statistical reporting — Retain per-task successes and attempts, the aggregation formula and uncertainty definition. Distinguish an equal-weight task mean from pooled successes divided by pooled attempts, and identify whether variability is across training seeds, episodes or both.

Conclude with one of three statements: conditions aligned for the stated comparison; specific differences limit the comparison; or evidence is insufficient. A larger mean alone does not resolve uncertainty or justify attributing a difference to the algorithm.

4. Trace the protocol through official configuration

These examples were checked on 12 October 2026 at LIBERO commit 8f1084e and RLBench commit 02720bb. The source links below pin the full commits. They illustrate how to complete the record; we did not run the benchmarks or measure model performance.

Read the configuration, the function that consumes it and the saved configuration of the reported run together. Defaults and teaching examples do not establish which settings a later paper actually used.

  • LIBERO trial count and policy-step limit: eval/default.yaml sets n_eval to 20 and max_steps to 600. In metric.py, evaluate_one_task_success loads fixed initial states, performs five zero-action settling steps, and then starts the policy loop limited by max_steps. It divides successful attempts by n_eval. These are settings and counting logic, not a reported model score. [5][6]
  • LIBERO evaluation entry point: the separate evaluate.py reads a saved checkpoint and its configuration, exposes --benchmark, --task_id, --seed, --ep and --load_task, and uses env_num = 20 for its rollout batch. It does not use n_eval as that denominator. Record the actual script and checkpoint; do not assume that changing one default affects every entry point. [7]
  • LIBERO observations: data/default.yaml lists agentview_rgb and eye_in_hand_rgb, plus gripper_states and joint_states; its depth list is empty. The observation conversion in metric.py follows cfg.data.obs.modality and obs_key_mapping. Check the run’s resolved configuration before describing a policy as image-only. [8][6]
  • RLBench observations and actions: ObservationConfig exposes camera modalities and low-dimensional fields. The imitation-learning example constructs MoveArmThenGripper with JointVelocity and Discrete, while its example learning batch selects left_shoulder_rgb. Environment outputs, policy inputs and action targets are separate fields to verify; this teaching example is not a standardized performance evaluation. [9][10]

If a paper reports only “LIBERO success rate” or “RLBench success rate,” leave unspecified fields as not reported. Retrieve its release, appendix, evaluation command and saved configuration before deciding which scores answer the same question.

5. Record what changed before calling a result transfer

A result in a new scene may involve existing model capabilities, extra demonstrations, or updated weights. Record these separately before interpreting the score. The following seven fields are an editorial reading template; “not reported” and “not independently checked” are valid answers.

  • Prior data and initialization: identify the starting checkpoint, robot-training mixture and pretrained vision/language components; distinguish the released dataset from the subset actually used.
  • Target-scene demonstrations: record their count, tasks and collection setting. Missing disclosure does not mean zero demonstrations.
  • Adaptation: record direct evaluation, joint training or fine-tuning; identify updated weights and data-mixture weights where available.
  • Test separation: distinguish training, validation/model selection and final evaluation; state which objects, scenes or tasks were held out.
  • Tested changes: list the actual changes in objects, background, camera, layout or hardware, and retain the trial and success definitions.
  • Unknowns: mark missing information and what you have not independently checked. Do not infer absence of training overlap.
  • Source location: retain the paper version, section/table/appendix and code commit. Separate the authors’ result from your interpretation.

Use one record per experiment. Simulation, physical trials and deployment evidence answer different questions. A tested variation does not establish robustness to every combination of changes.

6. Worked record: DROID co-training

This record concerns the paper’s policy experiments, not the size of the full dataset. [11]

  • Prior data: Appendix F-B specifies the first 40,000 successful, language-annotated DROID trajectories; the visual encoder is ImageNet-pretrained.
  • Target demonstrations: 50–150 per task in the described evaluation setups.
  • Adaptation: diffusion policies use 50/50 batches of target-domain and DROID data; this is co-training, not zero-target-data evaluation.
  • Test separation: the paper distinguishes in-distribution and out-of-distribution physical trials, with 10 rollouts per task setting and method.
  • Tested changes: task-specific distractors, replacement objects and, for Cook Lentils, a camera shift.
  • Unknowns in this record: trajectory-level training/test overlap has not been independently audited; mark it unknown.
  • Source locations: v2 §V-A/B and Appendices E, F-A/B. The official policy README also recommends target-domain demonstrations in the training mixture. [11] [3]

Reading consequence: any claimed benefit must retain the target-data condition. The dataset’s diversity alone does not establish transfer without target-domain adaptation to an arbitrary new site.

7. Worked record: OpenVLA adaptation to Franka

Keep the paper’s direct evaluations (§5.1) separate from the Franka adaptation experiments (§5.2). The official project likewise identifies its Franka demonstrations as fine-tuned rollouts. [12] [13]

  • Prior data: a pretrained Prismatic vision-language model followed by robot training on an Open X-Embodiment mixture.
  • Target demonstrations: 10–150 per task across seven Franka tasks.
  • Adaptation: §5.2 updates all parameters. LoRA is a separate §5.3 experiment; do not transfer its settings to this result.
  • Test separation: Appendix B.3 distinguishes in-distribution and out-of-distribution trials.
  • Tested changes: task-specific backgrounds, objects, distractors and object poses, rather than every possible deployment condition.
  • Unknowns: the base components’ complete pretraining data are not disclosed (§3.1), so universal absence of prior exposure cannot be established.
  • Source locations: v3 §§3.1, 3.3, 5.2 and Appendix B.3. The “OpenVLA (scratch)” comparison retains the base VLM and omits OpenX robot pretraining; it is not random initialization of all weights. [12]

Reading consequence: this experiment tests adaptation with target demonstrations. To attribute gains to prior robot training, inspect the matching ablation and task conditions, not only an aggregate score.

8. Keep conclusions within the evidence

The robomimic study reports sensitivity to demonstration quality, algorithm choices and stopping criteria. Its findings illustrate why a headline score needs a description of how the model was trained and selected. [4]

Before citing a paper, note its version, publication status and available code or evaluation protocol. An arXiv link alone does not establish peer review; check the stated venue separately. A compelling video illustrates behavior but does not reveal the full trial distribution. These are reading recommendations, not an independent reproduction of the cited studies.

  • Separate the authors’ observations from their explanations and future aims.
  • Label your own interpretation, and retain the experiment’s limitations.
  • Finish with three lines: what was tested, what was observed and what remains unknown.

This is an explanatory guide. Examples and reading advice are editorial synthesis; cited findings remain bounded by the original studies. Sources and corrections policy

Continue reading