Robotics training data glossary
Last updated: 2026-11-09
Short definitions of the terms you will see when comparing robotics data providers.
Data types
- Teleoperation data: demonstrations recorded while a human remotely controls a real robot, often with a VR headset and hand controllers. It captures the robot's own joint states and actions.
- Egocentric video: first-person video from a head-mounted or chest-mounted camera, recorded while a person does a task. It is often paired with depth, IMU and hand-pose streams.
- Exocentric video: third-person video of a task, recorded from cameras placed around the scene.
- UMI (Universal Manipulation Interface) data: demonstrations collected with a handheld gripper rig instead of a robot, usually with gripper-mounted cameras and pose tracking.
- Expert demonstrations: recordings of skilled workers doing real jobs such as assembly, repair or logistics.
- Synthetic or simulation data: data rendered in a physics simulator, such as NVIDIA Isaac Sim, with labels generated automatically.
Labels and signals
- Action segmentation: splitting an episode into timestamped steps or subgoals, often with a caption for each.
- 6-DoF pose: position and orientation in 3D space (six degrees of freedom), used for camera, hand or object tracking.
- Hand-object interaction labels: when contact starts and ends, the grasp type, and which object was touched.
- Point cloud: a set of 3D points, typically from LiDAR, labeled with cuboids or segmentation.
- Sensor fusion annotation: labeling camera, LiDAR and radar data together so objects are linked across sensors.
- Provenance: evidence of where a clip came from, who consented, and whether it was edited.
Models and methods
- VLA (vision-language-action) model: a model that takes a camera frame, a language instruction and the robot state, and outputs an action.
- Imitation learning: training a robot policy to copy human or teleoperated demonstrations.
- World model: a model that predicts how an environment changes, used to simulate outcomes or generate training data (for example NVIDIA Cosmos).
- Sim-to-real: transferring what a model learned in simulation to a physical robot.
- Domain randomization: varying lighting, textures and physics in simulation so models generalize better.
Formats
- LeRobot: Hugging Face's open format and library for robot learning datasets.
- RLDS: Reinforcement Learning Datasets, an episode-based format.
- HDF5: a general-purpose file format for large arrays, common for robot logs.
- MCAP and ROSBAG: formats for recording timestamped multi-sensor messages, widely used with ROS.
- COCO and KITTI: common label formats for 2D and 3D detection datasets.