The transition from chatbots to robots that follow natural language commands goes through one class of models. VLA models — visual language action models — combine visual perception, language comprehension, and action into a single neural network. Their power is real, but it depends almost entirely on the training data they import. This guide explains what the VLA training data actually contains, which groups it ignores, and how to organize the dataset that generates the model to use.
Key Takeaways
- Mapping vision of VLA models and language input directly to robot actions in a single network.
- Training data should include synchronized visual observations, language instructions, and actions.
- Various action tokens require large-scale display data to read properly.
- Human egocentric video is increasingly being mined as a low-cost VLA training resource.
- Robust test plots are as important as training data for reliable use.
- A good VLA configuration succeeds or fails on the intensity of the annotations, not the raw volume alone.
What is the VLA model?
A VLA model is a basic model of a robot that takes images and natural language commands such as input and robot actions. Unlike traditional pipelines that separate vision, programming, and control into separate modules, vision language action models learn end-to-end mapping to a single network.
VLA model: A neural network that takes synchronized visual observations and natural language commands and generates a sequence of robot actions or action tokens.
This unified design allows VLA models to gain cognitive abilities from pre-training visual language and expand on motor control. In practice, that means a single model can actually perform multiple tasks – but only if its training data covers it with the right structure.
What does VLA training data contain?
The VLA training data consists of four main ingredients per episode: observations, natural language instructions, action traces, and a success or failure label. Within that, teams add timestamps, eligibility status, and evaluation tags.
Four layers are required:
- Visuals – RGB frames, often paired with depth or wrist-cam views.
- Language guidelines — short natural language commands such as “pour water into a cup.”
- Methods of action – discrete or continuous action sequences mapped to degrees of freedom.
- Result labels – clear success, failure, or part completion markers for each episode.
The 7 billion open VLA model was trained on more than 1 million episodes taken from 22 robot embodiments (Stanford et al., 2024), showing the diversity expected to perform various tasks. Outside of this scope, VLA models tend to memorize specifics rather than generalize.
Why is action annotation more difficult than image annotation?
Action annotation is more difficult because actions reside in continuous, high-dimensional spaces and depend on the representation of the robot, not just the content of the frame. Labeling the binding box on the cup is straightforward; labeling a trajectory that successfully holds that cup with a particular holder at a particular point of contact is not.
Action token: The implicit representation of the robot’s movement or displacement results that the VLA model can predict as a language token.
Annotation teams need to synchronize each action token with its synchronized lookup, quickly mark the contact, capture failure detection, and mark the atomic boundaries of the language instruction. Shaip’s data annotation workflow handles this at scale, with systematic taxonomies mapped to robot action spaces and limits for accepting each task.
Where does egocentric human video fit into VLA training?
Egocentric human video fits as a quick training resource that fills in the gaps real robot data cannot. A video of a first person cooking, picking, and assembling captures behavior at a scale a robot can’t achieve.
A recent paper converted unstructured human videos into VLA-formatted episodes – 1 million segments and 26 million frames – by treating the human hand as the final sly operator (Wu et al., arXiv, 2025). This type of cross-embodiment data is now standard in VLA pretraining recipes.
Catch: raw video is not training data. It requires segmentation, language definitions, manual reconfiguration, and quality assurance before it reaches the VLA pipeline. Shaip’s Physical AI data ops include egocentric imaging, real2sim transformation, and VLA-aligned annotation in a single delivery.
How do you build test sets that capture VLA failure modes?
Tests set VLA failure modes when designed before training, not after. The three most important frameworks: deployment success benchmarks, non-deployment generalization probes, and risk-based security scenarios.
Consider a VLA indoor model highly trained in kitchen duties. A logical test set can test: known operations in known kitchens (distributed), known operations in unusual lighting (gentle OOD), unknown objects with known instructions (general logic), and rare events such as accidental spills (safety class). With each exception, the risk of deployment remains unquantifiable.
A useful neutral resource for planning risk category coverage is the NIST AI Risk Management Framework, which categorizes impact categories in a way that maps cleanly to test set design.
