Most practical AI teams know they need data. Few people know that they need a stack of it. The capabilities of an applied humanoid robot, AV, or warehouse need – perception, action, following instructions, execution of multi-step workflows – each map to a different layer of training data, with different methods of collection, depth of annotations, and quality controls. The AI data stack is a way to think of those layers as one integrated system rather than four disconnected purchasing decisions.
Key Takeaways
- The AI data stack consists of four layers tied to four real-world capabilities.
- Layer 1 includes human activities and visual and cognitive display data.
- Layer 2 captures data to manipulate the robot to perform repetitive tasks.
- Layer 3 aligns the idea, language, and action of teaching next to the scale.
- Layer 4 supports long-horizon, multi-step completions in real-world environments.
- Each layer provides the following; the weakness below spreads the stack.
Why think of virtual AI data as a stack?
AI virtual data behaves like a stack because each dynamic layer depends on the layers below it. Vision data without action data produces a model that sees but cannot move. Action data without language alignment produces a model that moves but cannot follow instructions. Long-horizon workflow data without strict instructions follows folding in the first multi-step task.
NVIDIA’s open body AI dataset, released to the developer community, includes thousands of hours of multi-camera video with unprecedented diversity (NVIDIA, 2025), and even at that scale, the teams below still need their task-specific layers on top of it. Prior training data is necessary, not sufficient.
Background 1: What does cognitive data include?
Human perception data is human activity and visualization data — first-person and third-person images of people performing tasks in real-world environments. It teaches the model what the world looks like and how people move through it.
Demographic data: Video and sensor recordings of people performing tasks, with annotations that align observations with actions, goals, or outcomes.
This layer feeds vision, scene understanding, and purpose interpretation. Quality questions to ask:
- Does the data include the areas your robot will operate in?
- Are the shows annotated at the atomic action level, or individual clips?
- Is participant consent documented and traceable?
Shaip’s L1 data collection layer captures real-world activity across kitchens, factories, warehouses, healthcare facilities, and roads — environments more like shipping than lab conditions.
Layer 2: What does performance data cover?
Performance data is robot manipulation data — the trajectories, joint positions, object interactions, and communication capabilities of physical repetitive tasks. It teaches the model how to do it, not just what to see.
Robot manipulation data: Time-stamped sequences of robot states, end-worker postures, and object interactions, captured during telework, scripting, or viewer playback.
This is where the structure of embodiment comes into play. Joint configurations, gripper geometries, and action spaces vary across robots, so manipulation data is rarely portable across embodiments without re-orientation. Cross-embodiment efforts – such as a dataset that includes 22 robots under a single action schema (DeepMind/Stanford et al., 2024) – have made this easier, but task-specific manipulation data remains a manual collection process.
Layer 3: What does VLA data add?
VLA data adds linguistic understanding to vision and action — every piece carries natural language instructions tied to the path it executes.
Vision-Language-Action (VLA) data: Episode-level training data containing synchronized visual observations, natural language instructions, and action trajectories with success labels.
This layer is what allows following instructions. Without it, the manipulation model can do one trained job; with it, the same backbone can include all hundreds of instructions. Practical: language definitions must be atomic, specific, and consistent with actual action parameters — not vague abstractions. The accuracy of the annotations in this layer determines whether a well-tuned VLA adapts to new information or memorizes the training set.
Layer 4: What does long-horizon activity data include?
Long-horizon task data includes multi-step workflows — sequences in which a robot must complete one subtask to begin the next. Cooking food, organizing a warehouse pallet, and assembling a set are all time-consuming tasks. Each requires a model to track state, recover from the failure of an underlying task, and chain capabilities.
A research data set focusing on long-horizon tablet manipulation includes 200 episodes across 20 multi-step tasks with dense scenes (LHManip authors, arXiv, 2024) – small in scale but tightly structured. Production teams often build test sets with hundreds to thousands of episodes for a long time horizon, and special tracking for failure detection.
