The Physical AI Stack

Humanoid robots, autonomous mobile robots, and self-regulating production systems may seem like standalone technologies at first glance. In reality, they are the result of many technological building blocks that must work together in close coordination. For a machine to perceive its environment, make decisions, and perform physical tasks, it needs far more than just an AI model: computing power, data, simulation, sensor technology, learning models, and precise mechanics. This interplay is known as the Physical AI stack. In this article, we break the stack down into six layers, explain what each one does, and identify where we believe the industry's biggest bottleneck lies. To learn what Physical AI is in principle, read our article Physical AI: The Next Evolutionary Stage of Artificial Intelligence.

What is the Physical AI Stack?

The Physical AI stack is a layered model that describes all the technologies required by an autonomously acting system: from computing power and training data to physical movement. Unlike traditional software, which only processes digital information, a Physical AI system must capture its environment, interpret it, make decisions, act, and learn from the results.

• Technically, two cycles can be distinguished

• The training cycle takes place before deployment, usually in a data center. Here, AI models learn from real and synthetic data, often within simulations.

• The runtime cycle takes place within the robot itself, in real time: perceive, decide, act, measure the result, and perceive again.

Both are connected: data from real-world operation flows back into the training process to improve the next model version.
‍

Why a Layered Model Helps

Physical AI is neither a single product nor a single technology. Understanding the layers helps you better evaluate product announcements, market trends, and investment decisions: a new robotics model, a new edge chip, and a new robotic hand are advancements on completely different levels. The model makes it clear where a company is actually innovating and where only one building block among many has been replaced.

The Six Layers of the Physical AI Stack

The bottom three layers make a system capable of learning, while the top three make it capable of acting. Compute, data, and simulation/world models form the foundation for training. Perception, robotics foundation models, and control/actuation form the cycle that runs in real time within the robot.

1. Compute: Computing Power for Training & Real-Time Operation

Physical AI requires two fundamentally different types of computing power:

• In the data center large GPU clusters train models using enormous amounts of data and perform complex simulations.
• Inside the robot itself a compact, energy-efficient edge computer runs the trained models in real time.

The requirements differ significantly. In training, raw power is what counts; in the robot, low latency, energy efficiency, and reliability are key. A robot cannot wait for a response from the cloud when it is in the middle of grasping an object or needs to avoid a person.

2. Data: The Raw Material for Every Model

Robotics models learn from data, and this is precisely where the biggest difference from language models lies. For text, there is the internet; for physical actions, there is no comparable archive. Every movement from which a robot is to learn must first be captured. The most important sources are:

• Teleoperation: A human remotely controls the robot, and the system learns from the recorded movements. This provides high-quality, albeit expensive, data.
• Human demonstrations: Videos and motion data of humans performing a task.
• Real-world process data: Measurement data on how work actually takes place in production and logistics—specifically, which activities occur, how often, for how long, and under what conditions.
• Synthetic data: Data generated in simulations or with world models that supplements and multiplies real-world data.

The quality and realism of this data significantly determine how well a model will later perform on the actual factory floor.

3. Simulation & World Models: Learning Before Acting

Because real-world data is scarce, expensive, and sometimes safety-critical, the industry is shifting a large portion of learning into virtual environments. Two different approaches are used here:

• Physics simulation and digital twins calculate the behavior of objects using a physics engine based on fixed rules. Robots can practice tasks thousands of times in parallel, for example, using reinforcement learning.
• World models are, by contrast, learned, generative AI models. They predict how a scene will develop next and can generate realistic training scenarios. Compact world models now even run directly on the robot.

The central challenge of this layer is the sim-to-real gap. No simulation perfectly replicates friction, material properties, or sensor noise. What works in simulation must first prove itself in reality. Techniques such as domain randomization, where simulation parameters are deliberately varied, and fine-tuning with real-world data help bridge this gap.

4. Perception: How Machines Perceive the World & Themselves

A robot must perceive two things simultaneously: its environment and its own body.

• Environment: Cameras, depth sensors, and LiDAR.
• Own body: Proprioception measures joint positions, speeds, and orientation, for example via encoders and inertial measurement units (IMUs).
• Gripping: Force-torque sensors and tactile sensors indicate how firmly an object is being held and whether it is slipping.

Only the combination of these sources—sensor fusion—provides a reliable situational awareness. In modern systems, perception is often no longer a separate module, as robotics foundation models process camera images and sensor data directly.

5. Robotics Foundation Models: From Perception to Action

This layer is the actual "intelligence" in the stack. The current state of the art is Vision-Language-Action (VLA) models. They receive camera images and a natural language instruction, such as "put the box on the top shelf," and generate motion commands directly from them. Well-known examples include NVIDIA Isaac GR00T, π0 from Physical Intelligence, and Gemini Robotics from Google DeepMind.

Many architectures combine two levels:

• A slower model, which understands a task and breaks it down into sub-steps.
• A fast layer, which generates fluid movements from those steps.

These models are primarily trained using imitation learning—learning from demonstrations—and reinforcement learning in simulation.

The difference from traditional automation: instead of hard-coding every movement, the system learns general skills that it can apply to new objects and situations. This generalization is the industry's stated goal, though it remains limited in practice. Robust performance over long shifts and changing conditions is one of the biggest remaining challenges.

6. Control $ Actuation: When Decisions Become Movement

The top layer physically implements decisions. It consists of two parts:

• Control (Software): Motion planning and real-time control ensure that a planned movement is executed precisely, stably, and without collisions. These control loops run significantly faster than the AI models above them. Middleware such as ROS 2 connects the components.
• Actuation (Hardware): Motors, gearboxes, joints, and grippers or robotic hands. Especially for humanoid robots, dexterous hands with many degrees of freedom and compact, cost-effective actuators are considered one of the major technical bottlenecks.

Functional safety is also part of this layer. Errors here do not have a digital effect, but an immediate impact in the real world. Standards such as ISO 10218 for industrial robots and ISO/TS 15066 for human-robot collaboration, as well as the future EU Machinery Regulation, set the framework.

This is precisely where the difference from generative AI becomes apparent: a language model can describe an action, but a physical AI system executes it.

Which Layer will Determine the Winners of the Next Decade?

In our view, the biggest bottleneck lies in the data: those who better understand how physical work actually takes place will train better models. If you view the stack as a map, the layers are contested to varying degrees:

• Compute is an established, capital-intensive market with a few dominant providers.

• Actuation is mature for classic industrial robots, but remains a real bottleneck for humanoid robots, especially regarding hands and costs.

• Foundation models are developing rapidly, but they are only as good as the data used to train them.

• Simulation and world models multiply data, but they do not replace real-world data: without a real-world reference, the sim-to-real gap remains.

This puts the data layer at the center. Data showing what work looks like in real warehouses and factories is particularly scarce: which manual tasks occur how often, how long they take, and where humans have to dodge, wait, or lift heavy loads. Such data helps on several levels: in selecting tasks that are worthwhile for robots, in building realistic simulation scenarios, and as a benchmark against which a robot's performance must be measured.

This is exactly where we come in. For years, MotionMiners has been using process and motion analysis to capture how people actually work in production and logistics.

Conclusion

Many technologies in the physical AI stack have existed for years. What is new is that all six layers are simultaneously reaching a level of maturity that makes the move from the lab to the factory floor possible. The real innovation, therefore, does not lie in a single component, but in their interaction. And this interaction stands or falls with data from the real working world.

Would you like to know which of your processes are suitable for physical AI?

Find out together with us now!

‍