Foundation models have expanded the range of instructions and scenes that robots can handle. Sustained physical tasks also require robots to detect when their actions have unexpected effects and respond before small errors accumulate. This survey uses Agentic Embodied Intelligence (AEI) to describe systems that ground decisions in physical observations, plan and execute actions, and use their outcomes to guide recovery or revision. We trace how this cognitive closed loop connects high-level reasoning to real-time control. We then examine the computing infrastructure, data, safety mechanisms, and evaluation methods needed to support it, with particular attention to what current systems demonstrate and what remains open.
Embodied intelligence develops through interaction between an agent, its body, and its environment. A robot's sensors, actuators, and physical limits determine both what it can learn and which actions are feasible. The survey traces this idea from feedback control and situated robotics to learning-based systems.
Imitation learning and visuomotor policies broadened the range of behaviors robots could learn from data. Foundation models brought wider language and visual capabilities to that line of work. The autonomy ladder puts these advances in context by asking how much of the task the robot can take responsibility for.
The L0–L5 ladder describes how responsibility shifts between humans and robots as autonomy increases. At L0, a person controls the robot directly. At L1, the robot executes individual primitives on command. L2 covers autonomous execution of predefined tasks in structured settings. At L3, vision-language-action (VLA) policies extend the range of tasks a robot can attempt from language instructions, with performance still dependent on training and operating conditions.
The transition from L3 to L4 shifts the focus from executing a task to sustaining a mission. Within a specified operating domain, an L4 system would need to gather missing information, coordinate skills, verify physical outcomes, and recover from routine failures. It would also need to recognize when to stop or escalate. L5 extends this research ambition to open-world operation.
The framework distinguishes the model, the execution infrastructure, and the robot itself. The model interprets goals and proposes plans or actions. The infrastructure schedules and dispatches those actions through skill and tool interfaces. Sensors, actuators, and low-level controllers carry out physical interactions and return observations, contact signals, and measured outcomes.
Perception and grounding estimate the state relevant to the task and identify what is uncertain. Planning uses that estimate to choose subgoals and check whether more evidence is needed. Execution translates a plan into skill calls or control commands. Closed-loop evolution compares the observed outcome with the intended one and uses the difference to revise state, memory, or the next action.
These functions can be distributed across modules or partly combined in a single model. In either case, their interfaces must carry information the next component can use. Perception needs to provide references and uncertainty that a planner can check. Plans need executable subgoals, and execution must report partial completion and failure. Data and evaluation support improvements across episodes, while safety constraints apply throughout operation and updating.
A robot asked to put a mug in a dishwasher must identify the intended mug, locate it relative to the robot, and check whether the mug and dishwasher are in suitable states. The survey distinguishes semantic grounding, spatial grounding, and physical grounding. Together, they describe what an object is, where it is, and how it can be used.
A useful state estimate has to remain current as the robot and its surroundings change. Passive belief maintenance associates new observations with previously seen objects and revises estimates when evidence changes. A cup missing from a camera image may have moved or may simply be hidden. Those possibilities have different implications for planning.
Active perception chooses an action specifically to resolve uncertainty. The robot may change viewpoint, probe an object, or ask the user to clarify an instruction. The choice depends on whether the information could change the next decision and on the cost of obtaining it, including time, effort, and physical risk.
Perception also supports execution and verification. For example, visual observations and contact signals can help estimate an object's pose when the gripper hides part of it. The resulting estimate can guide the next movement or help determine whether the intended physical change occurred.
Embodied planning connects a goal to a sequence of actions the robot can carry out from its current state. The survey treats goal formulation, hierarchical decomposition, candidate search, and plan representation as connected decisions. Candidate plans can differ in feasibility, anticipated effects, cost, and risk, especially when observations are incomplete or intermediate outcomes are uncertain.
The available skills determine how finely a task should be divided. A capable executor may handle a broad subgoal, whereas another may require separate steps for preparation and manipulation. The planner must check that the outcome of one skill provides the starting conditions for the next. This matters when a skill completes successfully but leaves the scene in a different state than the planner expected.
Language plans, generated programs, and callable skills expose different information to the executor. A useful representation preserves the target, the conditions required for action, and a way to check the expected result. When execution produces new evidence or invalidates an assumption, that feedback should inform which part of the plan is revised.
Action execution turns a plan or subgoal into robot commands. A policy may use images, the robot's own sensor readings, and a language instruction or goal image to predict motion. The survey separates how actions are represented, such as joint targets or end-effector poses, from how they are generated, such as discrete tokens, continuous predictions, diffusion, or flow matching.
In receding-horizon control, the controller executes part of a predicted action sequence before the policy makes a new prediction from updated observations. This lets the robot correct deviations as they arise. Predictions from successive sequences can disagree, so the system must balance responsiveness with smooth motion.
Learned policies often run more slowly than motor controllers. Action chunking provides several commands per prediction, and asynchronous execution can prepare the next chunk while the current one is being carried out. These approaches reduce waiting, but their effectiveness depends on coordinating new predictions with actions already in progress and with the robot's state when the new commands take effect.
The survey uses coding agents to illustrate a feedback loop in which actions can be tested, revised, and reused. Physical interaction makes this loop harder to validate. Resetting a scene can be costly, sensor evidence can be ambiguous, and a simulator may miss the conditions that caused a failure on the real robot.
The chapter distinguishes changes to control, reflection, memory, models, and the system itself. A control correction or a revised plan can help within the current episode. Updating a memory, skill, or policy makes a broader claim about future behavior and calls for evidence that the benefit carries over.
A useful update starts with a diagnosis. An incorrect stored object location may only require clearing that memory, while recurring grasp failures may justify revising a skill or collecting new data. The survey proposes recording the interaction, selecting a change that matches the failure, and evaluating it through replay, simulation, and constrained physical trials as appropriate.
Existing systems demonstrate pieces of this process, including feedback-based replanning, skill reuse, and policy adaptation. A complete update process would also separate proposing a change from approving it. Evaluation should test on new cases, check that earlier capabilities are retained, and record the released version and retain a way to roll back to the previous one.
The computing infrastructure connects models to the robot's sensors, actuators, and control software. It coordinates deliberative models, faster policies, memory, and tool calls. Its design affects which models can run within a robot's compute and power limits and whether commands can reach the controller in time.
The survey compares onboard, cloud, and hybrid deployments. Onboard computation avoids dependence on a network but must fit the hardware and energy budget. Cloud services offer access to larger models and introduce communication delays. A nearby edge server offers another place to run computation, with different limits on capacity and communication. Hybrid systems can keep time-sensitive control onboard and request more expensive reasoning from the cloud when needed.
For a fast control loop, occasional delays and connection failures can be more consequential than average latency. The runtime must schedule work across these different time scales. Interfaces such as the Model Context Protocol (MCP) can standardize access to tools and data, while the runtime and control stack remain responsible for meeting deadlines and handling interruptions.
Safety depends on how perception, planning, execution, and memory work together. A hazard may come from excessive contact force, an incorrect assumption about the scene, a missing prerequisite, or an action outside the robot's authority. A safety specification should make clear what is protected, when a constraint applies, and which actions the robot is permitted to take.
Figure 11 illustrates different responses to the same mug-transfer request. The robot may proceed if the route is clear and the action is authorized. It may remove an obstruction and recheck the route, inspect an unseen region, or decline a transfer it lacks permission to perform. Safety checks apply to the intermediate steps and the final state as well as to the transfer itself.
Safety also matters when changes persist across episodes. Figure 12 illustrates a candidate update that works in the recovered case but violates a constraint on an earlier task. Task success and safety therefore need separate checks, and the scope of any approved update should match the conditions under which it was validated.
The studies reviewed demonstrate specific safety capabilities under particular assumptions about hazards, robots, and environments. System-level assurance requires evidence that those protections still work together across a complete episode, including when observations are uncertain, a task is interrupted, or control is handed back to a person.
The embodied data pyramid organizes sources by their role in learning, from broad web and human video data to simulation and trajectories from the target robot. The Agentic Embodied Experience schema focuses on what a record captures about a particular interaction.
The schema links eight fields covering the goal, context, thought, subgoal, affordance, motion, feedback, and reflection. These fields link the evidence available at decision time to the intended interaction, the motion that was executed, and the effects that were observed. Failure and recovery records help explain where this sequence broke down.
The chapter reviews active exploration, human videos, simulation, and human corrections as ways to collect or annotate experience. In the agentic data flywheel, experience guides task selection, outcome assessment adds labels or diagnoses, and learning uses that feedback to improve policies. The resulting capabilities and remaining gaps inform the next round of collection.
The survey separates the roles of simulators, benchmarks, and evaluation protocols. A simulator provides an environment in which actions have observable effects. A benchmark defines the capability a task is meant to test. A protocol specifies what evidence is collected and how performance is judged.
The C1–C5 cognitive capability ladder organizes tasks by what makes them difficult. For example, clearing a spill may require choosing a tool by its absorbency even when grasping the tool is straightforward. The C1–C5 tiers describe demands placed on the agent in a benchmark. The L0–L5 levels describe the scope of its autonomy.
For AEI, a simulator should represent both physical dynamics and persistent task-relevant states. A test may depend on whether a container is filled, a door remains open, or an action changed the scene in a way that matters later. Results from simulation also need checks against physical robots when they are used to support claims about real-world performance.
A final success score can miss collisions, unnecessary retries, or a poor response to failure. The survey recommends reporting goal completion alongside decision quality, recovery, physical execution, and safety. These measures help distinguish a planning error from a control error and show when completing the task came at an unacceptable cost.
Perception needs to expose object identity, spatial references, and uncertainty in a form the planner can inspect. An open problem is keeping that uncertainty calibrated as the scene changes, particularly when relevant evidence is hidden, old, or available only through touch or force sensing.
Planning needs to allocate search to alternatives for which more evidence could change the decision. Comparisons should account for the motor policies available to each planner and for differences in computation, observations, and retries. These factors can all affect the final result.
Generalist policies still face challenges in dexterous manipulation and transfer across robot bodies. Action representations must capture useful similarities across platforms while respecting differences in kinematics and contact. Large policies also need to run with delays that the physical control loop can tolerate.
Experiments on evolution should match the scope of the claimed improvement. A local correction can be assessed within an episode, while persistent updates should be evaluated on held-out cases. Interaction costs, human resets, and unsuccessful candidates matter when comparing methods over time.
Distributed infrastructure needs ways to handle missed deadlines and network loss. Open questions include which functions can continue safely onboard, what the cloud is allowed to request through tool interfaces, and how local and remote state should be synchronized after a disruption.
Safety contracts need to stay consistent as information moves between components. A planner's assumptions about a target object, its state, and the robot's authority must still hold when the controller acts. Independent tests should check these relationships across embodiments, tasks, and operating conditions.
Combining data sources requires aligned timestamps, consistent object references, and traceable annotations. A wrong outcome label can affect both the next policy update and the next collection round. Datasets should capture rare failures and full recovery attempts, including cases where the attempted correction did not work.
Deployment evaluation should describe the operating conditions under which a robot can work unattended. Resets between trials can hide accumulated changes in the environment, calibration, or memory. Longer runs can reveal failures that isolated episodes miss.
When a robot reaches the boundary of its operating domain, evaluations should distinguish missed exceptions from unnecessary escalations. Reporting time to reach a safe state and whether the operator receives enough information to recover would make intervention rates more informative.
The survey presents agentic embodied intelligence as a system design problem. Vision-language-action models connect observations and instructions to control. World-action models help anticipate what actions will change, and programmatic approaches combine tools and skills into longer procedures. Dependable operation also requires the models, controls, infrastructure, and safeguards to work together under the conditions in which they will be used.
The literature reviewed does not establish sustained, unattended L4 operation across a broad, independently validated domain. L5 remains an open research goal. The strongest evidence for progress will come from systems that can verify their results, recover from disturbances, and stop or hand over control when they reach their limits.
The framework connects these requirements to the way systems are built and evaluated. It links changes in perception, planning, control, or learning to observable consequences in the physical world, so that improvements can be judged at the level of the complete system.