The LLM Picks the Verb, the Robot Owns the Motion

Someone typed "wave at me". A language model chose the verb `wave`, and aseparate process it cannot reach decided what waving means.
That loop took a while to earn, and almost none of the work was in the waving.

The obvious version, and why we did not build it

The tempting design is to let the model do robotics. Give it the joint angles,
describe the arm in the prompt, and let it emit motion. Or one step back, let it
write a short script and execute that.

Both fail for the same reason, and it is not that the model is not smart enough.
It is that correctness becomes sampled. A language model produces a plausible
token sequence. A servo does exactly what the number says. Put them together and
every failure mode of the model becomes a physical event in a room with people
in it.

Latency lives at the wrong layer. A control loop wants a decision every few
milliseconds. A model round trip is hundreds of milliseconds on a good day.

Safety becomes prompt engineering. If the model chooses velocities, the only
place to put a speed limit is the prompt, and a limit you can talk your way past
is not a limit.

Nothing is reviewable. "Do not move faster than 0.8 m/s" in English cannot
be unit tested. The same constraint as a number in a contract can.

One sentence

The model picks the verb. The robot owns the motion.

The model's entire authority is choosing a capability by name. Everything after
that, trajectory, speed, arrival, preemption, is deterministic code on the
robot. Poses, velocities, and actuator commands never cross.

That constraint is the whole design. The rest is the mechanism.

Two processes and a contract

Two processes, exchanging intents downward and events upward.

The whole of Phase 0. The agent runs inside our Mindroid runtime, the adapter
turns capabilities into tools and back into bytes, and Zenoh carries them out to
the Smart Creatures platform.

One thing the picture flattens: the adapter is not a single component. Half of
it lives in each process, and the two halves never share memory.

The brain half is not bespoke. It is Mindroid, our Rust
agent runtime, doing what it already does: a Pipeline of stages, a
ToolRegistry, and a Tool trait. This project was small because a robot
capability is just another Tool. The same trait that wraps a web API wraps a
servo, so giving an agent a body is a matter of what you register, not a fork of
the runtime.

The brain is allowed to be slow, to retry, and to be wrong, because it
cannot do anything directly. The robot has no model in it at all: a state
machine, an interpolator, and a set of trajectories. It is boring by design, and
boring is what you want holding the servos. Splitting them costs a serialisation
hop and buys a boundary you can review instead of a convention someone violates
in six months.

The robot tells the brain what it can do

We never wrote a list of tools for the model. The robot publishes a manifest at
startup, and the brain compiles each entry into a mindroid::Tool and registers
it before it will accept a single word of input.

message Capability {
string name = 1; // "wave"
string description = 2; // model-facing text
string target_schema_json = 3; // JSON Schema for the target
string params_schema_json = 4; // JSON Schema for constraints
bool interruptible = 5;
bool blocks_base = 6; // owns the movement lease
string safety_json = 7; // {"max_speed_dps": 90}
}

name and description are the boring fields. The last three are the point:
interruptible says whether "stop" can preempt this mid-motion, blocks_base
whether the capability takes exclusive control of movement, and safety_json
carries the limits as data, not as prompt text. The model never sees a speed
limit it could reason around, because the limit is not addressed to it.

Discovery paid off in a way we did not plan. The pick primitive only works in
simulation, where a magnetic constraint holds the card, so hardware does not
advertise it. The same binary against the real arm registers fewer tools, and
the model cannot call a verb that would have it grab at air. Nobody edited a
prompt to make that true.

The pick primitive, finished. A magnetic constraint holds the card, not a
real grasp.

The model's output is untrusted input

A tool call is not a command. It is a request from an untrusted client that is
very good at sounding confident. The robot checks it against what this robot
will actually do right now, and rejects anything else.


def factory(mission_id, req, on_event):
if req.capability not in PRIMITIVES:
return None, f"rejected_unknown_capability:{req.capability}"
if allowed is not None and req.capability not in allowed:
return None, f"rejected_unknown_capability:{req.capability}"
return ArmMission(mission_id, world, req.capability,
on_event=on_event, origin=origin), ""


Two checks, not one. The first rejects a verb that does not exist, the second a
verb that exists but is not permitted in this configuration. Rejection is a
MissionAck with accepted false and a reason string. Nothing moves, and the
brain finds out why.

A comment above it says what this check does not do. The robot ignores the
model-supplied target entirely, because every capability is a scripted gesture
whose keyframes come from its own configuration. The day one starts reading that
target is the day this becomes fail-open, and fail-closed validation is easy to
write and easy to quietly lose.

Missions, not calls

The other half is that a tool call does not return when the motion is done. It
returns immediately with a mission ID, and the robot streams events back.

This is what makes "stop" work. If a tool call blocked until the arm finished
waving, "stop" would arrive after the thing it was meant to prevent. Because the
mission is a handle, the brain can cancel mid-motion, and cancellation returns
the arm to its origin pose as an invariant rather than a best effort.

"stop" never reaches the model. It is matched in the input loop and dispatched
straight to the cancel path, because the one instruction that must never be
reinterpreted is the one that stops the machine.

Those events print alongside the conversation today. Folding them into the
agent's context is another stage, on the brain side.

We swapped the transport, and the design did not notice

The first version was a mobile base in simulation, one capability, gRPC. The
second was a real six-axis arm, scripted gestures, and Zenoh, which already
carried the camera frames elsewhere in the stack.

Transport, robot, and capability set all changed at once. What survived:

• the protobuf messages, byte for byte
• the manifest and the discovery handshake
• the mission state machine and its states
• every test that did not name a transport

The four request/reply methods became queryables, the server stream became
publish and subscribe on a per-mission key, and the change stayed inside one
transport module on each side. There is no server in the new arrangement at
all.

Every process is a peer. The brain does not hold the robot's address, it holds
the robot's name, a segment in the key expression, so starting the robot
after the brain works as well as the other order.

A boundary that survives having both of its sides replaced is a real boundary.

What broke

The model kept doing two things at once. Ask it to wave and it would call
greet and then wave, or wave and then decide to nod about it. Each is a
separate mission, so one sentence turned into three gestures back to back while
the person who typed it waited.

Nothing unsafe happened, which is the part worth noticing. Each call was a
well-formed request for a verb that exists, so the robot accepted it and played
it correctly. The bug was in how many verbs one utterance could produce, and it
lived entirely on the brain side. We capped the runtime's tool executor at one
iteration per turn.

.add_streaming_stage(
ToolExecutorStage::new(llm, self.registry.clone()).with_max_iterations(1),
)

The moral is not the prompt fix. It is that a model doing something we did not
intend showed up as too much correct motion, on the side of the line we could
reach, instead of as an arm doing something nobody designed.

What this is not

This is a proof of concept, and the parts that make it fast to build are the
parts that make it unsafe to deploy.

The arm motion is open-loop scripted trajectories played through an
interpolator. There is no learned policy, no vision in the control loop, no
whole-body control, no real-time guarantees. The camera grounding and voice loop
are designed and not built, so today it is typed text in and printed events
out. The robot process is Python, written to be thrown away and replaced
by something that speaks the same contract. One robot, one session, no audit
trail.

What we set out to prove was the seam, and the seam is what we are claiming.
Everything on either side of it is replaceable, which was rather the point.

A body is a tool boundary

The arm is not the point. An agent got a body without the runtime learning
anything about robotics, and the same seam decides where the next model goes.
Vision is the test, and it lands in two places.

A vision-language model that describes a scene is brain-side. It becomes a
pipeline stage that writes what the camera sees into the agent's context, so the
agent knows a person is present before deciding to greet one. Nothing about the
robot changes. Stages compose.

A vision-language-action model that emits motion is robot-side. It replaces
the scripted keyframes behind a verb. Today they are six lines a person tuned by
watching the arm, after the first version twisted the wrist too fast for the
servo to track.

"wave": [
(HOME, 0.3),
(_PAN_L, 0.5), # slow strokes, small per-tick delta the servo can track
(_PAN_R, 0.5),
(_PAN_L, 0.5),
(_PAN_R, 0.5),
(HOME, 0.4),
],

That table is what a policy would replace. The agent still calls wave, still
gets a mission ID, still gets ARRIVED, and the tool list does not change by
one character.

That is the return on drawing the line where we drew it: you can adopt a better
model on either side without renegotiating the contract between them. Neither is
built yet, but the camera keys are already on the plane, which is usually the
part that takes longest.

What we would keep

If we rebuilt this tomorrow, three decisions carry over unchanged.

Write the contract first. The protobuf file existed before either process
did, and both sides generate from it. It is the only artifact that survived
every other decision being reversed.

Let the robot describe itself. Hand-written tool lists drift from the robot
they describe, and the drift is silent until something moves. Discovery is why
plugging in real hardware quietly removed a verb the model used to have.

Keep the model on one side of a line you can point at. Not a guideline, a
process boundary with a typed contract across it. When someone asks what the
model is allowed to do, the answer should be a file, not a paragraph.

The model picks the verb. The robot owns the motion. Everything else was
implementation.