Driving · 2026-10-01
Code to Control: Synthesizing Parameterized Reactive Controllers
Zergham Ahmed, Joshua B. Tenenbaum, Chris Bates, Samuel J. Gershman
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Code to Control synthesizes Python reactive controllers by separating program structure from numerical parameters. An LLM proposes the controller structure; derivative-free search fits parameters from environment returns. After learning, controllers need no LLM inference or planning at decision time, enabling real-time action selection. The authors report better results than planning-based program synthesis on Atari, competitive results with deep RL with fewer interactions, robust transfer in Flappy Bird, and strong MuJoCo returns versus planning baselines.
Why it matters
The authors target the latency bottleneck of LLM policies (inference each step) and world-model planning (planning at each decision). By making policies executable Python programs and fitting only numerical parameters after structure synthesis, the approach aims for planning-free decision-time computation. The paper also probes robustness to dynamics changes (Flappy Bird parameter refitting without program changes) and scaling to continuous control via feature-map structure plus derivative-free parameter fitting.
Method
- LLM synthesizes an interpretable Python controller structure; the controller executes directly as the policy (no planning at decision time).
- Discrete domains: LLM revises the executable reactive controller using its own gameplay trajectories and returns; keep the best (champion) program over the run.
- Continuous domains: LLM proposes a feature map ϕ(o,t); derivative-free search (ARS) fits a matrix W mapping normalized features to clipped actuator commands.
Limitation
The authors report that reversing the sign of gravity in Flappy Bird is an exception where parameter refitting does not recover successful play, since that would require a structural program change.
Abstract (from arXiv)
Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.
Related papers
- Player-1: Learning to Play from Synthetic Experience
- DashVMC: Real-Time Discrete World Model Control in Geometry Dash
- EmbodiedEvo: Self-improving Coding Agents for Physical Intelligence
- Self-Improving Agents with World Model Predictive Paths
- FARL: A Fully Automated Pipeline of RL Interface and Solver Design
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
Code to Control: Synthesizing Parameterized Reactive Controllers
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.38733
Explicit source versions (1)
- v1 · 2026-09-30
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.