Driving · 2026-10-20
FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference
Liang Su, David Chen, Jason Li, Kevin Guo, Teresa Lv
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
FLASHRT-PAI is a low-latency physical AI inference engine that focuses on model–hardware pipeline co-design to reduce system overhead in small-batch, real-time loops. It uses deep kernel fusion with launch overlap, an execution contract for cross-decision state reuse/refresh, and execution-state capsules for snapshot/restore/fork. Across six policies on RTX 5090 and four on Jetson AGX Thor, it reports 1.3–3.7× faster than official compiled paths at matched 16-bit precision and up to 3.38–12.66× over official eager.
Why it matters
The authors argue that real-time single-batch inference bottlenecks in VLA/VA/WAM are dominated by system overhead (many short launches, intermediate data movement, redundant recomputation), which general-purpose graph compilers can’t eliminate. FLASHRT-PAI targets cross-call pipeline structure and state semantics, aiming to reduce latency while preserving action fidelity through bit-exact reused/restored results and checked drop-in kernel replacements.
Method
- Deep kernel fusion with launch overlap to remove inter-operator boundaries/launch gaps (hand-written producer–consumer paths; CUDA Programmatic Dependent Launch at safe boundaries).
- Execution contract that formalizes the stage DAG, update rates, and named buffers to enable cross-decision state reuse and selective refresh.
- Execution-state capsules enabling fast snapshot, restore, and fork of persistent pipeline state; supports parity/latency-gated drop-in PyTorch kernel replacement.
Abstract (from arXiv)
Vision-language-action (VLA), video-action (VA), and world action models (WAMs) form the backbone of modern physical AI systems, yet their real-time single-batch inference remains a defining bottleneck. Latency is dominated not by individual kernel computation, but by system-level overhead: thousands of short kernel launches, intermediate data movement across operator boundaries, and redundant recomputation across consecutive decisions. General-purpose graph compilers cannot address these costs, as they optimize individual calls in isolation and lack visibility into cross-call state semantics and global pipeline structure. We present FLASHRT-PAI, a low-latency inference engine built around model-hardware pipeline co-design rather than isolated kernel optimization. FLASHRT-PAI combines three core mechanisms: deep kernel fusion with launch overlap eliminates inter-operator boundaries and launch gaps; an execution contract formalizes the pipeline's stage DAG, update rates, and named buffers to enable cross-decision state reuse and selective refresh; and execution-state capsules enable fast snapshot, restore, and fork of persistent pipeline state. The system also supports drop-in acceleration of unmodified PyTorch models via parity- and latency-gated kernel replacement. Evaluation covers six policies on RTX 5090 and four on Jetson AGX Thor. At matched 16-bit precision, FLASHRT-PAI is 1.3-3.7x faster than official compiled paths. Combining pipeline co-design with reduced precision delivers 3.38-12.66x speedup over official eager implementations. In LIBERO-10, four-bit pi0.5 achieves 94.04% success on Thor and 93.32% on RTX 5090, compared with 92.4% for the measured official FP16 reference. FLASHRT-PAI is open source, distributed via Hugging Face Kernel Hub, and its opt-in pi0.5 kernels have been adopted into the LeRobot framework.
Related papers
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC
FlashRT-PAI: Fast Kernels Are Not Fast Pipelines for Physical AI Inference
Preprint posted: 2026-10-17 · Original record ↗
- arxiv
- 2610.24349
Explicit source versions (1)
- v1 · 2026-10-17
- Preprint available
Source date: 2026-10-17 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-language-action (VLA), video-action (VA), and world action models (WAMs) form the backbone of modern physical AI systems, yet their real-time single-batch inference remains a defining bottleneck. Latency is dominated not by individual kernel computation, but by system-level overhead: thousands of short kernel launches, intermediate data movement across operator boundaries, and redundant recomputation across consecutive decisions. General-purpose graph compilers cannot address these costs, as they optimize individual calls in isolation and lack visibility into cross-call state semantics and global pipeline structure. We present FLASHRT-PAI, a low-latency inference engine built around model-hardware pipeline co-design rather than isolated kernel optimization. FLASHRT-PAI combines three core mechanisms: deep kernel fusion with launch overlap eliminates inter-operator boundaries and launch gaps; an execution contract formalizes the pipeline's stage DAG, update rates, and named buffers to enable cross-decision state reuse and selective refresh; and execution-state capsules enable fast snapshot, restore, and fork of persistent pipeline state. The system also supports drop-in acceleration of unmodified PyTorch models via parity- and latency-gated kernel replacement. Evaluation covers six policies on RTX 5090 and four on Jetson AGX Thor. At matched 16-bit precision, FLASHRT-PAI is 1.3-3.7x faster than official compiled paths. Combining pipeline co-design with reduced precision delivers 3.38-12.66x speedup over official eager implementations. In LIBERO-10, four-bit pi0.5 achieves 94.04% success on Thor and 93.32% on RTX 5090, compared with 92.4% for the measured official FP16 reference. FLASHRT-PAI is open source, distributed via Hugging Face Kernel Hub, and its opt-in pi0.5 kernels have been adopted into the LeRobot framework.