A Single Decoder Turns Small Models into Action: TurboVLA turns to a throwback architecture to attend to language and vision

The vision-language-action architecture that increasingly drives robots relies on a large language model to combine visual observations with text instructions before an action decoder produces motor commands. This approach can achieve strong semantic understanding, but it’s costly in computation and memory. Researchers built a streamlined alternative.

What’s new: Hengyi Xie, Chenfei Yao, and colleagues at Huazhong University of Science and Technology developed TurboVLA, a system in which separate encoders handle vision and text, cross-attention connects them, and a decoder turns the result into motor commands. TurboVLA achieved better performance than competing systems with a fraction of the memory and execution time.

Key insight: Using an LLM to combine visual observations and text instructions provides two benefits to robotics: (i) It can break down general instructions like “clean the room” into subtasks, and (ii) it can use the LLM’s knowledge to understand instructions and connect them to scenes. If we forego the first capability, a small encoder-only model can learn to understand instructions like “stack bowls,” and a model with cross-attention can learn to connect words to objects in a scene. Using much smaller models for these capabilities greatly reduces the time and memory requirements.

How it works: TurboVLA paired a DINOv3 vision encoder with the lightweight text encoder BERT, projected their outputs into the same embedding space, and passed them through six cross-attention layers (a pretrained Grounding DINO Swin-T). A separate transformer combined the result with the robot's current joint positions to produce motor commands. The authors trained the system to reproduce the robot arm movements in pre-existing datasets, including a version of LIBERO (single-arm tasks) and RoboTwin 2.0 (50 bimanual tasks). They fine-tuned it on demonstrations of a single-arm robot, AgileX Piper. All datasets included text commands, camera images, and joint positions.

• DINOv3 embedded each camera view separately (if multiple cameras were involved). The text encoder embedded each token individually instead of combining them into one vector, retaining object names, attributes, and words that described how objects related in space.
• The cross-attention layers fused the embeddings. This produced visual embeddings that reflected instructions and instruction embeddings that reflected scenes.
• The action transformer used a set of trainable vectors with the vision-language embeddings and the robot's current joint positions to produce a set of target joint positions. The positions specified seven values for each arm that covered an arm’s movement and gripper control. The authors trained the system by minimizing the difference between its predicted motor commands and ground-truth joint positions.

Results: TurboVLA matched or outperformed much larger models while using less memory. 

• On LIBERO, the system succeeded 97.6 percent of the time with 0.2 billion parameters, incurring 31.2 ms latency and using 0.9 GB inference VRAM on an Nvidia GeForce RTX 4090 GPU. The next-best model, CogVLA, succeeded 97.4 percent of the time with 8.3 billion parameters, 115.5 ms latency, and 16.1 GB VRAM. By comparison, π0.5 succeeded 96.9 percent of the time with 3.4 billion parameters, 238.8 ms latency, and 7.7 GB VRAM. 
• On RoboTwin 2.0, TurboVLA succeeded 88.1 percent of the time with 43.4 ms latency. The best competing model, LingBot-VLA, succeeded 87.6 percent of the time with 164.4 ms latency. π0.5 succeeded 79.8 percent of the time with 246.5 ms latency.
• Using the AgileX Piper arm on four real-world tasks, TurboVLA achieved 92.5 percent (“grab roller”), 80.0 percent (“move playing card”), 90.0 percent (“press stapler”), and 87.5 percent (“stack bowls”), outperforming π0.5, which achieved about 87.5 percent, 77.5 percent, 82.5 percent, and 85 percent, respectively.

Why it matters: Low-level robotic control can use small modules instead of a larger, general-purpose LLM. This paves the way for robots that use smaller models and run on less powerful hardware.

We're thinking: TurboVLA revives a pre-LLM pattern for vision-language-action systems. Such systems fused image and text embeddings circa 2022, but the LLM boom took that approach in a different direction, embedding different media types in the same space, but varying how each was encoded, learned, and decoded. Given advances in the meantime, the earlier approach offers a more efficient path to driving robots.