TL;DR

Researchers at Harvard’s Kempner Institute have built a robot control system that uses video generation instead of language models to plan physical actions. The robot creates short imagined video clips of possible future movements, then selects the best sequence — an approach that sidesteps the limitations of translating written instructions into physical behaviour.

Moving beyond words

Most current robot AI systems use what are known as vision-language-action (VLA) models. These take camera input, process it through a large language model, and output movement commands. The problem, as researcher Yilun Du puts it, is that language contains limited direct information about how the physical world actually behaves. Describing how to fold a towel in words is far less informative than watching someone do it.

Du’s system takes a different path. Trained on internet video data rather than text, the robot builds what researchers call a “world model” — an internal representation of physical reality. When faced with a task, it generates multiple short video clips showing possible future actions, evaluates which sequence is most likely to succeed, and then executes the chosen plan.

The technical foundation

The system runs on the Kempner AI Cluster supercomputer, reflecting the substantial computational requirements of real-time video generation for robotics. The research paper, titled “Large Video Planner Enables Generalizable Robot Control,” is available on arXiv and describes how the approach generalises across different tasks and environments better than language-based alternatives.

The key advantage is generalisability. A robot trained on broad video data about how objects move and interact can attempt tasks it has never specifically practised, because it understands the underlying physics rather than following scripted instructions.

Looking forward

The research is still at the laboratory stage, and the computational demands of generating video in real time remain a barrier to practical deployment. But the underlying idea — that robots should learn from watching the world rather than reading about it — represents a meaningful shift in how the field thinks about machine intelligence. For UK robotics firms and manufacturing companies watching developments in embodied AI, video-based planning could eventually produce robots that adapt to new environments without extensive reprogramming.