Research path
Studying the requirements that connect video generation to world modeling: spatial control, long-range temporal consistency, persistent state, and action-conditioned prediction.
Core question
Video generation provides a concrete test of temporal modeling, but visual realism alone does not establish a world model. A model intended for prediction or planning must also maintain state across time, represent how actions affect state transitions, and express uncertainty when future observations are not determined.
Compared with image generation, video generation must model dependencies across time. Entity identity, spatial layout, motion, and event order must remain consistent across frames. Longer sequences expose errors that a short clip can hide, including appearance drift, geometric inconsistency, and motion that is not supported by the preceding state.
Large-scale training improves coverage, but data scale alone does not define a world model. World modeling introduces additional requirements: a state representation that persists across observations, predictions conditioned on possible actions, and evaluation against observed transitions. The papers on this path address several prerequisites for that goal—consistent supervision, controllable generation, long temporal context, persistent reconstruction, and post-training—without claiming that any single paper solves the complete problem.
IcoCap changes the content density of video-captioning samples by compounding image semantics with video content. Visual-Semantic Guided Captioning then adapts caption learning to the compounded input. The result is a specific supervision principle: when augmentation changes the semantic content of an input, the target must be adjusted accordingly.
IcoCap establishes an early supervision lesson for video-language learning: when visual content is compounded, the target caption must be reconsidered as part of the same operation.
VAST and FreeLong address two different limits of text-to-video generation. VAST introduces a storyboard with human poses and object layouts, providing explicit spatial conditions before synthesis. FreeLong extends a pretrained short-video diffusion model by changing temporal feature blending during denoising, balancing low-frequency global consistency with high-frequency local detail over longer sequences.
VAST inserts an explicit storyboard between text and video, using human pose and object layout as conditioning signals before synthesis.
FreeLong extends pretrained short-video diffusion models through frequency-aware temporal feature blending, addressing duration without retraining the base generator.
TeleWorld connects video generation with dynamic 4D reconstruction and persistent memory. Generated observations update the reconstructed state, and that state conditions later synthesis. Unlike isolated clip generation, this formulation allows an error in one step to affect the state used by subsequent steps, making state consistency an explicit system concern.
TeleWorld closes a loop between video generation, dynamic reconstruction, and persistent 4D memory so later synthesis can depend on an accumulated world state.
Video post-training often combines feedback on visual quality, text alignment, motion, and temporal consistency. TeleBoost separates supervised, reinforcement-learning, and preference-refinement stages; TaRoS adapts the influence of reward components during training; OTCA assigns outcome credit across denoising steps and objectives; and RATS conditions teacher guidance on relative reward quality. These methods improve visual generators under defined rewards, but do not by themselves provide action-conditioned world modeling.
TeleBoost organizes supervised policy shaping, reward-driven reinforcement learning, and preference refinement as distinct training stages, supported by diagnostics and training infrastructure.
TaRoS keeps composite video rewards informative when components disagree, group comparisons are sparse, or a metric has saturated during optimization.
OTCA maps final image or video rewards back onto denoising time and multiple objectives instead of treating every generation decision as equally responsible.
RATS combines few-step trajectory distillation with preference feedback, relaxing teacher guidance when the student matches or exceeds the teacher under the selected reward.
The Embodied Brains roadmap considers the system requirements beyond prediction. A predictive model needs interfaces to tools and controllers, execution in the physical environment, verification of the resulting state transition, and a procedure for reusing verified interaction data. This is a forward-looking systems connection rather than an empirical result of the video-generation papers.
The Embodied Brains roadmap places predictive world and action models inside a larger stack of shared contracts, physical harnesses, verification, and closed-loop learning.
Controllable or temporally consistent video generation is not sufficient evidence of a world model suitable for planning. Such a model must also predict action-conditioned state transitions, represent uncertainty, and remain accurate under changes in state and environment. TeleWorld adds persistent reconstruction, but a persistent representation is still not an embodied agent. The path identifies technical connections among the papers without assigning them a shared architecture or evaluation protocol.
Open questions include how to represent state and action in a common predictive model, how to calibrate uncertainty over long horizons, and how to distinguish model-generated observations from measurements of the external world. A closed-loop system must also verify whether an executed action produced the predicted transition before that trajectory is reused for training.
This page connects collaborative research contributions; it does not replace the individual papers. Use each canonical paper record for evidence, source links, and citation downloads.