Research path
Studying how reward validity, uncertainty, and credit assignment determine the reliability of post-training for visual generative models.
Core question
Visual post-training can optimize perceptual quality, semantic alignment, motion, and preference objectives that are not fully specified by pre-training losses. Its reliability depends on three questions: whether the reward measures the intended property, how uncertainty in that reward affects the update, and how credit is assigned across samples, regions, timesteps, and objectives.
Pre-training optimizes a model on a fixed dataset with likelihood, reconstruction, or related surrogate objectives. Visual post-training instead samples outputs or denoising trajectories, evaluates them with reward models or preference signals, and updates the generator toward higher-scoring behavior. This changes the optimization target, but it does not by itself create an agent that learns from an external environment.
The central difficulty is that reward models are imperfect measurements. A scalar score may combine several objectives, may be unreliable for an unfamiliar prompt, and may provide no direct evidence about which region or denoising step caused the result. My recent work studies these failure modes separately: reward granularity, uncertainty-aware weighting, adaptation of multiple objectives, temporal credit assignment, efficient generation, and integration into a complete training pipeline.
VrR-VG, ELP, MHEM, and IcoCap address different training settings, but they establish a common diagnostic principle. Benchmark accuracy can depend on dataset shortcuts, classifier confidence can differ from feature discriminability, hard-example losses can promote memorization, and labels can become inconsistent after augmentation. None of these signals should be interpreted without checking how it was produced.
VrR-VG uses a no-image relationship predictor to expose benchmark shortcuts, establishing the principle that performance must be tested against the information a model actually used.
ELP repeatedly resets a detached-feature reader, distinguishing representation quality from the confidence accumulated by a persistent classifier during training.
MHEM formalizes bounded emphasis for hard examples, preserving useful learning pressure without allowing extreme samples to dominate indefinitely.
IcoCap shows that modifying visual inputs requires corresponding changes to text supervision, an early instance of treating target construction as part of the learning system.
Visual generation is evaluated by several properties that are only partially captured by likelihood or reconstruction loss, including text alignment, perceptual quality, motion consistency, and human preference. Post-training introduces rewards for these properties, but every reward has a domain of validity defined by its training data, model assumptions, and calibration. Reward design and reward evaluation therefore remain part of the optimization problem.
ViPO and BPGO address two distinct limitations of sample-level GRPO rewards. ViPO uses perceptual features to distribute relative advantages over spatial or spatiotemporal regions. BPGO uses a semantic prior to reweight comparisons according to reward confidence. ViPO changes where the update is applied; BPGO changes the contribution of uncertain comparisons.
ViPO uses perceptual features to convert a sample-level GRPO advantage into spatial or spatiotemporal advantages, changing where the optimization update is applied.
BPGO uses a semantic prior to weight group-level and within-group reward comparisons, reducing the influence of comparisons judged to be uncertain.
TaRoS adjusts the influence of multiple reward components when they disagree, provide weak group separation, or saturate during training. OTCA decomposes a final reward across denoising timesteps and allocates different objectives across the trajectory. The first addresses how a composite reward changes during optimization; the second addresses how an outcome-level reward is assigned to intermediate decisions.
TaRoS changes reward-component influence when comparisons become sparse, objectives disagree, or a component saturates as training progresses.
OTCA decomposes final reward responsibility over denoising time and allocates multiple objectives where they are most informative.
RATS uses relative reward quality to determine when a few-step student should follow a multi-step teacher, connecting preference optimization with trajectory distillation. TeleBoost addresses a different level of the problem: it organizes supervised policy shaping, reward-driven reinforcement learning, and preference refinement into a staged video post-training pipeline, with diagnostics and infrastructure supporting training stability.
RATS makes teacher-student trajectory guidance conditional on relative reward quality, connecting preference alignment with efficient few-step generation.
TeleBoost places supervised shaping, reward-driven reinforcement learning, preference refinement, diagnostics, and systems constraints in one staged post-training pipeline.
Here, trustworthy does not mean that a reward model is correct for every prompt or that an attributed region or timestep is the true causal source of quality. The methods specify how available rewards are weighted or assigned within particular image and video training settings. Their conclusions remain conditional on reward-model coverage, prompt distribution, generator architecture, and whether evaluation uses evidence independent of the training rewards.
Open questions include how to calibrate reward uncertainty under distribution shift, detect reward exploitation even when aggregate metrics increase, and compare process-level credit across generator architectures. For deployment, another question is how verified outcomes should be used to update the reward model, generator, and evaluation protocol without allowing all three to reinforce the same bias.
This page connects collaborative research contributions; it does not replace the individual papers. Use each canonical paper record for evidence, source links, and citation downloads.