IcoCap: Using Images to Enrich Video Semantics

What images contribute—and what they cannot

A well-captioned image often concentrates objects, attributes, and relations in one visual unit. A video caption compresses many frames into a sentence and may omit background details or intermediate actions. Images can therefore add dense semantic combinations that are underrepresented in video-caption data. They cannot substitute for temporal evolution. An image of a person beside a ball does not specify whether the person throws, catches, or merely observes it, so a method that treats static images as ordinary videos risks learning appearance without motion.

IcoCap respects this boundary by working with visual features extracted by a frozen CLIP model. Images act as controlled semantic compounds in a video feature sequence, not as claims about a true underlying motion. The captioner is pushed to handle richer and more ambiguous content while the base visual representation remains fixed. This positions the method as a learning strategy for video captioners, rather than a new motion encoder, optical-flow model, or claim that temporal information is unnecessary.

ICS changes the input distribution on purpose

ICS includes operations such as feature concatenation, inter-feature mixup, and frame swap. Each changes the feature sequence differently: adding content, interpolating representations, or replacing selected positions with an image or another video feature. The transformations expose the captioner to combinations beyond the original training pair. Their strength matters. A small compound can enrich context, while an aggressive mix or swap may produce a representation with no coherent semantic interpretation.

This resembles mixup only at the level of a particular operation. In classification, interpolating inputs and labels can be defined numerically. A natural-language caption is not a vector target that can be mixed into a valid sentence. A compounded video may overlap with several candidate descriptions or none. ICS therefore creates the need for the second half of the method. Describing IcoCap as “positive noise” is an intuition, not the formal method name; the research contribution is the Image-Video Compounding Strategy together with supervision selected for the result.

VGC keeps the label aligned with the changed content

Visual-Semantic Guided Captioning compares the compounded visual representation with candidate text representations and selects a description that is more consistent with the current input. Candidates can come from the original video, an auxiliary video, or the added image. VGC does not generate a new sentence or guarantee that every visual detail is described. It chooses among available ground truths, replacing the brittle rule that the original video caption must remain correct after augmentation.

The pairing creates a closed training loop. ICS without VGC can strengthen label noise because the input changes while the target stays fixed. VGC without new compounded samples has less reason to select an alternative target. Together, they let the captioner encounter richer visual combinations while retaining a plausible semantic objective. The approach depends on CLIP’s cross-modal similarity: if the frozen representation misses specialist objects, subtle actions, or domain language, the caption choice can be consistently wrong rather than randomly noisy.

Why more images or more frames did not always help

The paper evaluates IcoCap on MSVD, MSR-VTT, and VATEX and reports competitive or superior captioning results in the stated settings. Its ablations make the boundary clearer. Increasing external image data improves CIDEr up to a point and then yields diminishing benefit; the paper uses 10,000 images by default after observing saturation. In the reported frame-count comparison, 32 frames perform better than 64. Mixup ratios and frame-swap proportions also have non-monotonic optima.

These patterns are evidence against the slogan “more multimodal data is always better.” Additional images can drift from the target video domain, and more frames can add redundancy or unmatched semantics that a short caption cannot cover. Exact metrics depend on the captioner, frozen features, sampling, dataset, and candidate descriptions. The experiments support controlled compounding plus guided supervision under those protocols. They do not show that static images universally improve long-form video understanding or that temporal modeling can be discarded.

Where image–video compounding remains useful

IcoCap belongs in related work on image–video joint training, multimodal augmentation, and supervision selection for video captioning. It differs from simply appending image-caption data because it explicitly alters video features; from ordinary video augmentation because it reconsiders the text label; and from pseudo-caption generation because VGC selects an existing candidate. That three-way distinction remains useful even as larger video-language models absorb more heterogeneous data.

A current extension would report image–video domain overlap, the source distribution of selected captions, and human judgments of semantic faithfulness in addition to CIDEr. It could generate or edit captions when no candidate fits, while auditing hallucination. Motion-specific tests would show whether semantic enrichment helps or distracts from verbs and temporal order. The lasting idea is simple but easy to neglect: an augmentation is only valid relative to its supervision. Change the visual evidence, and the target sentence must be part of the design.

One revealing audit is to stratify samples by which source supplied the VGC target: original video, auxiliary video, or image. For each group, measure caption faithfulness, motion verbs, and the degree of visual alteration. If most gains come from selecting the unchanged caption, compounding may be too weak; if image captions dominate, temporal semantics may be diluted. Publishing this routing distribution would make the supervision mechanism inspectable and help explain why an image scale or swap ratio stops helping.

A counterfactual supervision test would keep the compounded feature fixed and train with the original caption, the VGC-selected caption, and a randomly chosen candidate. Differences among those runs isolate label selection from visual augmentation. Human reviewers can then judge whether the selected sentence describes the resulting semantics, rather than only whether an automatic metric rewards it. This is especially important when CLIP similarity prefers nouns and scene categories but underweights direction, count, or temporal order.

Evidence map

The locations below point to the primary paper so readers can verify the method and reported evidence directly.

Reuse

Original commentary in this note is licensed under CC BY 4.0. The paper title and bibliographic material retain their original rights.