ViDiC: Video Difference Captioning

Abstract

ViDiC Teaser: Comparison of compositional, spatial, and temporal changes

Figure 1: Review of Video Difference Captioning framework.

Understanding visual differences between dynamic scenes requires the comparative perception of compositional, spatial, and temporal changes—a capability that remains underexplored in existing vision-language systems. While prior work on Image Difference Captioning (IDC) has enabled models to describe semantic changes between static images, these approaches fail to capture motion continuity, event evolution, or editing consistency over time.

We introduce the ViDiC (Video Difference Captioning) task and its corresponding ViDiC-1K dataset, designed to evaluate the ability of Multimodal Large Language Models (MLLMs) to provide fine-grained descriptions of similarities and differences between video pairs. ViDiC comprises 1,000 curated video pairs annotated with over 4,000 comparative checklist items, covering seven categories: Subject, Style, Background, Cinematography, Motion, Location (Position), and Playback Techniques.

To ensure reliable evaluation, we propose a dual-checklist framework that measures the accuracy of similarity and difference separately, using a scalable LLM-as-a-Judge protocol. Experiments on twelve state-of-the-art multimodal models reveal a significant performance gap in their comparative description and difference perception abilities, with all models underperforming on playback- and camera-related dimensions.

Leaderboard

We evaluate video comparison using Accuracy over a set of Dual-Checklist questions.
Avg: Average Score Diff: Difference Accuracy Sim: Similarity Accuracy

Fine-grained Categories:
Subj (Subject), Motion, Pos (Position), Backgr (Background), Cam (Camera Work), Style, Tech (Playback Technique).

💡 indicates specific "thinking" or reasoning modes activated in models.

#	Model	Param	Overall Metrics (%)			Category Performance (%)
#	Model	Param	Avg	Diff.	Sim.	Subj	Motion	Pos.	Backgr.	Cam.	Style	Tech.
-	Gemini-2.5-Pro💡	-	66.72	63.73	75.33	67.71	62.78	68.24	70.65	59.97	75.79	74.32
-	GPT-5💡	-	62.94	57.32	79.17	61.52	57.78	65.31	69.15	57.39	77.60	54.66
-	Gemini-2.5-Flash💡	-	58.87	52.11	78.37	59.63	51.29	57.23	63.98	52.82	81.58	55.41
-	Gemini-2.0-Flash💡	-	53.71	50.26	63.66	58.90	48.71	57.86	57.11	47.30	55.79	18.92
-	GPT-4o💡	-	49.95	39.14	81.12	46.79	43.53	51.89	53.73	49.18	77.89	27.03
-	Qwen3-VL-32B	32B	61.38	58.54	71.50	64.60	51.77	62.00	68.62	52.66	74.86	47.83
-	InternVL-3.5💡	38B	52.44	46.25	70.30	52.66	43.04	53.77	59.80	47.80	72.63	20.27
-	Qwen2.5-VL-Inst	72B	49.71	42.56	70.30	48.07	44.82	48.11	55.92	46.42	68.95	22.97
-	Mimo-VL-SFT	7B	52.59	46.51	70.17	54.39	46.55	51.25	57.31	48.37	67.71	25.33
-	GLM-4.1V💡	9B	40.95	33.99	61.08	42.60	34.35	38.13	47.26	33.83	64.58	14.67
-	Llama-3.2	11B	19.43	5.23	61.01	14.48	20.31	17.84	13.44	29.56	40.00	11.70

Video Comparison Pairs

Each example in ViDiC consists of a video pair and a fine-grained checklist.

Video A

Video B

Similarity Question: Are the man's clothes different between the two videos?

Correct Answer: No

Difference Question: Does the Source video feature diffuse, cool-toned light, while the Target video features bright, warm-toned backlighting from the upper left?

Correct Answer: Yes

Difference Question: Is the sky in the Source video completely grey and cloudy, in contrast to the Target video where the sky is blue with a visible sun?

Correct Answer: Yes

Video A

Video B

Similarity Question: Do the clothing attributes of the shooter and goalkeeper vary between the Source and Target videos?

Correct Answer: No

Difference Question: Regarding the camera's position, is it true that the Source video is filmed more to the side, while the Target video is filmed from more centrally?

Correct Answer: Yes

Difference Question: Is the building in the background more clearly visible in the Target video compared to the Source video?

Correct Answer: Yes

Video A

Video B

Similarity Question: Are the camera movements between the Source and Target videos different?

Correct Answer: No

Difference Question: Regarding the lighting, is the Source video illuminated by bright, white daylight, in contrast to the Target video which is illuminated by the warm, colored light of a sunrise or sunset?

Correct Answer: Yes

Difference Question: In terms of atmosphere, does the Source video show a blue sky with white clouds, while the Target video shows a sky with warmer colors?

Correct Answer: Yes

Benchmark Statistics

We propose a Dual-Checklist Evaluation Framework to ensure reliable measuring of comparative captioning. Traditional metrics measure textual similarity rather than factual correctness. ViDiC overcomes this by quantifying factual accuracy using a human-annotated checklist composed of binary questions derived from predefined dimensions (Subject, Style, Background, etc.).

Similarity Questions: Framed inversely to penalize hallucinations. A response is correct if it confirms similarity or omits the attribute.
Difference Questions: Framed as verifiable propositions about specific differences. The model must correctly affirm these true statements.

We leverage an LLM-as-a-Judge protocol (using GPT-5-Mini or equivalent high-capability models) to compare the generated captions against the ground-truth checklist without accessing the video pixels directly. This ensures scalable and interpretable benchmarking.


    @misc{wu2025vidicvideodifferencecaptioning,
      title={ViDiC: Video Difference Captioning}, 
      author={Jiangtao Wu and Shihao Li and Zhaozhou Bian and Yuanxing Zhang and Jialu Chen and Runzhe Wen and An Ping and Yiwen He and Jiakai Wang and Jiaheng Liu},
      year={2025},
      eprint={2512.03405},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2512.03405}, 
    }

ViDiC

A benchmark for Video Difference Captioning

Abstract

Leaderboard

Dataset Examples

Video Comparison Pairs

Benchmark Statistics

Evaluation Methodology

Citation