We introduce Video-DeepResearch (Video-DR), extending multimodal agents from static images to continuous video streams — a setting that demands dense spatiotemporal grounding coupled with open-web exploration. Preliminary evaluations reveal two critical bottlenecks in current models: (1) modality bias, where agents bypass visual tools in favor of textual search, and (2) parametric knowledge leakage, where models rely on internal memory rather than genuine tool-augmented execution.
To address these challenges, we propose Video-DeepResearch, featuring a decoupled perception-exploration pipeline with stage-wise tool unlocking that compels exhaustive cross-frame visual grounding prior to web retrieval. Our framework adopts a two-stage training recipe — supervised fine-tuning followed by Group Relative Policy Optimization (GRPO) — enabling autonomous exploration that breaks the imitation-learning ceiling.
Empirical results demonstrate that our Video-DeepResearch-35B-A3B establishes a new state-of-the-art of 68.0% accuracy on Video-DR, surpassing proprietary Claude-4.5-Sonnet (63.0%) by 5.0 points and significantly outperforming GPT-5 (57.0%) and Gemini 2.5 Pro (62.0%). The 30B-A3B variant reaches 62.0%, matching Gemini 2.5 Pro at a fraction of the parameter count and demonstrating the effectiveness of our training paradigm even at compact scale.
Before designing Video-DR, we stress-tested three representative agents on the existing Video-DR benchmark. Two systematic failure modes emerged.
Even the strongest open-source agent calls a visual tool only 0.10 times per task, while text search fires 1.27 times — a systematic aversion to active visual exploration.
GPT-5 reaches 57% accuracy with virtually zero tool calls. Current evaluations are largely solvable through memorized world knowledge alone, not genuine tool-augmented reasoning.
| Model | Acc. (%) | Vision Tool | Text Tools |
|---|---|---|---|
| Qwen3.5-35B-A3B | 41 | 0.04 | 0.58 |
| Qwen3.5-397B-A17B | 58 | 0.10 | 1.27 |
| GPT-5 | 57 | 0.00 | 0.12 |
Only select_crop_search is unlocked. The agent must scan frames,
crop regions of interest, and reverse-image search — no textual escape hatch.
This forces cross-frame perception before any web retrieval.
Once visual evidence is gathered, search and visit
unlock. The agent composes text queries grounded in visual findings and
synthesizes web knowledge into a final answer.
Supervised fine-tuning bootstraps trajectory quality; Group Relative Policy Optimization then enables autonomous exploration that breaks the imitation ceiling — the RL agent surpasses its own SFT teacher.
| Tool | Stage | Behavior |
|---|---|---|
select_crop_search |
1 · 2 | Batch (frame, bbox) selections → cropped reverse-image / visual web search. Returns top web hits and recognized entities (people, products, logos, landmarks). |
search |
2 | Parallel text queries over Google / Serper / Zhipu with automatic fallback. Returns ranked {title, url, snippet}. |
visit |
2 | Fetch URL(s), summarize via LLM into goal-conditioned JSON ({rational, evidence, summary}). Raw HTML never exposed. |
Our decoupled pipeline produces 30K video-grounded QA pairs and 7K curated trajectories. Each QA is guaranteed to require both visual localization and external knowledge lookup — closing the loophole that lets models cheat via parametric memory.
Phase I filters raw videos with rule- and agent-based stages. Phase II selects keyframes, crops entities for visual search, and synthesizes VQA pairs through single- and multi-entity patterns with parametric-leakage filtering. Phase III constructs trajectories via the decoupled perception-exploration pipeline; only correct trajectories survive reject sampling.
We report results across two benchmarks: the existing Video-DR (single-metric accuracy) and our VideoDR-Bench, a 200-instance multi-hop VQA benchmark broken down by six video categories. Video-DeepResearch-35B-A3B sets a new SOTA on both, with an overall 64.0% Avg. — surpassing Claude-4.5-Sonnet by 5.0 pp, Gemini 2.5 Pro by 6.5 pp, and GPT-5 by 11.5 pp. The compact 30B-A3B variant reaches 59.3% Avg., ahead of every open-source competitor and matching Claude while using a fraction of the parameters.
| Model | Video-DR | VideoDR-Bench (% Acc.) | Avg. | ||||||
|---|---|---|---|---|---|---|---|---|---|
| KNL | ENT | DLY | G&S | NWS | OTH | Overall | |||
| Closed-source | |||||||||
| Gemini 2.5 Pro | 62.0 | 54.2 | 52.3 | 51.4 | 51.7 | 54.2 | 57.1 | 53.0 | 57.5 |
| GPT-5 | 57.0 | 50.8 | 45.5 | 48.6 | 48.3 | 45.8 | 42.9 | 48.0 | 52.5 |
| Claude-4.5-Sonnet | 63.0 | 55.9 | 54.5 | 54.1 | 58.6 | 54.2 | 42.9 | 55.0 | 59.0 |
| Open-source | |||||||||
| Qwen3.5-397B-A13B | 58.0 | 49.2 | 61.4 | 40.5 | 55.2 | 29.2 | 14.3 | 47.5 | 52.8 |
| Kimi K2.5 | 61.0 | 54.2 | 52.3 | 51.4 | 51.7 | 54.2 | 57.1 | 53.0 | 57.0 |
| Qwen3-VL-30B-A3B-Instruct | 38.0 | 44.1 | 43.2 | 35.1 | 51.7 | 41.7 | 42.9 | 43.0 | 40.5 |
| Video-DeepResearch-30B-A3B (Ours) | 62.0 | 62.7 | 61.4 | 40.5 | 58.6 | 58.3 | 42.9 | 56.5 | 59.3 |
| Qwen3.5-35B-A3B | 42.0 | 45.8 | 50.0 | 40.5 | 44.8 | 33.3 | 28.6 | 43.5 | 42.8 |
| Video-DeepResearch-35B-A3B (Ours) | 68.0 | 66.1 | 65.9 | 56.8 | 62.1 | 41.7 | 42.9 | 60.0 | 64.0 |
Bold = best · Underlined = second-best. Our two rows are highlighted.
On the two-benchmark Avg. our 35B-A3B surpasses Claude-4.5-Sonnet by 5.0 pp (64.0 vs 59.0), Gemini 2.5 Pro by 6.5 pp (64.0 vs 57.5), and GPT-5 by 11.5 pp (64.0 vs 52.5). Under the same tool budget, training recipe matters more than raw parameter count.
Same base weights: Qwen3-VL-30B jumps 40.5 → 59.3 Avg. (+18.8 pp) and Qwen3.5-35B jumps 42.8 → 64.0 Avg. (+21.2 pp) under our SFT + GRPO pipeline — larger than the gap between neighbouring model sizes, and larger than the closed-vs-open gap.
On VideoDR-Bench our 35B leads on 4 of 6 categories (KNL 66.1, ENT 65.9, DLY 56.8, G&S 62.1) and ties or lags only on the two smallest slices (NWS, OTH). No single easy bucket props up the headline number.
Because search and visit are locked at
Phase 1, the agent cannot dodge visual grounding. This turns the
0.10 vision-tool failure mode from our motivation into a hard
structural constraint rather than a soft reward signal — and the
broad per-category lift in insight 3 is its downstream effect.
While Video-DeepResearch pioneers the first comprehensive pipeline integrating data construction and model training for the complex Video-DR task, this rigorous approach introduces certain trade-offs.
Primarily, achieving our current level of performance incurs considerable computational overhead. To ensure high-quality data synthesis and robust model training, the framework demands substantial GPU resources, largely due to the concurrent requirements of large-scale model deployment and dynamic web search operations.
In future work, we aim to mitigate these constraints by exploring computationally efficient pipelines, lightweight architectures, and automated LLM-based evaluation metrics to reduce human dependency.
@misc{fang2026videodeepresearch,
title={Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent},
author={Zhen Fang and Yu Zeng and Wenxuan Huang and Yiming Zhao and Shiting Huang and Tianfei Ren and Qi Lu and Qingnan Ren and Qisheng Su and Lionel Z. Wang and Qingyu Yin and Shuang Chen and Zehui Chen and Lin Chen and Zhenfei Yin and Yao Hu and Shaohui Lin and Wanli Ouyang and Shaosheng Cao and Feng Zhao},
year={2026},
eprint={2608.03979},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2608.03979},
}