StreamVibe: Proactive Video Agents for Evolving User Intent

4,100evaluation samples

20,546annotated help points

183.4hours of video

85Ktraining samples

10intent subtypes

01Motivation

Why proactive assistance is hard

Today’s streaming assistants wait for a request that names the event to watch for. Real users state goals and change them as they go.

1

The intent is often implicit

“Help me when I get stuck” names no trigger; the assistant must decide when to help.

2

The task state keeps changing

Without tracking progress, assistants repeat, give outdated advice, or miss steps.

3

Requests are revised mid-task

Users add, replace or interrupt requests; assistants keep following the first one.

**Figure 1.** Overview of StreamVibe. Click to enlarge.

02Task

Three levels of user intent

Graded by how explicit the intent is and whether it changes mid-task.

Level 1 · Explicit

The user names a condition or event; the assistant responds when it occurs.

Let me know when the water starts boiling.

silent

The water is boiling now.

Single EventTransitionRecurring

Level 2 · Implicit

The user gives a goal; the assistant infers when and how to help from the video and task progress.

Help me make these mini jam rolls.

silent

Trim off the crusts before spreading the jam.

Now flatten the bread with a rolling pin.

GuidanceLearningMonitoringTool use

Level 3 · Dynamic

The request is revised, extended or interrupted while the task is under way.

Recap my progress after each stage.

Stage 1 done: centre crease made.

Actually, just tell me when I make a mistake.

Fold the corners in first, then the tab.

RevisionCompositionInterruption

04StreamVibe-Bench

Benchmark construction and evaluation

Physical and digital task videos. Gemini-3.8-Flash proposes help points with time windows and reference replies; a sample is kept only if four reviewers all approve it.

600 explicit · 2,000 implicit · 1,500 dynamic samples. Scored on timing (P / R / F1 / FPR) and quality (answer score, content F1).

Comparison with existing proactive benchmarks

Benchmark	User intent type			Dur. (min)	Evid.	Environment		Eval setting		Traj.
Benchmark	Explicit	Implicit	Dynamic	Dur. (min)	Evid.	Physical	Digital	Pointwise	Trajectory	Traj.
ProactiveVideoQA	●	○	○	2.1	3.5K	●	○	○	●	1.4K
EgoProactive	◐	●	○	3.5	5.4K	●	○	●	○	700
EgoPro-Bench	●	●	○	3.0	2.4K	●	○	○	●	2.4K
IPIBench	●	○	●	1.0	3.7K	●	○	●	◐	1.8K
EgoServe	◐	●	◐	6.2	3.4K	●	○	○	●	293
GuideMe	●	●	○	5.5	47.8K	●	○	●	◐	2.5K
GUIDE	●	●	◐	33.8	0.7K	○	●	●	○	120
StreamVibe-Bench (ours)	●	●	●	8.3	20.5K	●	●	●	●	4.1K

● full ◐ partial ○ no support. Dur.: average sample duration; Evid./Traj.: numbers of evidence points and evaluation trajectories.

**Figure 2.** Construction and statistics of StreamVibe-Bench.

Task. Each second the agent sees only the past and chooses to stay silent or respond.

Evaluation. Pointwise scores single decisions; trajectory runs the whole video with the model’s own history.

05Method

StreamVibe-VL and StreamVibe-Harness

A streaming model decides when and what to say; a harness keeps track of the requests.

One second of inference. The Harness updates the state; the model stays silent until a help point.

**Figure 3.** Harness at inference, adaptive SFT, localized trajectory RL.

HStreamVibe-Harness

Tracks every request, task progress and past replies; retrieves the relevant history.

1Adaptive streaming SFT

Balances silence and response so the model neither stays mute nor over-talks.

70K + 768K samples

2Localized trajectory RL

GRPO on short rollouts around help points, rewarding timing, content and consistency.

15K tasks

06Results

Results

Against 11 proprietary, offline and streaming models.

Content F1 on StreamVibe-Bench

vs. the best open-source baseline

StreamVibe-VL (RL)MMDuet2 (best baseline)

Other proactive benchmarks

EgoProactive and IPIBench

General video understanding

accuracy before → after training

StreamVibe-VL (RL)Qwen3.5-9B (base)

All models

StreamVibe-VL (ours)baselines

01Best open-source model at every intent level

02Implicit and dynamic intents remain hard for all models

03Training and the Harness add up

07Citation

bibtex

@article{liu2026streamvibe,
  title   = {StreamVibe: Proactive Video Agents for Evolving User Intent},
  author  = {Liu, Chengzhi and Lai, Yuxiang and Yang, Yuzhe and Liu, Yanqing and
             Hou, Jiheng and Cao, Yue and Wang, Haonan and Fan, Ruixi and
             Wang, Lan and Liu, Sheng and Peng, Songyou and Song, Yale and
             Lin, Kevin Qinghong and Wang, Xin Eric and Lai, Jui-Hsin and Tang, Youbao},
  year    = {2026}
}

StreamVibe Proactive Video Agents for Evolving User Intent