Google 发布 StreamVibe：让视频 AI 学会在用户需要时主动提供帮助

A research team led by Google has introduced StreamVibe, a video agent designed to decide for itself when to speak up. The tests show above all how hard that remains for today's AI models.

Researchers from Google, Stanford, Oxford, and several U.S. universities are working on an assistant that watches you cook or assemble furniture and speaks up only when needed. With StreamVibe, they've built a video agent designed to understand even vague requests such as "Please help me if I run into any difficulties."

According to the paper, existing video assistants are built for clear-cut commands like "Let me know when the water starts boiling," so they rarely pick up on unspoken intentions. They also have a hard time tracking what's already been done, which causes them to repeat advice or respond too late, and when users change their requests midway through, the assistants often stick to the original instructions.

Diagram showing how StreamVibe distinguishes between explicitly stated, unspoken, and changing user intentions during a task — StreamVibe distinguishes between explicitly stated, unspoken, and changing user intentions during a task. | Image: Google Deepmind

Three building blocks for a more attentive assistant

To measure the problem, the team compiled a benchmark of about 4,100 examples from more than 183 hours of everyday life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, while four human evaluators had to unanimously approve each example. The test covers three stages including clear instructions, vague goals, and requests that change over time. In one example, a user taking apart a toy excavator asks for progress updates and then switches to requesting a quality check after three minutes.

From there, the researchers trained StreamVibe-VL, a model apparently based on Alibaba's Qwen3.5-9B that decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make the model more reserved so it doesn't constantly interrupt. A separate memory system tracks which requests are pending and which steps are complete, though the model alone decides whether to speak and what to say.

Diagram of StreamVibe's architecture: a separate memory system tracks requests and task status while the model decides when and what to say. — A separate system tracks requests and task status, while the model itself decides when and what to say. | Image: Google Deepmind

All tested models lose track of the conversation in continuous operation

According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models, while the memory system is designed to boost proactive interaction by another 12.3 percent and cut down on unnecessary interruptions, among other things.

StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google's proprietary Gemini 3.8 Flash scores higher in several tests. According to the paper, the specialized training doesn't hurt general video comprehension.

Chart showing that as video length increases, models are significantly less likely to time responses correctly, with the harness mainly reducing unnecessary interruptions. — As video length increases, the models are significantly less likely to time their responses correctly; the harness primarily reduces unnecessary interruptions. | Image: Google Deepmind

Models often make the right call when they only have to decide at specific moments whether a prompt is appropriate. But when they process an entire video and have to build on their own earlier responses, performance across all tested systems plummets as errors compound, and a key content metric for the responses drops by up to 61.5 percent based on the system's own history. In videos longer than five minutes, the models rarely pick the right moment and find it especially hard to track a changed instruction over a long stretch.

In a GIMP tutorial on converting a photo to black-and-white documented in the paper and on the project page, the comparison model JoyAI says the finished image has already been saved even though the user hasn't opened it yet, while StreamVibe-VL gives the next step at the right moment, such as "Colors" and then "Desaturate." In a step-by-step guide for making rolled toast with jam, other models just describe what's happening on screen, such as the bread being flattened, while StreamVibe-VL advises cutting off the crust first.

Side-by-side comparison showing that while other models only describe on-screen action, StreamVibe provides specific next steps during a task. — While comparison models often only describe what is happening in the video, StreamVibe provides specific next steps. | Image: Google

The researchers acknowledge that real-world situations are likely more complex than their test setup, and future work will examine how user preferences and the environment influence each other over time.

Google has been working on AI systems that actively analyze videos for a while now. In September, Google introduced agent-based video analysis for Gemini, where the model itself decides which scenes to examine more closely. With Gemini 4 Argon, Google also recently showed how AI can track longer action sequences in robot videos and spot errors. StreamVibe's assistant is designed to understand what's happening at any moment and recognize when its help is actually needed.

On the ProactiveBench benchmark, AI models often guess or refuse to answer rather than ask specific questions when visual information is missing. A separate video comprehension benchmark revealed major weaknesses in spatial and temporal reasoning. And for autonomous AI agents, the conversation itself can hurt performance, as Microsoft researchers have shown.