Visual Prompt Engineering: A Simple Way to Make Video Models Smarter

In the age of foundation models, prompt engineering has become a key technique for improving language model performance. As video models become foundational models for visual reasoning, researchers ask: can we optimize input images similarly to text prompts to enhance video model reasoning? The paper 'Visual prompt engineering for video models' answers affirmatively. The authors propose Visual Prompt Engineering (VIPE), which automatically uses image editing models to transform task images into styles more conducive to model understanding (e.g., converting abstract sketches into photorealistic scenes). Experiments across multiple visual reasoning tasks show that VIPE not only significantly boosts performance but in some tasks outperforms classic text prompt engineering and test-time scaling strategies. More importantly, VIPE has extremely low computational overhead, requiring only a single image editing operation to gain benefits. The project page provides numerous example videos demonstrating the method's effectiveness.