Context Engineering for Images: MIT researchers build VISTA, an agent harness that keeps images in memory for faster retrieval

Managing an agent’s context by storing all the input text it receives and giving it tools to search the text boosts its performance. For vision-language agents, the same is true of storing images.

What’s new: Qiushi Han, Keya Hu, Linlu Qiu, and colleagues at MIT released VISTA, an agent that completed the ARC-AGI-3 benchmark of interactive visual-reasoning puzzles using 57.4 percent fewer actions than humans who had not encountered them before. The VISTA harness is freely available under the MIT license.

Key insight: Given an image frame from a videogame or other interactive environment, agentic systems typically embed it and keep the embedding until the context is full, at which point they delete it. This raises two issues: Embeddings are lossy and may not retain valuable information, and once an embedding has been deleted, the model no longer has access to information it contained. Storing all frames in a persistent memory retains all the information they carry, and enabling an agent to retrieve and zoom in on them makes it possible to embed details that otherwise may not have been captured.

How it works: VISTA works with Anthropic Claude Opus 5.0 or OpenAI GPT-5.6 Sol. Its harness works similarly to other popular harnesses, except it retains all input frames and includes a tool that retrieves frames. Given a new frame and list of available keyboard actions (such as left, right, up, down, spacebar, or click), the agent tells the model to decide whether to (i) read or update notes, (ii) retrieve previous frames, (iii) retrieve previous actions and their results, or (iv) execute an action. It repeats this choice until the model takes action. After the environment updates (during which it may produce a series of frames), it adds to the model’s input the last frame and a new list of available actions and executes the loop again.

• The agent maintains two notes: information it deduces about the environment that can be reused across levels or sessions (for example, the difference between a brown block and a black background), and information deduced about the level or problem at hand (for example, that the character currently has a power-up or conditions under which the model failed previously). The agent decides when to read these notes.
• When the agent decides to retrieve earlier frames, the tool enables it to retrieve any number of frames and zoom in to any part of a frame.
• During an attempt to complete a puzzle or game level, VISTA stores in memory not only images but also a list of actions and their results. It can decide to retrieve any set of previous actions or summaries of all previous attempts.

Results: The authors tested VISTA on ARC-AGI-3 (25 games), GameWorld (170 tasks in 34 games), and AI GameStore (10 games). It enabled Claude Opus 5.0 and GPT-5.6 Sol to outperform earlier agents as well as humans who encountered the same challenges for the first time.

• On ARC-AGI-3, VISTA with Claude Opus 5.0 achieved a perfect score of 100 Relative Human Action Efficiency (RHAE), meaning it completed all puzzles in fewer actions than first-time human players. (Earlier projects that achieved perfect RHAE scores, such as Tycho, generated programs that simulated the environments.) VISTA with GPT-5.6 Sol achieved 99 RHAE. Denied the ability to store frames, VISTA with GPT-5.6 Sol achieved 70 RHAE, which shows that the frames were useful.
• On GameWorld, VISTA with GPT-5.6 Sol completed 63.3 percent of tasks on average. First-time human players completed 55.3 percent, and the native harness with GPT-5.6 Sol completed 40 percent.
• On AI GameStore, VISTA with GPT-5.6 Sol achieved a much higher score than first-time players and the native harness, 40 percent better than humans on average.

Why it matters: Games and other interactive environments generate potentially endless streams of frames, each of which contain information an agent may not have deduced earlier. Only so many frames fit in its context, and not all frames are relevant at any given moment. By storing all frames and giving the agent tools to retrieve them, agents can find the relevant frames and re-examine them within the current context.

We're thinking: Apparently context engineering helps whether the medium is text or images. Multimodal systems benefit from multimodal context management.