Vision–language models / Test-time memory

ReAccessLook Once, Stay Focusedwith Efficient Test Time Memory

The evidence is already there.
Keep it within reach.

Tianhao Chen1Yilong Fan2Yuheng Wu1Xiaobin Hu3,*Dongman Lee1,*
1 KAIST2 Tianjin University3 National University of Singapore

* Co-corresponding authors

1visual prefill
64 × 64memory per layer
8VLM backbones
12understanding benchmarks

01 The core idea

Seeing the evidence
is only the beginning.

A VLM can encode the right visual details and still fail to use them. The missing link is sustained access to the evidence that matters.

Answering a visual question often depends on a few small clues, interpreted within the broader scene. Repeatedly inspecting the image adds cost, while long textual reasoning can draw attention away from the very evidence needed for the answer.

ReAccess connects evidence preservation with evidence utilization. It builds a compact, question-conditioned memory during a single visual prefill, then progressively retrieves complementary evidence to steer the frozen VLM’s attention and hidden representations.

01 / Preserve

Remember what matters.

Store question-relevant visual associations while keeping the original image context.

02 / Retrieve

Return to the evidence.

Use successive memory reads to recover complementary details as computation evolves.

03 / Guide

Make memory actionable.

Let retrieved evidence guide attention queries and refine the model’s representations.

The motivation

Encoded does not mean utilized.

From representation to reliable access

Our analysis reveals why richer visual information alone does not guarantee better answers.

Six-panel motivation analysis: global-local complementarity, evidence reuse versus image magnification, gains by evidence attention, explicit reasoning cost, evidence attention decay, and text-vision attention competition. Enlarge
Figure 1. Visual evidence accessibility in VLMs. Focused evidence complements global context, yet its influence can fade as reasoning text accumulates.Original PDF ↗
Observation 01

Small clues need a bigger picture.

Relevant regions occupy only 2.8% of visual tokens. Isolating them hurts accuracy; combining them with the full image helps.

Observation 02

More reasoning can mean less grounding.

As reasoning text grows, attention shifts away from the image. Evidence can remain encoded while losing influence on the answer.

Observation 03

Useful evidence can stay compact.

Probing low-dimensional hidden states recovers substantial visual information, motivating a small, queryable evidence memory.

02 The method

A small memory. An active role.

One prefill · Progressive evidence recall

ReAccess turns visual memory into a mechanism for evidence-grounded reasoning, while preserving the VLM’s original visual pathway.

ReAccess framework. Question-conditioned memory construction writes visual associations during one prefill. Two progressive reads steer attention queries and outputs at each frozen Transformer layer. Enlarge
Figure 2. Memory construction preserves question-relevant visual associations; memory utilization retrieves complementary evidence and steers the frozen VLM.Original PDF ↗
Memory construction

Write once, with the question in mind.

Spatially aware features and question-conditioned weights organize visual key–value associations into a compact 64 × 64 memory at each layer.

Memory utilization

Read progressively, guide directly.

Two successive reads retrieve complementary evidence. Their outputs steer both the native attention query and the hidden representation used for prediction.

Frozen backbone, lightweight learning. On Qwen2.5-VL-7B, ReAccess trains 58M memory-module parameters (about 0.7% of the full model) for one epoch. Inference uses a single visual prefill, without explicit intermediate reasoning traces.

03 The results

Better access. Better answers.

Full experimental details

Consistent gains across model families, followed by a closer look at challenging visual perception and reasoning.

Table 2

Generalization across VLM backbones

Eight backbones. Twelve benchmarks. Improvements across every model–benchmark pair.

Table 2 from the paper. Baseline and ReAccess scores for eight VLM backbones on GQA, MMBench, MMStar, TextVQA, DocVQA, InfoVQA, ChartQA, AI2D, OCRBench, MMMU, LogicVista, and POPE. Every backbone improves on all twelve benchmarks. Enlarge
General visual understanding. The original paper table, including all backbone scores and absolute improvements.Table PDF ↗

Up to 16.6% relative improvement — from 30.7 to 35.8 on InfoVQA with LLaVA-NeXT-7B. Gains extend across perception, documents, charts, and reasoning.

Table 3

Competitive perception with a single pass

A comparison with multi-turn, two-stage, and single-pass methods on Qwen2.5-VL-7B.

Table 3 from the paper. ReAccess is compared with multi-turn, two-stage, and single-pass methods on HR-Bench 4K and 8K, V-star attributes and spatial reasoning, and CV-Bench. ReAccess scores 73.3, 71.8, 84.4, 82.9, 83.8, and 78.8 in the six reported columns. Enlarge
Fine-grained perception and reasoning. All methods use Qwen2.5-VL-7B. The dagger (†) marks single-prefill variants.Table PDF ↗
+4.9pp

HR-Bench 8K
66.9% → 71.8%

+5.3pp

V* spatial accuracy
77.6% → 82.9%

78.8%

CV-Bench accuracy
77.7% → 78.8%

The efficiency

Stay grounded.
Keep moving.

ReAccess retrieves evidence directly from hidden states, keeping the inference procedure consistent across tasks.

≈6%

additional latency over the frozen model, on average

The paper reports approximately 10× average speedup over reasoning-intensive methods such as Thyme-RL and DeepEyes.

Inference latency in seconds per sample across HR-Bench 8K, HR-Bench 4K, V-star, and MMMU. ReAccess stays close to the frozen backbone and is faster than the compared methods. Enlarge
Inference latency across four evaluation settings.Original PDF ↗

04 Inside the memory

Watch the evidence come into focus.

Successive memory reads attend to complementary clues, connecting the details needed for a grounded answer.

Qualitative results for a bird and a bench. Original attention overlooks the bird's leg and misidentifies the bench's surroundings. ReAccess read one attends to the bill or bench; read two attends to the leg or surrounding path, producing the correct answers Yes and Path. Enlarge
Evidence preservation and utilization. Original attention and two successive ReAccess memory reads, shown on the same image–question pairs.Original PDF ↗
Example 01 / Fine-grained attributes

From the bill to the leg.

The original model focuses on the bird’s bill and overlooks its leg. Two complementary reads recover both details, changing the answer from No to Yes.

Example 02 / Spatial context

From the object to its surroundings.

ReAccess first localizes the bench, then retrieves evidence from the surrounding path, refining the answer from Park to Path.

Look once. Keep the evidence in reach.

Reference

Cite this work

ReAccess manuscript

Read the full paper
@article{chen2026reaccess,
  title={ReAccess: Look Once, Stay Focused with Efficient Test Time Memory},
  author={Chen, Tianhao and Fan, Yilong and Wu, Yuheng and Hu, Xiaobin and Lee, Dongman},
  journal={arXiv preprint arXiv:2611.20936},
  year={2026}
}

Figure

Click the image to zoom · Scroll to explore · Esc to close