Visual Layer Selection

Background

Multimodal language models commonly use visual features from the final layers of a vision encoder. This convention assumes that deeper representations are always the most useful input for language models.

Motivation

Visual encoder layers capture different kinds of information. Deeper features can be more semantic, while shallower features may retain details that later layers discard. This project studies how the choice of visual layer affects multimodal model performance.

What We Study

• The information available at different visual encoder depths.
• How layer selection changes downstream multimodal behavior.
• Practical guidance for choosing visual features instead of defaulting to the final layer.

Personal Notes