Background
Different visual encoder layers represent complementary information, yet many multimodal language models rely on features from a single layer. Multi-layer fusion offers a way to retain a broader range of visual signals.
Motivation
There are many possible places and mechanisms for combining visual features. This project compares fusion strategies and analyzes how their design choices affect multimodal language models.
What We Study
- Methods for combining features across visual encoder layers.
- The difference between external, internal, direct, and modular fusion designs.
- Practical considerations for selecting an effective fusion strategy.
Personal Notes
I will use this section to record the design choices, experiments, and lessons that were most important to this work.