All projects

Project report · 2026

Glimpse

A controlled look at multimodal fusion

In one line

Glimpse compares eight lightweight CLIP fusion heads, then trains a larger ViT–BERT co-attention model. The work separates architecture ideas from implementation mistakes by recording both results and the bugs found along the way.

Question

When an image encoder and a language encoder must answer a visual question together, which fusion design earns its complexity?

What I built

  • Benchmarked eight fusion heads on frozen CLIP features so the comparison focused on how visual and text representations were combined.
  • Built GeometryFusion, a 657K-parameter head that uses similarity structure instead of simply concatenating embeddings.
  • Trained an end-to-end ViT-B/16 and BERT model with co-attention, while correcting soft-label, masking, pooling, and tokenization issues.

Main result

best frozen-CLIP validation accuracy
63.3%
VQA soft score for the ViT–BERT model
67.6

Result figures

Ranked validation accuracy, F1 score, and parameter count for eight frozen CLIP fusion models
Frozen CLIP track: GeometryFusion leads despite matching the smallest parameter count. The scale begins at the documented 50% chance baseline.
Validation scores for ViT and BERT model iterations v2, v3, and v4
ViT+BERT iterations: the 10-point rise is an iteration history, not a controlled ablation—multiple implementation fixes changed between versions.

What the work showed

  • GeometryFusion led the frozen-feature track while staying small enough to train as a focused fusion experiment.
  • The larger co-attention model improved the documented VQA soft score, but also made data and masking bugs much more consequential.
  • Correct soft targets and attention masks changed the trustworthiness of the experiment as much as the architecture choice did.

What it does not prove

  • The 63.3% accuracy and 67.6 soft score come from different documented experiment tracks and should not be compared as if they were the same metric.
  • These are validation results from the recorded runs, not a claim of state-of-the-art performance.

Conclusion

The project showed that a thoughtful fusion head can beat simpler combinations without becoming large. It also reinforced a less glamorous result: clean labels and masks are prerequisites for believing any architecture comparison.