Vision Transformers: ViT vs Differential ViT

A comparative implementation of Vision Transformer (ViT) and Differential Vision Transformer (Diff-ViT) on CIFAR-10, exploring how differential attention mechanisms compare to standard multi-head attention in the vision domain.

---

Results

Vision Transformer (ViT)	Differential Vision Transformer (Diff-ViT)
Attention Rollout
	Differential attention cancels noise through dual softmax maps, producing more focused attention patterns on relevant regions.
Attention Maps (CLS Token → Patches)
	Noise cancellation amplifies attention to relevant context—some heads show stronger, more sparse activation patterns compared to standard attention.
Position Embedding Test Accuracy

For more details, see the full reports: ViT Report | Diff-ViT Report

---

Setup

# Install dependencies using uv
uv sync

# Dataset (CIFAR-10) downloads automatically on first run to {vit,dvit}/data/

---

Usage

cd vit # or dvit
uv run python -m src.vit --mode train # or vis