๐Ÿ”ฌ MoE Expert Subspace Analysis

Research tool: Measuring Head (shared) vs Tail (expert-specific) subspace similarity across OLMoE-1B-7B experts.

Hypothesis: Tail subspace similarity < Head subspace similarity, and tail similarity โ‰ˆ random baseline โ†’ expert-specific directions are geometrically complementary, not redundant.

Compare two specific experts in detail.

0 15
0 63
0 63
Projection
5 40