The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups
Teaching AI to pay attention using pure geometry instead of learned rules
A new attention mechanism for AI treats tokens as geometric transformations—rotations, reflections, shearing—rather than vectors with learned features. The system scores relationships using intrinsic distance between these transformations, not learned kernels, and handles complex geometric groups (like rotations in 3D space or 2D affine transformations with scaling) that existing methods cannot. In tests on sequence completion, it matched learned approaches with 50–80 times fewer parameters and broke no geometric rules, while standard vector-based attention failed by trillions of times over.
Most AI attention mechanisms are built on learned, data-dependent rules that can violate the geometric structure they're meant to preserve. This construction builds attention directly from mathematical geometry, guaranteeing that transformations remain valid by design rather than by luck. That matters for any system working with structured spatial data—robotics, 3D vision, medical imaging, physical simulations—where breaking geometric consistency causes failures downstream.