UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
Sharing expert capacity across layers instead of duplicating it per layer
A new design for mixture-of-experts neural networks treats expert capacity as a shared resource rather than giving each layer its own separate experts. Across five model sizes, this approach reduces validation loss by up to 3.86% and matches the performance of traditional designs while using only 42–67% as many expert parameters, suggesting that experts don't need to multiply linearly as models get deeper.
Current large language models waste capacity by requiring each layer to have its own set of experts, forcing model size to balloon as networks grow deeper. This work shows you can build more efficient models by pooling experts globally, which directly reduces the computational and memory cost of training and running massive AI systems.