Two-Level Softmax Sampling Done Right: Correcting Bias from Size Imbalance and Dispersion
Fixing hidden bias in a popular machine learning speed trick
When machine learning systems need to quickly sample from millions of items, they use a shortcut called two-level softmax sampling that groups items into clusters. Researchers discovered this method systematically favors certain clusters because it ignores how cluster sizes and internal similarities differ, and they developed two corrected versions that eliminate this bias with almost no extra computational cost.
Two-level softmax sampling powers recommendation systems, language models, and other large-scale applications where exact sampling is too slow. The hidden bias means these systems were subtly skewing results — recommending some items more or less often than they should be. Using the corrected versions means fairer, more accurate recommendations and predictions without sacrificing the speed gains that made the shortcut attractive in the first place.