What Improves During Pretraining? Tracking Knowledge Acquisition with Point-in-Time Models
How AI models learn rare versus common knowledge during training
Language models learn knowledge differently depending on how consistently it appears across years of internet data. Knowledge that shows up sporadically across different time periods improves faster during later training stages and ends up performing 7.3 percentage points better than consistently-appearing knowledge, especially for factual tasks.
Understanding what AI models actually learn during training helps researchers design better pretraining methods and predict which types of knowledge will be reliable versus unreliable in deployed systems. This is particularly important for knowledge-intensive applications like question-answering and fact-checking, where knowing which facts a model learned well versus poorly affects whether you can trust its answers.