No Filter LLM Pretraining: Why Removing Data Filtering Boosts Large Language Model Performance

Data Filtering for LLM Pretraining: Why “No Filter” Might Be the Best Choice 「Core question this article answers:」 Is spending massive compute and human effort to filter “low-quality” data actually necessary when pretraining large language models? 「When compute is sufficiently large and model parameters are high enough, the best data filter is no data filter at all.」 Although there is a widely accepted belief in the industry that filtering data to include only high-quality information is essential, new scaling studies targeting high-compute, data-scarce regimes suggest otherwise. Sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in …