Data Filtering for LLM Pretraining: Why “No Filter” Might Be the Best Choice

「Core question this article answers:」 Is spending massive compute and human effort to filter “low-quality” data actually necessary when pretraining large language models?

「When compute is sufficiently large and model parameters are high enough, the best data filter is no data filter at all.」 Although there is a widely accepted belief in the industry that filtering data to include only high-quality information is essential, new scaling studies targeting high-compute, data-scarce regimes suggest otherwise. Sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally “poor” data, challenging the very foundation of traditional data curation pipelines.

Image Source: Unsplash

The Core Question: Do Large Models Really Need Low-Quality Data Filtered Out?

「Core question this section answers:」 Do mainstream data filtering strategies help or hinder performance as models scale to extreme sizes?

「Data filtering is necessary in the low-compute era, but at the large compute limit, it becomes a performance ceiling.」 The standard approach to selecting pretraining data for language models involves heavily filtering text from massive web scrapes like Common Crawl (CC). In compute-constrained regimes—where one must train on a strict subset of CC—different data selection strategies have a massive, measurable impact on performance. Intuitively, if all else is equal, it seems natural to train on “higher-quality” data.

However, filtering inherently removes data. This directly conflicts with scaling trends that prescribe ever-increasing amounts of data to improve model performance. For example, heavily filtered datasets keep only about 1% of the original Common Crawl, resulting in roughly 3.8 trillion tokens. While this sounds enormous, it falls drastically short of the optimal token budget for a 1 trillion parameter model, even after accounting for the diminishing returns that occur when repeating data over multiple epochs. The current industry trend is to over-train relative to optimal compute-to-data ratios, which prescribes even more tokens to allow for financially feasible, relatively smaller models to be deployed.

Author’s Reflection: The Intuition Trap in Resource Allocation

In engineering practice, we easily fall into the intuitive trap that “dirty data inevitably produces poor models.” This leads countless teams to pour precious engineering hours into designing complex, heuristic filtering rules. Yet, if we redirected that same effort into increasing model parameters or extending training steps, we would likely achieve much more guaranteed gains. The stark contrast between our human intuition and these empirical results is something every AI practitioner should deeply consider.

Experimental Design: How Do We Scientifically Measure a Dataset’s True Value?

「Core question this section answers:」 How can we ensure a fair comparison that unlocks a dataset’s full potential when comparing datasets of varying quality?

「To measure a dataset’s true value, you cannot fix model size and training steps; you must seek the best achievable performance under an unconstrained compute budget.」 Traditional comparative experiments often fix the model scale, which artificially suppresses the advantages of high-capacity datasets. To extract all the “juice” out of a dataset, experimental design must allow model parameters and training steps to be jointly optimized across several orders of magnitude, continuing until performance improvements plateau or compute runs out.

Experimental Setup and Benchmark Configurations

To rigorously test this hypothesis, the experiments relied on the following specific configuration baselines:

Regarding training hyperparameters, all experiments uniformly used a context length of 1024 tokens, a batch size of roughly 524,288 tokens, and a 500-step warmup phase. For every individual model, both training step count and weight decay were explicitly tuned to maximize stability when repeating data. The learning rate was set to decay with model size—a standard practice determined during an initial tuning stage.

Image Source: Unsplash

Filtering vs. No Filtering: The Divide Between Low and High Compute

「Core question this section answers:」 In direct comparison tests, how does an unfiltered raw data pool eventually overtake meticulously cleaned datasets?

「An unfiltered raw pool performs worst initially, but given a large enough model and long enough training, it will ultimately surpass all filtered datasets.」 The primary experiments operated on the smallest CC pool size of about 670 million tokens, where ablations are the cheapest. Five different filtered versions of this pool were curated, ranging from loose to extremely strict, to see if models could extract signal from the data that is typically thrown away.

The Logic and Retention Rates of Five Filtering Strategies

Understanding the mechanisms of these filters reveals exactly what they discard:

1. 「English Filter」: Uses a fastText classifier to score documents and applies a language threshold. It retains only 28.2% of the data, discarding massive amounts of non-English text that might still contain highly valuable, universal logical structures.
2. 「Repetition Filter」: Originating from the Gopher model’s curation stage, this operates on the premise that “excessive repetition is often linked with uninformative content.” It checks duplicate fractions across lines, paragraphs, and n-grams. It retains only 45.3% of the data.
3. 「Stop Word Filter」: Requires a document to contain at least 2 occurrences of specific English stop words (“the”, “be”, “to”, “of”, “and”, “that”, “have”, “with”). It retains 50.4% of the data.
4. 「RefinedWeb」: Combines the above and similar filters to reproduce a high-quality dataset. It retains a strict 13% of the data.
5. 「DCLM-Baseline」: Applies aggressive deduplication and quality-based filtering on top of RefinedWeb. This represents the pinnacle of standard cleaning pipelines, retaining only 2.1% of the original pool.

The Emergence of Crossover Points

On the extremely small 15-million parameter model, no matter how many tokens were processed (even up to 100 billion), the raw pool’s performance remained dead last. It never surpassed the first four loosely filtered datasets. This proves that 「small models genuinely cannot digest the noise in raw data.」

However, as model parameters increased to 330 million and then 1 billion, a remarkable shift occurred: clear “crossover points” appeared on the loss curves. For the 1-billion parameter model, the raw pool ultimately achieved a best average loss of 3.37, and its performance had not even begun to visibly plateau. Furthermore, when these runs are translated into a compute-performance Pareto frontier, the raw pool smoothly transitions from being the worst-performing dataset at low compute to the absolute best-performing dataset at high compute.

「Application Scenario:」 Imagine your team is pretraining a 1-billion parameter model for a specialized vertical domain with a limited compute budget. If you apply strict data cleaning, you will hit a decent baseline quickly with less compute. But if you have a sufficient compute budget and choose to use raw, “dirty” crawled data containing massive amounts of noise—and train that 1-billion parameter model for a prolonged period—you will ultimately break through a performance ceiling that the cleaned dataset simply cannot reach.

Extreme Stress Testing: Are Random Gibberish and Shuffled Words Useful?

「Core question this section answers:」 If conventional “low-quality” web pages aren’t bad enough, will feeding the model completely illogical random strings or text with scrambled word orders cause it to break?

「Even when faced with seemingly useless randomly generated strings, large models do not collapse; and when faced with real documents with shuffled word orders, sufficiently trained large models actually perform better than using the original documents.」 To probe the absolute limits of pretraining robustness, two extreme types of “junk data” were deliberately constructed and injected into the raw pool. The injection ratios ranged from a mild 20% up to an extreme 800% (meaning junk data was 8 times the size of the normal pool, leaving only about 10% untouched CC documents).

Constructing Two Extreme Junk Datasets

• 「Randomly Generated Strings」: A vocabulary of 10,000 words was built by uniformly sampling 3 to 8 characters from the lowercase English alphabet. Words were then uniformly sampled and concatenated with spaces to form documents. This data contains absolutely zero natural language logic or semantics.
• 「Shuffled Word Documents」: Extra CC documents not included in the base subset were taken, and the order of words in every single document was completely randomized. This destroys all syntax and word co-occurrence relationships, preserving only the document’s unigram (single word) distribution.

Counterintuitive Experimental Results

First, even with 8 times the amount of injected junk data, the model’s performance never degraded to random guessing levels (which would result in a theoretical worst-case loss of roughly 10.8).

For the randomly generated strings, the 15-million parameter model showed a clear performance gap. But by the time the model reached 330 million parameters, this gap had largely closed. On the 1-billion parameter model, the dataset with 20% random strings even slightly outperformed the pure raw pool. This is likely attributed to either an unintended regularization effect or the fact that random strings share a superficial similarity with natural text (short character sequences separated by spaces).

For the shuffled words, the results were even more staggering. On the 330-million parameter model, with the exception of the extreme +800% injection, all shuffled-word datasets surpassed the pure raw pool performance after processing around 11 billion training tokens.

「Why do shuffled-word documents actually help?」
Intuitively, random strings come from a completely alien distribution, which small models can easily identify but cannot utilize. Shuffled documents, however, are more “confusing” in a productive way. While the sentence “The capital of France is Paris” might be scrambled into “Paris capital the France of is,” the words “France” and “Paris” still appear within the same document boundary. This helps the model establish a latent statistical connection between these two words at the unigram level, even though all grammar is destroyed.

Author’s Reflection: Redefining “Noise”

We often arrogantly dismiss data that fails to meet grammatical standards as pure noise. This experiment strikes a heavy blow to that mindset: large models possess enough capacity to piece together statistical regularities from shattered lexical fragments. It made me realize that in the face of absolute large-scale statistical learning, the human cognitive bias toward the importance of “syntax” has likely been vastly overestimated.

Image Source: Unsplash

Scaling Law Projections: When Will Raw Common Crawl Beat Top-Tier Filtered Data?

「Core question this section answers:」 Can conclusions drawn from tiny pools of a few hundred million tokens be extrapolated to the full 240-trillion-token internet scale?

「By building scaling laws, the compute required for the full Common Crawl pool to outperform top-tier filtered datasets like RefinedWeb is predicted to be around 1e+30 FLOPs, which is well within future compute growth expectations.」 A 670-million token experiment is highly suggestive, but it is quite far from the real internet stock of 200 to 500 trillion tokens. Scale effects could fundamentally change these conclusions. To address this, pool sizes and model sizes were scaled simultaneously across 2 orders of magnitude.

The Mathematical Quest for the “Crossover Point”

To quantify this, a specific function was defined: given a fixed model parameter count and pool size, what is the minimum number of training steps required for the raw pool to first outperform the filtered pool? By fixing a 1-billion parameter model and increasing the pool size, two critical observations were made:

1. 「Super-linear Step Growth」: The training steps required to achieve a crossover grow super-linearly. A 670-million token pool needs roughly 1 epoch to win, a 2-billion token pool needs about 3 epochs, and a 10-billion token pool needs about 10 epochs.
2. 「The Salvation of Model Scale」: While an 80-million parameter model completely fails to find a crossover point at the 10-billion pool size (meaning it likely never will due to non-monotonic extreme epochs), increasing the model parameters to 1 billion or 7 billion causes the required epoch count to rapidly decrease.

The 1e+30 FLOPs Prediction Logic

Based on these observations, two distinct compute scaling laws were constructed to predict when the full 240-trillion token DCLM-Pool would outperform RefinedWeb:

1. 「Token-to-Parameter Ratio Fit」: Assuming a ratio of 600 tokens per non-embedding parameter (following certain frontier model architectures), combined with polynomial interpolation of observed data points, the required compute was derived.
2. 「Epoch Constraint Fit」: Setting a fixed 4-epoch training constraint (based on data-constrained scaling research) and finding where this linear constraint intersects with the crossover function.

Remarkably, both completely different fitting methods exhibited highly linear behavior in log-log space (R² > 0.99) and yielded nearly identical predictions: between 9.0e+29 and 3.6e+30 FLOPs. Given that current frontier pretraining compute is estimated near 5e+26 FLOPs, and existing forecasts predict 1e+29 FLOP training runs by 2030, this compute threshold for “no filtering” is far from an outlandish, unreachable figure.

The True Danger Boundary: What Data Actually Harms Large Models?

「Core question this section answers:」 If random gibberish and low-quality web pages are harmless, what exactly constitutes “poison” for LLM pretraining?

「Large models are highly resistant to covariate shifts (like word order scrambling), but are truly harmed by “conditional distribution shift” data—information that looks like high-quality text but contains factual errors.」 We should not expect “more data is better” to be a universal law in machine learning. In classic problems of domain adaptation and distribution shift, bad data absolutely hurts performance.

The Potential Threat of Factual Errors

If a model sees enough instances of “The capital of France is Copenhagen” during pretraining, it will learn the wrong capital. To evaluate the proportion of this “actively harmful” content in Common Crawl, a targeted corpus analysis was conducted. By matching keywords from specific knowledge domains (world religions, astronomy, college biology, medical genetics) and using an advanced model to classify whether the document supports, refutes, is related to, or is unrelated to the correct answer, a clear picture emerged.

The analysis showed that explicitly “supporting” documents in Common Crawl outnumbered explicitly “refuting” documents by at least an order of magnitude across all tested categories. This indicates that in the full CC pool, malicious data actively spreading factual errors is extremely low frequency.

A Minor Edge Case: Predicting the Start of a Document

Despite overall extreme robustness, a specific micro-test revealed a localized degradation caused by shuffled-word data: predicting the very first token of a document. When comparing the average validation loss of a pure CC pool against a dataset injected with 200% shuffled words, the shuffled dataset won. However, when evaluating only the loss on the first 1, 4, or 16 tokens of a document, the shuffled dataset immediately lost its advantage.

The logic is straightforward: faced with an empty prefix, a model has absolutely no way to detect if the upcoming document has had its word order scrambled. This distributional inconsistency causes confusion right at the sequence boundary. However, since the vast majority of language model use cases involve generating well beyond a few tokens, this localized edge degradation does not constitute a meaningful practical impact.

Theoretical Explanation: Why Can Large Models “Digest” Junk Data?

「Core question this section answers:」 From the bottom-up logic of model capacity and theory, what is the mechanism that allows large parameter models to absorb poor-quality data without penalty?

「When a model’s hidden layer capacity (such as matrix rank) is large enough, it can process different data distributions through independent, non-interacting components, leaving “good” and “bad” data undisturbed; small models lack this capacity and suffer from representational interference.」

Mapping the Theory via Low-Rank Matrix Factorization

The exact behavior observed in the experiments can be perfectly explained at the population level using the simplest single-hidden-layer linear neural network: low-rank matrix factorization.

Imagine predicting vector-valued outputs (tokens) using a rank-r linear transformation of an input. Assume the data comes from K different tasks, and these tasks have strictly orthogonal input distributions—meaning their features occupy entirely separate, non-overlapping dimensions. The theoretical proposition proves the following:

• 「When Capacity is Sufficient (Model rank r ≥ number of tasks K)」: The matrix factorization can perfectly represent the prediction problem. Gradient descent and other first-order optimization methods are mathematically guaranteed to find the globally optimal matrices. In this state, noise or “junk” tasks are seamlessly absorbed into their own dedicated dimensions, causing absolutely zero interference with the learning of “useful” tasks.
• 「When Capacity is Insufficient (Model rank r < effective rank ρ)」: Model performance necessarily degrades due to interference between tasks in the representation space. The features of different tasks are forcibly compressed into a lower-dimensional space, making it impossible to accurately reconstruct the targets.

This exceptionally clean theoretical model maps precisely onto the empirical reality: large models (high rank) can isolate the noise in an unfiltered pool into non-interfering pathways, while small models (low rank) struggle desperately, causing the signal and noise to clash and degrade performance.

Limitations and Engineering Implications

「Core question this section answers:」 How should these conclusions be rationally viewed and applied in real-world, industrial-scale LLM development?

「The “no filter” conclusion is strictly limited to standard pretraining of dense Transformers; in today’s compute-constrained environment, data filtering remains valuable, but we must reduce our excessive anxiety over “low quality.”」

Essential Limitations to Acknowledge

Before applying this “bitter lesson” in production, its strict boundaries must be recognized:

• 「Architecture and Training Paradigms」: The conclusions are rooted in dense Transformers without data curricula, data weights, or post-training. For potentially more unstable architectures like Mixture of Experts (MoEs), or during later post-training phases, data selection might remain critically sensitive.
• 「Impact of Synthetic Data」: If low-quality data primarily acts as a regularizer, synthetic data might be strictly superior. If low-quality data provides unique information, synthetic data will merely delay the crossover point where “no filter” becomes optimal.
• 「Influx of AI-Generated Content」: The proportion of AI-generated content in Common Crawl is expected to rise. Its impact on this dynamic remains entirely unclear.
• 「Duplicate Document Variables」: While weight decay was tuned, the expected fraction of duplicate documents inherently increases with subset size. Though unlikely to change core conclusions, it remains a variable not heavily tested here.

Strategic Shifts for Engineering Practice

Despite these limitations, the research provides a clear roadmap for a mindset shift in engineering teams. When you are operating in an extremely compute-constrained environment and must get a small model (e.g., tens of millions of parameters) to a usable state in limited steps, traditional data filtering (like deduplication and language classification) remains your best option.

However, if you are planning large-scale training (billions of parameters or more) and your compute budget allows for multi-epoch, long-cycle training, you should decisively loosen your filtering thresholds. Stop wasting engineering effort weeding out web pages that “look a bit messy.” Your large model has more than enough capacity to extract statistical value from them at the unigram level. Shifting engineering resources from “data cleaning” to “increasing parameters” and “optimizing long-cycle training stability” is the wise move that aligns with true scaling laws.

Practical Summary and Actionable Checklist

Based on the analysis above, here is a directly actionable checklist for technical teams to evaluate their data pipelines:

• [ ] 「Evaluate Compute and Parameter Scale」: Clearly define the parameter scale of your training task. If it is below 100 million parameters, continue executing strict heuristic filtering; if planning for 1 billion parameters or more, strongly consider loosening thresholds.
• [ ] 「Stop Over-Deduplicating」: Unless facing extreme repetition (e.g., infinite loops from spam sites), do not sacrifice data scale merely to eliminate marginal duplicates.
• [ ] 「Value the Unigram Distribution」: Recognize that even if grammar is completely destroyed, as long as a document’s vocabulary distribution is relevant to your target domain, that data holds potential regularization or statistical value for large models.
• [ ] 「Beware of Factual Poison」: Shift the focus of data cleaning from “format and grammatical quality” to “factual verification.” Focus on catching text that reads smoothly but contains counter-factual statements.
• [ ] 「Monitor Initial Sequence Loss」: When training with datasets containing massive amounts of out-of-distribution noise, separately monitor the prediction loss on the first few tokens of documents to assess potential localized degradation from distribution shifts.
• [ ] 「Plan for Long-Cycle Training」: If you decide to adopt loose filtering or no filtering, you must simultaneously adjust regularization hyperparameters like weight decay to handle the inevitable non-monotonic fluctuations of multi-epoch training.

One-Page Summary

Frequently Asked Questions (FAQ)

「Q1: Does this mean we never need to clean our data when training large models in the future?」
No. These findings specifically target the standard pretraining phase of dense Transformers. Until compute reaches extremely massive scales, or when using potentially unstable architectures like Mixture of Experts, moderate deduplication and basic cleaning remain beneficial.

「Q2: Why do documents with shuffled word orders actually make the model perform better?」
Because shuffling destroys syntax but preserves the document’s unigram distribution. For example, even scrambled, “France” and “Paris” still appear in the same document. This helps the large model establish a latent statistical connection at the basic word-co-occurrence level, a connection that smaller models simply cannot extract.

「Q3: Are randomly generated gibberish strings really free of negative impact?」
Negative impacts do exist on very small models (like 15 million parameters). However, as parameters scale to 330 million and 1 billion, this impact is entirely absorbed. In some specific low-injection ratios (like 20%), it even provides a slight performance boost, likely due to an unintended regularization effect.

「Q4: What does the predicted 1e+30 FLOPs compute threshold actually mean in reality?」
It represents several thousand times the compute of current top-tier frontier model pretraining. While out of reach today, based on established compute growth forecasts, this level of compute is projected to be achievable around the year 2030, making the “no filter” strategy practically relevant in the near future.

「Q5: What kind of “poor quality” data is absolutely intolerable to models?」
Models are highly robust to covariate shifts (like changing word order), but they cannot tolerate “conditional distribution shifts.” This refers to data that is flawlessly written and looks like a high-quality encyclopedia, but states completely wrong facts (e.g., identifying the wrong city as a country’s capital).

「Q6: Why were negative log-likelihood metrics chosen over common QA benchmarks for evaluation?」
Because common QA benchmark test sets are quite small, resulting in performance curves that are too noisy. Negative log-likelihood on datasets like C4 correlates highly with downstream performance and captures subtle differences between datasets during prolonged training much more smoothly and reliably.

「Q7: What do “orthogonal inputs” in the theoretical model correspond to in reality?」
In the theoretical explanation, orthogonal inputs mean the features of different tasks can be perfectly separated via linear transformations. In real-world pretraining, this approximates a model possessing a high-enough-dimensional hidden space to assign the “logical features of normal web pages” and “surface features of gibberish text” to completely separate, non-interfering dimensions for independent processing.