Fresh Data for Coding Models: Hugging Face updated The Stack, the largest crawl of public GitHub repositories

The biggest open dataset of source code went years without an update. A new GitHub crawl gives it a much-needed overhaul.

What’s new: Anton Lozhkov and colleagues at Hugging Face releasedThe Stack v3, a snapshot of public GitHub code gathered to pretrain large language models. The team calls the new version of The Stack the largest, most up-to-date open dataset of source code. Unlike earlier versions, The Stack v3 delivers both whole repositories and their code files, so models can learn how parts of a codebase fit together.

• Scope: Source code and Jupyter notebooks only (no GitHub issues, pull requests, or documentation)
• Contents: Two releases: stack-v3-train, 15.9 terabytes of code, roughly 4.9 trillion tokens, 713 programming languages from 173 million repositories (deduplicated, quality-filtered, and scrubbed of personal information); and stack-v3-full, 113.7 terabytes of code in 770 languages from 224 million repositories (raw data, designed for teams that want to apply their own filters)
• Knowledge Cutoff: August 7, 2025
• Availability/license: Free to download under an Open Data Commons Attribution license v1.0, which permits commercial and noncommercial uses that credit the source; users must also honor the licenses of the original repositories
• Opt-out: Developers can check whether their code is included and ask for it to be removed. The code is removed in subsequent patch releases

How it works: Hugging Face crawled GitHub directly, unlike earlier versions of The Stack v2, which pulled files from the Software Heritage archive. The team filtered the raw crawl for licenses, duplicates, personal information and for files unlikely to be useful for training.

• The team built its crawl list by combining every repository that appears in GH Archive, a record of public activity on GitHub or recorded by Software Heritage. The crawler downloaded a single snapshot of each repository’s default branch at its most recent commit, without git history. It skipped files larger than 5 megabytes, binary files, and forks that had fewer than five stars. This crawl totaled 43.9 billion files.
• The team used ScanCode, a license-detection tool, to scan files whose names suggest legal content (LICENSE, COPYING, MIT.txt, and the like) and mapped any licenses it found to all files in the same directory prefix. Files that carried non-permissive licenses were excluded from both datasets. Files with no detected license were kept.
• During the crawl, each unique file was stored once under a hash of its contents, collapsing exact duplicates. The team removed near-duplicate files by reducing each one to a short MinHash signature, which estimates how much two files overlap without comparing them line by line. An index of those signatures proposed pairs likely to be near copies.
• Unlike The Stack v2, which deduplicated each programming language separately, this pass ran across all languages at once and, rather than trusting the index, measured each proposed pair before grouping into clusters of files whose estimated overlap reached at least 70 percent, which weeded out false matches. For each cluster, the training set keeps the file from the repository with the most stars, breaking ties by fork count, license permissiveness, and earliest creation date. While building v3’s deduplication pipeline, the team discovered that a bug had caused The Stack v2 to discard more files than intended, which v3’s approach corrects.
• For the training set, quality filters drawn from the StarCoder2 pipeline dropped files that contain fewer than 25 percent alphabetic characters (alphanumeric characters for Assembly code), contain at least one line longer than 1000 characters or an average line length greater than 100 characters, auto-generation markers, or large blobs of encoded data. StarPII, a model trained to detect personal information, found emails, keys, names, passwords, and IP addresses in the remaining files and replaced them with placeholders.

Yes, but: The company issued a few important warnings. The license labels come from automated detection and repository metadata, which is an error-prone approach. The training set includes files in which the tools found no license, which grant no public rights for re-use. Emails, keys, and IP addresses that developers published to public repositories may remain. Finally, the corpus may include malicious code.

Behind the news: Each version of The Stack has grown along with developers’ demand for training data. The raw corpus swelled from 6.4 terabytes in the 2022 original to 67.5 terabytes in The Stack v2 in 2024, and the training subsets from roughly 200 billion tokens to 550 billion tokens. The latest release expanded both again to 113.7 terabytes and around 4.9 trillion tokens. BigCode, a collaboration led by Hugging Face and ServiceNow, produced the two earlier versions, training StarCoderBase on the first and the StarCoder2 family on the second. The Stack v3 comes from Hugging Face’s code research team, which credits BigCode’s groundwork on data filters, personal-information detection, and dataset governance. Other companies that trained models on earlier versions of The Stack include Nvidia and Jetbrains.

Why it matters: Developers increasingly ask coding models to work across whole projects, tracing how a function in one file depends on definitions in another, rather than completing isolated snippets. Training for that skill requires data organized the way developers organize software. The Stack v2 recorded which files made up each repository but delivered only identifiers rather than the files themselves, leaving users to fetch them from a separate archive. The Stack v3 delivers each repository whole, files included, which is the context agentic coding assistants need. Code also goes stale quickly. A model trained on a 2023 GitHub snapshot knows nothing of new frameworks, language versions, and APIs. By adding around two additional years of open-source code, The Stack v3 allows code models to learn current software.

We’re thinking: A crawl of GitHub in 2025 includes substantial amounts of code that AI models wrote or helped write, which means that tomorrow’s code models will learn partly from the output of today’s. In some cases, the GitHub repositories themselves will indicate what AI assistants were used in their creation. In others, AI use can be inferred or detected. Researchers can compare code committed before and after coding assistants became widely used, and study the effect of such data on the models that were trained on it.