CommonCrawl-Creative Commons (C5)
Collection
Raw CommonCrawl crawls, annotated with Creative Commons license information • 3 items • Updated • 1
Datasets:
Tasks:
Text Generation
Modalities:
Text
Formats:
parquet
Sub-tasks:
language-modeling
Size:
10M - 100M
License:
Dataset Viewer
The dataset viewer is not available for this dataset.
Unexpected token '<', "<html>
<h"... is not valid JSON
Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Common Crawl Creative Commons Corpus Strict (C5s)
A filtered version of the Common Crawl Creative Commons Corpus (C5), only retaining samples that:
- are also present in the FineWeb or FineWeb-2 datasets;
- have no license disagreement (all found licenses have the same type; version number might differ);
- are not "non-commercial" ("nc" in license);
- are not "cc-unknown";
- do not have "wiki" in their name (the idea is that you should include Wikipedia and other Wikidata from other, high-quality resources, parsed with a better parser).
A less strict filtered version called C5f is also available that is only based around the first criterion: FineWeb data.
Created with this script.
For more information, see C5.
Progress
In the v1 release, the following crawls are included
- CC-MAIN-2019-30
- CC-MAIN-2020-05
- CC-MAIN-2023-06
- CC-MAIN-2024-51
- CC-MAIN-2024-46
- CC-MAIN-2025-05 (included in the original but not here because FineWeb and FineWeb-2 do not include it)
- CC-MAIN-2022-05
Languages
The following languages are included. This is a limited set due to computational and storage limitations.
- Afrikaans: afr
- German: deu
- English: eng
- French: fra
- Frysian: fry
- Italian: ita
- Dutch: nld
- Spanish: spa
Quantity
Counts for the strict release. Detailed number of tokens (Llama 3.3 tokenizer) and number of documents are given in the counts.json file.
Note: when comparing the original with this version, understand that: i. this version does not include crawl 2025-05 because FineWeb(-2) did not include it; ii. both 2024-46 and 2024-51 only contain English because FineWeb-2 did not include those crawls. So calculating quantity reduction after filtering should take this information into account.
| Language | No. Docs (original) | No. Docs (C5s) | No. Tokens (original) | No. Tokens (C5s) |
|---|---|---|---|---|
| afr | 312,262 | 350 | 358,873,448 | 913,178 |
| deu | 9,530,746 | 89,340 | 11,362,859,534 | 84,408,955 |
| eng | 92,635,372 | 7,843,160 | 87,537,859,958 | 7,035,305,977 |
| fra | 9,234,900 | 44,824 | 12,366,480,025 | 43,143,952 |
| fry | 230,910 | 1 | 197,430,774 | 1092 |
| ita | 10,734,597 | 68,418 | 11,913,669,333 | 58,765,829 |
| nld | 2,827,636 | 18,266 | 2,757,074,705 | 18,957,134 |
| spa | 22,226,944 | 123,301 | 22,515,709,432 | 113,258,753 |
| Total | 147,733,367 | 8,187,660 | 149,009,957,209 | 7,354,754,870 |
- Downloads last month
- 338