Data Attribution
Collection
Leave-1%-out retrained models, per-query loss changes, and datasets for computing LDS. • 42 items • Updated
Datasets:
Dataset Viewer
The dataset viewer is not available for this subset.
Cannot get the split names for the config 'default' of the dataset.
Exception: SplitsNotFoundError
Message: The split names could not be parsed from the dataset config.
Traceback: Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 290, in _generate_tables
pa_table = paj.read_json(
io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)
)
File "pyarrow/_json.pyx", line 342, in pyarrow._json.read_json
File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
return check_status(status)
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
raise convert_status(status)
pyarrow.lib.ArrowInvalid: JSON parse error: Column() changed from object to string in row 0
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.14/site-packages/datasets/inspect.py", line 286, in get_dataset_config_info
for split_generator in builder._split_generators(
~~~~~~~~~~~~~~~~~~~~~~~~~^
StreamingDownloadManager(base_path=builder.base_path, download_config=download_config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 101, in _split_generators
pa_table = next(iter(self._generate_tables(**splits[0].gen_kwargs, allow_full_read=False)))[1]
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 304, in _generate_tables
batch = json_encode_fields_in_json_lines(original_batch, json_field_paths)
File "/usr/local/lib/python3.14/site-packages/datasets/utils/json.py", line 111, in json_encode_fields_in_json_lines
examples = [ujson_loads(line) for line in original_batch.splitlines()]
~~~~~~~~~~~^^^^^^
File "/usr/local/lib/python3.14/site-packages/datasets/utils/json.py", line 20, in ujson_loads
return pd.io.json.ujson_loads(*args, **kwargs)
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
ValueError: Expected object or value
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/src/services/worker/src/worker/job_runners/config/split_names.py", line 68, in compute_split_names_from_streaming_response
for split in get_dataset_split_names(
~~~~~~~~~~~~~~~~~~~~~~~^
path=dataset,
^^^^^^^^^^^^^
config_name=config,
^^^^^^^^^^^^^^^^^^^
token=hf_token,
^^^^^^^^^^^^^^^
)
^
File "/usr/local/lib/python3.14/site-packages/datasets/inspect.py", line 340, in get_dataset_split_names
info = get_dataset_config_info(
path,
...<6 lines>...
**config_kwargs,
)
File "/usr/local/lib/python3.14/site-packages/datasets/inspect.py", line 291, in get_dataset_config_info
raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.
Pile held-out query set (1000 x 2048 tokens) (Warning: documentation is by Claude)
Held-out evaluation sequences for training data attribution on Pythia-family pretraining runs. Drawn from a permanently reserved pool of Pile documents that no training set at any scale may contain, then decontaminated against the full deduplicated Pile.
Reserved pool
Dataset: pile_0.87_deduped_text_document.bin (Pythia deduplicated Pile, GPT-NeoX tokenizer,
134,318,121 documents). Document index i is reserved iff
splitmix64(doc_index XOR 2026) % 1000 == 0 -> 135,098 documents, 205,331,803 tokens
(reserved_doc_ids.npy). Training sets must be drawn from the complement.
Windows
Packed like GPT-NeoX training samples: a random reserved start document (random
offset if longer than 2048), then following reserved documents joined by
EOD (id 0) up to 2048 tokens. Columns: input_ids, length, text,
doc_ids (source document indices), n_docs, pile_coverage, pile_longest.
Quality filters
Control characters <= 0.01, unique-token ratio >= 0.3, English stopword ratio >= 0.05, alphabetic fraction >= 0.6.
Query deduplication (decontamination) against the training corpus
Reservation vs. deduplication. The reservation rule above removes the windows' source documents from every training set. Deduplication additionally rejects windows whose text also appears, in whole or in large part, in other documents of the corpus (the deduplicated Pile still contains near-duplicates across documents), so that no training set can contain a copy of a query.
Detection unit. A token-level 13-gram exact-match scan over all 207,170,414,058 tokens of the 134,183,023 non-reserved documents (GPT-NeoX tokenizer, rolling polynomial hash, verified against the exact candidate 13-gram set). For every candidate window this yields
pile_coverage: the fraction of the window's tokens that lie inside at least one 13-gram also present in some non-reserved document, andpile_longest: the longest exact shared substring in tokens (a run ofrconsecutive shared 13-grams is a shared substring ofr + 12tokens).
Rejection rule (Lee et al. 2022 ExactSubstr thresholds). A window is
rejected if pile_longest >= 50 or pile_coverage > 0.20. The same rule is
applied between candidates in order (earlier candidate wins), so the kept
windows are mutually non-duplicate.
What the rule does and does not guarantee. Shared substrings shorter than 13 tokens are below the detection unit and are neither measured nor penalised; shared substrings of 13-49 tokens are tolerated as long as they cover at most 20% of the window. Both are deliberate: sub-13-token overlaps (names, phrases, boilerplate fragments) occur in essentially every window, and the median random 2048-token Pile window shares an 85-token exact substring with some other document, so demanding zero shared 13-grams keeps only 1.4% of candidates and selects for atypical text.
Outcome. Of 26,000 candidates, 22,199 passed the quality filters; of those 5,724 passed this rule (5,323 after the between-window check), and 1000 were drawn uniformly (seed 2026). Residual overlap of the selected 1000 windows, all measured against the full non-reserved Pile:
| statistic | value |
|---|---|
| longest shared substring, median / p90 / max | 27 / 44 / 49 tokens |
| coverage by shared 13-grams, median / p90 / max | 6.3% / 15.4% / 20.0% |
| windows sharing no 13-gram with any non-reserved document | 61 |
Per-window values are the pile_coverage and pile_longest columns, so a
stricter subset can be selected without rescanning; among the 22,199
quality-passing candidates the survivors under stricter rules are:
| rule (longest < L and coverage <= C) | candidates surviving |
|---|---|
| L = 50, C = 20% (this release) | 5,724 |
| L = 32, C = 10% | 2,939 |
| L = 25, C = 5% | 1,736 |
| L = 13, C = 0% (no shared 13-gram at all) | 311 |
decontamination_summary.json lists every rejected candidate with its reason.
- Downloads last month
- 70