verify-ppt/ppt-c4_c4_3b_re
Updated
Verify PPT
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
Organization Card
Welcome to the verify-ppt project!
In this space, we release all the model checkpoints and pre-processed datasets used in the paper: Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior. For the codebase, please refer to our GitHub repository: https://github.com/gucci-j/verify-ppt-at-scale.
Model Checkpoints
500M
Click to expand the table for 500M models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | C4 | verify-ppt/c4_500m |
| k-Shuffle Dyck | C4 | verify-ppt/ppt_c4_500m_v2 |
| Control (C4) | C4 | verify-ppt/ppt-c4_c4_500m |
| - | Olmo3 | verify-ppt/olmo3_500m |
| k-Shuffle Dyck | Olmo3 | verify-ppt/ppt_olmo3_500m_v2 |
| Control (Olmo3) | Olmo3 | verify-ppt/ppt-olmo3_olmo3_500m |
| - | SmolLM3 | verify-ppt/smollm3_500m |
| k-Shuffle Dyck | SmolLM3 | verify-ppt/ppt_smollm3_500m_v2 |
| Control (SmolLM3) | SmolLM3 | verify-ppt/ppt-smollm3_smollm3_500m |
| - | Marin | verify-ppt/marin_500m |
| k-Shuffle Dyck | Marin | verify-ppt/ppt_marin_500m_v2 |
| Control (Marin) | Marin | verify-ppt/ppt-marin_marin_500m |
1B
Click to expand the table for 1B models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | C4 | verify-ppt/c4_1b |
| k-Shuffle Dyck | C4 | verify-ppt/ppt_c4_1b_v2 |
| Control (C4) | C4 | verify-ppt/ppt-c4_c4_1b_re |
| - | Olmo3 | verify-ppt/olmo3_1b |
| k-Shuffle Dyck | Olmo3 | verify-ppt/ppt_olmo3_1b_v2 |
| Control (Olmo3) | Olmo3 | verify-ppt/ppt-olmo3_olmo3_1b_re |
| - | SmolLM3 | verify-ppt/smollm3_1b |
| k-Shuffle Dyck | SmolLM3 | verify-ppt/ppt_smollm3_1b_v2 |
| Control (SmolLM3) | SmolLM3 | verify-ppt/ppt-smollm3_smollm3_1b_re |
| - | Marin | verify-ppt/marin_1b |
| k-Shuffle Dyck | Marin | verify-ppt/ppt_marin_1b_v2 |
| Control (Marin) | Marin | verify-ppt/ppt-marin_marin_1b_re |
3B
Click to expand the table for 3B models
7B
Click to expand the table for 7B models
| PPT Task | PT Data | Hugging Face Model Link |
|---|---|---|
| - | Marin (75.5B) | verify-ppt/marin_100b_7b |
| k-Shuffle Dyck | Marin (75.5B) | verify-ppt/ppt_marin_100b_7b |
Preprocessed Datasets
Pre-pretraining
Click to expand the table for pre-pretraining datasets
| Settings | Hugging Face repository |
|---|---|
| k-Shuffle Dyck | verify-ppt/ppt-shuff_dyck_v2 |
| Control (C4) | verify-ppt/c4-c4_baseline |
| Control (SmolLM3) | verify-ppt/smollm3-baseline_v3 |
| Control (Olmo3) | verify-ppt/olmo3-dolma3_mix_baseline_v2 |
| Control (Marin) | verify-ppt/marin-baseline_v3 |
| Set | verify-ppt/ppt-set_v2 |
| MP-Struct Core | verify-ppt/ppt-mpstructcore_v2 |
| NCA | verify-ppt/ppt-nca |
Pre-training
Click to expand the table for pre-training datasets
Citation
If you find these artifacts useful for your research, please consider citing the following paper:
@article{yamaguchi2026syntheticprepretrainingsurvivesscale,
title={Synthetic Pre-pretraining Survives Scale, but Not as a Grammatical Prior},
author={Atsuki Yamaguchi and Tatsuro Inaba and Joel Niklaus and Michal Štefánik and Aline Villavicencio and Nikolaos Aletras},
year={2026},
eprint={2609.39827},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2609.39827},
journal={arXiv preprint},
volume={arXiv:2609.39827}
}
models 97
verify-ppt/ppt-olmo3_olmo3_3b_re
Updated
verify-ppt/ppt-smollm3_smollm3_3b_re
Updated
verify-ppt/ppt-mix_marin_3b
Updated
verify-ppt/ppt-marin_marin_3b_re
Updated
verify-ppt/marin_3b_seed2
Updated
verify-ppt/ppt_marin_3b_v2_seed3
Updated
verify-ppt/ppt-olmo3_olmo3_1b_re
Updated
verify-ppt/ppt-smollm3_smollm3_1b_re
Updated
verify-ppt/ppt-marin_marin_1b_re
Updated
datasets 145
verify-ppt/ppt-logic_mixed
Updated • 109
verify-ppt/ppt-logic_first_order
Updated • 147
verify-ppt/ppt-logic_propositional
Updated • 185
verify-ppt/ppt-logic_term
Updated • 586
verify-ppt/ppt-nca
Updated • 119
verify-ppt/ppt-mpstructcore_v2
Updated • 131
verify-ppt/ppt-mpstructcore
Updated • 368
verify-ppt/finewebedu-finewebedu_30b
Updated • 59
verify-ppt/marin-dclm_marin_103b
Updated • 540
verify-ppt/c4-c4_65
Updated • 66