Pieter Delobelle | SynthCommonCorpus

Abstract

This talk introduces SynthCommonCorpus, an open synthetic pretraining corpus designed for agentic models and planned for release in autumn 2026. It incorporates tool calls, multi-step task decomposition, and adversarial examples into pretraining data to study how models learn to act. The session explains how synthetic generation and traceable, licensed seed data support reproducibility and the measurement of memorization and data contamination.

This talk is about how we go from SYNTH, our 80B-token corpus, to 20 trillion synthetic tokens: a much faster generator, hundreds of GPUs on public European supercomputers, all 24 EU languages and a lot more seed text.

Technical talk, in English. The first release of SynthCommonCorpus is planned for November 2026.

Outline

What the talk covers

01 Where we start. SYNTH, the 80B-token synthetic corpus from our NeurIPS 2026 paper, and the Baguettotron models trained on it. Baguettotron-600M comes close to Qwen3-0.6B with 230 times fewer training tokens.
02 A small, fast generator. Synthotron-8B is a mixture of experts with 1.5B active parameters that learned from three teacher models. It writes a full conversation from one Wikipedia paragraph, and one H100 writes 19,000 tokens per second (the SYNTH v1 generator wrote about 3,500).
03 Scaling out. We run on Jean Zay and MareNostrum 5, shared clusters with SLURM and no internet on the nodes. Without retrieval in the pipeline we ran on 368 H100 GPUs at the same time, so 20T tokens needs about 520k GPU-hours.
04 24 EU languages. Nine languages are done, 830B tokens, each with its own seeds, queries and reasoning. The other 15 EU languages are next.
05 From 830B to 20T. More copies of a seed don't teach more facts, so we need more seeds. Common Corpus gives licensed, traceable seed text, which keeps the data reproducible and lets us measure memorization and contamination.
06 First agentic data. An early result with single-turn tool calls, written by two Gemma-3-12B models from Wikipedia seeds. With this data, Baguettotron-600M scores 4.6 points more than FunctionGemma on BFCL v2. Multi-step tasks are next.

SynthCommonCorpus

What the talk covers

The deck

Further reading