Our Dutch language model RobBERT is almost 6 years old and still quite a popular model (approaching 5M downloads!). When we developed it, the world was a little different: there was no AI Act, shuffling a dataset every few sentences sufficed for copyright according to Inria's OSCAR corpus, and COVID-19 didn't exist yet (something we later caught RobBERT-2022 up on). Back then we had enough data for Dutch, but just barely so.

Nowadays there's enough data that we can afford to be picky: can we make a model with only permissible data?

That's RobBERT-2026. It's trained from scratch with the modern NeoBERT^[1] architecture on ~27B tokens of provenance-clean Dutch, so entirely permissibly licensed or synthetic from permissibly licensed data. Common Corpus^[2] from Pleias, GPT-NL's data, some synthetic data, and Dutch case law, with no web crawl whatsoever. So no OSCAR, no CommonCrawl, not even a cleaned-up crawl subset or a translated FineWeb.

As far as I know, that also makes RobBERT-2026 the first released Dutch model that is explicitly compliant with the EU AI Act. GPT-NL was set up with the same goal, but that model was never released. Its public corpus did make it into RobBERT-2026, so some of that work lives on.

Training data

The pretraining corpus is ~27B tokens from four sources. Each of them is either permissibly licensed per document or synthetic.

There's no web crawl in any of these, and also no crawl hiding inside them, like a CC-licensed CommonCrawl subset or a machine-translated FineWeb-Edu. That way, every document in the corpus can be traced back to a license, which is what we need for the AI Act's provenance and transparency requirements.

Architecture

The architecture is new as well. RobBERT, RobBERT-2022 and RobBERT-2023 were all RoBERTa models, but RobBERT-2026 is a NeoBERT-base model: 28 layers, hidden size 768, 12 attention heads, SwiGLU, RMSNorm and rotary position embeddings, 275M parameters in total. We pretrained it with masked language modeling (20% masking) at a sequence length of 512, for 13k steps of ~2.1M tokens each. The tokenizer is a byte-level BPE with a 50k vocabulary.

Since the AI Act asks for it (for larger models than this one, but still), here is the compute. The released run saw 27.3B tokens, which comes down to ~4.5×10¹⁹ FLOPs with the usual 6ND estimate, at about 39% MFU. The whole project, including a first stage and two runs that didn't make the cut, was ~191B tokens or ~3.1×10²⁰ FLOPs, roughly 1,300 GPU-hours. The longest of those runs, on ~91B tokens, actually gave the worst downstream model, which is part of why we don't think more tokens would fix the NLI gap below.

Results

Does clean data cost you? A little, but it's still competitive. We evaluated on DUMB^[3], a suite of Dutch benchmarks from GroNLP, on the test split and with the same finetuning recipe for both models (2 epochs, learning rate 5e-5, bf16).

RobBERT-2026 matches RobBERT-2023 on sentiment and on question answering F1. The big jump on exact match is not because RobBERT-2026 is that much better at QA, though. RobBERT-2023's byte-level pre-tokenizer produces offset mappings that are off by one on word-initial tokens, so the extracted answer spans are often shifted by a character, which exact match counts as wrong. RobBERT-2026 has a canonical byte-level pre-tokenizer with correct offsets, so its spans line up. This is also why we report DUMB rather than EuroEval, since its SQuAD-NL pipeline hits the same bug and reports too low extractive QA scores for both models.

Natural language inference is the weak spot. 82.4 on SICK-NL is below RobBERT-2023, and below every Dutch base-size encoder on the DUMB leaderboard. We couldn't attribute this to pretraining scale: SICK-NL accuracy stayed flat at around 82 for every token budget and data mixture we tried, so this is something we still need to figure out.

For context, here are DUMB's published scores for the other Dutch base-size encoders, next to ours.

The other rows come from DUMB's own tuning protocol rather than our fixed recipe, but our RobBERT-2023 run reproduces their numbers to within 0.7 on every task, so they are comparable. Note that these are three of DUMB's nine tasks, so you can't compute a leaderboard-style average from them. Our numbers are also single-seed: a learning-rate sweep with two seeds puts RobBERT-2026's mean QA F1 at 68.9, so the 69.6 above is about 0.7 optimistic.

Retrieval model

Since retrieval is such a common task, we also release a retrieval model finetuned on top of RobBERT-2026, together with the synthetic training data it's built on. The queries in cc-nl-retrieval are generated over Common Corpus NL, and we mix in the three CC-BY-SA similarity sets from the Netherlands Forensic Institute (NFI): simplewiki, StackExchange duplicate questions and wiki atomic edits, all machine-translated to Dutch. The recipe is pretty standard: a cached multiple-negatives ranking loss on (query, positive, 4 hard negatives) triples, CLS pooling, cosine similarity, and query: and document: prefixes, for one epoch. Because the NFI sets are CC-BY-SA, the retrieval model is CC-BY-SA-4.0 as well. The base model stays MIT.

Zero-shot on BEIR-NL^[4], the machine-translated Dutch version of BEIR, it doesn't beat the big multilingual retrievers in general. Models like gte-multilingual-base and multilingual-e5-base are trained on far more data, often in-domain, and they win on the biomedical, scientific and argument retrieval sets that are out of domain for our Wikipedia-heavy training data. Against older multilingual embedders like LaBSE and mContriever it holds up fine, beating both on nfcorpus and scidocs. On fact-checking retrieval (FEVER) it's actually competitive with the big ones: clean, Wikipedia-heavy data turns out to be a good fit for matching claims to evidence.

Usage

RobBERT-2026 uses the NeoBERT architecture and the modeling code is included in the repository, so you need trust_remote_code=True:

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("pdelobelle/robbert-2026")
model = AutoModelForMaskedLM.from_pretrained("pdelobelle/robbert-2026", trust_remote_code=True)

For a downstream task, swap in AutoModelForSequenceClassification and finetune like any other HuggingFace encoder. You don't need xformers: without it, the code falls back to a pure PyTorch SwiGLU with the same outputs. One thing to keep in mind is that the model was pretrained at a sequence length of 512. The RoPE table goes up to 8192 positions so longer inputs will run, but we didn't train or test beyond 512.

The retrieval model is a sentence-transformers model. The prefixes are stored as prompts, so you can pass prompt_name and they get added automatically:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("pdelobelle/robbert-2026-retrieval", trust_remote_code=True)

docs = model.encode(["Het Atomium is een bouwwerk in Brussel."], prompt_name="document")
query = model.encode(["Waar ligt het Atomium?"], prompt_name="query")
print(model.similarity(query, docs))

Don't forget the prefixes, retrieval quality drops a lot without them.

Wrapping up

A paper on RobBERT-2026 is in the works. RobBERT remains joint work with Thomas Winters, Bettina Berendt and François Remy. I hope that this permissible model will make some Dutch use-cases easier! If you build something with it, let me know.

References

1. Le Breton et al., “NeoBERT: A Next-Generation BERT”, arXiv. ↩
2. Pleias, “Common Corpus”, Hugging Face. ↩
3. de Vries, Wieling and Nissim, “DUMB: A Benchmark for Smart Evaluation of Dutch Models”, EMNLP 2023. ↩
4. Banar, Lotfi and Daelemans, “BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language”, arXiv. ↩