Strafrecht-RAG: semantic search over Dutch court rulings

Own project, built to show how I approach AI systems. The design choices are the actual story: they decide whether a RAG system becomes trustworthy or a black box.

What it is

A RAG (Retrieval-Augmented Generation) system over Dutch court rulings: search rulings from Rechtspraak.nl semantically and have a language model formulate answers — based solely on the retrieved passages, and always with a reference to the source.

The two rules the whole design revolves around:

1. Every claim gets an ECLI reference. From answer back to the exact passage in the exact ruling. No black box.
2. No relevant source, no answer. The model isn't allowed to make anything up; "no source found" is a first-class outcome, not an error.

These are the same requirements I put on any AI system I build. Legal text is just more unforgiving about them: a fabricated reference is caught immediately.

The design

Three layers, each testable on its own:

• Ingest — fetch rulings through the Rechtspraak.nl Open Data interface, with rate limiting and retries, and parse the raw XML into one normalised model: ECLI, court, date, area of law and the text sections.
• Index — split text into overlapping chunks while preserving section and paragraph numbers, so every chunk traces back to its exact place in the source. Chunks are embedded in batches and stored in a vector store, with metadata for filtering on court, date and area of law.
• Query — embed the question, retrieve the best-matching passages, and a prompt that forces the model to answer solely from those passages — with an explicit "no source found" path.

The whole chain runs from the command line: ingest, index and ask as separate commands, from fetching to answer.

Quality is measured, not hoped for

The question "does it work?" is notoriously vague for RAG systems, so it has been replaced by a measurement. A fixed evaluation set — questions paired with the ECLIs the system is supposed to find — runs in CI on every change and measures two things: does retrieval find the right rulings (recall), and do the source references in the answer match the passages that were provided. A change to chunking, the embedding model or the prompt is no longer a matter of "it feels better", but of a number going up or down — the yardstick exists before the first line of code.

The result

Every answer is clickably traceable to its underlying passages, claim by claim. Questions without a relevant source get a clean "no source found" instead of a convincing fabrication — and that refusal is tested in the evaluation set just as thoroughly as the answers themselves. The quality of the chain is expressed as a number, again on every commit.

Points of attention

• Rulings are public, but contain pseudonymised personal data — handling that is part of the design, not an appendix afterwards.
• Answers are not legal advice; source citation is mandatory in every output.

In the repo

• Ingest: fetching and parsing rulings, with rate limiting and retries
• Index: chunking with section numbers preserved, embeddings and vector store
• Query: retrieval and answer generation with mandatory citations
• CLI for the whole chain: ingest, index, ask
• Evaluation set with recall and citation-correctness measurement in CI

Technology

Python, embeddings and vector store, Rechtspraak.nl Open Data, GitHub Actions