Cosine similarity is not a relevance score

strafrecht-rag has one rule: no source, no answer. If the corpus of criminal-law rulings has no passage that answers the question, the system says so and the language model is never called. That sounds like a threshold on a similarity score, and that is how I first built it. It did not work, and the reason is a mistake that sits in a lot of RAG pipelines: treating the number a bi-encoder gives you as if it meant "relevant".

What a bi-encoder score actually is

The embedding model is sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, served locally through fastembed. It turns a text of at most 128 tokens into a 384-dimensional vector. The question is embedded once, every chunk was embedded at index time, and retrieval is the cosine between the two. That is the point of a bi-encoder: question and passage never meet, so the passage side can be precomputed and a query is one matrix multiplication over 124,952 chunks.

The cosine is the angle between two vectors, nothing more. The model was trained so that texts that say roughly the same thing point in roughly the same direction; its name says "paraphrase", not "answers this question". Two consequences follow.

First, the ordering is useful. If chunk A is closer to the question than chunk B, A is more often the better passage. Cosine is good enough for ranking, and the eval numbers (hit@6 of 100 %) confirm that.

Second, the absolute value means very little. Nothing in training says "0.3 is unrelated and 0.7 is relevant". A small multilingual model puts all Dutch legal prose in the same region of the space: same register, overlapping vocabulary, the same sentence structure in every ruling. So the cosine between an arbitrary question and the closest of 124,952 legal paragraphs is high no matter what the question is about. The score ranks well; it is not calibrated.

The measured problem on this corpus

This is not a theoretical concern. The docstring at the top of the reranker module records what I measured:

"""Cross-encoder reranking: a calibrated relevance score for the top candidates.
 
Bi-encoder cosine similarity ranks well but is not calibrated: with a small multilingual
embedder, an unrelated question still scores 0.6 against some Dutch legal paragraph. A
cross-encoder reads question and passage together and answers "is this relevant?"; its
scores separate real hits from noise, which is what the no-source threshold needs.
 
Rerankers return probabilities in [0, 1] (sigmoid of the model logit).
"""

Think of a question about doping in the Tour de France: it has nothing to do with any of the 1,906 rulings in the corpus. Questions like that still ended up with a best cosine around 0.6, because out of 124,952 chunks there is always one that happens to sit at a similar angle. The answerable questions did not score reliably higher than that, so, as the README puts it, a cosine threshold "cannot tell 'no source' from 'source'". Whatever value I picked, it either refused real questions or let unrelated ones through to the model. And once an unrelated question reaches the model with six vaguely legal passages, the model does what models do: it writes something plausible.

What a cross-encoder does differently

A cross-encoder does not embed anything. It takes question and passage as one input sequence and runs a single transformer pass over the pair, so every token of the question can attend to every token of the passage. The output is not a vector but one logit, trained to answer "is this passage relevant to this query?". That is a different task from paraphrase similarity, and it produces a different kind of number.

The model is jinaai/jina-reranker-v2-base-multilingual, again through fastembed. The wrapper is small:

def _sigmoid(x: float) -> float:
    return 1.0 / (1.0 + math.exp(-x))
 
 
class FastEmbedReranker:
    def __init__(self, model: str) -> None:
        ...
        from fastembed.rerank.cross_encoder import TextCrossEncoder
 
        self.name = model
        ...
            self._model = TextCrossEncoder(model_name=model)
 
    def score(self, query: str, texts: Sequence[str]) -> list[float]:
        if not texts:
            return []
        return [_sigmoid(float(logit)) for logit in self._model.rerank(query, list(texts))]

The sigmoid turns the logit into a probability in [0, 1]: a number whose absolute value was trained to mean something. The reranker sees section title and chunk text together, and the candidates are re-sorted by that probability, with cosine only as a tie-breaker:

    def _rerank(self, vraag: str, hits: list[Hit]) -> list[Hit]:
        scores = self.reranker.score(vraag, [f"{h.sectie_titel}\n{h.tekst}" for h in hits])
        scored = [replace(h, relevantie=s) for h, s in zip(hits, scores, strict=True)]
        return sorted(scored, key=lambda h: (-(h.relevantie or 0.0), -h.score))

The catch is cost. A cross-encoder cannot precompute anything, so scoring is one forward pass per (question, passage) pair, and the model is about 560 MB against 120 MB for the embedder. Running it over the whole index is out of the question; running it over a dozen candidates is not. The retriever fuses the cosine top-18 and the BM25 top-18 with reciprocal rank fusion, and only the top 12 of that fused list go through the reranker:

    def __init__(
        self,
        store: VectorStore,
        embedder: Embedder,
        *,
        k: int = 6,
        threshold: float = 0.35,
        hybrid: bool = True,
        per_ecli: int = 2,
        reranker: Reranker | None = None,
        rerank_top: int = 12,
    ) -> None:

On a laptop CPU that stage costs a few seconds per question. Against the alternative, a confidently wrong answer, that is cheap. The bi-encoder does what it is good at (narrowing 124,952 chunks to 12 in milliseconds) and the cross-encoder does what it is good at (judging 12 pairs carefully).

Where the threshold lives now

The threshold did not disappear; it moved. _relevance returns the cross-encoder probability when there is one and falls back to cosine when the reranker is off (RAG_RERANKER=none), so one code path serves both modes and the tests run without downloading a model:

def _relevance(hit: Hit) -> float:
    return hit.relevantie if hit.relevantie is not None else hit.score

The decision itself is a handful of lines in retrieve:

        beste = max((_relevance(h) for h in fused), default=0.0)
        if beste < self.threshold:
            return Retrieval(
                vraag, [], beste, self.threshold, filters, len(fused), soort, herordend, fused[:3]
            )
        # what scores under half the threshold is noise for the model, not a source
        usable = [h for h in fused if _relevance(h) >= self.threshold / 2]
        hits = _diversify(usable, k, self.per_ecli)
        return Retrieval(vraag, hits, beste, self.threshold, filters, len(fused), soort, herordend)

Three things happen here.

The empty retrieval keeps the closest candidates. When the best score is below the threshold (0.35 by default, RAG_SCORE_THRESHOLD), hits is empty but dichtstbij holds the three best candidates. The generator turns this into a GeenBron answer whose reason says what happened:

        if not retrieval.boven_drempel:
            return GeenBron(
                retrieval.vraag,
                reden=(
                    f"beste score {retrieval.beste_score:.2f} ligt onder de drempel "
                    f"{retrieval.drempel:.2f}; het taalmodel is niet aangeroepen"
                ),
                beste_score=retrieval.beste_score,
                drempel=retrieval.drempel,
                kandidaten=retrieval.dichtstbij,
            )

The CLI shows those three under "dichtstbijzijnde passages, ter controle", with scores and ECLIs. A refusal that shows what it almost picked is one you can check; a bare "no source found" is one you have to trust.

The LLM is never called below the threshold. "No source" is a property of retrieval, decided in code, before any prompt exists. The model can still answer GEEN_BRON when the passages do not answer the question, but that is a second safety net, not the primary one.

Half the threshold is the noise floor. Above the threshold the question is answerable, but that does not make all twelve candidates sources. A passage scoring below threshold / 2 is dropped before the model sees it. The test for this uses a threshold of 0.5 and a fixture corpus in which the unrelated rulings score 0; the result has fewer than k hits because the noise is left out rather than padded in. Fewer, better passages beat a context window of filler.

What the eval says

The evaluation set holds 15 questions, and three of them are there specifically to hit the no-source path:

    {"id": "q13", "type": "geen_bron", "vraag": "Wat zegt de rechter over de octrooiaanvraag voor een kernfusiereactor?", "verwacht": [], "geen_bron": true},
    {"id": "q14", "type": "geen_bron", "vraag": "Hoe hoog mag de huurverhoging van een bedrijfspand in Utrecht zijn?", "verwacht": [], "geen_bron": true},
    {"id": "q15", "type": "geen_bron", "vraag": "Welke uitspraken gaan over doping tijdens de Tour de France?", "verwacht": [], "geen_bron": true}

A fusion-reactor patent, commercial rent in Utrecht, doping in the Tour: all plausible Dutch legal questions, none of them criminal law, none in the corpus. The eval scores them on the retrieval outcome:

            geen_bron_gegeven=not retrieval.boven_drempel,
            geen_bron_correct=(not retrieval.boven_drempel) if q.geen_bron else None,

It reports the mirror image too: onterecht_geen_bron, the share of answerable questions that were refused. That number matters as much as the first. A threshold of 0.99 scores 100 % on no-source correctness and refuses everything else; a test asserts exactly that, so a "safe" threshold cannot quietly break the system.

In the committed eval output (docs/eval_results.json, default configuration, k = 6) the three no-source questions end with best relevance scores of 0.266, 0.303 and 0.090. The twelve answerable questions range from 0.412 to 0.775. The threshold of 0.35 sits in the gap: no-source correctness 100 %, false no-source 0 %. Under cosine there was no gap to put a threshold in.

Fifteen questions is a small set and three data points do not prove calibration. What they show is that the cross-encoder's number behaves like a probability of relevance across question types, and the cosine did not.

Takeaways

• A bi-encoder's cosine is trained to order, not to mean. Use it to rank candidates; do not use it to decide whether any candidate is good.
• A cross-encoder's sigmoid output is trained on "relevant or not". That is the number a no-source threshold needs.
• The cost argument works in your favour: the expensive model only ever sees the top 12, so it adds seconds, not minutes, and it never touches the index.
• Put the refusal in code, before the model, and show the closest candidates when you refuse. Then measure both directions: refused when it should, and not refused when it should not.