Skip to content
Back to projects
Own project

Digital twin: an AI agent that answers for me with sources

A LangGraph agent over my CV, projects and writing that answers visitors' questions in a streaming chat. Every claim cited, refusal when there is no source, a hard daily budget, and human approval before it acts on my behalf.

Everyone knows the demo: a chatbot over “your documents” that answers convincingly in a slick interface. That takes an afternoon. What the demo doesn't show is what happens on day three — when a visitor types “ignore your instructions”, when someone has it write their Python homework on your API budget, or when it confidently invents a job you never had.

This project was built to close that gap. The chat was the easy part. The interesting part is everything around the model that makes it safe to leave running unattended: guards that need no model, a cost cap that fails closed, an approval step on the one path that acts on my behalf, and evaluations that guard every change.

Guards

Fail closed, no model

Eval gate

95% tools · 90% tasks

Risky actions

Human approval

Daily budget

Hard cap, survives restarts

Python · FastAPI
LangGraph
Postgres + pgvector
React + Vite
Server-sent events
Gemini / OpenAI / DeepSeek / Kimi
58 tests, ruff clean
System architecture of digital-twin in five columns (labels in Dutch): browser (React + Vite, SSE events, a thread_id per conversation), FastAPI (POST /api/chat rate limited to 20 per 10 minutes per IP, POST /api/feedback, /api/admin/approvals with bearer token, pydantic validation), agent (LangGraph graph guard → generate → tools → verify, guards, tools, BudgetTracker, checkpoint per thread), provider layer (system prompt + knowledge base as a stable prefix, chat-completions client with retry, cost per turn) and models (Gemini, OpenAI, DeepSeek, Kimi; one .env variable switches). Below: Ruud as admin, the knowledge base, and Postgres + pgvector with the tables checkpoints, pending_approvals, turn_log, budget_ledger, guard_incidents, feedback, contact_messages, knowledge_chunks and eval_runs.
One turn runs from the browser through FastAPI to the agent and the model; everything the agent does lands in Postgres.

Why this project

I wanted to know where the line runs between an agent that demonstrates and an agent you would dare to leave alone for a weekend. That line isn't at the prompt and isn't at the model. It's at the questions every organisation meets as soon as an assistant over its own knowledge goes public: how do you guarantee that answers are grounded, how do you stop it spending money or acting without oversight, and how do you know a change made it better rather than just different?

My own CV, project write-ups and blog posts make a good proving ground: small enough to fit in a system prompt, precise enough to check every claim, and with one action — drafting a message to me — that actually does something and can therefore actually go wrong.

The design

The chain is deliberately thin. A small React app talks to FastAPI over server-sent events; FastAPI hands the turn to a LangGraph agent; the agent touches a model exactly once per step through an OpenAI-compatible client; and everything that happens is recorded in Postgres. There is no framework magic in between: routing, tool execution and verification are plain Python.

The model provider is a configuration switch. The same client talks to Gemini, OpenAI, DeepSeek or Kimi; one variable in .env decides which. That isn't a luxury but risk control: the knowledge base and the guards are mine, the model is replaceable.

Three layers, each testable on its own

01

Browser and API

The client sends a thread_id and the full history with every conversation, and reads back a stream of SSE events: text, citations, tool metadata, trace, approval status, done. FastAPI validates every payload with pydantic, limits to 20 requests per 10 minutes per IP address, and offers a feedback endpoint and an admin endpoint for approvals that stays off as long as no bearer token is configured.

02

Agent and provider layer

The graph guard → generate → tools → verify, with three tools (search the knowledge base, report availability, draft a contact message) and a BudgetTracker with a daily cap in dollars. The provider layer puts system prompt and knowledge base into every call as a stable prefix, parses the model's SSE stream, and only retries as long as nothing has been sent to the visitor yet.

03

Postgres as memory and log

Checkpoints per thread, the approval queue, a turn log with cost and latency, a budget ledger, guard incidents, feedback and unanswered questions, and the knowledge base as chunks with embeddings in pgvector. Without DATABASE_URL everything runs in memory — handy for tests, and nothing then survives a restart.

The agent graph: one turn, from question to answer

One chat turn is a small state machine. Of its six nodes exactly one talks to a model; the other five are deterministic Python and can simply be tested with a fake model. That is the design choice the rest of the project follows from.

The agent graph as a flowchart (labels in Dutch): START → guard_input (daily budget reached? prompt injection? card or account number? clearly off-topic?). If a guard fires → refuse (polite refusal without spending a single token) → END. Otherwise → generate, the only node that calls a model; it streams text, collects citations and adds up tokens and cost. A regular tool goes to execute_tools (deterministic Python with a timeout), a high-risk tool goes to request_approval (interrupt: the graph pauses and is checkpointed, the visitor is notified) and only after Ruud's decision to execute_tools. Tool results go back to generate, at most three rounds. No tools → verify (cost booked to the budget, a long answer without citation is flagged, trace to the client) → END.
One node talks to a model; routing, guards, tool execution and verification are plain Python.

What each node does

guard_input

Screen before a single token is spent

Is the daily budget reached? Are there prompt-injection markers in the input? A card or account number? A request that is clearly off-topic (“write me a script”)? Each check is a few lines of Python without a model, costs nothing and can't crash. The node also resets the per-run state, so a previous turn never leaks through.

refuse

Polite refusal as a first-class outcome

A fired guard ends in a tidy redirect — “that's not in my knowledge base, ask Ruud directly” — and in a row in guard_incidents. The model was not called. Refusing isn't an error path but one of the tested answers.

generate

The only node that calls a model

Receives the system prompt plus the full knowledge base as a stable prefix, streams text to the visitor as it arrives, collects citations from the answer and adds up tokens and cost — real numbers from the provider's usage report, cache hits included.

execute_tools

Deterministic execution with a timeout

Three tools, each with a strict JSON schema and a timeout. Arguments are always parsed, never matched on text. A tool that fails returns a clean error instead of crashing the turn. A high-risk tool without approval is refused here, not executed.

request_approval

Pause until a human has looked

If the model reaches for draft_contact_message, the graph calls interrupt(): the state is checkpointed, the request goes into pending_approvals, and the visitor is told it's waiting for review. The thread stays put until I've made a decision.

verify

Post-check and bookkeeping

Books the turn's cost against the daily budget, flags a long and assertive answer without a single citation as a hallucination risk, and sends the trace — the latency per node — to the browser as a diagnostic event.

Tool results go back to generate, but not forever. After three tool rounds, tool_choice is set to “none”, so the model has to answer in text — even if it keeps insisting. Every node writes its own latency into the trace, so when a turn is slow, you can see where.

The model is the only component that can improvise. Everything before and after it is deterministic Python — and fails closed: when in doubt, no answer instead of a guess.

Nothing irreversible without a human

Of the three tools, one acts on my behalf: drafting a contact message. It is marked high-risk, and that label isn't an instruction in the prompt but a property of the tool that the graph enforces. The path below is the reason LangGraph is in this project — an interrupt that durably persists the state and later picks it up again, in another process, is not something you want to build yourself.

Sequence diagram of the approval flow (labels in Dutch). Visitor → browser: “Can you pass this on to Ruud?”. Browser → FastAPI: POST /api/chat with thread_id. FastAPI → agent: astream. guard_input: no guard fired. generate → model, which returns a tool call draft_contact_message. Note: high-risk tool, never without approval. request_approval calls interrupt(), the thread is checkpointed in Postgres and put into pending_approvals. FastAPI sends SSE “submitted for approval” plus approval pending and done; the browser shows a notice. Later — the thread stays paused. Ruud → FastAPI: POST /api/admin/approvals/{thread}/decision with bearer token. FastAPI → agent: Command(resume = approved). The agent loads the checkpoint, execute_tools now runs, generate finishes the answer, contact_messages and turn_log are written, and Ruud gets 200 — decision processed.
The visitor gets an answer right away; the action itself waits for my decision, even if that comes days later.

What exactly happens

  1. 01

    The model asks for draft_contact_message. The graph sees the risk label and goes to request_approval instead of execute_tools.

  2. 02

    interrupt() pauses the graph. The full state is checkpointed in Postgres and the request lands in pending_approvals.

  3. 03

    The visitor is told over SSE that the message has been submitted for approval — the turn ends cleanly, without anything being sent.

  4. 04

    I approve or reject through the admin endpoint, with a bearer token. A Command(resume) loads the checkpoint and lets the graph continue where it stood.

  5. 05

    Only now does execute_tools run. generate finishes the answer, contact_messages and turn_log are written.

Without a bearer token the admin endpoint is off. Then nothing can be approved — and therefore nothing sent.

The question with an agent isn't “can it do this?”, but “what happens when it shouldn't have?”

What stands around the model

The same rule applies on every layer: when in doubt, no answer instead of a guess. Three layers sit before the model and cost no tokens at all. Two sit after it and look at what the model said — without asking the model whether it was right.

Citations are native to the answer: for every claim the model names the document it came from. The part that matters comes after. A citation is only passed on if the named title really exists in the knowledge base; otherwise it is silently dropped and flagged. And a long, assertive answer without a single citation and without a tool result is logged for manual review. The model doesn't get to decide whether it was grounded.

0

tokens spent before transport, budget and input guards have been passed

20

requests per 10 minutes per IP address; at most 40 turns per conversation

3

tool rounds per turn; after that the model has to answer in text

Five layers around the model, left to right (labels in Dutch): 1 transport (20 requests per 10 minutes per IP, pydantic validation on role, length, at most 40 turns, last turn from the visitor), 2 budget (daily cap in dollars, reached means polite refusal before a token is spent, survives a restart), 3 input guards (prompt injection, card and account numbers, off-topic; plain Python, no model judging itself), the model (system prompt plus full knowledge base in a fixed order; the only component that can improvise), 4 citation check (only pass on if the title really exists in the knowledge base, otherwise drop silently and flag) and 5 groundedness (a long, assertive answer without citation and without tool result is flagged for manual review). Below: the tool contract, what is recorded per turn, and the evals with thresholds 95% and 90% and failure injection.
Three layers before the model cost no token; two layers after it check without asking the model.

Five layers, outside in

01

Transport

Rate limiting per IP address and pydantic validation on every payload: role, length, at most 40 turns, and the last turn must be the visitor's. A public endpoint that calls an LLM is otherwise an open wallet.

02

Budget

A daily cap in dollars. Once reached, the agent politely refuses until midnight — before a single token is spent. The ledger lives in Postgres, so a restart doesn't reset the meter.

03

Input guards

Prompt injection, card and account numbers, clearly off-topic requests. Plain Python — no model judging itself, and so no model that can be talked around.

04

Citation check

Every citation in the answer is compared with the titles in the knowledge base. What doesn't exist doesn't reach the visitor and is flagged. An invented source thereby becomes a measurable event, not a surprise.

05

Groundedness

A long, factual-sounding answer without any citation and without a tool result is suspect. It is flagged as a hallucination risk and logged, so I can check it and, if it's structural, add it to the evaluation set.

The tool contract

  • Strict JSON schema per tool: every property required, no extra fields.

  • Arguments are always parsed, never matched on text.

  • Every tool has a timeout; a tool that fails returns a clean error instead of crashing the turn.

  • draft_contact_message is high-risk and never runs without explicit approval.

Recorded per turn: outcome, tokens, cost and latency in turn_log, plus guard_incidents, unanswered_questions, budget_ledger and visitor feedback.

Knowledge base and data platform

The knowledge base is small and deliberately kept that way: CV, projects, skills and an about-me, as Python modules in knowledge/. At startup they are loaded in a fixed order as markdown into the system prompt — a stable prefix, so follow-up turns consist largely of cached tokens.

The same documents are chunked by heading, hashed and embedded into pgvector. The sync is incremental: only paragraphs whose hash changed are re-embedded. The search tool ranks by cosine distance, so a question phrased differently from the document still finds the right passage.

Everything the agent does lives in Postgres — with alembic migrations, so the schema is in git just like the code. Without DATABASE_URL the whole thing runs in memory; then nothing survives a restart, and that is exactly the difference between a test and production.

The unanswered questions are the most useful table of all: that's where you see what visitors wanted to know and therefore what the knowledge base is missing.

Tables in Postgres
checkpointsLangGraph state per thread — the pause at an approval survives a restart
pending_approvalsqueue of high-risk actions, with decision and timestamp
turn_logevery turn with outcome, model, tokens, cache hits, cost and latency
budget_ledgerdaily cap that survives a restart
guard_incidentswhich guard fired when, and on what
feedback · unanswered_questionsthumbs up or down per answer, and questions the knowledge base had no answer to
contact_messagesapproved messages to me
knowledge_chunksknowledge base per heading, with hash and embedding, updated incrementally
eval_runs · eval_casesevery evaluation run kept, so quality can be followed over time

Quality is measured, not hoped for

“Does it work?” is a vague question for an agent. It has been replaced by two labelled datasets that run against the real graph, with a threshold below which the run fails.

WhatGateMeaning
Tool selection12 cases · ≥ 95%does the agent reach for the right tool — and never a high-risk one it shouldn't
Task completion11 cases · ≥ 90%does the answer contain what it must, cite when it must, refuse when it must
Failure injectionevery 2nd requestevery other HTTP request is dropped; the client's retries must absorb it, or the run fails
Unit and integration58 testsguards, budget, tool contract, graph routing and the SSE stream, with a fake model
Lintruff, every pushCI runs on every push via GitHub Actions
Historyeval_runsevery run is kept, so a change can be compared with the previous one

The task-completion cases deliberately include questions where the right answer is a refusal. Refusing is measured as thoroughly as answering — otherwise the agent is eventually rewarded for inventing a source.

The failure-injection mode exists because a streaming endpoint can fail in two ways: before anything has been sent, and midway. The provider layer only retries in the first case; in the second, the visitor gets a clean abort instead of half an answer that starts twice.

What the checks do and don't guarantee

A guard that looks impressive but promises more than it delivers is more dangerous than no guard. So what each layer does is written down precisely.

01

The citation check verifies existence, not content

A citation only passes if the named title really exists in the knowledge base. Whether the claim is actually in that document, the layer doesn't check. That catches invented sources, not misattributed claims; the task-completion cases are there for those.

02

Groundedness flags, it doesn't block

A long answer without a citation is logged for manual review, not stopped. A hard block would hit too many good answers — “no, I don't know anything about that” has no citation either. The choice is deliberate: measure first, block only once the data shows it's needed.

03

The input guards are patterns, not a classifier

Prompt injection and off-topic requests are screened on patterns. That's fast, deterministic and can't be talked around, but it isn't a complete defence. The real defence is that the model can't do anything irreversible: the only tool that acts waits for me.

04

In-memory is not production

Without DATABASE_URL everything works, but nothing survives a restart — not the budget, not the approval queue. That's kept on purpose for tests and local runs; in production Postgres isn't an option but a requirement.

Design choices

A few trade-offs I thought about the longest.

01

Guards without a model

It's tempting to ask the model whether an input is safe. But a model judging itself can be talked around with the same trick as the model that answers. Deterministic Python is duller, costs no tokens, can't crash and can simply be unit-tested.

02

Knowledge base in the prompt and in pgvector

The full knowledge base fits in the system prompt, and thanks to prompt caching that's cheap: the prefix is stable. The pgvector search tool is for targeted questions — it finds the right passage even when the visitor phrases it differently from how I wrote it. Neither alone was enough.

03

Provider as configuration, retries only before streaming

One OpenAI-compatible client over httpx for Gemini, OpenAI, DeepSeek and Kimi. Retry with backoff only applies as long as nothing has been sent to the visitor; an aborted stream is never silently restarted. A cheap default provider plus caching keeps a day of conversations under a couple of dollars — and the cap makes sure of it.

04

LangGraph for the interrupt, not for the chat

For an ordinary chat turn LangGraph is overkill. It's there for one thing: interrupt() with a checkpointer in Postgres, so a paused thread can be resumed days later in another process. Building that yourself is exactly the kind of code that goes subtly wrong.

What this project shows

LLM agent in production: LangGraph, streaming, tools
Deterministic guards and fail-closed design
Human-in-the-loop with durable interrupts
Cost control: budget, caching, per-turn telemetry
Postgres + pgvector as memory and log
Evaluation sets with thresholds and failure injection
FastAPI, SSE and a React client
CI with tests and lint on every push

Every organisation that wants an assistant over its own knowledge runs into the same questions — not “which model”, but how to keep grounding, cost and actions under control. Deterministic guards, a budget that fails closed, human approval on the risky path and evaluations that guard the pipeline transfer one to one.

The assistant speaks about me, never for me: anything that reaches my inbox passes through my approval first.

Background article on the blog: “A digital twin that only says what it can prove” (in Dutch).