A LangGraph agent over my CV, projects and writing that answers visitors' questions in a streaming chat. Every claim cited, refusal when there is no source, a hard daily budget, and human approval before it acts on my behalf.
Everyone knows the demo: a chatbot over “your documents” that answers convincingly in a slick interface. That takes an afternoon. What the demo doesn't show is what happens on day three — when a visitor types “ignore your instructions”, when someone has it write their Python homework on your API budget, or when it confidently invents a job you never had.
This project was built to close that gap. The chat was the easy part. The interesting part is everything around the model that makes it safe to leave running unattended: guards that need no model, a cost cap that fails closed, an approval step on the one path that acts on my behalf, and evaluations that guard every change.
Fail closed, no model
95% tools · 90% tasks
Human approval
Hard cap, survives restarts
Why this project
I wanted to know where the line runs between an agent that demonstrates and an agent you would dare to leave alone for a weekend. That line isn't at the prompt and isn't at the model. It's at the questions every organisation meets as soon as an assistant over its own knowledge goes public: how do you guarantee that answers are grounded, how do you stop it spending money or acting without oversight, and how do you know a change made it better rather than just different?
My own CV, project write-ups and blog posts make a good proving ground: small enough to fit in a system prompt, precise enough to check every claim, and with one action — drafting a message to me — that actually does something and can therefore actually go wrong.
The design
The chain is deliberately thin. A small React app talks to FastAPI over server-sent events; FastAPI hands the turn to a LangGraph agent; the agent touches a model exactly once per step through an OpenAI-compatible client; and everything that happens is recorded in Postgres. There is no framework magic in between: routing, tool execution and verification are plain Python.
The model provider is a configuration switch. The same client talks to Gemini, OpenAI, DeepSeek or Kimi; one variable in .env decides which. That isn't a luxury but risk control: the knowledge base and the guards are mine, the model is replaceable.
Three layers, each testable on its own
01
Browser and API
The client sends a thread_id and the full history with every conversation, and reads back a stream of SSE events: text, citations, tool metadata, trace, approval status, done. FastAPI validates every payload with pydantic, limits to 20 requests per 10 minutes per IP address, and offers a feedback endpoint and an admin endpoint for approvals that stays off as long as no bearer token is configured.
02
Agent and provider layer
The graph guard → generate → tools → verify, with three tools (search the knowledge base, report availability, draft a contact message) and a BudgetTracker with a daily cap in dollars. The provider layer puts system prompt and knowledge base into every call as a stable prefix, parses the model's SSE stream, and only retries as long as nothing has been sent to the visitor yet.
03
Postgres as memory and log
Checkpoints per thread, the approval queue, a turn log with cost and latency, a budget ledger, guard incidents, feedback and unanswered questions, and the knowledge base as chunks with embeddings in pgvector. Without DATABASE_URL everything runs in memory — handy for tests, and nothing then survives a restart.
The agent graph: one turn, from question to answer
One chat turn is a small state machine. Of its six nodes exactly one talks to a model; the other five are deterministic Python and can simply be tested with a fake model. That is the design choice the rest of the project follows from.
What each node does
guard_inputScreen before a single token is spent
Is the daily budget reached? Are there prompt-injection markers in the input? A card or account number? A request that is clearly off-topic (“write me a script”)? Each check is a few lines of Python without a model, costs nothing and can't crash. The node also resets the per-run state, so a previous turn never leaks through.
refusePolite refusal as a first-class outcome
A fired guard ends in a tidy redirect — “that's not in my knowledge base, ask Ruud directly” — and in a row in guard_incidents. The model was not called. Refusing isn't an error path but one of the tested answers.
generateThe only node that calls a model
Receives the system prompt plus the full knowledge base as a stable prefix, streams text to the visitor as it arrives, collects citations from the answer and adds up tokens and cost — real numbers from the provider's usage report, cache hits included.
execute_toolsDeterministic execution with a timeout
Three tools, each with a strict JSON schema and a timeout. Arguments are always parsed, never matched on text. A tool that fails returns a clean error instead of crashing the turn. A high-risk tool without approval is refused here, not executed.
request_approvalPause until a human has looked
If the model reaches for draft_contact_message, the graph calls interrupt(): the state is checkpointed, the request goes into pending_approvals, and the visitor is told it's waiting for review. The thread stays put until I've made a decision.
verifyPost-check and bookkeeping
Books the turn's cost against the daily budget, flags a long and assertive answer without a single citation as a hallucination risk, and sends the trace — the latency per node — to the browser as a diagnostic event.
Tool results go back to generate, but not forever. After three tool rounds, tool_choice is set to “none”, so the model has to answer in text — even if it keeps insisting. Every node writes its own latency into the trace, so when a turn is slow, you can see where.
The model is the only component that can improvise. Everything before and after it is deterministic Python — and fails closed: when in doubt, no answer instead of a guess.
Nothing irreversible without a human
Of the three tools, one acts on my behalf: drafting a contact message. It is marked high-risk, and that label isn't an instruction in the prompt but a property of the tool that the graph enforces. The path below is the reason LangGraph is in this project — an interrupt that durably persists the state and later picks it up again, in another process, is not something you want to build yourself.
What exactly happens
01
The model asks for draft_contact_message. The graph sees the risk label and goes to request_approval instead of execute_tools.
02
interrupt() pauses the graph. The full state is checkpointed in Postgres and the request lands in pending_approvals.
03
The visitor is told over SSE that the message has been submitted for approval — the turn ends cleanly, without anything being sent.
04
I approve or reject through the admin endpoint, with a bearer token. A Command(resume) loads the checkpoint and lets the graph continue where it stood.
05
Only now does execute_tools run. generate finishes the answer, contact_messages and turn_log are written.
Without a bearer token the admin endpoint is off. Then nothing can be approved — and therefore nothing sent.
The question with an agent isn't “can it do this?”, but “what happens when it shouldn't have?”
What stands around the model
The same rule applies on every layer: when in doubt, no answer instead of a guess. Three layers sit before the model and cost no tokens at all. Two sit after it and look at what the model said — without asking the model whether it was right.
Citations are native to the answer: for every claim the model names the document it came from. The part that matters comes after. A citation is only passed on if the named title really exists in the knowledge base; otherwise it is silently dropped and flagged. And a long, assertive answer without a single citation and without a tool result is logged for manual review. The model doesn't get to decide whether it was grounded.
0
tokens spent before transport, budget and input guards have been passed
20
requests per 10 minutes per IP address; at most 40 turns per conversation
3
tool rounds per turn; after that the model has to answer in text
Five layers, outside in
01
Transport
Rate limiting per IP address and pydantic validation on every payload: role, length, at most 40 turns, and the last turn must be the visitor's. A public endpoint that calls an LLM is otherwise an open wallet.
02
Budget
A daily cap in dollars. Once reached, the agent politely refuses until midnight — before a single token is spent. The ledger lives in Postgres, so a restart doesn't reset the meter.
03
Input guards
Prompt injection, card and account numbers, clearly off-topic requests. Plain Python — no model judging itself, and so no model that can be talked around.
04
Citation check
Every citation in the answer is compared with the titles in the knowledge base. What doesn't exist doesn't reach the visitor and is flagged. An invented source thereby becomes a measurable event, not a surprise.
05
Groundedness
A long, factual-sounding answer without any citation and without a tool result is suspect. It is flagged as a hallucination risk and logged, so I can check it and, if it's structural, add it to the evaluation set.
The tool contract
Strict JSON schema per tool: every property required, no extra fields.
Arguments are always parsed, never matched on text.
Every tool has a timeout; a tool that fails returns a clean error instead of crashing the turn.
draft_contact_message is high-risk and never runs without explicit approval.
Recorded per turn: outcome, tokens, cost and latency in turn_log, plus guard_incidents, unanswered_questions, budget_ledger and visitor feedback.
Knowledge base and data platform
The knowledge base is small and deliberately kept that way: CV, projects, skills and an about-me, as Python modules in knowledge/. At startup they are loaded in a fixed order as markdown into the system prompt — a stable prefix, so follow-up turns consist largely of cached tokens.
The same documents are chunked by heading, hashed and embedded into pgvector. The sync is incremental: only paragraphs whose hash changed are re-embedded. The search tool ranks by cosine distance, so a question phrased differently from the document still finds the right passage.
Everything the agent does lives in Postgres — with alembic migrations, so the schema is in git just like the code. Without DATABASE_URL the whole thing runs in memory; then nothing survives a restart, and that is exactly the difference between a test and production.
The unanswered questions are the most useful table of all: that's where you see what visitors wanted to know and therefore what the knowledge base is missing.
| checkpoints | LangGraph state per thread — the pause at an approval survives a restart |
|---|---|
| pending_approvals | queue of high-risk actions, with decision and timestamp |
| turn_log | every turn with outcome, model, tokens, cache hits, cost and latency |
| budget_ledger | daily cap that survives a restart |
| guard_incidents | which guard fired when, and on what |
| feedback · unanswered_questions | thumbs up or down per answer, and questions the knowledge base had no answer to |
| contact_messages | approved messages to me |
| knowledge_chunks | knowledge base per heading, with hash and embedding, updated incrementally |
| eval_runs · eval_cases | every evaluation run kept, so quality can be followed over time |
Quality is measured, not hoped for
“Does it work?” is a vague question for an agent. It has been replaced by two labelled datasets that run against the real graph, with a threshold below which the run fails.
| What | Gate | Meaning |
|---|---|---|
| Tool selection | 12 cases · ≥ 95% | does the agent reach for the right tool — and never a high-risk one it shouldn't |
| Task completion | 11 cases · ≥ 90% | does the answer contain what it must, cite when it must, refuse when it must |
| Failure injection | every 2nd request | every other HTTP request is dropped; the client's retries must absorb it, or the run fails |
| Unit and integration | 58 tests | guards, budget, tool contract, graph routing and the SSE stream, with a fake model |
| Lint | ruff, every push | CI runs on every push via GitHub Actions |
| History | eval_runs | every run is kept, so a change can be compared with the previous one |
The task-completion cases deliberately include questions where the right answer is a refusal. Refusing is measured as thoroughly as answering — otherwise the agent is eventually rewarded for inventing a source.
The failure-injection mode exists because a streaming endpoint can fail in two ways: before anything has been sent, and midway. The provider layer only retries in the first case; in the second, the visitor gets a clean abort instead of half an answer that starts twice.
What the checks do and don't guarantee
A guard that looks impressive but promises more than it delivers is more dangerous than no guard. So what each layer does is written down precisely.
01
The citation check verifies existence, not content
A citation only passes if the named title really exists in the knowledge base. Whether the claim is actually in that document, the layer doesn't check. That catches invented sources, not misattributed claims; the task-completion cases are there for those.
02
Groundedness flags, it doesn't block
A long answer without a citation is logged for manual review, not stopped. A hard block would hit too many good answers — “no, I don't know anything about that” has no citation either. The choice is deliberate: measure first, block only once the data shows it's needed.
03
The input guards are patterns, not a classifier
Prompt injection and off-topic requests are screened on patterns. That's fast, deterministic and can't be talked around, but it isn't a complete defence. The real defence is that the model can't do anything irreversible: the only tool that acts waits for me.
04
In-memory is not production
Without DATABASE_URL everything works, but nothing survives a restart — not the budget, not the approval queue. That's kept on purpose for tests and local runs; in production Postgres isn't an option but a requirement.
Design choices
A few trade-offs I thought about the longest.
01
Guards without a model
It's tempting to ask the model whether an input is safe. But a model judging itself can be talked around with the same trick as the model that answers. Deterministic Python is duller, costs no tokens, can't crash and can simply be unit-tested.
02
Knowledge base in the prompt and in pgvector
The full knowledge base fits in the system prompt, and thanks to prompt caching that's cheap: the prefix is stable. The pgvector search tool is for targeted questions — it finds the right passage even when the visitor phrases it differently from how I wrote it. Neither alone was enough.
03
Provider as configuration, retries only before streaming
One OpenAI-compatible client over httpx for Gemini, OpenAI, DeepSeek and Kimi. Retry with backoff only applies as long as nothing has been sent to the visitor; an aborted stream is never silently restarted. A cheap default provider plus caching keeps a day of conversations under a couple of dollars — and the cap makes sure of it.
04
LangGraph for the interrupt, not for the chat
For an ordinary chat turn LangGraph is overkill. It's there for one thing: interrupt() with a checkpointer in Postgres, so a paused thread can be resumed days later in another process. Building that yourself is exactly the kind of code that goes subtly wrong.
What this project shows
Every organisation that wants an assistant over its own knowledge runs into the same questions — not “which model”, but how to keep grounding, cost and actions under control. Deterministic guards, a budget that fails closed, human approval on the risky path and evaluations that guard the pipeline transfer one to one.
The assistant speaks about me, never for me: anything that reaches my inbox passes through my approval first.
Background article on the blog: “A digital twin that only says what it can prove” (in Dutch).