Code: labs/04/ · Uses: data/docs/*.md (7 policy docs), data/eval/support_qa.jsonl (15 questions)

Business Problem — 商業問題

Half of Kettle & Kiln's support email asks questions the policies already answer:

• "Do gift cards expire?"
• "How much is shipping to Taiwan?"
• "Can I return a clearance mug?"

Staff answer from memory and sometimes get it wrong. A generic chatbot is worse: it confidently invents a 60-day return window.

We need a bot that answers only from our documents, cites the policy section, and says "I'm not sure — I'll pass this to our team" when the documents don't cover the question. And we need a number that tells us whether it's good enough to put in front of customers.

Architecture — 系統架構

flowchart TB
    subgraph Ingest["Ingest (startup or POST /v1/ingest)"]
        D[data/docs/*.md] --> CH[chunking.py<br/>split on headings, stable ids]
        CH --> EM1[embed with doc prefix] --> VS[(Qdrant<br/>or pgvector)]
    end
    subgraph Ask["POST /v1/ask"]
        Q[question] --> EM2[embed with query prefix] --> VS
        Q --> BM[BM25 keyword search]
        VS --> RRF[Reciprocal-rank fusion]
        BM --> RRF
        RRF --> TH{best vector score ≥ RAG_MIN_SCORE?}
        TH -- no --> REF[refusal]
        TH -- yes --> LLM[LLM → JSON: answer, citations, can_answer]
        LLM --> GC{grounding check:<br/>cites only real passages}
        GC --> OUT[answer + citations + retrieved]
    end

Design choices:

• Heading-aware chunks. Policies are written in sections, so we split on # headings (max CHUNK_MAX_CHARS = 900, with overlap). Each rule stays with its heading, and the heading is prepended to the chunk text. Chunk ids are stable (doc#heading-slug-n), so re-ingesting only re-embeds changed chunks.
• Hybrid retrieval. Vector search misses exact terms ("RMA", "VAT", SKU codes); BM25 catches them. Reciprocal-rank fusion (1/(60+rank)) combines both without calibrating their scores against each other.
• Structured, checkable citations. The model returns JSON {answer, citations: [passage numbers], can_answer}. Code checks that every cited number refers to a passage that was actually retrieved; an answer with no valid citation is marked grounded: false.
• Collection per embedding model. The collection name includes the embedding model and dimension. Switching models can't mix incompatible vectors.

Prerequisites — 先決條件

• Labs 01–03. An embedding model: ollama pull nomic-embed-text.
• Qdrant via Docker (docker compose up -d qdrant), or no Docker at all: leave QDRANT_URL empty and Qdrant runs embedded in-process at QDRANT_PATH.
• Optional: Postgres with pgvector (docker compose --profile pgvector up -d pg, VECTOR_BACKEND=pgvector, PG_DSN=...).

Step-by-Step Setup — 逐步設定

cd hands-on/labs/04
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e ../common -r requirements.txt
cp .env.example .env
docker compose up -d qdrant          # or: set QDRANT_URL= (empty) for embedded mode
python -m app.ingest                 # {"chunks": N, "embedded": N, ...}; run again → 0 re-embedded
uvicorn app.main:app --port 8004

Ask:

curl -s localhost:8004/v1/ask -H 'content-type: application/json' \
  -d '{"question":"Do gift cards expire?"}' | python -m json.tool
curl -s localhost:8004/v1/ask -H 'content-type: application/json' \
  -d '{"question":"What is your CEO'\''s favourite film?"}' | python -m json.tool   # should refuse

Evaluate:

python eval_rag.py                   # reports/rag-<provider>-<time>.md/.json

The eval reports four metrics:

• hit@k: the expected source doc was retrieved.
• cite_ok: the answer cited the expected doc, or correctly refused.
• keywords: the must-include facts are in the answer.
• grounded: every non-refusal answer cites only real passages.

Calibrate RAG_MIN_SCORE for your embedding model. Cosine scores are not comparable across models. Print retrieved[].vector_score for 10 on-topic and 10 off-topic questions, then set the threshold between the two groups. If the groups overlap (they did on the build box with nomic-embed-text on short policy docs), don't rely on the threshold alone; the model's can_answer=false and the grounding check are the second and third lines of defence.

Using nomic-embed-text? Keep the task prefixes from .env.example (search_query: / search_document:). That model is trained with them. Remove them for other embedding models.

How to Test — 測試方法

LLM_PROVIDER=mock pytest -q          # 8 tests

The tests cover:

• stable chunk ids and headings
• incremental ingest
• an answer citing the right policy
• refusing an off-topic question
• the eval harness
• collection names separating embedding models
• the pgvector backend (runs a real Postgres via pgserver)
• the API end to end

What was run on the build box — 實測紀錄

• ✅ pytest: 8 passed, including pgvector against a real embedded Postgres + pgvector.
• ✅ Live eval, Ollama qwen2.5:1.5b + nomic-embed-text, CPU, embedded Qdrant. Single run, not a benchmark; report in labs/04/reports/sample-run-qwen2.5-1.5b-cpu/:
  • hit@k 100% · cite_ok 87% · keywords 73% · grounded 100% (n=15).
  • Retrieval was perfect. The misses were the small model refusing answerable questions (q06, q11) and paraphrasing away required facts. That points to the generator model, not the index.
• ⚠️ Not run: the Qdrant server container and the rag-api Docker image (no Docker daemon); the embedded Qdrant and pgvector paths were run.

Cost & Performance Notes — 成本與效能

• Ingest cost is one embedding call per chunk, and only for changed chunks thanks to stable ids. For 7 short docs that is a few dozen chunks.
• Each question costs: one query embedding, plus RAG_TOP_K × ~900 chars of context in the prompt, plus the answer. Prompt tokens dominate. Lowering RAG_TOP_K from 4 to 3 cuts context by about a quarter (an estimate from the arithmetic; measure with the usage field).
• BM25 runs in memory over all chunks. That's fine for hundreds or thousands of chunks; for a much larger corpus, use the vector DB's own sparse/hybrid search.
• Embedded Qdrant (QDRANT_PATH) is single-process. Use the server (QDRANT_URL) as soon as more than one worker or service reads the index.

Common Failures & Fixes — 常見錯誤與修正

Stretch Exercise — 延伸練習

1. Add a reranking step: retrieve 10, ask the model (or a cross-encoder) to score relevance, keep 4. Compare cite_ok and latency in eval_rag.py.
2. Answer in the customer's language: detect zh-TW questions, keep retrieval on the English docs, and add two zh-TW cases to support_qa.jsonl.

Deliverable Checklist — 交付清單

• [ ] Index built (Qdrant or pgvector), re-ingest shows 0 re-embedded
• [ ] RAG_MIN_SCORE calibrated, with the on/off-topic scores you measured
• [ ] Eval report on your production models with hit@k, cite_ok, keywords, grounded
• [ ] Off-topic question refused; screenshot of /docs answer with citations
• [ ] pytest -q passes