Code: labs/10/ · Combines Labs 03–09

Business Problem — 商業問題

Kettle & Kiln now has working pieces:

• listing generator (03)
• support knowledge bot (04)
• refund-capable agent (05)
• n8n automations (06)
• data copilot (07)
• a self-hosting recipe (08)
• a gateway (09)

The capstone turns them into one platform the business can run and trust:

• one docker compose up
• every model call through the gateway, so there is one bill, one audit trail and one place to redact PII
• one public entry point behind Cloudflare Access
• an end-to-end smoke test
• a load test with your numbers
• a production checklist with a named owner for every area

The question to answer at the end is the owner's: "Is this ready for customers, and what will it cost us to run?" Answer it with measurements, not hopes.

Architecture — 系統架構

flowchart TB
    subgraph Internet
        S[Staff browser] & W[Website chat widget] & SH[Shopify / review / form webhooks]
    end
    S & W & SH -->|HTTPS| CF[Cloudflare edge + Access]
    CF -->|tunnel| CFD[cloudflared]
    subgraph Host["One host (TrueNAS / Linux / Mac), no inbound ports"]
        CFD --> CAD[Caddy<br/>/rag /agent /listings /copilot /healthz]
        CFD --> N8N[n8n own hostname]
        CAD --> RAG[rag-api] & AG[agent-api] & LS[listing-api] & CP[copilot-api]
        N8N --> SVC[ai-services]
        RAG & AG & LS & CP & SVC -->|per-app key| GW[AI gateway<br/>auth · limits · budgets · redaction · traces · cost]
        GW --> OL[(Ollama, GPU optional)]
        GW -.optional.-> CLOUD[(Cloud model API)]
        RAG --> QD[(Qdrant)]
        GW --> USE[(usage.sqlite + traces.jsonl)]
    end

What's new compared with Labs 08 and 09:

• Apps never hold a model key. Each runs with LLM_PROVIDER=compat, LLM_BASE_URL=http://gateway:8009/v1 and its own gateway key (KEY_RAG, KEY_AGENT, …), so usage is attributed per app. The gateway alone knows the upstream (UPSTREAM_*).
• The gateway and ai-services are not routed by Caddy. They're reachable only on the internal network.
• n8n gets its own hostname through the tunnel and reuses Lab 06's workflows.

Prerequisites — 先決條件

• All previous labs. For the Docker path, see Lab 08 prerequisites. For the no-Docker smoke test, you only need Python 3.12 and the labs' requirements in one venv.

Step-by-Step Setup — 逐步設定

1. Smoke-test the whole platform without Docker (works on any laptop).

cd hands-on/labs/10
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e ../common -r ../03/requirements.txt -r ../04/requirements.txt -r ../05/requirements.txt \
  -r ../06/requirements.txt -r ../07/requirements.txt -r ../09/requirements.txt -r requirements.txt
python smoke_local.py                                     # offline mock upstream
python smoke_local.py --upstream ollama --model qwen3:8b --embed-model nomic-embed-text   # real model

smoke_local.py does the following:

1. Starts the gateway and the five services as local processes on ports 180xx. Each service talks to the gateway over HTTP with its own key, using the real OpenAI SDK.
2. Runs one business check per service:
   • listing: valid title for KK-MUG-001
   • rag: "Do gift cards expire?" answered from faq.md or refused, grounded
   • agent: KK-20001 damaged teapot → lookup_order used, or escalated
   • automation: T-511 fire ticket → urgent
   • copilot: AOV by channel → 3 rows, no error
3. Checks that the gateway attributed traffic to all five clients.
4. Tears everything down and exits non-zero if anything failed.

2. Run the real stack.

cp .env.example .env               # set UPSTREAM_*, generate every KEY_* and secret
python - <<'PY' > secrets/gateway_keys
import secrets; print(",".join(f"{n}:{secrets.token_urlsafe(24)}" for n in ["rag","agent","listing","copilot","automation"]))
PY
# copy each generated key into the matching KEY_* in .env (same values!)
docker compose --profile local-ollama up -d --build
docker compose exec ollama ollama pull qwen3:8b && docker compose exec ollama ollama pull nomic-embed-text
docker compose ps
bash ../08/scripts/smoke.sh http://127.0.0.1:8080
docker compose exec n8n n8n import:workflow --separate --input=/workflows   # then publish them in the n8n UI
docker compose --profile tunnel up -d          # Tunnel public hostnames: ai.<domain> → http://caddy:80, n8n.<domain> → http://n8n:5678

3. Measure.

python load_test.py --url http://127.0.0.1:8080/rag/v1/ask --concurrency 4 --requests 20
docker compose exec gateway python report.py --days 1

Write the p50/p95 and tokens per request into PRODUCTION_CHECKLIST.md. Those are your numbers.

4. Work through PRODUCTION_CHECKLIST.md (security, privacy, quality gates, reliability, cost, ownership). Every line needs a tick and every area an owner.

How to Test — 測試方法

pytest -q          # 2 tests: the full no-Docker platform smoke (mock), checklist completeness

Config validation (no daemon needed):

cp .env.example .env && touch secrets/gateway_keys
docker compose config -q && docker compose --profile tunnel --profile local-ollama config -q
caddy validate --config caddy/Caddyfile --adapter caddyfile

What was run on the build box — 實測紀錄

• ✅ pytest: 2 passed. smoke_local.py (mock upstream) passed all checks: 5 health checks, 5 e2e checks, gateway attribution to 5 clients. The run took ~3 s.
• ✅ Live: smoke_local.py --upstream ollama --model qwen2.5:1.5b --embed-model nomic-embed-text, CPU only, single run, saved in labs/10/reports/sample-live-qwen2.5-1.5b-cpu.txt. Results:
  • 5/5 services healthy.
  • 4/5 end-to-end checks passed (listing ~23 s, rag ~3 s, agent ~4 s, automation ~1 s).
  • The copilot failed. The 1.5B model added a date filter nobody asked for (WHERE order_date > … - INTERVAL '1' DAY). This is the same weakness as the Lab 07 golden eval, and exactly what the smoke test exists to catch.
  • The gateway saw all 5 clients.
  • Timings are single observations on a shared CPU box, not benchmarks.
• ✅ docker compose config for both profile sets and caddy validate: OK.
• ⚠️ Not run:
  • docker compose up / image builds (no Docker daemon)
  • the tunnel and Access
  • n8n inside this stack (n8n itself was run in Lab 06)
  • load_test.py against a real stack, so no latency numbers are claimed
  • GPU and cloud upstreams

Cost & Performance Notes — 成本與效能

Build your monthly estimate from measured inputs. Label it as an estimate.

monthly_cloud_estimate = Σ_features requests × tokens/request ÷ 1e6 × price_per_mtok. Put the sum next to the value side (support hours saved, listings shipped, faster lead replies) before deciding cloud vs local vs hybrid. A common pattern:

• local for PII-heavy and high-volume tasks
• cloud for the few tasks where quality clearly pays

The gateway lets you switch per feature without touching apps. Treat this as a pattern to evaluate, not a recommendation for your numbers.

Performance: the model server is the bottleneck. Scale by:

1. a GPU
2. a smaller or quantised model where the evals still pass
3. OLLAMA_NUM_PARALLEL if VRAM allows
4. moving low-priority work (reviews, listings) to off-hours batches through n8n schedules

Common Failures & Fixes — 常見錯誤與修正

Stretch Exercise — 延伸練習

1. Hybrid routing in the gateway: route X-Feature: listing-gen to a cloud model and everything carrying PII to local. Re-run the Lab 02/04/07 evals through the gateway for both routes, and compare quality, latency and (estimated) cost in one table.
2. CI for the platform: a GitHub Actions workflow that runs every lab's pytest in mock mode, labs/08/tests/validate.sh, and labs/10/smoke_local.py on each push.

Deliverable Checklist — 交付清單

• [ ] python smoke_local.py passes (mock) and on your real upstream
• [ ] Full stack running with docker compose ps all healthy; smoke.sh passes through the tunnel
• [ ] Load-test p50/p95 recorded from your hardware
• [ ] One-page cost estimate built from report.py tokens and dated prices (labelled as an estimate)
• [ ] PRODUCTION_CHECKLIST.md fully ticked with owners
• [ ] A 5-minute demo: new SKU listing → policy question → refund for KK-20001 → urgent ticket alert → revenue question, all visible in report.py