Code: labs/09/ · Every earlier lab can point at it with LLM_PROVIDER=compat

Business Problem — 商業問題

After Labs 03–07 the store has five services and n8n all calling models directly, each with its own API key in its own .env. Three months in, the owner asks:

• "What are we spending on AI, per feature?"
• "Did the support bot send customer emails to a cloud provider?"
• "Why did the listing job make 4,000 calls on Sunday?"

Nobody can answer.

The fix is one internal AI gateway: every app calls it with its own key, and the gateway handles in one place:

• authentication and per-client rate limits
• daily budgets (USD and tokens)
• reversible PII pseudonymisation before anything goes to a cloud model
• tracing (JSONL locally, optionally Langfuse)
• a usage and cost report by client and feature

It's OpenAI-compatible, so apps change one URL and one key.

Architecture — 系統架構

flowchart LR
    A1[listing-api] & A2[rag-api] & A3[agent-api] & A4[copilot] & A5[n8n / ai-services] -->|Bearer client-key<br/>X-Feature| GW
    subgraph GW["Gateway :8009 (/v1/chat/completions, /v1/embeddings, /v1/usage)"]
        AU[auth: sha256 key → client] --> RL[token-bucket rate limit per client]
        RL --> BU[daily budget check<br/>USD + tokens]
        BU --> RD[PII → placeholders EMAIL_1, PHONE_1, CARD_1, ADDRESS_1<br/>when upstream is cloud]
        RD --> UP[llmkit → upstream]
        UP --> RS[restore tokens in reply<br/>cards only as last 4]
        UP --> TR[trace JSONL / Langfuse]
        UP --> US[(usage SQLite<br/>tokens, latency, cost)]
    end
    UP --> M[(Ollama / cloud API)]
    US --> REP[report.py]

How each part works:

• Keys:
  • GATEWAY_KEYS="client:key,...", or in production a Docker secret file (GATEWAY_KEYS_FILE=/run/secrets/gateway_keys).
  • Keys are stored as sha256 hashes and compared with hmac.compare_digest.
  • One key per app, so you can revoke one without touching the others, and usage is attributed.
• Rate limit: in-memory token bucket, RATE_LIMIT_PER_MIN per client, returning 429 with Retry-After. It's single-process; with several replicas use Redis or rate limiting at the edge.
• Budgets:
  • DAILY_TOKEN_BUDGET works for every model, including local ones.
  • DAILY_BUDGET_USD only counts models that have prices in pricing.yaml.
  • Over budget → 429 for everyone until the next day (the gateway's local date: containers use UTC unless you set TZ).
• Redaction (REDACT_PII=auto|always|never):
  • auto redacts only when the upstream is a cloud/compat provider.
  • Card numbers are only treated as cards if they pass the Luhn check, so order numbers survive.
  • Placeholders are restored in the reply, but a card is restored only as "card ending 1234".
• Pricing: pricing.yaml ships with no cloud prices on purpose. Copy them from your provider's page, with the date you checked. Unpriced models are reported as unpriced and still counted in tokens.

Prerequisites — 先決條件

• Labs 01–08. Optional: a Langfuse project (Langfuse Cloud keys, or self-hosted via Langfuse's own compose file, which needs Postgres, ClickHouse, Redis and S3/MinIO and is not bundled here).

Step-by-Step Setup — 逐步設定

cd hands-on/labs/09
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e ../common -r requirements.txt
cp .env.example .env              # LLM_* = the upstream the gateway calls
uvicorn gateway.main:app --port 8009

Call it like any OpenAI endpoint:

curl -s localhost:8009/v1/chat/completions \
  -H 'authorization: Bearer change-me-n8n-key' -H 'x-feature: review-reply' -H 'content-type: application/json' \
  -d '{"messages":[{"role":"user","content":"Write a 1-line thank-you to chloe.lin@example.com"}]}' | python -m json.tool

The response has an x_gateway block with request_id, client, cost_usd and the list of redacted placeholders. From Python:

from openai import OpenAI
c = OpenAI(base_url="http://localhost:8009/v1", api_key="change-me-n8n-key", default_headers={"X-Feature": "listing-gen"})

Point any lab at it:

LLM_PROVIDER=compat LLM_BASE_URL=http://localhost:8009/v1 LLM_API_KEY=change-me-listing-key uvicorn app.main:app --port 8003

See what happened:

python report.py --days 7          # day × client × feature × model: calls, tokens, cost (or "unpriced")
tail -n 3 var/traces.jsonl         # what the model actually saw (redacted when redaction is on)

Production (Docker secret):

mkdir -p secrets
printf 'n8n:%s,listing-svc:%s' "$(openssl rand -hex 24)" "$(openssl rand -hex 24)" > secrets/gateway_keys
docker compose up -d --build

How to Test — 測試方法

LLM_PROVIDER=mock pytest -q        # 9 tests

The tests cover:

• auth
• redaction reaching the model as placeholders and being restored
• OpenAI SDK compatibility
• rate limit
• usage + unpriced report
• pricing math
• budget blocking
• the Luhn check and street addresses
• embeddings passthrough

What was run on the build box — 實測紀錄

• ✅ pytest: 9 passed.
• ✅ Live through Ollama qwen2.5:1.5b with REDACT_PII=always: the trace showed the model received <EMAIL_1>, and the client got the real email back in the reply. report.py printed the usage table, with local models at 0 and nothing priced.
• ✅ Langfuse: the client (SDK v3 API, start_observation(as_type="generation")) was constructed with dummy keys. Export to an unreachable host failed quietly without breaking requests, which is the intended behaviour.
• ✅ Lab 10's smoke test ran five services through this gateway; usage was attributed to all five clients.
• ⚠️ Not run: a real Langfuse server, a cloud upstream (no key), the Docker image/secret mount (no daemon).

Cost & Performance Notes — 成本與效能

• The gateway adds one local HTTP hop and a SQLite write per call. That's small next to model latency, but measure it with x_gateway.request_id timings if it matters to you.
• Cost formula: prompt_tokens/1e6 × input_per_mtok + completion_tokens/1e6 × output_per_mtok. Embeddings report estimated input tokens (chars/4) when the upstream doesn't return them. That's an estimate, labelled as such in code.
• Local models: the marginal cost is power and hardware wear, not tokens. To compare with cloud honestly, estimate (power draw × hours × electricity rate) ÷ tokens served per month, using your meter and your bill.
• Use the report to kill waste: a feature with high tokens and low business value (e.g. long prompts for tag generation) is the first to optimise.

Common Failures & Fixes — 常見錯誤與修正

Stretch Exercise — 延伸練習

1. Add per-client budgets (CLIENT_BUDGETS="listing:500000,rag:2000000" tokens/day) and a test.
2. Add a response cache for identical, temperature-0 requests (hash of model + messages). Measure the hit rate on a replay of Lab 06 events.

Deliverable Checklist — 交付清單

• [ ] Gateway running with one key per app, from a secret file
• [ ] At least two labs pointed at the gateway (LLM_PROVIDER=compat)
• [ ] Trace proving PII was pseudonymised for a cloud (or always) upstream
• [ ] pricing.yaml filled for any paid model, with source and date
• [ ] report.py --days 7 output reviewed; one optimisation identified
• [ ] pytest -q passes