Code: labs/08/ · Packages Labs 03–05 into one stack

Business Problem — 商業問題

The prototypes run on a laptop with uvicorn. The business needs them running 24/7 on the office TrueNAS box (or a Mac mini), reachable by staff at home and by the website's chat widget, without:

• opening ports on the router
• paying for a cloud VM
• the owner ever SSH-ing in at midnight

Requirements:

• one command to start everything
• data on snapshotted storage
• HTTPS
• staff-only access to internal tools
• GPU acceleration when there is one
• backups that have actually been restored once

Architecture — 系統架構

flowchart LR
    U[Staff / website] -->|HTTPS| CF[Cloudflare edge<br/>TLS + Access policy]
    CF -->|outbound-only tunnel| CFD[cloudflared]
    subgraph Host["TrueNAS / Linux / Mac — no inbound ports"]
        subgraph edge[edge network]
            CFD --> CAD[Caddy :80<br/>/rag /agent /listings /healthz]
        end
        subgraph backend[backend network]
            CAD --> RAG[rag-api :8004]
            CAD --> AG[agent-api :8005]
            CAD --> LS[listing-api :8003]
            RAG --> QD[(Qdrant + snapshots)]
            RAG & AG & LS --> OL[Ollama<br/>GPU optional]
            OW[Open WebUI<br/>own hostname]
        end
    end
    CFD -.->|chat.example.com| OW
    DS[(DATA_ROOT on a ZFS dataset<br/>periodic snapshots + replication)] --- QD & OL & AG & LS

Key decisions:

• Cloudflare Tunnel, not port forwarding. cloudflared dials out to Cloudflare, so nothing listens on the internet. Put Cloudflare Access (SSO/email OTP, or service tokens for machines) in front of every hostname.
• Caddy as the single internal router. It handles path routing, security headers, a 1 MB body limit, JSON logs and a /healthz. TLS terminates at Cloudflare and the tunnel carries traffic to Caddy over the private network, so Caddy runs auto_https off. On a LAN-only setup without a tunnel, let Caddy do automatic HTTPS instead.
• Two networks. Only caddy and cloudflared share edge; model server, vector DB and apps sit on backend only. Caddy's host port is bound to 127.0.0.1:8080 for local testing.
• Profiles:
  • local-ollama: Ollama in a container. On a Mac, skip it and run Ollama natively for Metal.
  • tunnel: cloudflared.
  • ui: Open WebUI for staff chat. It needs its own hostname, because it doesn't support being served under a sub-path.
• GPU overrides live in separate files: docker-compose.gpu.yml (NVIDIA) and docker-compose.rocm.yml (AMD).

Prerequisites — 先決條件

• Docker Engine + Compose v2 on the host. On TrueNAS SCALE 24.10+, apps run on Docker; see labs/08/truenas/README.md.
• A domain on Cloudflare (free plan is enough for Tunnel + Access at small scale — check Cloudflare's current plan limits).
• Optional GPU:
  • NVIDIA: driver + nvidia-container-toolkit.
  • AMD: ROCm-supported card + /dev/kfd, /dev/dri.

Step-by-Step Setup — 逐步設定

1. Configure.

cd hands-on/labs/08
cp .env.example .env
# DATA_ROOT=./volumes            (TrueNAS: /mnt/tank/apps/kk-ai — create the datasets first)
# LLM_BASE_URL=http://ollama:11434/v1   (Mac with native Ollama: http://host.docker.internal:11434/v1)

2. Start the stack.

docker compose --profile local-ollama up -d --build      # NAS/Linux
# NVIDIA:  docker compose -f docker-compose.yml -f docker-compose.gpu.yml --profile local-ollama up -d --build
# AMD:     docker compose -f docker-compose.yml -f docker-compose.rocm.yml --profile local-ollama up -d --build
bash scripts/pull-models.sh                              # pulls LLM_MODEL + LLM_EMBED_MODEL into the container
docker compose ps                                        # wait for "healthy"
bash scripts/smoke.sh http://127.0.0.1:8080              # health of every route + one real RAG question

3. Publish with a Cloudflare Tunnel (token mode).

1. In Cloudflare Zero Trust, go to Networks → Tunnels → Create tunnel (cloudflared), and copy the token into TUNNEL_TOKEN in .env.
2. Add public hostnames:
   • ai.example.com → http://caddy:80
   • chat.example.com → http://open-webui:8080 (if using the ui profile)
3. Under Access → Applications, add a self-hosted app for each hostname, with a policy (e.g. emails ending in your domain). For the website widget or n8n, create a service token and send CF-Access-Client-Id/Secret headers.
4. Start the tunnel and test from outside the LAN:

docker compose --profile tunnel up -d
CF_ACCESS_CLIENT_ID=... CF_ACCESS_CLIENT_SECRET=... bash scripts/smoke.sh https://ai.example.com

Prefer config-as-code? Use cloudflared/config.example.yml (a locally managed tunnel) and validate it with cloudflared tunnel --config cloudflared/config.example.yml ingress validate.

4. Back up and restore once.

• Turn on ZFS periodic snapshots + replication of DATA_ROOT.
• Add scripts/backup.sh (Qdrant snapshots via API + tarballs of agent records and listings) as a nightly cron job.
• Restore on a spare machine at least once. Copy DATA_ROOT, start the stack, and run smoke.sh. A backup you haven't restored is a hope, not a backup.

5. Validate config without starting anything (what CI and the build box run):

DOCKER_COMPOSE="docker compose" bash tests/validate.sh

How to Test — 測試方法

tests/validate.sh checks:

• docker compose config for 5 combinations (default; local-ollama; tunnel+ui+local-ollama; the GPU override; the ROCm override)
• caddy validate on the Caddyfile
• cloudflared ... ingress validate on the example config
• bash -n on all scripts

After deployment, scripts/smoke.sh is the runtime test.

What was run on the build box — 實測紀錄

• ✅ tests/validate.sh: all OK. The checks used Docker Compose v2.29.7 (config mode), Caddy v2.8.4 and cloudflared 2026.10.1, and covered all 5 compose combinations, the Caddyfile, the tunnel ingress rules and the script syntax.
• ⚠️ Not run on the build box: docker compose up (no Docker daemon), image builds, a GPU, a real TrueNAS system, a live Cloudflare Tunnel/Access, Open WebUI, and backup.sh/smoke.sh against a running stack. The services themselves were run and tested as plain processes in Labs 03–05 and together in Lab 10's smoke_local.py.

Cost & Performance Notes — 成本與效能

• Software cost: all components here are open source. Cloudflare Tunnel and Access have a free tier; check current limits and pricing on Cloudflare's site before relying on it.
• Hardware:
  • A GPU with enough VRAM for your chosen model is the single biggest latency improvement.
  • CPU-only works for embeddings and 1–3B models. The build box's CPU runs of 1.5B models took seconds per answer, and 7–8B models will be slower.
  • Measure with Lab 01's compare.py on your box rather than trusting any published number (including ours).
• OLLAMA_KEEP_ALIVE=30m avoids reloading the model on every request, at the cost of holding its memory. OLLAMA_NUM_PARALLEL=2 lets two requests share the loaded model. Raise it only if VRAM allows.
• Power is a real running cost for an always-on GPU box. Measure it at the wall with a plug meter and multiply by your electricity rate.

Common Failures & Fixes — 常見錯誤與修正

Stretch Exercise — 延伸練習

1. Add Uptime Kuma (or Cloudflare health checks) that hits /healthz and each /…/health every minute and alerts on failure.
2. Move Lab 06 (n8n + ai-services) and Lab 07 (copilot) into this stack, with n8n on its own Access-protected hostname. Lab 10 does this; try it first yourself.

Deliverable Checklist — 交付清單

• [ ] validate.sh passes; docker compose ps all healthy on your host
• [ ] smoke.sh passes on LAN and through the tunnel with a service token
• [ ] No router port forwards (screenshot of router config or external port scan)
• [ ] Access policy on every hostname
• [ ] Snapshot/backup schedule set and one restore rehearsed
• [ ] GPU visible in the Ollama container (nvidia-smi / rocm-smi) or a documented decision to run CPU-only