Lab 01 — Model Access: Local vs Cloud, One Client — AI Infrastructure for Real Business Problems
Last updated 10/17/2026
Course › Hands-on track › Lab 01

Lab 01 — Model Access: Local vs Cloud, One Client

模型接入：本地與雲端 · Hands-on track: AI Infrastructure for Real Business Problems

Track: AI Infrastructure for Real Business Problems · Code: labs/01/ · Shared client: labs/common/

Business Problem — 商業問題

Kettle & Kiln (an online tea-ware shop in Oakland; our running example) wants AI help with product copy, customer emails and support. The owner has three questions before spending anything:

  1. Can we run a model on our own hardware (a Mac mini or the TrueNAS box in the office) so customer emails never leave the building?
  2. When is a cloud model worth paying for?
  3. How do we avoid rewriting every script when we switch?

The store's privacy policy (data/docs/privacy.md) says customer personal data must not be sent to a third-party AI service unless approved. So the decision is not only "which model is smartest", it is also where each request is allowed to go.

In this lab you build the foundation every later lab uses: one client (llmkit) that talks to any OpenAI-compatible backend, a doctor script, a latency comparison, and a privacy router that keeps PII on the local model.

Architecture — 系統架構

flowchart LR
    subgraph App["Your scripts / services"]
        C[chat.py / compare.py] --> K[llmkit.LLM]
        R[router.py] -->|PII found| L
        R -->|no PII| F[FallbackLLM cloud→local]
    end
    K --> L[(Local: Ollama :11434 / llama.cpp :8080 / LM Studio :1234)]
    K --> CL[(Cloud: OpenAI or any compatible API)]
    K --> M[(mock: offline, for tests/CI)]
    F --> CL
    F -.fallback.-> L

Every backend speaks the same OpenAI-compatible /v1/chat/completions and /v1/embeddings API, so switching is a .env change:

LLM_PROVIDER Default base URL Notes
ollama http://localhost:11434/v1 easiest local option
llamacpp http://localhost:8080/v1 llama-server from llama.cpp
lmstudio http://localhost:1234/v1 desktop app
openai OpenAI default needs LLM_API_KEY
compat set LLM_BASE_URL any other compatible server (e.g. the Lab 09 gateway)
mock in-process deterministic, offline; every test suite uses it

Prerequisites — 先決條件

Step-by-Step Setup — 逐步設定

cd hands-on/labs/01
uv venv --python 3.12 && source .venv/bin/activate
uv pip install -e ../common -r requirements.txt
cp .env.example .env

# local model
ollama serve &                      # skip if the Ollama app is already running
ollama pull qwen3:8b                # or any chat model your hardware can hold
ollama pull nomic-embed-text        # used from Lab 04 on

python doctor.py                    # checks every configured prefix (LLM_, LOCAL_, CLOUD_)
python chat.py "Write a 1-line tagline for a celadon tea cup set"
python chat.py --json "Classify this message: 'my mug arrived cracked'"
python chat.py --prefix CLOUD_ "Same tagline, please"   # if you set CLOUD_* in .env

Then compare backends and try the router:

python compare.py --prefixes LOCAL_ CLOUD_ --repeat 3   # writes compare_results.csv
python router.py

router.py sends two prompts. The policy summary (no PII) goes cloud-first, falling back to local if the cloud call fails. The draft reply to chloe.lin@example.com contains an email address, so it is forced local.

Code Walkthrough — 程式碼導覽

How to Test — 測試方法

LLM_PROVIDER=mock pytest -q          # offline: 8 tests
RUN_LIVE=1 pytest -q                 # + 1 live test against the backend in .env

The tests cover:

What was run on the build box — 實測紀錄

Cost & Performance Notes — 成本與效能

Common Failures & Fixes — 常見錯誤與修正

Symptom Cause Fix
Connection refused localhost:11434 Ollama not running ollama serve (or start the app); curl localhost:11434/api/tags
model "qwen3:8b" not found not pulled ollama pull qwen3:8b; ollama list shows exact tags
Container can't reach Ollama on the host localhost inside Docker is the container use http://host.docker.internal:11434/v1 (+ extra_hosts: host-gateway on Linux)
401 from cloud wrong/missing key, or key for another org/project re-create key; check CLOUD_API_KEY is read (python doctor.py)
StructuredOutputError model too small or prompt unclear larger model, lower temperature, simpler schema; read the repair errors in the exception
Very slow local replies model larger than VRAM → CPU offload smaller/quantised model, shorter context, check ollama ps
ollama pull fails behind a corporate proxy/firewall registry blocked set HTTPS_PROXY, or download a GGUF and ollama create <name> -f Modelfile (this is what the build box had to do)

Stretch Exercise — 延伸練習

  1. Add a fourth backend (llama.cpp llama-server -m model.gguf --port 8080) to .env as ALT_ and include it in compare.py --prefixes LOCAL_ ALT_.
  2. Extend router.py so that when the cloud is used, a X-Data-Class: public header is logged. Write a test proving a message with a phone number never reaches the cloud backend (hint: give the router a cloud LLM whose mock handler raises).

Deliverable Checklist — 交付清單

← OverviewTrack overview
↑