LLM Apps in Production
Observability, guardrails, latency, and cost optimisation
Skills taught: llms.production
The Production LLM Checklist
Running LLMs in production requires much more than a working prompt. You need to manage latency, costs, safety, and observability — and each is a separate engineering concern.
- Observability: Log every call with user ID, model, prompt version, tokens in/out, latency, cost
- Model routing: Route simple tasks to cheap models (Haiku), complex tasks to powerful models (Opus)
- Semantic caching: Return cached responses for semantically identical queries — huge cost savings
- Guardrails: Validate inputs (block injections) and filter outputs (block harmful content)
- Rate limiting: Protect against abuse and runaway costs per user/account
- Fallbacks: If primary model fails, fall back to secondary (e.g., Sonnet → GPT-4)
import anthropic
import time
import logging
import uuid
from dataclasses import dataclass
client = anthropic.Anthropic()
logger = logging.getLogger("llm_calls")
PRICING = {
"claude-haiku-5-5": {"in": 0.25, "out": 1.25},
"claude-sonnet-5-5": {"in": 3.00, "out": 15.00},
}
def call_with_observability(
prompt: str,
model: str = "claude-haiku-5-5",
user_id: str = "anonymous"
) -> str:
call_id = str(uuid.uuid4())[:8]
start = time.perf_counter()
try:
response = client.messages.create(
model=model,
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
)
text = response.content[0].text
latency_ms = int((time.perf_counter() - start) * 1000)
# Cost calculation
p = PRICING.get(model, PRICING["claude-haiku-5-5"])
cost = (response.usage.input_tokens / 1e6 * p["in"] +
response.usage.output_tokens / 1e6 * p["out"])
logger.info({
"call_id": call_id, "user_id": user_id, "model": model,
"tokens_in": response.usage.input_tokens,
"tokens_out": response.usage.output_tokens,
"latency_ms": latency_ms,
"cost_usd": round(cost, 6),
"status": "success"
})
return text
except Exception as e:
logger.error({"call_id": call_id, "user_id": user_id, "error": str(e), "status": "error"})
raise
Model Routing by Complexity
import anthropic
client = anthropic.Anthropic()
# Claude Haiku is 60x cheaper than Opus — route simple tasks to it
SIMPLE_PATTERNS = [
"what is", "define", "when was", "how many",
"yes or no", "true or false", "translate"
]
def select_model(prompt: str) -> str:
prompt_lower = prompt.lower()
# Simple factual lookup → Haiku
if any(p in prompt_lower for p in SIMPLE_PATTERNS) and len(prompt) < 200:
return "claude-haiku-5-5"
# Medium complexity → Sonnet
elif len(prompt) < 1000:
return "claude-sonnet-5-5"
# Long/complex → Opus
else:
return "claude-opus-5-5"
def smart_call(prompt: str, user_id: str) -> str:
model = select_model(prompt)
return call_with_observability(prompt, model=model, user_id=user_id)
# Simple query → Haiku ($0.0001)
print(smart_call("What is the capital of Germany?", "user123"))
# Complex query → Sonnet ($0.012)
print(smart_call("Analyse this 500-word contract and identify all obligations...", "user123"))
Semantic Caching
import numpy as np
from openai import OpenAI
oai = OpenAI()
class SemanticCache:
"""Cache LLM responses for semantically similar queries."""
def __init__(self, threshold: float = 0.95):
self.threshold = threshold
self.entries = [] # List of {embedding, response}
def _embed(self, text: str) -> np.ndarray:
vec = oai.embeddings.create(input=text, model="text-embedding-3-small").data[0].embedding
return np.array(vec)
def _cosine(self, a: np.ndarray, b: np.ndarray) -> float:
return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
def get(self, query: str):
q_emb = self._embed(query)
for entry in self.entries:
sim = self._cosine(q_emb, entry["embedding"])
if sim >= self.threshold:
return entry["response"] # Cache hit!
return None
def set(self, query: str, response: str):
self.entries.append({"embedding": self._embed(query), "response": response})
cache = SemanticCache(threshold=0.95)
def cached_call(prompt: str) -> str:
# Check cache first
cached = cache.get(prompt)
if cached:
print("[CACHE HIT] Returning cached response")
return cached
# Miss: call API
response = call_with_observability(prompt)
cache.set(prompt, response) # Store for future similar queries
return response
cached_call("What is the capital of France?") # API call
cached_call("What's the capital city of France?") # Cache hit! Sim ~0.97
Input Guardrails
import anthropic
client = anthropic.Anthropic()
# Simple LLM-based content classifier (you can also use Llama Guard)
def is_safe_input(user_message: str) -> tuple[bool, str]:
"""Returns (is_safe, reason)."""
response = client.messages.create(
model="claude-haiku-5-5", # cheapest model for this check
max_tokens=50,
temperature=0,
system="""Classify if the input is safe for a customer support chatbot.
Output ONLY 'SAFE' or 'UNSAFE: <reason in 5 words'.""",
messages=[{"role": "user", "content": user_message}]
)
result = response.content[0].text.strip()
is_safe = result.startswith("SAFE")
return is_safe, result
def safe_chat(user_message: str) -> str:
safe, reason = is_safe_input(user_message)
if not safe:
return f"I can't help with that request. ({reason})"
return call_with_observability(user_message)
print(safe_chat("How do I track my order?")) # Normal support query → OK
print(safe_chat("How do I hack my bank account?")) # Flagged → Refused
Tip: Use the cheapest model that meets quality requirements. Claude Haiku is 60x cheaper than Opus for the same call. Route simple queries to Haiku and reserve Opus for genuinely complex tasks.
Warning: Never expose raw LLM errors to end users — they may leak system prompt details. Catch all API errors and return a generic user-facing message. Log the full error internally.
Frequently Asked Questions
What are guardrails?
Pre/post-processing checks that filter harmful inputs or outputs. Tools like Llama Guard (open-source), NeMo Guardrails (rule-based), and simple LLM classifiers can prevent misuse. Output guardrails catch hallucinations or policy violations before showing responses to users.
What LLM observability tools are available?
Langfuse (open-source, self-hostable), Helicone (cloud), LangSmith (LangChain ecosystem), Traceloop (OpenTelemetry-compatible). All provide prompt versioning, trace logging, cost tracking, and latency dashboards.
How do I handle LLM API outages in production?
Use a circuit breaker pattern: after N consecutive failures, fail fast for 30 seconds. Have a fallback model (Anthropic → OpenAI). Keep a degraded mode that shows a 'service temporarily unavailable' message rather than hanging. The Anthropic SDK auto-retries on 529/503 up to 2 times.