Integrating LLM APIs
Authentication, rate limits, retries, and cost management
Skills taught: llms.api
SDK Setup and Authentication
Most LLM providers expose a REST API. Use the official SDK rather than raw HTTP — it handles retries, error types, and streaming automatically. Never hardcode API keys in source code.
# pip install anthropic
import anthropic
import os
# Option 1: SDK auto-reads ANTHROPIC_API_KEY from environment
client = anthropic.Anthropic()
# Option 2: Explicit key (use env var or secrets manager)
client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
# Option 3: Azure OpenAI or custom base URL
from openai import AzureOpenAI
azure_client = AzureOpenAI(
api_key=os.environ["AZURE_OPENAI_KEY"],
api_version="2024-02-01",
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"]
)
Error Handling
import anthropic
import time
import logging
client = anthropic.Anthropic()
logger = logging.getLogger("llm")
def call_with_retry(prompt: str, max_retries: int = 3) -> str:
"""Call Claude with exponential backoff on rate limits."""
for attempt in range(max_retries):
try:
response = client.messages.create(
model="claude-haiku-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": prompt}]
)
return response.content[0].text
except anthropic.RateLimitError as e:
wait = 2 ** attempt # 1s, 2s, 4s
logger.warning(f"Rate limited. Waiting {wait}s (attempt {attempt+1}/{max_retries})")
time.sleep(wait)
except anthropic.APIStatusError as e:
if e.status_code >= 500:
logger.error(f"Server error {e.status_code}. Retrying...")
time.sleep(1)
else:
raise # 4xx client errors won't get better on retry
except anthropic.APIConnectionError:
logger.error("Network error. Retrying...")
time.sleep(1)
raise RuntimeError("All retries exhausted")
Cost Tracking and Usage Monitoring
import anthropic
from dataclasses import dataclass, field
from threading import Lock
client = anthropic.Anthropic()
# Claude pricing (Nov 2026, per 1M tokens)
PRICING = {
"claude-haiku-5-5": {"input": 0.25, "output": 1.25},
"claude-sonnet-5-5": {"input": 3.00, "output": 15.00},
"claude-opus-5-5": {"input": 15.00, "output": 75.00},
}
@dataclass
class UsageTracker:
total_input_tokens: int = 0
total_output_tokens: int = 0
total_cost_usd: float = 0.0
_lock: Lock = field(default_factory=Lock)
def record(self, model: str, usage):
p = PRICING.get(model, PRICING["claude-haiku-5-5"])
cost = (usage.input_tokens / 1e6) * p["input"] + (usage.output_tokens / 1e6) * p["output"]
with self._lock:
self.total_input_tokens += usage.input_tokens
self.total_output_tokens += usage.output_tokens
self.total_cost_usd += cost
tracker = UsageTracker()
MODEL = "claude-haiku-5-5"
response = client.messages.create(
model=MODEL,
max_tokens=512,
messages=[{"role": "user", "content": "What is the capital of France?"}]
)
tracker.record(MODEL, response.usage)
print(f"Answer: {response.content[0].text}")
print(f"Total cost so far: ${tracker.total_cost_usd:.6f}")
Prompt Caching (Anthropic)
Prompt caching stores frequently-used prompt prefixes (like system prompts or large documents) on Anthropic's servers for 5 minutes. Cache hits cost 90% less than uncached input tokens — critical when you have a large system prompt used by thousands of requests.
import anthropic
client = anthropic.Anthropic()
# Large system prompt (must be >= 1024 tokens to cache)
large_system_prompt = """You are an expert financial advisor...
""" + "(much more context here — policies, examples, domain knowledge)" * 50
# Mark the system prompt for caching with cache_control
response = client.messages.create(
model="claude-sonnet-5-5",
max_tokens=1024,
system=[
{
"type": "text",
"text": large_system_prompt,
"cache_control": {"type": "ephemeral"} # cache for 5 minutes
}
],
messages=[{"role": "user", "content": "What is the best ETF for a 30-year-old?"}]
)
# Second request hits cache — 90% cheaper on the system prompt
print(f"Cache creation input tokens: {response.usage.cache_creation_input_tokens}")
print(f"Cache read input tokens: {response.usage.cache_read_input_tokens}")
Tip: Never hardcode API keys. Load from environment variables (os.environ['ANTHROPIC_API_KEY']) or a secrets manager like AWS Secrets Manager, HashiCorp Vault, or Doppler.
Warning: Rate limits are per model and per tier. Hitting a rate limit on claude-opus-5-5 doesn't affect your claude-haiku-5-5 quota. Use different models for different priority levels in the same application.
Frequently Asked Questions
How do I reduce API costs?
Use caching for repeated prompts, choose smaller/cheaper models for simpler tasks (Haiku is 60x cheaper than Opus), reduce max_tokens, implement prompt caching for large system prompts, and batch requests where possible.
What is a good default timeout for LLM API calls?
Set a generous timeout — 60-120 seconds for long generations. The Anthropic SDK defaults to 10 minutes but you can tune with timeout parameter. For streaming, consider first-token timeout (10s) separately from full response timeout.
Should I use the Anthropic SDK or raw HTTP?
Always use the official SDK. It handles retries, streaming, type safety, and SDK updates that track API changes. Raw HTTP requires you to reimplement all of this.