Streaming Responses
Server-sent events, real-time display, and latency
Skills taught: llms.streaming
Why Stream?
By default, the API waits until the full response is generated before returning. Streaming sends tokens as they are generated, dramatically improving perceived latency. Users start reading immediately instead of waiting for the full response.
import anthropic
client = anthropic.Anthropic()
# Streaming with the Anthropic SDK
with client.messages.stream(
model="claude-sonnet-5-5",
max_tokens=1024,
messages=[{"role": "user", "content": "Tell me a short story about a robot learning to paint."}]
) as stream:
for text in stream.text_stream:
print(text, end="", flush=True) # Prints each token as it arrives
# Get final message with usage stats
final = stream.get_final_message()
print(f"\n\nTokens: in={final.usage.input_tokens}, out={final.usage.output_tokens}")
Streaming in JavaScript / Node.js
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic(); // reads ANTHROPIC_API_KEY
async function streamResponse(prompt) {
const stream = client.messages.stream({
model: "claude-sonnet-5-5",
max_tokens: 1024,
messages: [{ role: "user", content: prompt }],
});
// Stream text as it arrives
for await (const event of stream) {
if (
event.type === "content_block_delta" &&
event.delta.type === "text_delta"
) {
process.stdout.write(event.delta.text);
}
}
const finalMsg = await stream.finalMessage();
console.log(`\n\nStop reason: ${finalMsg.stop_reason}`);
console.log(`Tokens: ${finalMsg.usage.input_tokens} in, ${finalMsg.usage.output_tokens} out`);
}
awStreamResponse("Explain async/await in JavaScript.");
Streaming to a Next.js / Web API Response
// app/api/chat/route.ts (Next.js App Router)
import Anthropic from "@anthropic-ai/sdk";
import { AnthropicStream, StreamingTextResponse } from "ai"; // Vercel AI SDK
export const runtime = "edge";
export async function POST(req: Request) {
const { messages } = await req.json();
const client = new Anthropic();
const response = await client.messages.create({
model: "claude-sonnet-5-5",
max_tokens: 1024,
messages,
stream: true,
});
// Vercel AI SDK converts Anthropic SSE to a web ReadableStream
const stream = AnthropicStream(response);
return new StreamingTextResponse(stream);
}
Handling Streaming Events
import anthropic
client = anthropic.Anthropic()
# Full event-level control (not just text)
with client.messages.stream(
model="claude-haiku-5-5",
max_tokens=512,
tools=[{"name": "search", "description": "Search the web",
"input_schema": {"type": "object", "properties": {"query": {"type": "string"}},
"required": ["query"]}}],
messages=[{"role": "user", "content": "Search for the latest news on AI."}]
) as stream:
for event in stream:
if event.type == "content_block_start":
print(f"Block started: {event.content_block.type}")
elif event.type == "content_block_delta":
if hasattr(event.delta, "text"):
print(event.delta.text, end="", flush=True)
elif hasattr(event.delta, "partial_json"):
print(event.delta.partial_json, end="")
elif event.type == "message_stop":
print("\n--- Stream complete ---")
Tip: For user-facing chat UIs, always use streaming. The perceived latency improvement is dramatic — users rate streamed responses as 'faster' even when total generation time is identical.
Warning: Don't try to parse JSON from a streaming response mid-stream. Buffer the full text and parse after stream.get_final_message(). Partial JSON will throw parse errors.
Frequently Asked Questions
Can I cancel a streaming request mid-way?
Yes. Break out of the for loop or call stream.close() to cancel. The API stops generating and you're only charged for tokens generated up to that point.
What is time-to-first-token (TTFT)?
The delay between sending the request and receiving the first token. This is the 'prefill' time — the model processes your input before starting generation. For long prompts, TTFT can be 2-5 seconds even with streaming.
How do I stream to multiple clients simultaneously?
Make separate API calls per client. You can't broadcast one LLM stream to multiple subscribers directly — but you can store the full response as it streams and use a pub/sub system (Redis, SSE broadcast) to fan it out.