A 200 Does Not Mean It Worked: LLM Streaming Failures in Production

Non-streaming requests have a comfortable property: one call, one outcome. It worked or it did not, and the status code says which. Streaming discards that, and most stream-handling code is written as though it did not.

A streaming response is a sequence of discrete events arriving over an open connection. It can begin correctly, deliver two thirds of an answer, and then send an error. The HTTP status was decided long before that happened and will never be revised.

The event flow you are actually consuming

Worth stating precisely, because three of the four failure modes below are consequences of its shape. The documented flow is: a message_start carrying a Message with empty content; then a series of content blocks, each opening with content_block_start, emitting one or more content_block_delta events, and closing with content_block_stop; then one or more message_delta events carrying top-level changes; then message_stop.

message_start                      Message, content: []
  content_block_start   index 0    type: text
  content_block_delta   index 0    text_delta        x N
  content_block_stop    index 0
  content_block_start   index 1    type: tool_use
  content_block_delta   index 1    input_json_delta  x N   <- partial JSON
  content_block_stop    index 1
message_delta                      stop_reason, CUMULATIVE usage
message_stop                       <- the only proof of completion

  ping events may appear anywhere
  error events may appear anywhere

Two properties of that structure matter more than they look. Every content block carries an index corresponding to its position in the final content array, and a single response routinely contains several blocks — text, then a tool call, then more text. Their events are distinguished only by that index.

Failure one: the error that arrives after the 200

The API can send an error inside the event stream. The documented example is an overloaded_error during periods of high usage — the same condition that would have produced an HTTP 529 on a non-streaming request, arriving instead as an event on a connection that already returned 200. The full set of error types is worth reading, because each one implies a different retry decision and a stream can deliver any of them.

event: error
data: {"type": "error", "error": {"type": "overloaded_error", "message": "Overloaded"}}

Handlers that only inspect the status, or that wrap the whole call in a try/except expecting an exception, treat the partial output as the complete answer. Downstream, that becomes a truncated summary stored as final, or a half-written record committed as though it were whole. Nothing raises.

Assert on message_stop rather than on the loop ending

The correction is small and structural: track completion explicitly, and treat its absence as a failure regardless of how the loop ended.

class StreamIncomplete(RuntimeError):
    """The stream ended without message_stop."""


def consume(stream) -> dict:
    blocks: dict[int, list[str]] = {}
    completed = False
    stop_reason = None

    for event in stream:
        etype = event.get("type")

        if etype == "error":
            # A 200 was already returned. This is the real outcome.
            raise StreamIncomplete(event["error"]["type"])

        elif etype == "content_block_delta":
            blocks.setdefault(event["index"], []).append(delta_text(event))

        elif etype == "message_delta":
            stop_reason = event["delta"].get("stop_reason") or stop_reason

        elif etype == "message_stop":
            completed = True

        # Unknown event types are ignored deliberately — the API adds new
        # ones, and a handler that throws on them breaks on the next release.

    if not completed:
        raise StreamIncomplete("stream ended without message_stop")

    return {"blocks": blocks, "stop_reason": stop_reason}

Failure two: usage counts are cumulative

This one is called out in a warning box in the documentation and is still routinely missed, because the mistake produces plausible numbers rather than obviously broken ones. Token counts in the usage field of message_delta are cumulative, not per-event.

Summing them across events therefore does not give you the total. It gives you a triangular number of the total, and the size of the error scales with how many message_delta events arrived — so short responses look nearly right and long ones look wildly expensive.

# Wrong: usage is cumulative, so this sums a running total.
total = 0
for event in stream:
    if event["type"] == "message_delta":
        total += event["usage"]["output_tokens"]   # inflates with each delta

# Right: take the last value, which already includes everything before it.
latest = 0
for event in stream:
    if event["type"] == "message_delta":
        latest = event["usage"]["output_tokens"]

The consequence is not cosmetic if those numbers drive anything. Cost dashboards over-report, per-tenant billing over-charges, and a spend ceiling built on the summed figure trips early and kills healthy runs — which is the opposite of the failure described in why agent bills explode, arriving from the same field read the wrong way.

Failure three: tool inputs arrive as unparseable fragments

Streaming and tool use do not compose as neatly as either does alone. A tool call's input field arrives through input_json_delta events carrying partial JSON strings, while the final tool_use.input is always an object. Between those two states the payload is not valid JSON and cannot be parsed.

content_block_delta  index 1  partial_json: ""
content_block_delta  index 1  partial_json: "{\"location\":"
content_block_delta  index 1  partial_json: " \"San"
content_block_delta  index 1  partial_json: " Francisc"
content_block_delta  index 1  partial_json: "o,"
content_block_delta  index 1  partial_json: " CA\"}"
content_block_stop   index 1          <- only now is it parseable

Accumulate by index, not into one buffer

Accumulate the fragments and parse once the block closes. Accumulate them keyed by index rather than into a single buffer, because a response contains several content blocks and nothing but the index distinguishes their events — concatenating everything produces a string that is neither tool call.

pending: dict[int, dict] = {}
calls: list[dict] = []

for event in stream:
    etype = event["type"]

    if etype == "content_block_start":
        block = event["content_block"]
        if block["type"] == "tool_use":
            # Key on index: several blocks are open across one response.
            pending[event["index"]] = {
                "id": block["id"], "name": block["name"], "buf": [],
            }

    elif etype == "content_block_delta":
        entry = pending.get(event["index"])
        if entry and event["delta"]["type"] == "input_json_delta":
            entry["buf"].append(event["delta"]["partial_json"])

    elif etype == "content_block_stop":
        entry = pending.pop(event["index"], None)
        if entry:
            # Complete only now. Parsing earlier fails on partial JSON.
            entry["input"] = json.loads("".join(entry["buf"]))
            calls.append(entry)

Failure four: resume semantics differ by model generation

When a stream is interrupted, the documented recovery is to capture what arrived and issue a continuation request rather than repeating the whole generation. The mechanism for that continuation is not the same across model versions, which makes it a genuine migration hazard.

Carrying the older pattern forward onto a newer model does not raise an error. It produces a response that reads slightly wrong — restating, or resuming from the wrong place — which is far harder to attribute than an exception would be.

The second constraint is sharper: tool use and extended thinking blocks cannot be partially recovered. Only the most recent text block can be resumed. If the interruption landed mid tool-input, that call has to be re-run rather than repaired, which is worth knowing before designing a resume path that assumes every block is patchable.

Common mistakes

1. Treating a 200 as proof of a complete response. Errors arrive as events after the status is fixed; only message_stop proves completion.
2. Summing usage across message_delta events. The counts are cumulative, so summation inflates spend by an amount that grows with response length.
3. Buffering all input_json_delta fragments into one string. Multiple blocks interleave and only the index separates them; concatenating produces invalid JSON for every call.
4. Parsing tool input before content_block_stop. Fragments are deliberately partial, and a parse attempt mid-block fails on valid data.
5. Throwing on unknown event types. New ones are added by design, and a strict switch turns a forward-compatible change into an outage.
6. Carrying pre-4.6 continuation logic onto newer models. The failure is a subtly wrong answer, not an exception.
7. Assuming a silent connection is a dead one. Gaps are expected while the model assembles tool input.
8. Relying on browser EventSource auto-reconnect for LLM streams. It restarts the request rather than resuming the generation, so the user sees the answer begin twice and you pay for both.

Security and performance considerations

Streaming shifts what your users see before you have inspected it. In a non-streaming flow you can validate, redact, or reject a response before rendering; in a streaming flow the first tokens are on screen while the rest is still generating. Any moderation or PII filtering that assumed a complete document now has to work incrementally, or it is not running at the point where it matters.

Proxy rather than streaming straight to the browser

That also constrains what may be streamed directly to a browser. Proxying through your own server rather than exposing the provider endpoint keeps the API key server-side and gives you a place to enforce budget, apply filtering, and terminate a stream you have decided to stop — none of which is possible if the client holds the connection.

On performance, the honest benefit of streaming is perceived latency rather than total latency: the full response takes the same time, the first token simply arrives sooner. Weigh that against the handling complexity above for any non-interactive path. A background extraction job gains nothing from streaming and inherits every failure mode in this article — and if it is parsing the result, the guarantees discussed in what structured outputs actually guarantee are the more useful tool.

Troubleshooting

1. Responses occasionally truncated with no error logged — a mid-stream error event being ignored. Assert on message_stop and log the error type.
2. Recorded token usage far exceeds the provider invoice — cumulative usage being summed. Take the last value rather than accumulating.
3. Tool calls fail with JSON decode errors — parsing before content_block_stop, or buffering multiple blocks together without keying on index.
4. Tool calls work singly and break when the model makes two — interleaved blocks sharing one buffer. The index is the discriminator.
5. Streams abort during tool use but not during text — an idle timeout shorter than the gaps the model takes while assembling tool input.
6. Continuation after an interruption produces repeated or misaligned text — pre-4.6 continuation logic on a 4.6-or-later model.
7. Mobile users see responses vanish mid-answer — the connection dropped on backgrounding, often with no error surfaced by the client.

The through-line is that streaming converts one atomic outcome into a protocol with its own state machine, and most of the bugs come from code that still assumes the atomic version. Handle the events the API documents — including the ones that only appear when something goes wrong — and the failure modes stop being mysterious.

If you are shipping a streaming assistant and want the failure modes handled before users find them, that is the sort of work our applied AI team does. Where the streaming boundary sits inside a larger service you already run, our backend and integration work covers that layer.