The 504 that was never a timeout

For six weeks I had a line in my known-issues file that read, more or less: action router 504s, seventy five of them, about fifteen seconds each, suspect a function timeout on the hosting platform, cheap to confirm from the logs.

I never confirmed it. That is the whole story, really, but the details are worse and better than that.

When I finally sat down with the diagnostics export from the session that produced those failures, almost every clause in my own sentence turned out to be wrong. There were not seventy five 504s. There were four. The hosting platform was not in the request path at all, because the session ran on my laptop against a dev server. And the timeout was mine. I had written the number myself, months earlier, and never looked at it again after the thing behind it got four times more expensive.

The defect was real. The size of it and the blame for it were both invented, by me, in a hurry, and then quoted back to me by my own documentation every time I read the file.

The setup

The app is a voice whiteboard. You talk, and a diagram appears, and you can still drag the boxes afterwards. Speech comes in as a stream of transcript chunks, and each finalized chunk has to be sorted into one of two buckets: is this content to draw, or is this an instruction about the board? "The three layers are ingest, transform, serve" is content. "Tidy this up and make it look like a lesson" is an instruction.

That sorting is done by a small agent behind one HTTP request. The request is what I called the action router.

Behind the router sits a tool-calling loop. The agent has twelve tools, including a hosted web search, and it is configured to run the model again after any tool call so it can decide what to do with the result. Tools are not allowed to run in parallel. Turn limit is ten.

Read that configuration back slowly and the cost falls out of it. Any turn that touches a single tool is at least two model calls in sequence: one to choose the tool, one to look at what it returned. Two tools means three calls. A web search adds a whole search round trip in the middle.

The wall clock budget on the whole thing was twelve seconds.

const DEFAULT_ACTION_TIMEOUT_MS = 12_000;
// ...
setTimeout(() => controller.abort("action_router_timeout"), timeoutMs);

That constant is also the origin of the 504. My own handler aborts the agent, catches the abort, and returns a 504 with a small JSON body. I wrote the status code. I wrote the budget. I then spent six weeks attributing both to somebody else's infrastructure.

Worth saying where the twelve came from: it was correct once. The route started life as a single classification call, one model round trip to sort a sentence into a bucket, and twelve seconds was generous for that. The classifier then grew tools, twelve of them, one of them web search. The budget stayed at twelve because nothing forces you to revisit a constant. The sibling routes in the same app, added later, are budgeted at 30, 90 and 120 seconds. The router hosts the most expensive shape of work in the codebase on the tightest budget in the codebase. Nobody decided that. It just accumulated.

What I measured

The fixture is a diagnostics file from one live session, 1,716 records over about ten minutes. That session made 81 router calls.

Here is the breakdown my documentation had collapsed into a single number:

• 6 calls returned a plan
• 4 returned 504
• 71 returned 429, quota exhausted

Seventy five failures. Four of them were timeouts. The other seventy one were the metering ledger refusing to let the request through at all, which is a completely different problem with a completely different fix. I had written the status of the failure into a free text message field instead of a structured key, so when I later counted failures I got one number, attached the most alarming label I had seen to all of it, and filed it.

The second correction is more embarrassing. All four 504s land in a tight band: 13.9, 14.1, 14.3 and 15.2 seconds, measured from the client. A platform timeout clusters on the platform's limit. This band sits two to three seconds past my 12 second budget, which is exactly what a server-side deadline looks like once an auth check and a metering round trip are stacked in front of it. The arithmetic only fits one suspect.

And the successes give it away completely. The six calls that returned took 5.6, 7.0, 7.4, 10.1, 10.7 and 15.3 seconds. That last one is a success that took longer than three of the four timeouts. Successes and failures drawn from one continuous distribution, with a threshold cutting through the middle of it, is not what a gateway looks like. It is what a self imposed deadline looks like.

Then the timeline, which is the part that made the 4.9 percent failure rate meaningless:

Every timeout happened inside the first 111 seconds, because that is the only window in the session where quota still allowed a request to reach the agent at all. After that the route was returning 429 before the handler ran.

So the honest denominator is not 81. It is 10, the number of calls that actually ran the agent. Four of ten timed out. Forty percent, not five.

The number that explains the forty percent came later, from a trace of a much simpler run on the same code path months after the incident. Root span 10.82 seconds: a model call at 6.69 s, a tool that returned instantly, then a second model call at 4.13 s. Nothing pathological. No search. One tool, the cheapest possible tool-calling shape the agent can produce.

10.82 against a 12 second budget is 90 percent of the budget consumed by the floor. Not the median. The floor. Anything richer than a single tool call times out deterministically, and the only reason the failure rate was forty percent rather than higher is that some utterances needed no tool at all.

I should also admit what was not measured. The trace evidence and the failure evidence come from different days, because the agent emitted no traces during the incident window at all. A sibling agent logged 89 traced runs the day before, and this one logged zero for six weeks. The one artifact that would have shown me what the agent was doing for twelve seconds did not exist while it mattered, which is precisely why my note could only say "suspect".

What failed underneath

The timeout was the visible defect. Two things behind it were worse.

The first is billing. Quota is consumed before the handler runs, as a pre-flight check against the metering service, and there is no refund path anywhere in the codebase. So a request that burns twelve seconds and returns nothing is billed as a full turn against the user's allowance. It costs real money and real quota to fail.

This is not specific to the router. Every metered route in the app charges first and runs second, and several of them charge five units instead of one. There is a note elsewhere in my issues file about realtime sessions charging five turns on a failure, filed as its own bug. It is not its own bug. It is this mechanism, seen from a different route.

There is one small mercy: the client never retries a 504. So the timeouts did not multiply load or quota. The failure is expensive but not self amplifying.

The second thing is worse than billing, because the user cannot see it happening.

When the router fails, the client catches the error, records a diagnostic event, and returns null. No retry, no status change, nothing on screen. Downstream, a speculative plan built from the raw transcript is already in flight, and the gate that decides whether to commit it asks one question: did the router produce a plan? No plan means commit the speculative one.

That gate was written for a real problem. If the router says "this is ordinary content", you want the raw transcript drawn, otherwise the board livelocks waiting for instructions that are never coming. Correct behaviour, correct fix, still correct today.

But null means two different things and the code only understands one of them. "The router says this is ordinary content" and "the router never answered" arrive at the gate wearing identical clothes. So a transport failure gets read as a verdict, and the app confidently draws your instruction as prose.

You say "tidy this up and make it presentable". Twelve seconds later, the words "tidy this up and make it presentable" appear on the board as a sticky note. Every instruction in that utterance is gone, and the app looks like it worked.

There is even a branch further down that was meant to handle exactly this case by re-queueing the raw text. It is unreachable. The gate above it already returned.

And because router calls are serialized, one at a time, each timeout blocks the queue for twelve seconds while speech keeps producing new chunks every one to three seconds. Median time from someone speaking to anything appearing on the board in that session was 105 seconds. A twelve second stall that yields nothing is the worst possible unit of that backlog.

The tests, by the way, were green the entire time. Seven cases on that handler, all passing a fake resolver that returns instantly. Not one of them constructs a timeout. A fake that never takes time cannot fail a deadline, so the single parameter that was wrong was the only parameter never under test.

The takeaway

Budget for the floor latency of the chain, not for the median of the thing you originally built.

Twelve seconds was a fine number for one model call. The moment "run the model again after a tool call" entered the configuration, the floor doubled by construction, and that is arithmetic I could have done at my desk without any telemetry at all. Two chained calls where there used to be one. I did not do it, because the budget lived in one file and the agent configuration lived in another, and nothing connects a constant to the thing it constrains except somebody remembering.

The cheapest guard I know now is to emit the observed latency next to the configured budget on every response, and alert when p95 crosses seventy percent of it. On the trace I have, that alert fires at 90 percent. It would have predicted the incident instead of explaining it.

The other half is louder fallbacks. A fallback that cannot distinguish "the system answered no" from "the system did not answer" is not a fallback, it is a silence. Return a typed outcome, not a null. Keep committing the speculative plan if that is what keeps the product moving, but put something on the screen that says the instruction did not land, because the alternative is a user who thinks they were understood and a log line that agrees with them.

And write the status code down as a field. Seventy one quota refusals and four timeouts spent six weeks as one number with the scarier label, in a file I read every week, because I recorded the failure as prose instead of data.