One click on our feature turned into 10 to 15 model calls. The server behind it was built to answer in 100 to 200 milliseconds. Some of these requests now ran for minutes.
You can guess what happened next. The load balancer gave up while the work was still running (on AWS, an Application Load Balancer closes an idle connection after 60 seconds by default). The user saw an error and clicked again. The same work ran twice. We paid for both runs and showed the user neither.
Move the slow part out of the request
The fix is old, and it works. The request does three small things: it checks the input, saves a job row and returns a job id. A separate worker process takes the job from a queue, makes the model calls and writes the result.
What stays in the request is anything fast that the user must see at once: "this file is too big", "you can't do that". What goes to the worker is every model call, and anything that can fail and need a retry.
That is the easy half. The hard half starts the first time a worker dies in the middle of a job.
The job that said "processing" forever
The first time one of our workers crashed mid-job, the job stayed "processing". Nothing marked it as failed. The user waited for a result that was never coming, and nobody on our side knew.
Later, on a production AI platform for U.S. legal documents, the same bug came back in a quieter form. The queue was set to give up after five deliveries, and so was our code. The queue counted first. On the fifth failure it moved the message to the dead-letter queue, so our "give up and mark it failed" branch never got a turn. Files sat at "Extracting" with no error anywhere. We made the same mistake again in a second service before we wrote the rule down: the queue's limit must be one more than the code's.
The lesson fits in one line. Every job must end in a state a person can see: done, failed, or "needs a human". Never "processing" for a week.
Three things that close the gap
A heartbeat. A queue like Amazon SQS hides a message while a worker holds it. That window is the visibility timeout. If the worker doesn't finish in time, the message comes back and another worker takes it. The SQS docs recommend a heartbeat that keeps extending the window while the job runs. Long model calls need this, or a second worker starts the same job while the first one is still busy.
A limit on the heartbeat. This one surprised us. A call that hangs throws no error, so the heartbeat keeps the message hidden forever, with no log line. Our fix stops the heartbeat after a fixed wall-clock limit and releases the message for a retry. The stuck thread can't be killed, but the job is free to run again. A hang is worse than a crash, because a crash at least tells you.
A sweeper. A small scheduled job that finds jobs nobody is working on and marks them failed, or sends them again. It is the last line of defence for the day everything above has a bug.
Every step must be safe to run twice
Retries mean repeats. The SQS docs say it plainly: you might get the same message again, so design your applications to be idempotent. Celery, by default, acknowledges a task just before it runs. If the worker crashes halfway, that job is gone. Turn on acks_late and the task is acknowledged after it finishes instead, which means a crash makes it run again.
So you choose between losing jobs and repeating them. For slow, paid work, choose repeats and make them harmless. Martin Kleppmann's Designing Data-Intensive Applications calls this idempotence: running an operation twice leaves things as if it ran once.
In practice:
- Give each job a stable id. We build ours from the project and the file, so the same upload always maps to the same job.
- Before every step, ask "is this already done?". For model work, ask before you pay for the call, not only before you save the result.
There is one more gap. Saving the job and sending the message are two separate writes. On that same platform, an upload request returned HTTP 200 while the message never reached the queue, because the database connection pool ran out halfway through. The user saw "OK" and nothing happened. The fix is the transactional outbox: save the job and its message in one database transaction, and let a small relay send whatever is still unsent. The relay can also send twice, which is one more reason every step must be safe to repeat.
How the user finds out
The worker writes the status to the database first, then sends an event. The browser listens over server-sent events (the reasons are in why chat needs SSE). If an event gets lost, the database is still right, and a refresh shows the truth.
The order matters. Send the event first, let the database write fail, and the screen says "done" for a job the system thinks is still running.
None of this is new. Queues, retries and outboxes are older than LLMs. What is new is that AI features hit all of them on day one, because a single feature now waits on a slow, paid, sometimes failing service, many times per click. The same rules carry over when the worker is an agent; security for agentic workflows picks up from here, and agents at scale covers what happens when the queue fills faster than the provider lets you drain it. If you are taking a feature like this from demo to real users, it is the work we describe under prototype to production and AI document workflows.
Build the small version this week
One endpoint that creates a job row and puts a message on a queue. One worker that picks it up, calls the model and writes the result. One status endpoint.
Then kill the worker halfway through a job and see what happens. Fixing what you see teaches you most of this article. Then kill it again while it is waiting on the model, and fix that too.
A demo shows you what happens when everything works. A worker you killed on purpose shows you what your users would have seen.