The cover illustrates a fictional store, not a deployed system or real test result.
A customer confirms an order. The system records it, but keeps the page waiting for PDF generation and email until it times out, so the customer clicks again. At fictional Ready Parts, I would separate the result the customer needs immediately from work that can follow, and make order status checkable. A fast response without knowing where the follow-on work was stored merely moves the problem behind the screen.
A queue lets intake and processing happen at different times and can absorb a temporary burst. It does not create more processing capacity. If arrivals continually exceed completions, the queue grows, along with wait time and cost. The important questions are what work we accept, whether its status is trustworthy, and who owns failed jobs.
What must finish on screen, and what can follow?
The customer should know whether the order has been recorded under the rules. A document based on that committed order may follow if the business accepts the delay and shows clear status. But do not defer authorization or stock reservation and then say “confirmed” when the last unit might already have been sold.
Separating work can begin with a background process in the same system or a managed queue; several new technologies are not required at once. What matters is durable acceptance under the requirements, traceability, and continuation after failure. A letter placed in a temporary tray that disappears when the machine stops has not been reliably accepted.
Accepted and completed are different states.When the system durably acknowledges a job, provide a reference that can show pending, running, succeeded, or failed. Do not use HTTP 202 or a “processing” message as proof that the job was actually retained.
Reference and required data exist
User and operator can check status
Real outcome or exception owner
Close the gap between committing a transaction and queuing its work
Suppose the app commits an order but stops before sending the document job. The customer has an order without a document. If it queues the job first and the main transaction fails, it may produce a document for an order never confirmed. Design their relationship for the actual system instead of assuming two separate commands will always succeed back to back.
One approach stores the transaction and an outgoing job record in the same transaction, then a dispatcher sends that record to the queue. This is the transactional outbox pattern. Dispatch can still repeat, so duplicate outcomes and stranded records need checks. The pattern does not put every outside service in one transaction with our database.
Whatever design we choose, reconcile it: orders that should have documents but do not, queue jobs without referenced records, and documents created while status remains stale. Alerts alone are insufficient. Name who checks, what rights they use to fix it, and how evidence is kept without unnecessary customer data.
A redelivered job must not duplicate the business result
Some queues, such as standard SQS queues, deliver at least once and may redeliver the same item. A retry after a timeout may also start after the first attempt succeeded. Separate “I did not see an answer” from “the job did not happen,” especially for payments, stock reservations, or numbered documents.
Idempotency here: retrying the same intended operation must not duplicate its business effect. Use a request key tied to intent, retain outcome and status durably, and coordinate duplicate prevention with the data change in a step that succeeds or rolls back together in that system. If the same key arrives with a different quantity or buyer, reject or handle it under the defined contract; do not silently return an old result.
Matching payloads alone can mistakenly merge two separate orders that a customer intentionally made alike. Distinguish a retry of the same order from a new order, set a lifetime for remembered keys, and check the outside provider’s duplicate-control method. If it has none, inspect or reconcile the outcome before trying again.
Check parameters and prior state
Prevent duplication with the data change
Check outcomes and reconcile by service
When will the queue drain?
Suppose 120 document jobs arrive per second while workers finish 100 per second for 60 seconds. The backlog grows by 1,200 jobs, with no retries or discarded work. Arrivals then fall to 80 per second while completions stay at 100. The queue shrinks by a net 20 jobs per second, so clearing that backlog takes another 60 seconds.
These rates are fixed for a hypothetical calculation, not a real queue forecast. Average time and oldest-job age also depend on order and variability. If arrivals stay at 120 indefinitely while capacity stays at 100, the queue never drains. Buying more queue storage does not close that gap.
Track oldest-job age, backlog size, intake and completion rates, and time from acceptance to success. Do not watch only worker execution time: customers also wait before that. If a document has a business deadline, alert on the deadline rather than assuming a small-looking message count means everything is fine.
Bound intake and concurrency to what downstream services can handle
Backpressure pushes back when downstream capacity is insufficient. It may limit intake, shed optional work, or explicitly reject new work. Choose by business priority. Users should know whether a job was not accepted or was accepted with a longer wait. Do not show success just to improve the page metric.
More workers help when jobs can be divided and shared services have headroom. If every worker builds a PDF with a heavy read or calls the same API, extra workers may exhaust connections or quotas. Set concurrency by the weakest relevant service, separate urgent work from large reports where needed, and guard against one class waiting forever.
| Situation | Response to design | What operators must see |
|---|---|---|
| Destination temporarily unavailable | Retry within a bounded count and total time | Error, job age, and attempt count |
| Data fails validation | Route for correction; do not retry unchanged | Problem item and authorized fixer |
| May have succeeded but the response was lost | Check the result or prevent a duplicate effect | Intent key, prior outcome, and reconciliation history |
| Queue over capacity | Bound intake and shed work under business rules | Deadline impact and named decision owner |
Give retries a budget so they do not become a storm
Increasing delay between retries, with jitter to spread attempts, helps stop every request returning at once. Set attempt count, total time, and retryable error types. Bad permissions or incomplete data will not improve through faster repeats. Follow the service’s rate-limit response rules too.
If three layers each permit three total attempts, and all exhaust them, one request can trigger up to 27 downstream attempts. That is a worst-case calculation, not what happens every time. Define whether “three attempts” includes the first one; three *additional* retries would create a base of four instead.
Jobs past the threshold can move to an inspection queue or dead-letter queue. Give it an owner, retention period, and replay method that still honors duplicate prevention. Moving work out of the main queue can make its graph look better, but the customer has not received the result. Count that failure in service outcomes and cost.
Practice stopping and recovering before it is needed
Test a worker stopping after an outside effect but before acknowledging the message. Test duplicate delivery and restoring the queue with related transaction data. Use a test destination, not a real charge. Run a controlled volume with stop criteria and verify that final status matches the business result.
Disposable work and work that must not vanish need different rules
A preview-image request superseded by a newer version might be canceled under the rules. A document for an already confirmed order should not be discarded because the queue is long. Set retention and cancellation rights by job type. For work that cannot be dropped, limit new acceptance before promising more than can be delivered instead of accepting everything and deleting jobs later.
Queue messages and error queues need read permissions and retention limits. Pass a reference ID when that is enough; justify any stored details. Do not copy whole sets of secrets into logs and several queues until deletion or access revocation no longer works as designed.
With several customer accounts, test whether one account’s huge report delays another account’s essential job beyond its deadline. Separate queues or quotas are options to assess against actual load; one queue per person is not always needed. We still need fairness and an aggregate view, not a new queue that hides the old backlog.
Grows 20 jobs/second × 60 seconds = 1,200 jobs
Net drain 20 jobs/second; another 60 seconds
The queue builds and drains as arrival rate changes
Backlog (jobs)
Time since start (seconds)
- Hypothetical backlog
- Starts with no backlog
- 60 seconds / 1,200 jobs
- 120 seconds / fully drained
120 arrive / 100 finish per second · 80 arrive / 100 finish per second
Assumes no starting backlog, constant rates, no retries, and no dropped work. The queue grows when arrivals exceed completions and shrinks when completions exceed arrivals. These are not real queue test results.
I would let the screen say a job is accepted only when there is traceable evidence of acceptance. I would move work to a queue only when the business accepts the wait and someone owns exceptions. A faster page should not create invisible work.
Once work is clearly separated, we can choose where it runs. Some work belongs near users, some near data, and some should avoid an AI call on every request. The next episode uses those boundaries to assess edge and serverless options without outsourcing correctness to a technology label.
Read the sources and limits of the example
The following sources explain the concepts and limitations and were checked on October 18, 2026. The fictional store numbers and choices are my synthesis, not provider prices, test results, or guarantees. Check the current version, plan, and terms before applying them.