The first surprise with SQS is that ReceiveMessage doesn’t take a message off the queue. It lends it to you. The message stays in the queue, hidden from other consumers for a while, and if you don’t delete it before that while is up, it comes back. Everything else about reliable SQS processing follows from that one fact.
Receive, hide, delete: the three-step contract
A consumer’s loop has three moves, and each one has a specific meaning:
- Receive. SQS hands you up to 10 messages, each with a body and a receipt handle. Each message is now invisible to other
ReceiveMessagecalls for the queue’s visibility timeout (30 seconds by default, configurable from 0 seconds to 12 hours). - Process. You do the work: write to a database, call a payment API, resize an image.
- Delete. You call
DeleteMessagewith the receipt handle. That’s the acknowledgement. Until you do it, SQS assumes you might have crashed.
If the visibility timeout expires before step 3, the message becomes visible again and the next consumer to poll gets it. SQS doesn’t know or care whether your first worker is still running. It has no heartbeat from you unless you send one.
This is why standard queues are described as at-least-once delivery. Besides the timeout case, SQS stores messages redundantly across servers, and on rare occasions a receive can return a copy that a different server still thought was undeleted. You can’t configure duplicates away on a standard queue. You design for them.
A detail that catches people out: every receive produces a new receipt handle. The docs say you must delete with the most recently received handle; an older one can return success without actually deleting the message. So “the delete call returned 200” is not proof the message is gone if someone else received it in between.
Watching a duplicate happen
Here’s a toy model in Python. It isn’t SQS, just the timing rules above: a message becomes visible again when its timeout lapses, and whoever polls next takes it. The job charges a card and takes 45 seconds.
# Toy model of one SQS message and its visibility timeout. Not the real service:
# it only shows when the message becomes visible again and who receives it.
def run(visibility, job_seconds, heartbeat=None):
invisible_until, deleted = 0, False
running, log = {}, []
for t in range(200):
for w, start in list(running.items()):
if heartbeat and t > start and (t - start) % heartbeat == 0:
invisible_until = t + visibility # ChangeMessageVisibility
log.append(f"t={t:<3} {w} extends visibility to t={invisible_until}")
if t == start + job_seconds:
log.append(f"t={t:<3} {w} charges the card and deletes")
deleted = True
del running[w]
if not deleted and t >= invisible_until:
w = f"worker-{len(log) + 1}"
running[w] = t
invisible_until = t + visibility
log.append(f"t={t:<3} {w} receives the message")
if deleted and not running:
break
return log
for label, kw in [("visibility 30s, job takes 45s", dict(visibility=30, job_seconds=45)),
("same, plus a heartbeat every 20s", dict(visibility=30, job_seconds=45, heartbeat=20))]:
print(f"# {label}")
print("\n".join(run(**kw)))Running it with python3:
# visibility 30s, job takes 45s
t=0 worker-1 receives the message
t=30 worker-2 receives the message
t=45 worker-1 charges the card and deletes
t=75 worker-2 charges the card and deletes
# same, plus a heartbeat every 20s
t=0 worker-1 receives the message
t=20 worker-1 extends visibility to t=50
t=40 worker-1 extends visibility to t=70
t=45 worker-1 charges the card and deletesIn the first run the customer is charged twice. Nothing failed; no exception was thrown; both workers did exactly what the code said. And because worker-2 received the message after worker-1, worker-1’s delete used a stale receipt handle, so in the real service that first delete may not have removed anything.
The second run shows the fix for legitimately slow work: a heartbeat that calls ChangeMessageVisibility while the job is still going, pushing the deadline out each time. The total time a message can stay invisible is capped at 12 hours from the first receive, so a heartbeat buys time but not forever.
Choosing the timeout and extending it
The rule of thumb is to set the visibility timeout comfortably above the time a normal message takes at its slowest, not its average. If p99 processing is 20 seconds, 30 seconds is tight; 60 is reasonable. Too long has its own cost: when a worker really does crash, the message sits invisible for the whole timeout before anyone retries it.
When processing time varies a lot (a video that’s 10 seconds or 40 minutes), a fixed timeout won’t fit both. Keep the queue’s timeout modest and have the worker extend it. In boto3 that looks like this (illustrative; the queue URL is made up and nothing here was run against AWS):
import boto3, threading
sqs = boto3.client("sqs")
QUEUE = "https://sqs.eu-west-1.amazonaws.com/111122223333/transcode-jobs"
def keep_alive(receipt_handle, stop, every=60, extend_to=120):
while not stop.wait(every):
sqs.change_message_visibility(
QueueUrl=QUEUE, ReceiptHandle=receipt_handle, VisibilityTimeout=extend_to)
resp = sqs.receive_message(QueueUrl=QUEUE, MaxNumberOfMessages=1, WaitTimeSeconds=20)
for msg in resp.get("Messages", []):
stop = threading.Event()
threading.Thread(target=keep_alive, args=(msg["ReceiptHandle"], stop), daemon=True).start()
try:
transcode(msg["Body"]) # your work
sqs.delete_message(QueueUrl=QUEUE, ReceiptHandle=msg["ReceiptHandle"])
finally:
stop.set()Two things are worth pointing out. VisibilityTimeout in change_message_visibility sets a new timeout counted from now, not an addition to the old one. And WaitTimeSeconds=20 turns on long polling, which cuts down on empty responses and the cost of polling an idle queue.
If Lambda is your consumer, Lambda does the receiving and deleting for you. AWS recommends setting the queue’s visibility timeout to at least six times the function timeout, so that Lambda’s own retries on throttling don’t cause the message to reappear mid-flight.
Delete last, and make the work safe to repeat
Some teams “fix” duplicates by deleting the message right after receiving it. That trades duplicates for loss: if the worker dies after the delete and before the database write, the message is gone and nobody will ever process it. Delete after the result is durably stored.
That still leaves a window. The worker can write to the database and crash before DeleteMessage. The message returns, and the work runs again. So the side effect itself has to be idempotent:
- Put a business key in the message (
orderId: "ord_7731") and record processed keys with a conditional write, so the second attempt sees “already done” and just deletes. - Pass an idempotency key to external APIs that support one, such as payment providers.
- Prefer state-setting operations (“set status to SHIPPED”) over increments (“add 1 to shipped count”).
FIFO queues narrow the problem but don’t remove it. They de-duplicate messages sent with the same deduplication ID within a five-minute window, and they deliver messages in order within a MessageGroupId. While one message in a group is in flight, later messages in that group wait. But if your consumer exceeds the visibility timeout, the same message is still received again. Exactly-once processing in FIFO is a statement about sending and delivery, not about your worker’s side effects.
Poison messages and the dead-letter queue
Some messages will never succeed: a malformed payload, a reference to a deleted record. Without a limit they cycle forever, burning compute and hiding newer work. A redrive policy fixes that. You attach a dead-letter queue and set maxReceiveCount; once a message has been received that many times without being deleted, SQS moves it to the DLQ.
Pick maxReceiveCount with your timeout in mind. With a 60-second timeout and maxReceiveCount of 5, a failing message gets about five minutes of attempts. Set it too low (1 or 2) and a brief downstream outage dumps healthy messages into the DLQ. Then alarm on ApproximateNumberOfMessagesVisible for the DLQ, because a DLQ nobody watches is just a slower way to lose data. Once the cause is fixed, a redrive moves the messages back to the source queue.
Also remember retention. Messages are kept for 4 days by default and at most 14. A DLQ’s retention should be longer than the source queue’s, since a message’s age keeps counting from when it was first sent.
Interview practice: the double-charged order
An interviewer describes a system: an order service sends a message per order to a standard SQS queue, and a worker charges the card. Some customers are charged twice, mostly during busy periods. The visibility timeout is the default. What’s happening and how would you fix it?
Answer
The default visibility timeout is 30 seconds. During busy periods the payment call slows down, so some jobs take longer than 30 seconds, the message reappears, and a second worker charges again. That matches “mostly during busy periods” better than random SQS duplicates, which are rare.
I’d fix it in layers. First, measure processing time and raise the timeout above the slow tail, or add a heartbeat with ChangeMessageVisibility for jobs that can run long. Second, make the charge idempotent, because duplicates can still happen from crashes between charge and delete, or from SQS itself: send the order ID as the payment provider’s idempotency key, and record “charged” with a conditional write so a repeat sees it and skips. Third, make sure the delete happens after the charge is recorded, never before. Finally, add a redrive policy and a DLQ alarm so orders that fail repeatedly get human attention rather than looping.
If they ask “why not FIFO?”, the answer is that FIFO dedupes sends within five minutes and orders delivery per group, but a slow worker still sees the message again after the timeout. The idempotent charge is what actually protects the customer.