Certopolis

AWS Certified Solutions Architect – Associate

SQS standard vs FIFO queues, visibility timeout and DLQs

A standard queue gives nearly unlimited throughput with at-least-once delivery and best-effort ordering, so consumers must handle duplicates. A FIFO queue gives strict order within each message group and exactly-once processing, with lower throughput: 300 API calls per second per method, or 3,000 messages per second with batching.

Standard vs FIFO

Standard queues store copies of each message across multiple Availability Zones and can deliver a message more than once, so make processing idempotent. Order is best effort.

FIFO queues order messages within a message group ID; different groups can be processed in parallel. Every message needs a group ID. Duplicates are prevented with a MessageDeduplicationId or content-based deduplication, so a producer can safely retry a failed send with the same deduplication ID. High throughput mode raises the limit to 30,000 transactions per second with relaxed ordering across groups.

Visibility timeout

A received message stays in the queue but is hidden from other consumers for the visibility timeout. The default is 30 seconds. If the consumer does not delete it in time, it becomes visible again and another consumer can process it.

For jobs whose duration varies, AWS recommends a heartbeat that calls ChangeMessageVisibility to extend the timeout while work continues. The maximum is 12 hours from first receipt, and extending does not reset that limit. Setting the timeout to 0 releases a message at once.

Dead-letter queues and fan-out

A redrive policy on the source queue sets maxReceiveCount, the number of receives before a message moves to the dead-letter queue. The DLQ must be in the same account and Region. A redrive allow policy on the DLQ controls which source queues may use it; byQueue lists up to 10 source queue ARNs.

To fan out, subscribe several SQS queues to one SNS topic. SNS pushes a copy of each published message to every queue, and each consumer reads its own queue at its own pace.

Where the exam catches people

  • Believing a standard queue never delivers duplicates: delivery is at least once, so consumers must be idempotent.
  • Believing a long fixed visibility timeout is best for variable jobs: AWS recommends extending it with ChangeMessageVisibility as a heartbeat.
  • Believing the receive request attempt ID deduplicates sends: producers retry with the same message deduplication ID; the attempt ID is for consumer retries.
  • Believing maxReceiveCount is set on the dead-letter queue: it goes in the redrive policy on the source queue.
  • Believing a dead-letter queue can be in any Region: it must be in the same account and Region as the source queue.

Written from Amazon Web Services's documentation

Try 3 questions on this

Pick an answer to see why each option is right or wrong.

  1. An Amazon SQS consumer processes video files, and the work takes between two minutes and four hours depending on the file. Some long jobs are being processed twice, because the message reappears while the first worker is still running. The queue's visibility timeout is set to ten minutes. The company needs each file processed only once, and a message whose worker fails must become available for retry without a long delay. Which solution meets these requirements?

    Right. In the game that’s Renown for your Design Resilient Architectures building.

    Not that one: A is right. In the game your Design Resilient Architectures building would lose a little Renown, and this question would come back later.

    Show the full explanation and doc quotes

    And it names the instrument: "If your processing time varies or may exceed the initially set timeout, use the ChangeMessageVisibility action to extend the visibility timeout while processing the message." One caution travels with it - the ceiling is absolute: "the visibility timeout has a maximum limit of 12 hours from when the message is first received. Extending the timeout doesn't reset this 12-hour limit."

    Written from docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

  2. A producer sends a message to an Amazon SQS FIFO queue, receives no response because of a network fault, and retries the same send. A solutions architect must ensure the retry cannot place a duplicate on the queue or disturb the order of the messages around it. Which approach will meet these requirements?

    Right. In the game that’s Renown for your Design Resilient Architectures building.

    Not that one: D is right. In the game your Design Resilient Architectures building would lose a little Renown, and this question would come back later.

    Show the full explanation and doc quotes

    The retries "do not introduce duplicate messages" and "do not disrupt message order." The consumer has the mirror-image mechanism for its own failures - "the consumer can retry as many times as necessary using the same receive request attempt ID" - and the two identifiers are not interchangeable.

    Written from docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-understanding-logic.html

  3. Operations monitors one Amazon SQS dead-letter queue for two source queues, orders and payments. Last month a test queue's redrive policy was pointed at it by mistake and flooded it with test messages. Messages from orders and payments must move to the dead-letter queue after four failed receives, and no other queue may ever use it. Which combination of steps should a solutions architect take to meet these requirements? (Select TWO.)

    Right. In the game that’s Renown for your Design Resilient Architectures building.

    Not that one: C,D is right. In the game your Design Resilient Architectures building would lose a little Renown, and this question would come back later.

    Show the full explanation and doc quotes

    On the source side, "Use a redrive policy to specify the maxReceiveCount. The maxReceiveCount is the number of times a consumer can receive a message from a source queue before it is moved to a dead-letter queue." On the dead-letter side, "The redrive allow policy specifies which source queues can access the dead-letter queue", and "The default allows all source queues to use the dead-letter queue. If you choose to allow specific queues using the byQueue option, you can specify up to 10 source queues using the source queue Amazon Resource Name (ARN)."

    Written from docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html