- Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components
- https://aws.amazon.com/sqs/faqs/
- Queue message service. Can have EC2+AutoScalingGroup-hosted apps process messages in queue
- Provides decoupling of services – messages a written into a queue by one and read by another
- Polling service – check for messages. Not pushing (use SNS for pushing)
- Name – 80 char max; unique within region/account
- Can have SQS write into SNS (SNS subscribes) or Lambda
- Can have SNS publish to an SQS queue
- REGIONAL - can NOT share messages between queues in different regions. However, can have an EC2 write/read to/from an SNS queue in a different region. Charges apply to both reads and writes. Data transfer in same region is free
- Queue message service. Can have EC2+AutoScalingGroup-hosted apps process messages in queue
- Provides decoupling of services – messages a written into a queue by one and read by another
- Polling service – check for messages. Not pushing (use SNS for pushing)
- Name – 80 char max; unique within region/account
- Can have SQS write into SNS (SNS subscribes) or Lambda
- Can have SNS publish to an SQS queue
- REGIONAL - can NOT share messages between queues in different regions. However, can have an EC2 write/read to/from an SNS queue in a different region. Charges apply to both reads and writes. Data transfer in same region is free
Standard Queue
- Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action
- At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered (duplicates are possible)
- Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent
- Send data between applications when the throughput is important, for example:
§ Decouple live user requests from intensive background work: let users upload media while resizing or encoding it
§ Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.
§ Batch messages for future processing: schedule multiple entries to be added to a database.
- In-flight messages (read but not yet deleted) – 120,000 max
- Unlimited Throughput: Standard queues support a nearly unlimited number of transactions per second (TPS) per API action
- At-Least-Once Delivery: A message is delivered at least once, but occasionally more than one copy of a message is delivered (duplicates are possible)
- Best-Effort Ordering: Occasionally, messages might be delivered in an order different from which they were sent
- Send data between applications when the throughput is important, for example:
§ Decouple live user requests from intensive background work: let users upload media while resizing or encoding it
§ Allocate tasks to multiple worker nodes: process a high number of credit card validation requests.
§ Batch messages for future processing: schedule multiple entries to be added to a database.
- In-flight messages (read but not yet deleted) – 120,000 max
FIFO Queue
- High Throughput: FIFO queues support up to 300 messages per second (300 send, receive, or delete operations per second). When you batch 10 messages per operation (maximum), FIFO queues can support up to 3,000 messages per second. To request a limit increase, file a support request
- First-ln-First-out Delivery: The order in which messages are sent and received is strictly preserved.
- Exactly-Once Processing: A message is delivered once and remains available until a consumer processes and deletes it. Duplicates are not introduced into the queue - No duplicates guarantee.
- Available not in all regions
- In-flight messages (read but not yet deleted) – 20,000 max
- Priced per million requests
- Send data between applications when the order of events is important, for example:
§ Ensure that user-entered commands are executed in the right order
§ Display the correct product price by sending price modifications in the right order
§ Prevent a student from enrolling in a course before registering for an account
- Request – any SQS action
§ Can contain batches of 1-10 messages, up to 256KB
§ Each 64KB is a chunk, a chunk is one request
§ A message size can be 1-256KB
§ Can store message in S3 (charges apply)
§ Data transfer b/w EC2 and SQS in same region – free; across regions – chargeable
- Amazon SQS long polling is a way to retrieve messages from your Amazon SQS queues. While the regular short polling returns immediately, even if the message queue being polled is empty, long polling doesn’t return a response until a message arrives in the message queue, or the long poll times out
- Same price for million requests, Short or Long
- Short (default)
§ doesn’t query all servers at once – just a subset (uses weighted random distribution)
§ doesn’t wait for messages to appear in queue; no messages – comes back w nothing, ReceiveMessageWaitTime = 0
- Same price for million requests, Short or Long
- Short (default)
§ doesn’t query all servers at once – just a subset (uses weighted random distribution)
§ doesn’t wait for messages to appear in queue; no messages – comes back w nothing, ReceiveMessageWaitTime = 0
-
Long
§ Will query all servers
§ Will wait in the queue up to 20 sec for messages to appear:
ReceiveMessageWaitTime >0 but max 20 sec
§ Preferred to short. Fewer requests; eliminates false empty responses.
§ Do not use if the application expects immediate response to a query
§ Will query all servers
§ Will wait in the queue up to 20 sec for messages to appear:
ReceiveMessageWaitTime >0 but max 20 sec
§ Preferred to short. Fewer requests; eliminates false empty responses.
§ Do not use if the application expects immediate response to a query
- a message remains in the queue after writing for 1 min to 14 days
- afterwards message deleted automatically
- Messages can be written and read from queue at same time
- How many times a message can be
read and not processed before its is moved into a separate queue
- Source: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
- When a consumer receives and processes a message from a queue, the message remains in the queue. Amazon SQS doesn't automatically delete the message. Because Amazon SQS is a distributed system, there's no guarantee that the consumer actually receives the message (for example, due to a connectivity issue, or due to an issue in the consumer application). Thus, the consumer must delete the message from the queue after receiving and processing it
- Immediately after a message is received, it remains in the queue. To prevent other consumers from processing the message again, Amazon SQS sets a visibility timeout, a period of time during which Amazon SQS prevents other consumers from receiving and processing the message
§ The default is 30 seconds.
§ The minimum is 0 seconds.
§ The maximum is 12 hours.
- Reader can extend Visibility timeout to allow more time for processing
- After a message is read, two things are possible:
§ ACK from reader – SQS removes msg from queue
§ FAIL from reader OR Visibility timeout expires – SQS unlocks the message for reading
- all queues redundant, single region multi-AZ
- IAM policies can control who writes/reads
- Apps can keep the messages encrypted
- Supports HTTPS and TLS 1.0, 1.1, 1.2 in all regions
- PCI DSS (credit card payment) and HIPPA (health insurance) compliant
- Server-Side Encryption (SSE) – optional chargeable via KMS. Encrypted upon arrival into queue; AES-256
- Not available in ALL regions. Covers Standard and FIFO queues
- Messages encrypted only after encryption is enabled – no backdating
- Avail for active queues: within 6 hours of containing messages or an API call to access it
- Integrated, free. Metrics sent every 5 min
- Detailed monitoring (1 min) not available yet
CloudTrail
- All API calls: read/write, who, when
No comments:
Post a Comment