Use SQS to trigger functions in Lambda -- finally

With AWS Lambda, a service or resource can initiate a serverless function to perform a variety of tasks. And new support gives Lambda users something they've wanted for years.

Developers can now use Amazon Simple Queue Service (SQS) as a direct event source for Lambda to trigger functions. With this support, developers can eliminate workarounds, such as using a Lambda function to periodically poll for messages, and instead trigger functions immediately after a new SQS message is available.

Unlike other Lambda event sources, SQS uses a queue pattern, which enables events to throttle more easily and automatically retry if they fail. Unlike Amazon Kinesis, SQS provides visibility into how many messages -- or events -- still need to be processed and how many have been deleted. It also has support for dead-letter queues, which enables the Lambda function to automatically retry processing a message a certain number of times before sending it to a failure queue -- which could retry at a later date or alert developers to a potential issue.

SQS metrics enable all of these alarms and automatic retries. Now that SQS can directly trigger a Lambda function, we can make all of those deliveries serverless.

To get started, developers need to add SQS as a new event source from AWS Management Console or the API. Configure the name of the queue that will trigger functions and how many messages a single Lambda function should accept at a time:

For example, if a queue has 10 messages sent to it and the batch size is 1, then 10 Lambda executions would trigger simultaneously. If the batch size is 10, then only one Lambda execution would trigger, but all 10 messages would be sent to it.

Additionally, developers can use this batch size in connection with Reserved Concurrency settings in Lambda to set how many messages can process at once, which can be helpful to throttle usage or prevent it from flooding other systems, such as content delivery to an FTP server.

Before you can configure the SQS event source, you must add these permissions to your Identity and Access Management role for the Lambda function:

• sqs:DeleteMessage;
• sqs:ReceiveMessage; and
• sqs:GetQueueAttributes.

Once your Lambda function has access to those three permissions, simply add the event source in the console, and save. As with other events, SQS messages trigger functions with one Record for each message received. Each record has a:

• messageId;
• receiptHandle;
• body;
• attributes;
• optional messageAttributes;
• md5OfBody;
• eventSource of aws:sqs;
• eventSourceARN; and
• awsRegion.

If the function exits successfully, Lambda will delete all messages received. If an error pops up, the SQS messages will be kept in the queue and reattempted at a later time. By setting the default queue visibility timeout and maximum number of messages received, you can determine how many times the message will be retried, as well as the duration between retries.