Home / server / Questions / 744983
In Process
Chida
Asked: 2015-12-24 06:14:09 +0800 CST

AWS SQS + SNS + Lambda

  • 772

I was wondering if I can send a message to SQS queue and subscribe an SNS topic to it to trigger a lambda for sending an email.

SQS -> SNS -> (Lambda) -> SES

I know SNS messages can be sent to SQS but I'm curious if the other way around is possible

amazon-web-services

4 Answers

  1. One thing I did was to create a CloudWatch alarm on ApproximateNumberOfMessagesVisible (>= 1 for 5 minutes) for the SQS queue. The alarm publishes to an SNS topic that triggers the lambda function. The lambda function loops until it clears the queue.

    It can take up to 5 minutes to trigger from the alarm, but it works fantastically for batch-scheduled tasks without needing to poll the queue. (Alarm granularity is 5 minutes for active queues.)

    • 11

  2. You can't go SQS -> SNS, only SNS -> SQS.

    Lambda now supports scheduling so one option is to implement an SQS poller in a Lambda function and run it frequently.

    Another option to consider is whether you actually need a queue. Lambda supports asynchronous processing (via the Event invocation mode) and should transparently scale horizontally to handle parallel invocations. If your lambda function doesn't require access to a central state store which might constrain parallel execution then you could probably just run all your invocations in parallel. I believe there's a 100 concurrent execution limit per account though, so you may need to batch your messages to stay under that.

    • 7

  3. SQS queue can be subscribed to SNS topic and so to process received SNS messages. Currently, it is not doable in other direction without additional coding (see e.g. Lambda FAQ).

    I would say there is a couple of options how to do it but it is not so elegant as using more common event-driven system AWS event->SQS->Lambda. Otherwise you may need to customize/implement the code how SQS queues are processed:

    1. you can implement your own event sources
    2. you can have some intermediate EC2 instance to listen to SQS queues and then to trigger Lambda on SQS events
    • 1

  4. This was asked and answered a while ago, but having just thought about this myself, I thought I'd add an approach.

    As mentioned, Event Sources may be the best bet here. Alternatively, and I haven't tested this nor thought this through (so this is kind of academic), but it may be possible to accomplish this via a Fan-Out pattern with SNS as follows:

     1. Create a SNS topic.............................: SNS-topic-01
 2. Subscribe a SQS queue to that topic............: SQS-queue-01
 3. Subscribe a Lambda Function to that topic......: LAMBDA-func-01

    Using this configuration, submitting a message to the SNS topic will enqueue it to the SQS queue while simultaneously triggering a companion Lambda function. That Lambda function would be written to read that very same SQS queue but with Long Polling enabled (up to 20 seconds) so that it doesn't read the queue before the enqueue completes (i.e. race condition).

    In essence, this scheme just-in-time invokes one Lambda function for each enqueued SQS message. I don't know how simultaneous Long Poll readers work on SQS (... does one get dropped?), but this is just another way to consider solving this. =:)

    • 0