How to Extract S3 Payload from SQS + SNS Event in AWS Lambda

How to Extract S3 Payload from SQS + SNS Event in AWS Lambda

by

When S3 sends notifications through SNS and then SQS before reaching your Lambda function, the actual S3 event data is buried under multiple layers of JSON. Instead of a clean S3 event, you get an SQS message wrapping an SNS notification wrapping the S3 payload. This guide shows you how to extract the S3 payload from an SQS + SNS event in a Python Lambda function.

Why the Event Is Nested

In this architecture, the event passes through three services before reaching Lambda:

  1. S3 detects a file upload and sends a notification to an SNS topic
  2. SNS wraps the S3 event in its own message envelope and forwards it to an SQS queue
  3. SQS wraps the SNS message in its own record envelope and delivers it to Lambda

Each service adds its own metadata around the original payload. By the time Lambda receives the event, the S3 bucket name and object key are three JSON layers deep.

The Event Structure

Here’s what the Lambda event looks like when it comes through the S3 → SNS → SQS path. The layers from outside in:

{
  "Records": [                          // SQS layer
    {
      "messageId": "abc-123",
      "body": "{                         // SNS envelope (JSON string)
        \"Type\": \"Notification\",
        \"Message\": \"{                 // S3 event (JSON string inside JSON string)
          \\\"Records\\\": [{
            \\\"s3\\\": {
              \\\"bucket\\\": { \\\"name\\\": \\\"my-bucket\\\" },
              \\\"object\\\": { \\\"key\\\": \\\"uploads/file.csv\\\" }
            }
          }]
        }\"
      }"
    }
  ]
}

The key thing to notice: body is a JSON string (not a dict), and inside it, Message is also a JSON string. You need to call json.loads() twice to get to the S3 data.

Extracting the S3 Payload Step by Step

Step 1: Parse the SQS Record Body

Each SQS record has a body field that is a JSON string containing the SNS envelope.

import json

def lambda_handler(event, context):
    for sqs_record in event["Records"]:
        sns_envelope = json.loads(sqs_record["body"])

Step 2: Parse the SNS Message

Inside the SNS envelope, the Message field contains the original S3 event — again as a JSON string.

        s3_event = json.loads(sns_envelope["Message"])

Step 3: Extract Bucket and Key

Now you have the standard S3 event structure. Loop through its Records to get the bucket name and object key.

        for s3_record in s3_event["Records"]:
            bucket = s3_record["s3"]["bucket"]["name"]
            key = s3_record["s3"]["object"]["key"]
            print(f"File: s3://{bucket}/{key}")

Complete Lambda Function

Here’s the full handler with error handling and logging:

import json

def lambda_handler(event, context):
    for sqs_record in event["Records"]:
        try:
            # Layer 1: Parse SQS body to get SNS envelope
            sns_envelope = json.loads(sqs_record["body"])

            # Layer 2: Parse SNS Message to get S3 event
            s3_event = json.loads(sns_envelope["Message"])

            # Layer 3: Extract S3 details
            for s3_record in s3_event["Records"]:
                bucket = s3_record["s3"]["bucket"]["name"]
                key = s3_record["s3"]["object"]["key"]
                size = s3_record["s3"]["object"].get("size", 0)
                event_name = s3_record["eventName"]

                print(f"Event: {event_name}")
                print(f"File: s3://{bucket}/{key} ({size} bytes)")

                # Your processing logic here
                process_file(bucket, key)

        except (json.JSONDecodeError, KeyError) as e:
            print(f"Failed to parse record: {e}")
            print(f"Raw body: {sqs_record['body']}")
            raise

    return {"statusCode": 200, "body": "OK"}


def process_file(bucket, key):
    """Replace with your actual file processing logic."""
    print(f"Processing s3://{bucket}/{key}")

The raise after logging the error is intentional — it causes Lambda to report the SQS message as failed so it returns to the queue for retry. If you’re seeing messages get retried too many times, check How to Fix SQS Visibility Timeout in AWS Lambda.

Handling S3 Test Events from SNS

When you first set up an S3 notification to SNS, S3 sends a test event to confirm the subscription. This test event doesn’t contain S3 records — it’s just a confirmation message. If your Lambda doesn’t account for this, it will crash on deployment.

Add a check at the top of your parsing logic:

sns_envelope = json.loads(sqs_record["body"])

# Skip SNS subscription confirmations and test events
if sns_envelope.get("Type") == "SubscriptionConfirmation":
    print("Received SNS subscription confirmation, skipping")
    continue

s3_event = json.loads(sns_envelope["Message"])

# Skip S3 test events
if s3_event.get("Event") == "s3:TestEvent":
    print("Received S3 test event, skipping")
    continue

Helper Function for Reuse

If you have multiple Lambda functions processing the same S3 → SNS → SQS pattern, pull the unwrapping logic into a helper:

import json
from urllib.parse import unquote_plus


def extract_s3_records(event):
    """Extract S3 bucket/key pairs from an SQS + SNS + S3 event."""
    s3_records = []

    for sqs_record in event["Records"]:
        sns_envelope = json.loads(sqs_record["body"])

        if sns_envelope.get("Type") == "SubscriptionConfirmation":
            continue

        s3_event = json.loads(sns_envelope["Message"])

        if s3_event.get("Event") == "s3:TestEvent":
            continue

        for s3_record in s3_event["Records"]:
            bucket = s3_record["s3"]["bucket"]["name"]
            key = unquote_plus(s3_record["s3"]["object"]["key"])
            s3_records.append({"bucket": bucket, "key": key})

    return s3_records

The unquote_plus() call decodes URL-encoded characters in the object key. S3 encodes special characters (spaces become +, others become %XX), so without this you’ll get a wrong key when the filename has spaces or special characters.

Usage in your handler:

def lambda_handler(event, context):
    for record in extract_s3_records(event):
        print(f"Processing s3://{record['bucket']}/{record['key']}")
        # Your logic here

Quick Comparison: Direct S3 vs S3 + SNS + SQS

For reference, here’s how the extraction differs depending on how S3 triggers your Lambda:

Trigger Path Layers to Parse json.loads() Calls
S3 → Lambda (direct) 0 — event is already a dict 0
S3 → SQS → Lambda 1 — SQS body is a JSON string 1
S3 → SNS → SQS → Lambda 2 — SQS body + SNS Message 2

If you want a deeper look at how event and context work across different Lambda triggers, see Understanding Lambda’s Event and Context Parameters.

Conclusion

The S3 → SNS → SQS → Lambda pattern is common in event-driven architectures, but the nested JSON catches people off guard. The fix is straightforward: parse the SQS body to get the SNS envelope, then parse the SNS Message to get the S3 event. Add checks for test events and URL-decode the object key, and you’re set.

If you’re working with cross-account S3 notifications, see Setting Up Cross-Account SNS to SQS Subscription in AWS. For uploading files across accounts with Lambda, check out Setting Up Cross-Account S3 Upload with Lambda.