Retrieve AWS Secrets Manager Keys Across Accounts with a Python Script on EC2

When working with multiple AWS accounts, securely managing secrets can be challenging. In this tutorial, we’ll walk you through how to retrieve AWS Secrets Manager keys across accounts using a Python script on an EC2 instance. Looking to automate your secrets retrieval, this guide provides a practical, hands-on solution for secure access.

Scope

This blog covers how to use a Python script on an EC2 instance to securely retrieve AWS Secrets Manager keys across multiple AWS accounts. It covers the key setup steps, including proper IAM permissions, cross-account access, and Python code implementation, making it a hands-on guide for managing secrets in AWS.

Purpose

The purpose of this tutorial is to help developers and system administrators:

Understand how to set up cross-account access for AWS Secrets Manager.
Learn how to use Python to retrieve secrets on an EC2 instance.
Gain insight into best practices for secure secrets management in AWS environments.

By the end of the blog, readers will be able to securely manage AWS Secrets Manager keys across different accounts using Python scripts on EC2.

Prerequisites

Before you start, ensure you have the following:

Active AWS Accounts: Two separate AWS accounts set up.
Secrets in Secrets Manager: Secrets Manager created in AWS account A.
Running EC2 Instance: An EC2 instance created in AWS accounts B.
IAM User with Administrative permissions both AWS Account.

AWS Infrastructure Overview:

AWS Account A: Hosts the secret in Secrets Manager.
- Account ID: [AccountA-ID]
- Secret Name: dev/Project/ExampleSecretName
- Secret ARN: arn:aws:secretsmanager:[Region]:[AccountA-ID]:secret:dev/Project/ExampleSecretName-abc123
AWS Account B: Hosts the EC2 instance that will retrieve the secret.
- Account ID: [AccountB-ID]

Step-by-Step Process

Follow these steps to retrieve AWS Secrets Manager keys across accounts using a Python:

Step 1: Configure Permissions in AWS Account A

AWS Account A, navigate to the Secrets Manager console and update the resource policy for the secret:

Modify Secrets Manager Resource Permissions

{
  "Version": "2012-10-17", 
  "Statement": [
    {
      "Sid": "CrossAccountReadSecretsManagerKeys",
      "Effect": "Allow", 
      "Principal": {
        "AWS": "*"  // Allows any AWS account to access the secret
      },
      "Action": "secretsmanager:GetSecretValue",  // Grants permission to read the secret's value
      "Resource": "*",  // Applies to all secrets
      "Condition": {
        "StringEquals": {
          "aws:PrincipalOrgID": "o-s3xxxxx"  // Restricts access to a specific AWS Organization ID
        }
      }
    }
  ]
}

Note: Please remove the temporary comments within the JSON content to avoid syntax error.

Create an IAM role named EC2ReadSecretsManagerRole with the following trust policy:

Create an IAM Role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow", 
      "Principal": {
        "AWS": "arn:aws:iam::[AccountB-ID]:role/EC2InstanceAssumeRole"  // Specifies the role in Account B that can assume the role
      },
      "Action": "sts:AssumeRole"  // Grants permission to assume the specified role
    }
  ]
}

Attach the following IAM policy to the role created above:

Attach an IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAccountBAccess", 
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue",  // Grants permission to retrieve the secret's value
        "secretsmanager:DescribeSecret"  // Grants permission to view the secret's details
      ],
      "Resource": "arn:aws:secretsmanager:[Region]:[AccountA-ID]:secret:dev/Project/ExampleSecretName-abc123"  // Specifies the exact secret this policy applies to
    }
  ]
}

Provide your IAM Policy name e.g. EC2ReadSecretsManagerPolicy and save.

Step 2: Configure Permissions in AWS Account B

In AWS Account B, create an IAM role named EC2InstanceAssumeRole with the following trust policy:

Add EC2 in Trust Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow", 
      "Principal": {
        "Service": [
          "ec2.amazonaws.com"  // Allows EC2 instances to assume the role
        ]
      },
      "Action": "sts:AssumeRole"  // Grants permission to assume the role
    }
  ]
}

Attach the following IAM policy to the role:

Attach an IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow", 
      "Action": "sts:AssumeRole",  // Grants permission to assume the specified role
      "Resource": "arn:aws:iam::[AccountA-ID]:role/EC2ReadSecretsManagerRole"  // Specifies the role that can be assumed
    }
  ]
}

Provide your IAM Policy name e.g. EC2AssumeSecretsManagerRole and save.

Attach the Role to the EC2 Instance

Ensure this IAM role (EC2InstanceAssumeRole) is attached to your EC2 instance.

Step 3: Python Script to Retrieve Secrets

Now, create a Python script to retrieve the secrets from AWS Secrets Manager:

import os
import json
import boto3
from dotenv import load_dotenv
from botocore.exceptions import ClientError

# Load environment variables
load_dotenv()

def assume_role(role_arn):
    sts_client = boto3.client('sts')
    try:
        response = sts_client.assume_role(
            RoleArn=role_arn,
            RoleSessionName='EC2InstanceSession'
        )
        return response['Credentials']
    except ClientError as e:
        raise Exception(f"Error assuming role: {e}")

# Function to get secrets for EC2
def get_secrets(key):
    secret_name = os.getenv('SECRET_NAME')
    role_arn = os.getenv('ROLE_ARN')

    if not secret_name or not role_arn:
        raise ValueError("SECRET_NAME and ROLE_ARN must be set in the environment variables.")

    # Assume role to get temporary credentials
    credentials = assume_role(role_arn)

    # Create a Secrets Manager client with the temporary credentials
    client = boto3.client(
        'secretsmanager',
        region_name='us-east-1',
        aws_access_key_id=credentials['AccessKeyId'],
        aws_secret_access_key=credentials['SecretAccessKey'],
        aws_session_token=credentials['SessionToken']
    )

    try:
        response = client.get_secret_value(SecretId=secret_name)
        secret_value = response['SecretString']
        secret_dict = json.loads(secret_value)
        return secret_dict.get(key, None)
    except ClientError as e:
        raise Exception(f"Error retrieving secret: {e}")
    except json.JSONDecodeError:
        raise Exception("Error decoding the secret JSON.")

if __name__ == '__main__':
    # Retrieving the keys on Secrets Manager API keys or database credentials
    try:
        app_api_key = get_secrets('APP_API_KEY')
        database_credentials = get_secrets('DATABASE_CREDENTIALS')

        # Display Sample Output
        print("App API Key:", app_api_key)
        print("Database Credentials:", database_credentials)
    except Exception as e:
        print(f"An error occurred: {e}")

Save the above script on your EC2 instance and install the required libraries like boto3, dotenv, and Python. Then, run python script.py to validate access to Secrets Manager key values.

Final Step: Set Up Environment Variables

Ensure to create a .env file in your project directory before running the python script with the following required variables:

ROLE_ARN=arn:aws:iam::[AccountA-ID]:role/EC2ReadSecretsManagerRole
SECRET_NAME=dev/Project/ExampleSecretName

Conclusion

With this setup, your EC2 instance in Account B can securely retrieve secrets stored in Secrets Manager in Account A. This approach ensures that your sensitive information remains protected while allowing cross-account access where needed.