Taking The New Secrets Manager Lambda Extension For a Spin

If you have built automation utilizing Amazon Web Services (AWS) Lambda that needed to interact with an external service, it is likely you had to retrieve API keys from either Secrets Manager or Parameter Store. This pattern often looks something like this:

import boto3

def lambda_handler(event, context):
    client = boto3.client("secretsmanager")
    secret_arn = os.environ.get("MY_SECRET_ARN")
    response = client.get_secret_value(SecretId=secret_arn)
    secret = response['SecretString']

    make_external_call(secret)
    #profit

To summarize, I am importing the AWS Python SDK (Boto3), initializing a client for Secrets Manager, grabbing a provided Amazon Resource Name (ARN) from an environment variable, calling the Secrets Manager API to retrieve the secret, and calling our external service with it.

Back in October, AWS released Lambda extensions for AWS Secrets Manager and Parameter Store. This introduced a new pattern for grabbing application secrets for Lambda into the mix. In this blog, I walk through utilizing this method for retrieving secrets and compare it to the existing approach.

Context

A few key points to keep in mind:

• Secrets Manager charges per API call with a rate of $0.05/10k API calls.
• Making API calls to external systems (secrets manager) will always introduce some form of latency
• The AWS Python SDK (Boto3) is large, and initializing it likely has a performance impact
• AWS Lambda bills on a per-millisecond basis, shorter runtimes can result in large savings under the correct circumstances

The new Lambda Extension allows us to both retrieve secrets without initializing the SDK or calling the Secrets Manager API directly. It also implements configurable caching which helps reduce the total number of calls made to Secrets Manager altogether. With all of that in mind, let’s take it for a spin, and see what happens!

Setting Everything Up

After consulting the documentation, I learned that I need to:

1. Create a Lambda Function that retrieves a secret from the Lambda Extension instead of the SDK
2. Add the appropriate permissions to access the secret I want to the functions IAM Role
3. Attach the Lambda Extension as a Layer to the function

The Extension exposes the secrets via a local endpoint in the functions execution environment, requiring the AWS_SESSION_TOKEN environment variable as header on the request. It can be accessed at http://127.0.0.1:2773/secretsmanager/get by passing a query string parameter secretId with the name of the secret to retrieve. With that in mind, my resulting lambda looks like:

import json, requests, urllib.parse, os

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

patch_all()


def get_lambda_response(status_code: int, body: str):
    return {"statusCode": status_code, "body": body}


def get_secret(secret_id: str, version_stage: str = None) -> str:

    headers = {"X-Aws-Parameters-Secrets-Token": os.environ.get("AWS_SESSION_TOKEN")}
    request_uri = f"http://127.0.0.1:2773/secretsmanager/get?secretId={urllib.parse.quote(secret_id)}"
    if version_stage:
        request_uri = f"{request_uri}&versionStage={urllib.parse.quote(version_stage)}"
    response = requests.get(url=request_uri, headers=headers)

    return response.json()["SecretString"]


def lambda_handler(event, context):

    secret_name = "aquiablogsecret"
    region_name = "us-east-1"

    print(f"retrieved value: {get_secret(secret_id=secret_name)}")

    return {"statusCode": 200, "body": json.dumps("Hello from Lambda!")}

The secrets manager extension also has environment variables for configuration of things like the cache time-to-live, but I kept the defaults for this exercise.

In order to actually compare the methods, I also created a Lambda function that calls the secrets manager API directly upon each invocation.

import json
import boto3
from botocore.exceptions import ClientError
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

patch_all()


def get_secret():

    secret_name = "aquiablogsecret"
    region_name = "us-east-1"
    session = boto3.session.Session()
    client = session.client(service_name="secretsmanager", region_name=region_name)

    get_secret_value_response = client.get_secret_value(SecretId=secret_name)

    secret = get_secret_value_response["SecretString"]
    return secret


def lambda_handler(event, context):

    print(f"retrieved value: {get_secret()}")

    return {"statusCode": 200, "body": json.dumps("Hello from Lambda!")}

Both Lambda functions were created using the Python 3.9 runtime, arm64 architecture, and with 256 MB of memory.

I am instrumenting both Lambdas with the AWS X-Ray SDK. This will make testing performance easier in the next section, as I will be able to dig deeper into the durations of both Lambda functions.

Testing

Now that I have our two Lambda functions, I can test out the differences in function duration. To see if the Lambda Extension improves performance, I will continuously invoke both Lambda functions and monitor the durations. I wrote a small bash script to invoke both functions every second:

do
  sleep 1
  aws lambda invoke --function-name $MYFUNC results.txt
done

With our tests running, now I can take a look at the durations across the various invocations:

At first glance, there is a staggering difference between the duration of the functions. The Extension function floats in low 100 ms durations outside of its cold start (the ~300 ms outlier), with an average of around 12 ms. The SDK Lambda has an average of about 580 ms. I can utilize the AWS X-Ray instrumentation mentioned above to dig into where the differences in latency specifically are. See the image of the X-Ray service map below:

The X-Ray Service Map provides a nice visual representation of our two Lambda functions, the external services they make API calls to, and the average request time of each part of that process.

The service map for the AquiaBlog-SecretsManagerSDK lambda shows us that:

• The call to Secrets Manager introduces an average of 134 ms per invocation
• A majority of the additional latency appears to be from initializing the AWS SDK, as that is the only major difference between the two functions

The map for the Lambda utilizing the extension is much more straightforward, with the call being made to the local endpoint exposed by the extension.

I want to highlight a few points that could affect duration but were kept out for simplicity’s sake:

• I didn’t bundle the AWS SDK in the deployment package, instead just utilizing Boto3 that is provided in the runtime
• I avoided comparing a Lambda that utilizes caching across invocations (by implementing something like cachetools or lambda-cache) while still using the Secrets Manager API and SDK
• Your traffic patterns may not match the one request per second I used in the example, and that can yield different results based on Lambda’s concurrency model

Wrapping Up

The AWS Lambda Secrets Manager extension shows some promising performance benefits based on my testing above. The effort to implement is relatively low, and would likely result in less code vs. implementing your own caching. Definitely consider the unique aspects of your edge case/organization, but the Secrets Manager (or Parameter store) Lambda Extension is definitely worth a look!

If you have any questions, or would like to discuss this topic in more detail, feel free to contact us and we would be happy to schedule some time to chat about how Aquia can help you and your organization.