Using AWS Powertools in Fargate Containers

[dobeerman]

Hello everyone,

We're interested in using the Logger, Parameters and Datadog provider utilities across different environments.

Context

We have a mixed environment of Lambda functions (Python/TypeScript) and Fargate containers written in Python. We're currently integrating Datadog into our applications and migrating most of our Python/TypeScript Lambda loggers to aws-powertools.

In our containerized Lambdas, we're using code from our main application, which currently utilizes the standard Python logger. This raises questions about unifying our logging solution across different environments.

Goals:

1. Unified Logging: To maintain consistency and reusability, we'd like to use Powertools as a unified logger across all our services, including containerized Lambdas.
2. Datadog Metrics: We also use Datadog for custom metrics and would like to use Powertools for that as well. We're using the Datadog forwarder to push metrics to Datadog.
3. Parameters: We also use Powertools to fetch parameters, secrets, and AppConfig in our Lambdas. Would be nice to have the same ability in Fargate containers.

Questions:

1. Can the Logger utility be used in Fargate containers without any limitations?
2. Does the Datadog Metric provider support flushing metrics to standard output in a Fargate environment? I guess, yes, cuz under the hood it utilizes the same datadog packages.
3. Any limitations or upcoming features of Parameters we should be aware of using it in Fargate containers?

Looking forward to your insights and suggestions!

[leandrodamascena]

Hello @dobeerman! Thank you so much for opening this discussion and allowing us to help you with the adoption of Powertools for AWS Lambda. I'll try to cover as much as possible all your questions and let me know if I miss something.

We're interested in using the Logger, Parameters and Datadog provider utilities across different environments.

One of our tenants says: "AWS Lambda only. We optimise for AWS Lambda function environments and supported runtimes only. Utilities might work with web frameworks and non-Lambda environments, though they are not officially supported."

I started by bringing this statement to explain that you can use Powertools for AWS Lambda outside of the Lambda runtime, but it may have some limitations and unsupported features. This is because some functionality is closely linked to how AWS Lambda works and we can't modify these behaviours.

Questions:

Logger

Can the Logger utility be used in Fargate containers without any limitations?

You can use the Logger utility on Fargate containers, but it has some specific limitations. The Logger utility supports, among others, decorating the Lambda function with an @logger.inject_lambda_context and leveraging features such as: capturing Lambda context fields in the log and automatically logging the request. You can't use this decorator in Fargate as it is tightly coupled to the way Lambda works.
But basically, all the rest of the Logger utility works, which includes several features like: different log levels, adding extra fields, adding default keys, date formatting, standard log field formatting, and others. We can validate this in an example and verify that using these supported features meets the requirements of your application.

Metrics

Does the Datadog Metric provider support flushing metrics to standard output in a Fargate environment? I guess, yes, cuz under the hood it utilizes the same Datadog packages.

Yes, you can use Datadog Metric Provider and flush metrics to standard output using the standalone method flush_metrics(). You might not need to install any additional libraries for this if using Datadog Forwarder, as our provider will output the metric name, value, timestamp and tags in the format that Datadog expects.
Our provider does not support specifying the METRIC_TYPE and INTERVAL fields (as supported in the Datadog API), because this implementation is specific to Lambda and the Datadog extension does not yet support it. We are working to talk to the Datadog team and see how we can improve this further.

An important point here is that the @metrics.log_metrics decorator will not work in non-Lambda environments and consequently Cold Start captures (nor does it apply to Fargate environment) and automatic flush either. As mentioned in the previous paragraph, you will necessarily have to manually flush the metrics at the end of each request, but you can take advantage of middleware in frameworks like Flask, and FastAPI and do this automatically at the end of each request.

Parameters

Any limitations or upcoming features of Parameters we should be aware of using in Fargate containers?

I don't see any limitations on using the utility Parameter outside of Lambda environments, as long as you bring boto3 in as a dependency and your tasks/credentials are allowed to access the required resources in AWS it should work without any issues.

Example:

I created an example of a Flask application running in a container that we can use as a starting point for you to verify the viability of adopting Powertools for AWS Lambda in your Fargate containers.

Dockerfile:

FROM python:3.10
COPY ./* ./app/
WORKDIR /app/
RUN pip install -r requirements.txt
EXPOSE 3000
CMD ["python", "app.py"]

requirements.txt

aws-lambda-powertools==2.23.1
click==8.1.3
Flask==2.2.2
itsdangerous==2.1.2
Jinja2==3.1.2
jmespath==1.0.1
MarkupSafe==2.1.2
typing_extensions
Werkzeug==2.2.2

app.py:

from flask import Flask
from aws_lambda_powertools import Logger
from aws_lambda_powertools.metrics.provider.datadog import DatadogProvider
from aws_lambda_powertools.utilities import parameters



logger = Logger(service="PowertoolsAPP")
# Appending persistent keys across all future log messages
# See: https://docs.powertools.aws.dev/lambda/python/latest/core/logger/#appending-additional-keys
logger.append_keys(environment="Fargate")

# Using Datadog Forwarder method
# See: https://docs.powertools.aws.dev/lambda/python/latest/core/metrics/datadog/#integrating-with-datadog-forwarder
metrics = DatadogProvider(flush_to_log=True)

app = Flask(__name__)


@app.route('/')
def index():

    # Getting parameter
    powertools_hello: str = parameters.get_parameter("/lambda-powertools/powertools_hello")
    logger.info(powertools_hello)

    # Logging info log with ephemeral metadadata
    # See: https://docs.powertools.aws.dev/lambda/python/latest/core/logger/#ephemeral-metadata
    logger.info("Hello world", route="default", customer="dobeerman")

    # Adding a custom metric with additional tags
    # https://docs.powertools.aws.dev/lambda/python/latest/core/metrics/datadog/#adding-tags
    metrics.add_metric(name="HelloWorldPowertools", value=1, tag1="tag")

    # Flushing metrics manually 
    # See:https://docs.powertools.aws.dev/lambda/python/latest/core/metrics/datadog/#flushing-metrics-manually
    metrics.flush_metrics()

    return 'Hello World PowertoolsAPP'

app.run(host='0.0.0.0', port=3000)

Output:

❯ docker rm fargate-beyond; docker run --name fargate-beyond -p 3000:3000 fargate-beyond 
fargate-beyond
 * Serving Flask app 'app'
 * Debug mode: off
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
 * Running on all addresses (0.0.0.0)
 * Running on http://127.0.0.1:3000
 * Running on http://172.17.0.2:3000
Press CTRL+C to quit
{"level":"INFO","location":"index:25","message":"Hello world parameter","timestamp":"2023-08-27 21:04:51,569+0000","service":"PowertoolsAPP","environment":"Fargate"}
{"level":"INFO","location":"index:30","message":"Hello world","timestamp":"2023-08-27 21:04:51,685+0000","service":"PowertoolsAPP","environment":"Fargate","route":"default","customer":"dobeerman"}
172.17.0.1 - - [27/Aug/2023 21:04:51] "GET / HTTP/1.1" 200 -
{"m":"HelloWorldPowertools","v":1,"e":1693170291,"t":["tag1:tag"]}

Please let me know if you have any questions and we can evolve this scenario.

Thanks

[dobeerman]

Hey @leandrodamascena,

Thanks a lot for such a detailed breakdown!

Logger: Having no @logger.inject_lambda_context is ok for Fargate since we have no context. Instead, we can enrich the logger with some other data which are relevant to the container.

Datadog Metrics: I guess we can have the flush_metrics method called in a decorator of the main function when the process exits or terminates.

BTW, in Lambda, if I add capture_cold_start_metric=True to the @metrics.log_metrics decorator Pylance complains that Argument missing for parameter "context".

Parameters: No roadblocks? Awesome!

Any ETA on Datadog getting its act together with METRIC_TYPE and INTERVAL? Just fishing for some insider info. 😄

Thanks!

[leandrodamascena]

Hello @dobeerman! I am glad that this explanation can help you to clarify your questions.

Datadog Metrics: I guess we can have the flush_metrics method called in a decorator of the main function when the process exits or terminates.

Good idea, so you make it generic.

BTW, in Lambda, if I add capture_cold_start_metric=True to the @metrics.log_metrics decorator Pylance complains that Argument missing for parameter "context".

I tried to simulate here and it's working perfectly. Can you show me the code and Pylance warning?

Any ETA on Datadog getting its act together with METRIC_TYPE and INTERVAL? Just fishing for some insider info. 😄

Not yet, but we'll create an issue as soon as we start adding this feature, so stay tuned to the Powertools repository 😄

Let me know if I can help with any other requests, and if possible, let us know about your experience using our new Datadog Metric provider. It is very important for us to know from our customers what they think of the experience.

[dobeerman]

Here it is

[leandrodamascena]

I see! You have reportGeneralTypeIssues enabled in Pylance. It's complaining because this decorator implements the same signature as lambda_handler to capture the ColdStart metric AND flush metrics automatically after the Lambda execution is complete - https://github.com/aws-powertools/powertools-lambda-python/blob/develop/aws_lambda_powertools/metrics/provider/base.py#L202.

To be honest, I don't know if we can fix this or add something that Pylance doesn't complain about. Any idea @heitorlessa?

For now, you can disable checking for this line by adding # type: ignore?

Thank you