feat: add support for function logs streaming to sandbox #1492

Amplifiyer · 2024-05-10T16:03:07Z

Changes

Add support for streaming function logs in sandbox

CLI
- Default to disabled with an explicit opt-in --stream-function-logs
- Allow customers to specify a list of function names for which logs should be streamed
- Sandbox with --once should not stream function logs
- Allow specifying a file path to stream all the function's execution log events to.
Display
- Display log events with the "friendly name" of the function in a specific color. Total of 5 colors used in round robin
- Display a timestamp for each log event
- Colors are turned off for file writing.
Functionality
- While sandbox is idle, function logs are streamed in parallel for all the functions as they are executed.
- Log streaming is paused during a deployment and resumes from the time the deployment was initiated to avoid intermingling the logs
- New functions, deleted functions are automatically updated after a deployment, i.e. if customer adds a new function, it's logs will be automatically streamed (if no filter was used)

Validation

Unit tests added

Checklist

If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
If this PR requires a change to the Project Architecture README, I have included that update in this PR.
If this PR requires a docs update, I have linked to that docs PR above.
If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

changeset-bot · 2024-05-10T16:03:12Z

🦋 Changeset detected

Latest commit: c4d423e

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@aws-amplify/cli-core	Minor
@aws-amplify/sandbox	Minor
@aws-amplify/backend-cli	Minor
create-amplify	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

packages/sandbox/src/sandbox_singleton_factory.ts

edwardfoyle · 2024-05-10T16:49:28Z

packages/sandbox/src/lambda_function_log_streamer.ts

+    const backendOutput: BackendOutput =
+      await this.backendOutputClient.getOutput(sandboxBackendId);
+
+    const definedFunctionsPayload =
+      backendOutput[functionOutputKey]?.payload.definedFunctions;
+    const deployedFunctionNames = definedFunctionsPayload
+      ? (JSON.parse(definedFunctionsPayload) as string[])
+      : [];


Can we use DeployedBackendClient.getBackendMetadata() instead? That returns a list of FunctionConfiguration objects which should eliminate the need to depend on backend-output-schemas directly in this package

We shouldn't. DeployedBackendClient.getBackendMetadata() is a pretty heavy command as it loads all the nested stacks and all resources.

We need something as lightweight as possible since this method gets called every time sandbox gets idle.

Maybe we need some sort of "filter" prop for getBackendMetadata where it can be instructed to only load metadata from certain verticals? So in this case, we could use it to only load function metadata

We don't even need the function stack as technically what we are loading is not a metadata unless we start adding the "friendly name" in the metadata or outputs section. In that case we can continue to use getOutput here.

I also don't see any issues with depending on backend-output-schemas here. backendOutputClient.getOutput is the right abstraction here.

Taking a closer look at this, it looks like we're only using the functionOutputKey from that package. However, getOutput should already have typed keys so we can just use the string directly here and we'll get typechecking without having to import the schema package.

I will update it to using the key "AWS::Amplify::Function" directly, but wouldn't it just add duplicity. Not that we are ever going to change it but I don't understand the drawback of this dependency?

It's not a huge deal. In my head, the deployed-backend-client is the "entry point" by which all readers should get info about the backend. the schema package is meant to be a contract between the backend-client package and the places where the output is written (the constructs). Ideally the schema shouldn't be needed directly by consumers of deployed-backend-client

It makes sense to have one writer, but I believe it's fine to have multiple readers. We already have client-config and model-generator

packages/cli-core/src/format/format.ts

edwardfoyle · 2024-05-10T17:06:06Z

packages/cli-core/src/printer/printer.ts

    private readonly refreshRate: number = 500
-  ) {}
+  ) {
+    // if not running in tty, turn off colors


why is this class manipulating colors?

Since this is a central place for performing logging, it's better to turn off/on colors here rather than at the consumers place. Especially since this $ is a global variable, the colors will only change when one consumer changes it making the colors inconsistent.

Aren't the methods in format already a noop if colors are not enabled? Why do we need additional logic for it here?

format uses supportColors which only looks at the environment/terminal you are running in and decides whether to enable or disable colors. E.g. in CI/CD the colors might be disabled. See https://github.com/isaacs/color-support

In the printer, we might be piping the data in some other place as well (e.g. writing to a file) supportColors doesn't know that and will keep the colors enabled.

edwardfoyle · 2024-05-10T17:10:50Z

packages/cli/src/commands/sandbox/sandbox_command.ts

+        .option('stream-function-logs', {
+          describe:
+            'Whether to stream function execution logs. Default: false. Use --function-names to filter for specific functions to stream logs from',
+          boolean: true,
+          global: false,
+          group: 'Options to configure streaming of lambda function logs',
+        })
+        .option('function-name', {


Could stream-function-logs be a boolean or array? ie you can specify --stream-function-logs to stream all logs or you can specify --stream-function-logs name1 name2 to only stream those two lambdas. Then we wouldn't need a separate arg for function-name.

This is a TBD since requirements are pending (hence PR being in draft).

I think it is fine to merge the options, my reasoning is that it impacts discoverability as hints are hidden under the --stream-function-logs option's help description. With an explicit option, that shows up clearly in the help for easy discoverability.

packages/sandbox/src/cloudwatch_logs_monitor.ts

packages/sandbox/src/file_watching_sandbox.ts

packages/sandbox/src/lambda_function_log_streamer.ts

edwardfoyle · 2024-05-10T18:10:13Z

packages/sandbox/src/lambda_function_log_streamer.ts

+          Resource: entry.arn,
+        })
+      );
+      const friendlyFunctionName =


using getBackendMetadata() will do this filtering for us

Do we get the tagged names there? I thought it was only the physical Arn

amplify-backend/packages/deployed-backend-client/src/deployed_backend_client.ts

Lines 346 to 363 in 57a5138

customerFunctionNames.forEach((functionName) => {

const resource = functionResources.find(

(func) => func.physicalResourceId === functionName

);

if (resource) {

functionConfigurations.push({

status: this.stackStatusMapper.translateStackStatus(

resource.resourceStatus

),

lastUpdated:

resource.lastUpdated ??

functionStack.LastUpdatedTime ??

functionStack.CreationTime,

functionName,

});

}

});

ah, yeah if we need the friendly name, we'd need to do some extra processing or expand the interface the deployed backend client

I don't think we are at a point that warrants to expand the interface of deployed backend client. Also one can argue that tags (or other resource specific properties) are not considered metadata and shouldn't be part of this package. We can look into adding the friendly-name to stack outputs to avoid lambda:list-tags, but I don't think we are there yet.

edwardfoyle · 2024-05-13T16:56:50Z

packages/cli-core/src/format/format.ts

+export type Color = Colorize;
+export type ColorName = keyof typeof colors;
+
+export const colors = {


this is still exporting kleur functions. Why not have

export enum Color { GREEN = 'green'; YELLOW = 'yellow'; etc }

Then internally map those enum values to the kleur function.

Updated to not export the colors object but just a type and enumerated color names.

Amplifiyer added 3 commits May 10, 2024 17:44

feat: add support for function logs streaming to sandbox

74ccb5f

Merge branch 'main' into function_logs

4d30c77

update package lock

9141649

Amplifiyer added the run-e2e Label that will include e2e tests in PR checks workflow label May 10, 2024

Amplifiyer added 2 commits May 10, 2024 19:09

update package lock

11bc7b6

update package lock

3dacebd

edwardfoyle reviewed May 10, 2024

View reviewed changes

PR feedback updates

841a4c3

edwardfoyle reviewed May 13, 2024

View reviewed changes

Amplifiyer added 5 commits May 14, 2024 11:14

PR feedback updates

376de18

Merge branch 'main' into function_logs

6fd0a63

PR feedback updates

e8bbf91

try this

e15bcbf

try this

8fbeb3b

Amplifiyer force-pushed the function_logs branch from b50699a to 8fbeb3b Compare May 15, 2024 16:17

Amplifiyer added 2 commits May 15, 2024 18:34

Merge branch 'main' into function_logs

b589fdb

try this

c4d423e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support for function logs streaming to sandbox #1492

feat: add support for function logs streaming to sandbox #1492

Amplifiyer commented May 10, 2024

changeset-bot bot commented May 10, 2024 •

edited

edwardfoyle May 10, 2024

Amplifiyer May 13, 2024

edwardfoyle May 13, 2024

Amplifiyer May 14, 2024

edwardfoyle May 14, 2024

Amplifiyer May 14, 2024

edwardfoyle May 14, 2024

Amplifiyer May 16, 2024

edwardfoyle May 10, 2024

Amplifiyer May 13, 2024

edwardfoyle May 14, 2024

Amplifiyer May 14, 2024

edwardfoyle May 10, 2024

Amplifiyer May 13, 2024

edwardfoyle May 10, 2024 •

edited

Amplifiyer May 13, 2024

edwardfoyle May 13, 2024

Amplifiyer May 14, 2024

edwardfoyle May 13, 2024

Amplifiyer May 14, 2024

	customerFunctionNames.forEach((functionName) => {
	const resource = functionResources.find(
	(func) => func.physicalResourceId === functionName
	);

	if (resource) {
	functionConfigurations.push({
	status: this.stackStatusMapper.translateStackStatus(
	resource.resourceStatus
	),
	lastUpdated:
	resource.lastUpdated ??
	functionStack.LastUpdatedTime ??
	functionStack.CreationTime,
	functionName,
	});
	}
	});

feat: add support for function logs streaming to sandbox #1492

Are you sure you want to change the base?

feat: add support for function logs streaming to sandbox #1492

Conversation

Amplifiyer commented May 10, 2024

Changes

Validation

Checklist

changeset-bot bot commented May 10, 2024 • edited

🦋 Changeset detected

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edwardfoyle May 10, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

changeset-bot bot commented May 10, 2024 •

edited

edwardfoyle May 10, 2024 •

edited