Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

Open
Saberos opened this issue Jul 26, 2023 · 7 comments
Open

core: Stack with many resources combined with aspects results in StackOverflowErrors #26517

Saberos opened this issue Jul 26, 2023 · 7 comments
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/large Large work item – several weeks of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. p2

Comments

@Saberos
Copy link

Saberos commented Jul 26, 2023

Describe the bug

When using aspects in combination with stacks with a large number of resources we are running into StackOverflowError during synthesis in our Java CDK applications. The issue is somewhat system dependent as the default limits enforced by the JVM change depending on available system resources (and probably JVM vendor).

Expected Behavior

Stacks with many resources with aspects to synthesize successfully.

Current Behavior

Synthesis fails on a StackOverFlowError.

Reproduction Steps

I have managed to reproduce the issue in a minimal project, available here: https://github.com/Saberos/cdk-aspect-stackoverflowerror

cdk synth produces the error while the aspect is present. Once the aspect is removed synthesis completes as expected. Note that the stack size has been forcibly limited in the cdk.json to ensure the issue is reproducible independent of the system. Without the -Xss flag the issue may or may not appear on the system. Increasing the number of resources in the stack or the number of aspects should allow the issue to be reproduced on any system even without using this flag.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.86.0

Framework Version

2.86.0

Node.js Version

v18.12.1

OS

macOS Ventura 13.2.1

Language

Java

Language Version

Corretto-17.0.7.7.1

Other information

The StackOverflowError stacktrace:

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project cdk-aspect-stackoverflowerror: An exception occured while executing the Java class. null
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:347)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. null
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:311)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:342)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:330)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:213)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:175)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:76)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:163)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:160)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:910)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.lang.StackOverflowError
    at com.fasterxml.jackson.databind.node.NodeCursor$ObjectCursor.currentNode (NodeCursor.java:224)
    at com.fasterxml.jackson.databind.node.TreeTraversingParser.currentNode (TreeTraversingParser.java:374)
    at com.fasterxml.jackson.databind.node.TreeTraversingParser.getText (TreeTraversingParser.java:205)
    at com.fasterxml.jackson.databind.deser.std.BaseNodeDeserializer._deserializeAnyScalar (JsonNodeDeserializer.java:564)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize (JsonNodeDeserializer.java:93)
    at com.fasterxml.jackson.databind.deser.std.JsonNodeDeserializer.deserialize (JsonNodeDeserializer.java:20)
    at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue (DefaultDeserializationContext.java:323)
    at com.fasterxml.jackson.databind.ObjectMapper._readValue (ObjectMapper.java:4650)
    at com.fasterxml.jackson.databind.ObjectMapper.readTree (ObjectMapper.java:2941)
    at com.fasterxml.jackson.core.JsonParser.readValueAsTree (JsonParser.java:2365)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:130)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet (MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize (BeanDeserializer.java:314)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize (BeanDeserializer.java:177)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:149)
    at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet (MethodProperty.java:129)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize (BeanDeserializer.java:314)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize (BeanDeserializer.java:177)
    at software.amazon.jsii.JsiiObjectMapper$JsiiDeserializer.deserialize (JsiiObjectMapper.java:149)
    at com.fasterxml.jackson.databind.ObjectMapper._convert (ObjectMapper.java:4388)
    at com.fasterxml.jackson.databind.ObjectMapper.convertValue (ObjectMapper.java:4344)
    at software.amazon.jsii.JsiiObjectMapper.treeToValue (JsiiObjectMapper.java:46)
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:165)
    at software.amazon.jsii.JsiiRuntime.requestResponse (JsiiRuntime.java:121)
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:197)
... Above 2 lines repeating many times
    at software.amazon.jsii.JsiiRuntime.processCallbackResponse (JsiiRuntime.java:197)
    at software.amazon.jsii.JsiiRuntime.requestResponse (JsiiRuntime.java:121)
    at software.amazon.jsii.JsiiClient.callMethod (JsiiClient.java:184)
    at software.amazon.jsii.Kernel.call (Kernel.java:52)
    at software.amazon.awscdk.Stage.synth (Stage.java:109)
    at com.myorg.CdkAspectStackoverflowerrorApp.main (CdkAspectStackoverflowerrorApp.java:19)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:833)
@Saberos Saberos added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 26, 2023
@github-actions github-actions bot added the @aws-cdk/core Related to core CDK functionality label Jul 26, 2023
@peterwoodworth peterwoodworth added p1 effort/large Large work item – several weeks of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. and removed needs-triage This issue or PR still needs to be triaged. labels Jul 26, 2023
@peterwoodworth
Copy link
Contributor

Thanks for reporting, might be best in the jsii repo, but we can certainly track it here.

@github-actions github-actions bot mentioned this issue Aug 1, 2023
Closed
@mrgrain
Copy link
Contributor

mrgrain commented Aug 2, 2023

Hi @Saberos can you confirm why you can't use -Xss to give the app sufficient memory?
Aspects are indeed memory expensive.

@Saberos
Copy link
Author

Saberos commented Aug 3, 2023

@mrgrain just to ensure we're on the same page, the default stack size limit in the JVM seems to be commonly 1-2MB. It isn't so much that the program is incredibly memory expensive, just that aspects appear to be implemented using recursion which can exceed the stack size limit.

Increasing the limit is mostly inconvenient because we have to consider/manage the -Xss flag in all situations where the CDK app is ran, which is not always through cdk.json. Having to consider for example heap size for an application makes sense, but having to increase the stack size due to the implementation of a used library/framework seems a bit much to ask to me.

In my opinion the limit is also hit fairly easily, the scenarios do not seem that extreme. With -Xss1m, which appears to be a common default, I run into the issue at ~700 resources. It does not seem to matter if they are all in a single stack or spread our over multiple stacks, as long as it is a single CDK app. It is not a deeply nested structure in my example, just stacks with a completely flat list of resources. I fail to see the need for recursion instead of simply iterating over all these resources.

@mrgrain mrgrain added p2 and removed p1 labels Aug 3, 2023
@mrgrain
Copy link
Contributor

mrgrain commented Aug 3, 2023

Thanks for the clarification @Saberos ! I agree having to consider -Xss can be a pain, as is managing the heap size.
I'm sure there could be a more stack size friendly implementation for Aspects, which would require a relatively large refactor.

Depending on your scenario, you can also apply Aspects more locally on specific parts of the Construct tree, not just the root. Assigned p2 since there is a workaround.

@Saberos
Copy link
Author

Saberos commented Aug 3, 2023

A workaround for us for now was to refactor an aspect to set IAM permission boundaries to the CDK construct that has been introduced since we set that up. I do not know if that eliminated the recursion of just moves it to Node.js where the stack limit is higher, but for now it is sufficient for us to stay under the limit.

@gefloh
Copy link

gefloh commented Sep 18, 2023

We ran into the exact same problem when using an Aspect as a workaround for the issue described in #19114. Like in the scenario of the OP our stack is also not very big, so this issue came a little unexpected. Setting a higher -Xss in the cdk.json is fine for us and solved the problem but I agree that it is not ideal.

@lucasam
Copy link

lucasam commented Dec 21, 2023

I have a similar problem with a even more weird behavior.
With this aspect i have java.lang.StackOverflowError

public class EnvironmentStageAspect implements IAspect {

    private final StackStageType stackStageType;
    @Override
    public void visit(@NotNull IConstruct iConstruct) {
        if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }
}

IF i print the construct Id, the issue persists

    public void visit(@NotNull IConstruct iConstruct) {
        System.out.println(iConstruct.getNode().getId());
        if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }

But, if i print getAllContext the issue got fixed

    public void visit(@NotNull IConstruct iConstruct) {
    
    System.out.println(iConstruct.getNode().getAllContext());
     if (iConstruct instanceof Function) {
            Function function = (Function) iConstruct;
            function.addEnvironment("BLA", stackStageType.toString());
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
@aws-cdk/core Related to core CDK functionality bug This issue is a bug. effort/large Large work item – several weeks of effort jsii This issue originates in jsii, or this feature must be implemented in jsii. p2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mrgrain @Saberos @lucasam @peterwoodworth @gefloh