Stack corrupted by "parent coming after child..." error

[phillipedwards]

What happened?

During an update with a S3 backend stack, an error occurred and state became corrupted. The error is:

pulumi:pulumi:Stack aws-s3-stack running 
pulumi:providers:aws aws-provider-source  
pulumi:providers:aws aws-provider-destination  
pulumi:providers:pulumi default  error: post-step event returned an error: failed to save snapshot: .pulumi/stacks/aws-s3-stack.json: snapshot integrity failure; it was already written, but is invalid (backup available at .pulumi/stacks/aws-s3-stack.json.bak): child resource urn:pulumi:aws-s3-stack::aws-to-primary::pulumi:providers:pulumi::default's parent urn:pulumi:aws-s3-stack::aws-s3-stack::pulumi:pulumi:Stack::aws-s3-stack comes after it
pulumi:pulumi:Stack aws-s3-stack running error: update failed
pulumi:providers:pulumi default **failed** 1 error
pulumi:pulumi:Stack aws-s3-stack **failed** 1 error

Possibly related to: #5577

Steps to reproduce

Unknown...TBD

Expected Behavior

pulumi up succeeds

Actual Behavior

Stack state gets corrupted.

Output of pulumi about

No response

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[Frassle]

I suspect this isn't actually a problem with the S3 backend itself, but an issue elsewhere.

It looks like somehow the pulumi default provider was recorded to state before it's parent stack resource was. The stack must have been registered first because the step generator asserts that the parent for resources can be found.

It would help to know what language SDK this was using, and if they're doing anything special about how they're creating the stack resource.

[lukehoban]

Additional notes:

• A user is hitting this "pretty consistently"
• They are using Go
• They are using an httpstate backend, so as @Frassle notes above, this does not appear to be limited to S3 backend

[rh4ll]

I have experienced a similar thing:

• in a code branch
• change parent of resource with children (adding alias)
• run pulumi up on stack
• on main branch run pulumi up on stack
• Get error

Looking at the op's original urn's this looks to be the case here too

[dougludlow]

Just ran into the same issue, with the same steps @rh4ll mentioned. Pulumi version: 3.64.0. I'm pretty sure the issue had to do with the aliases that were added.

In a couple cases, I was able to simply revert to a previous version of the state (pulumi stack export --file out.json --version x, pulumi stack import --file out.json), after removing the new resources manually in AWS. In another case, I had to export the state and fix the json manually.

In all the above cases, I had renamed some resources and added aliases so that they would not be replaced. I deployed my branch to a shared nonprod environment to test the changes and then shortly after the main branch was deployed to the same environment and somehow the resources were duplicated (most likely because the main branch didn't know about the aliases?), but the new resources took on the original urns. At that point the state became corrupt and any attempt to run pulumi up from the main branch resulted in error: resource complete event returned an error: failed to verify snapshot: resource <x>'s dependency <y> comes after it. When I exported the state, the dependency did indeed come after the resource in the json.