Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle replacements of resources marked for deletion #11475

Merged
merged 1 commit into from Dec 7, 2022

Conversation

Frassle
Copy link
Member

@Frassle Frassle commented Nov 28, 2022

Description

This required getting state into a bit of unusual failure mode, but clearly people are hitting this (see #11391 and #11464).

If we replace a resource with create before delete semantics, and then fail to delete it we end up with two copies of the resource in state, but one copy will have the "delete" flag set.

If we then manage to trigger another resource to replace with delete before replace semantics that has the first resource as a dependent we would hit an assertion about Delete and PendingReplace both being set the same, if you just removed that asset then state management got confused and left the first resource in the state file even after it had really been deleted.

So now we track if we've seen a pendingDelete for a resource via the state pointer not the URN (so we can handle the multiple state objects) and don't trigger a DeleteReplacement, but a plain Delete if the resource is already flagged to Delete.

Fixes #11391

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • I have run make changelog and committed the changelog/pending/<file> documenting my change
  • Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

@pulumi-bot
Copy link
Contributor

pulumi-bot commented Nov 28, 2022

Changelog

[uncommitted] (2022-12-07)

Bug Fixes

  • [engine] Fix an assert for resources being replaced but also pending deletion.
    #11475

@Frassle
Copy link
Member Author

Frassle commented Nov 29, 2022

bors try

bors bot added a commit that referenced this pull request Nov 29, 2022
@bors
Copy link
Contributor

bors bot commented Nov 29, 2022

try

Build failed:

@Frassle Frassle marked this pull request as ready for review November 29, 2022 16:08
@Frassle
Copy link
Member Author

Frassle commented Dec 6, 2022

bors merge

bors bot added a commit that referenced this pull request Dec 6, 2022
11475: Handle replacements of resources marked for deletion r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This required getting state into a bit of unusual failure mode, but clearly people are hitting this (see #11391 and #11464).

If we replace a resource with create before delete semantics, and then fail to delete it we end up with two copies of the resource in state, but one copy will have the "delete" flag set.

If we then manage to trigger another resource to replace with delete before replace semantics that has the first resource as a dependent we would hit an assertion about Delete and PendingReplace both being set the same, if you just removed that asset then state management got confused and left the first resource in the state file even after it had really been deleted.

So now we track if we've seen a pendingDelete for a resource via the state pointer not the URN (so we can handle the multiple state objects) and don't trigger a DeleteReplacement, but a plain Delete if the resource is already flagged to Delete.

Fixes #11391

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Build failed:

@Zaid-Ajaj
Copy link
Contributor

bors retry

bors bot added a commit that referenced this pull request Dec 6, 2022
11475: Handle replacements of resources marked for deletion r=Frassle a=Frassle

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

This required getting state into a bit of unusual failure mode, but clearly people are hitting this (see #11391 and #11464).

If we replace a resource with create before delete semantics, and then fail to delete it we end up with two copies of the resource in state, but one copy will have the "delete" flag set.

If we then manage to trigger another resource to replace with delete before replace semantics that has the first resource as a dependent we would hit an assertion about Delete and PendingReplace both being set the same, if you just removed that asset then state management got confused and left the first resource in the state file even after it had really been deleted.

So now we track if we've seen a pendingDelete for a resource via the state pointer not the URN (so we can handle the multiple state objects) and don't trigger a DeleteReplacement, but a plain Delete if the resource is already flagged to Delete.

Fixes #11391

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Fraser Waters <fraser@pulumi.com>
@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Build failed:

@Frassle
Copy link
Member Author

Frassle commented Dec 6, 2022

Oh python lint is borked: #11542

@aq17 aq17 force-pushed the fraser/handleDeleteReplace branch from 0ed78b8 to 897f39d Compare December 6, 2022 19:04
@Frassle
Copy link
Member Author

Frassle commented Dec 7, 2022

bors merge

@bors
Copy link
Contributor

bors bot commented Dec 7, 2022

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

@bors
Copy link
Contributor

bors bot commented Dec 7, 2022

Build succeeded:

@bors bors bot merged commit 68308f5 into master Dec 7, 2022
@bors bors bot deleted the fraser/handleDeleteReplace branch December 7, 2022 20:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

assertion failure in pulumi cli
5 participants