Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(testing): Adds cached python venvs for integration tests #11532

Merged
merged 1 commit into from Dec 6, 2022
Merged

Conversation

kpitzen
Copy link
Contributor

@kpitzen kpitzen commented Dec 5, 2022

The cache key is based on hashing the contents of "requirements.txt" for each python test. This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

Description

Fixes #11485

Checklist

  • I have added tests that prove my fix is effective or that my feature works
  • I have run make changelog and committed the changelog/pending/<file> documenting my change
  • Yes, there are changes in this PR that warrants bumping the Pulumi Service API version

@pulumi-bot
Copy link
Contributor

pulumi-bot commented Dec 5, 2022

Changelog

[uncommitted] (2022-12-06)

Bug Fixes

  • [pkg/testing] Optionally caches python venvs for testing
    #11532

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 5, 2022

bors try

bors bot added a commit that referenced this pull request Dec 5, 2022
@bors
Copy link
Contributor

bors bot commented Dec 5, 2022

try

Build failed:

@@ -292,6 +293,12 @@ type ProgramTestOptions struct {
UseAutomaticVirtualEnv bool
// Use the Pipenv tool to manage the virtual environment.
UsePipenv bool
// Use a shared virtual environment for tests based on the contents of the requirements file. Defaults to false.
UseSharedVirtualEnv *bool
Copy link
Contributor Author

@kpitzen kpitzen Dec 5, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really don't love that I have to use a pointer here, but it seems to be the only way to disambiguate between false and an unset value at initialization time. I've used a couple convenience functions to try to alleviate the ergonomics issue this introduces, but I'm open to other approaches if anyone has any.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have a better answer here either, I expect we'll want to make this the default in the future because it'll speed up examples & templates tests, tests in providers.

Using a pointer for Maybe[bool] and an env var to set the default is a good approach for a testing harness, and we can change the default in the future after validating.

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 5, 2022

bors try

bors bot added a commit that referenced this pull request Dec 5, 2022
@bors
Copy link
Contributor

bors bot commented Dec 5, 2022

try

Build failed:

Copy link
Member

@AaronFriel AaronFriel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The improvement in test execution time speaks for itself.

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 5, 2022

bors merge

bors bot added a commit that referenced this pull request Dec 5, 2022
11532: fix(testing): Adds cached python venvs for integration tests r=kpitzen a=kpitzen

The cache key is based on hashing the contents of "requirements.txt" for each python test.  This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes #11485 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Kyle Pitzen <kylepitzen@Kyles-MBP.fios-router.home>
@bors
Copy link
Contributor

bors bot commented Dec 5, 2022

Canceled.

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 5, 2022

bors merge

bors bot added a commit that referenced this pull request Dec 5, 2022
11532: fix(testing): Adds cached python venvs for integration tests r=kpitzen a=kpitzen

The cache key is based on hashing the contents of "requirements.txt" for each python test.  This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes #11485 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Kyle Pitzen <kylepitzen@Kyles-MBP.fios-router.home>
@bors
Copy link
Contributor

bors bot commented Dec 5, 2022

Build failed:

The cache key is based on hashing the contents of "requirements.txt" for each
python test.  This should result in unchanged test results among tests
sharing venvs, and maintains environment isolation across requirement sets
@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 6, 2022

bors merge

bors bot added a commit that referenced this pull request Dec 6, 2022
11532: fix(testing): Adds cached python venvs for integration tests r=kpitzen a=kpitzen

The cache key is based on hashing the contents of "requirements.txt" for each python test.  This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes #11485 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Kyle Pitzen <kylepitzen@Kyles-MBP.fios-router.home>
@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 6, 2022

bors cancel

@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Canceled.

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 6, 2022

bors merge p=-1

bors bot added a commit that referenced this pull request Dec 6, 2022
11533: Allow opting out of PULUMI_OPTIMIZED_CHECKPOINT_PATCH r=t0yv0 a=t0yv0

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

Allow users to opt out of deltaCheckpointUpdates even if the backend indicates it should be used. This remains necessary while PULUMI_OPTIMIZED_CHECKPOINT_PATCH has higher memory requirements on the client and may cause out-of-memory issues in constrained environments.

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Part of https://github.com/pulumi/home/issues/2426

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11537: Fix panic in mapper support for map[string]*string r=t0yv0 a=t0yv0

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

`map[string]*string` does not turn around through mapper, causing a panic.

```go
go test -run TestReproduceMapStringPointerTurnaroundIssue                                                                                                  ~/code/pulumi/sdk/go/common/util/mapper
--- FAIL: TestReproduceMapStringPointerTurnaroundIssue (0.00s)
    mapper_test.go:608: encodedMap: map[args:map[key:value]]
panic: reflect: Elem of invalid type string [recovered]
        panic: reflect: Elem of invalid type string

goroutine 20 [running]:
testing.tRunner.func1.2({0x11cb480, 0xc0000992b0})
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/testing/testing.go:1396 +0x24e
testing.tRunner.func1()
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/testing/testing.go:1399 +0x39f
panic({0x11cb480, 0xc0000992b0})
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/runtime/panic.go:884 +0x212
reflect.(*rtype).Elem(0x11d1500?)
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/reflect/type.go:972 +0x134
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.(*mapper).adjustValueForAssignment(0xc0000a2240, {0x11d1500?, 0xc000099290?, 0x98?}, {0x126b940?, 0x11c6160}, {0x126b940?, 0x11dd060}, {0xc0000a4a80, 0xf})
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_decode.go:158 +0x2ea
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.(*mapper).adjustValueForAssignment(0xc0000a2240, {0x11d5c20?, 0xc00009d470?, 0x11de1e0?}, {0x126b940?, 0x11d5680}, {0x126b940?, 0x11dd060}, {0x11c0b24, 0x4})
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_decode.go:204 +0x1a15
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.(*mapper).DecodeValue(0xc0000a2240, 0xc0000a0060?, {0x126b940?, 0x11dd060}, {0x11c0b24, 0x4}, {0xc0000aac80?, 0xc0000a0060?}, 0x1)
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_decode.go:117 +0x8ee
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.(*mapper).Decode(0xc0000a2240, 0xc0000a8380?, {0x11c36a0?, 0xc0000a0060?})
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_decode.go:55 +0x8db
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.TestReproduceMapStringPointerTurnaroundIssue.func2(0xc000083520?)
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_test.go:592 +0xe9
github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper.TestReproduceMapStringPointerTurnaroundIssue(0xc000083520)
        /Users/t0yv0/code/pulumi/sdk/go/common/util/mapper/mapper_test.go:610 +0x11f
testing.tRunner(0xc000083520, 0x1231f88)
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
        /nix/store/0c30lcag5r6ahw3qj0x7lkshpry4yqwl-go-1.19/share/go/src/testing/testing.go:1493 +0x35f
exit status 2
FAIL    github.com/pulumi/pulumi/sdk/v3/go/common/util/mapper   0.674s
```


<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes # (issue)

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [ ] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


11539: Add `*[TYPE]` to `[TYPE]Ptr` methods r=aq17 a=aq17

Fixes #11536
Adds methods to convert pointer types to corresponding pulumi Ptr types

11540: ci: Remove page file tweak for Windows r=aq17 a=AaronFriel

The step to [configure the pagefile on Windows](https://github.com/pulumi/pulumi/actions/runs/3624853251/jobs/6112393713) began failing due to an update to the underlying runner image. Given that this is blocking CI, we'll remove the step for now.

Either #11532 or #11471 should address performance of integration tests and reliability and obviate the need for this step.

Co-authored-by: Anton Tayanovskyy <anton@pulumi.com>
Co-authored-by: aq17 <aqiu@pulumi.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
bors bot added a commit that referenced this pull request Dec 6, 2022
11532: fix(testing): Adds cached python venvs for integration tests r=kpitzen a=kpitzen

The cache key is based on hashing the contents of "requirements.txt" for each python test.  This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes #11485 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Kyle Pitzen <kylepitzen@Kyles-MBP.fios-router.home>
@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Build failed:

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 6, 2022

bors retry p=-1

bors bot added a commit that referenced this pull request Dec 6, 2022
11539: Add `*[TYPE]` to `[TYPE]Ptr` methods r=aq17 a=aq17

Fixes #11536
Adds methods to convert pointer types to corresponding pulumi Ptr types

11540: ci: Remove page file tweak for Windows r=aq17 a=AaronFriel

The step to [configure the pagefile on Windows](https://github.com/pulumi/pulumi/actions/runs/3624853251/jobs/6112393713) began failing due to an update to the underlying runner image. Given that this is blocking CI, we'll remove the step for now.

Either #11532 or #11471 should address performance of integration tests and reliability and obviate the need for this step.

Co-authored-by: aq17 <aqiu@pulumi.com>
Co-authored-by: Aaron Friel <mayreply@aaronfriel.com>
bors bot added a commit that referenced this pull request Dec 6, 2022
11532: fix(testing): Adds cached python venvs for integration tests r=kpitzen a=kpitzen

The cache key is based on hashing the contents of "requirements.txt" for each python test.  This should result in unchanged test results among tests sharing venvs, and maintains environment isolation across requirement sets

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

Fixes #11485 

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [x] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Kyle Pitzen <kylepitzen@Kyles-MBP.fios-router.home>
@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Build failed:

@kpitzen
Copy link
Contributor Author

kpitzen commented Dec 6, 2022

bors merge

@bors
Copy link
Contributor

bors bot commented Dec 6, 2022

Build succeeded:

@bors bors bot merged commit fbaf685 into master Dec 6, 2022
@bors bors bot deleted the KP/11485 branch December 6, 2022 23:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[sdks/python] Cache virtual environments for integration tests
3 participants