Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow more advanced test case support semantic #27373

Merged
merged 4 commits into from
Sep 17, 2021
Merged

Allow more advanced test case support semantic #27373

merged 4 commits into from
Sep 17, 2021

Conversation

lidizheng
Copy link
Contributor

@lidizheng lidizheng commented Sep 16, 2021 •
edited

After this PR, the framework can skip a test case for a specific language with a specific version range, or something more fancy.

Test runs:

@lidizheng lidizheng added area/test release notes: no Indicates if PR should not be in release notes labels Sep 16, 2021
menghanl
menghanl suggested changes Sep 16, 2021
View reviewed changes
tools/run_tests/xds_k8s_test_driver/framework/xds_flags.py Outdated
TESTING_VERSION = flags.DEFINE_string(
"testing_version",
default="master",
help="The testing gRPC version branch name.")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add example values to the help string.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Good idea.

tools/run_tests/xds_k8s_test_driver/framework/xds_url_map_testcase.py Outdated

Returns:
A tuple of strings contains the supported languages for this test.
A tuple of strings contains the supported versions for this test or
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this doc right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awkward, updated.

tools/run_tests/xds_k8s_test_driver/tests/url_map/retry_test.py
return 'cpp', 'java', 'go'
def is_supported(config: TestConfig) -> bool:
if config.client_lang in ['cpp', 'java', 'go']:
return config.version_ge('v1.40.x')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retry should be the same as affinity. go version >= 1.41.x

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

@lidizheng lidizheng marked this pull request as ready for review September 16, 2021 23:23
menghanl
menghanl reviewed Sep 17, 2021
View reviewed changes
tools/run_tests/xds_k8s_test_driver/tests/url_map/retry_test.py
return 'cpp', 'java', 'go'
def is_supported(config: TestConfig) -> bool:
if config.client_lang in ['cpp', 'java', 'go']:
return config.version_ge('v1.40.x')
Copy link
Contributor

@menghanl menghanl Sep 17, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this. Go >= 1.41.x

@lidizheng lidizheng merged commit f99c193 into grpc:master Sep 17, 2021
lidizheng added a commit to lidizheng/grpc that referenced this pull request Sep 17, 2021
@lidizheng
[Backport grpc#27373] add testing_version flag
082099f
@lidizheng lidizheng mentioned this pull request Sep 17, 2021
Merged
lidizheng added a commit to lidizheng/grpc that referenced this pull request Sep 17, 2021
@lidizheng
[Backport grpc#27373] add testing_version flag
58385cd
This was referenced Sep 17, 2021
Merged
Merged
@copybara-service copybara-service bot added the imported Specifies if the PR has been imported to the internal repository label Sep 17, 2021
lidizheng added a commit that referenced this pull request Sep 17, 2021
@lidizheng
[Backport #27373] add testing_version flag (#27385)
b31c03b
lidizheng added a commit to lidizheng/grpc-java that referenced this pull request Sep 17, 2021
@lidizheng
[Backport grpc/grpc#27373] add testing_version flag
d20ad2a
lidizheng added a commit that referenced this pull request Sep 17, 2021
@lidizheng
[Backport #27373] add testing_version flag (#27386)
07d9037
This was referenced Sep 17, 2021
Merged
Merged
lidizheng added a commit to lidizheng/grpc that referenced this pull request Sep 23, 2021
@lidizheng
Allow more advanced test case support semantic (grpc#27373)
7266ad0 
* Allow more advanced test case support semantic

* Fix comments and docs

* Correct the supported Golang version in retry test

* Make isort happy
markdroth added a commit that referenced this pull request Oct 25, 2021
@markdroth @donnadionne
@lidizheng @jtattermusch @sanjaypujare @sergiitk @veblush @temawi @gnossen
Upmerge from v1.41.x (#27821)
677b3c7 
* Bump version to v1.41.0-pre1 (#27371)

* Bump version to v1.41.0-pre1

* Regenerate projects

* [Backport #27373] add testing_version flag (#27385)

* Bump version to v1.41.0-pre2 (#27390)

* Bump version to v1.41.0-pre2

* Regenerate projects

* Core 19: bump core version from 18.0.0 to 19.0.0 (#27394)

* Bump core version to 19.0.0

* Regenerate projects

* fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398)

* Final release: bump up version to 1.41.0 (#27476)

* Bump version to 1.41.0

* Regenerate projects

* xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)

* Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590)

This reverts commit da0c7d6.

* Update root pem certs (backport of #27539) (#27619)

* Update boringssl to the latest (#27606) (#27625)

* Change boringssl branch name

* update submodule boringssl-with-bazel with origin/main-with-bazel

* update boringssl dependency to main-with-bazel commit SHA

* regenerate files

* Increment podspec version

* generate boringssl prefix headers

* Bumping up version to v1.41.1 (#27699)

* Bump version to v1.41.1

* Regenerate projects

* [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695)

Backports sourcing the test driver install script from master.

This is a backport of #27389, #27462 and #27658:


* Add missing quatation marks.

These were missed when  creating the Python virtual env.

* xds-k8s tests: Use test driver from master branch (#27462)

Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch.

This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches.

All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.

* xds_k8s tests: Fix xlang install script sourcing. (#27658)

This change sources the test driver install script correctly for the xlang tests.
This fixes a mistake in #27462 where this was missed.

* Fix Python Interop (#27620) (#27703)

* WIP. Attempt to fix interop

* Yapf

* Switch Python xDS Example Server to Listen on IPV4 Only (#27679)

* Switch to IPV4

* Update to all hosts

* Fix rvm ruby install failure (#27769)

Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
oontvoo pushed a commit to oontvoo/grpc that referenced this pull request Oct 26, 2021
@markdroth @donnadionne
@lidizheng @jtattermusch @sanjaypujare @sergiitk @veblush @temawi @gnossen @oontvoo
Upmerge from v1.41.x (grpc#27821)
abb46b8 
* Bump version to v1.41.0-pre1 (grpc#27371)

* Bump version to v1.41.0-pre1

* Regenerate projects

* [Backport grpc#27373] add testing_version flag (grpc#27385)

* Bump version to v1.41.0-pre2 (grpc#27390)

* Bump version to v1.41.0-pre2

* Regenerate projects

* Core 19: bump core version from 18.0.0 to 19.0.0 (grpc#27394)

* Bump core version to 19.0.0

* Regenerate projects

* fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (grpc#27398)

* Final release: bump up version to 1.41.0 (grpc#27476)

* Bump version to 1.41.0

* Regenerate projects

* xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580)

* Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (grpc#27580)" (grpc#27590)

This reverts commit da0c7d6.

* Update root pem certs (backport of grpc#27539) (grpc#27619)

* Update boringssl to the latest (grpc#27606) (grpc#27625)

* Change boringssl branch name

* update submodule boringssl-with-bazel with origin/main-with-bazel

* update boringssl dependency to main-with-bazel commit SHA

* regenerate files

* Increment podspec version

* generate boringssl prefix headers

* Bumping up version to v1.41.1 (grpc#27699)

* Bump version to v1.41.1

* Regenerate projects

* [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (grpc#27695)

Backports sourcing the test driver install script from master.

This is a backport of grpc#27389, grpc#27462 and grpc#27658:


* Add missing quatation marks.

These were missed when  creating the Python virtual env.

* xds-k8s tests: Use test driver from master branch (grpc#27462)

Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch.

This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches.

All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.

* xds_k8s tests: Fix xlang install script sourcing. (grpc#27658)

This change sources the test driver install script correctly for the xlang tests.
This fixes a mistake in grpc#27462 where this was missed.

* Fix Python Interop (grpc#27620) (grpc#27703)

* WIP. Attempt to fix interop

* Yapf

* Switch Python xDS Example Server to Listen on IPV4 Only (grpc#27679)

* Switch to IPV4

* Update to all hosts

* Fix rvm ruby install failure (grpc#27769)

Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
veblush added a commit that referenced this pull request Oct 27, 2021
@oontvoo @markdroth
@donnadionne @lidizheng @jtattermusch @sanjaypujare @sergiitk @veblush @temawi @gnossen
Upgrade benchmark to 1.6.0 and remove previous hacks. (#27778)
70b26a1 
* Upgrade benchmark to 1.6.0 and remove hacks.

Details:
 - GRPC currently uses an old version of benchmark (from Sept 2020). It should probably upgrade because downstream, in google3, everyone is already using 1.6.0)
 - Removed the hack added in PR/27629 to allow benchmarks in GRPC to continue to work with both pre-1.6.0 and 1.6.0 benchmarks.
   (This was needed to allow importing benchmarks 1.6.0 into google3 without breaking GRPC)

* fix typo

* update third_party/benchmark and check_submodules.sh

* Upmerge from v1.41.x (#27821)

* Bump version to v1.41.0-pre1 (#27371)

* Bump version to v1.41.0-pre1

* Regenerate projects

* [Backport #27373] add testing_version flag (#27385)

* Bump version to v1.41.0-pre2 (#27390)

* Bump version to v1.41.0-pre2

* Regenerate projects

* Core 19: bump core version from 18.0.0 to 19.0.0 (#27394)

* Bump core version to 19.0.0

* Regenerate projects

* fix use-after-free metadata corruption in C# when receiving response headers for streaming response calls (#27398)

* Final release: bump up version to 1.41.0 (#27476)

* Bump version to 1.41.0

* Regenerate projects

* xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)

* Revert "xds_k8s_test: increase timeout to 3 hours due to recent timeout failure (#27580)" (#27590)

This reverts commit da0c7d6.

* Update root pem certs (backport of #27539) (#27619)

* Update boringssl to the latest (#27606) (#27625)

* Change boringssl branch name

* update submodule boringssl-with-bazel with origin/main-with-bazel

* update boringssl dependency to main-with-bazel commit SHA

* regenerate files

* Increment podspec version

* generate boringssl prefix headers

* Bumping up version to v1.41.1 (#27699)

* Bump version to v1.41.1

* Regenerate projects

* [Backport][v1.41.x] xds-k8s tests: Use test driver from master branch (#27695)

Backports sourcing the test driver install script from master.

This is a backport of #27389, #27462 and #27658:


* Add missing quatation marks.

These were missed when  creating the Python virtual env.

* xds-k8s tests: Use test driver from master branch (#27462)

Instead of directly sourcing the test driver provisioning script from the same branch, the script is downloaded (with curl) and sourced from the master branch.

This allows changes made to the test driver to be reflected in all future release branches. A separate PR will backport this change to existing release branches.

All cluster definitions are also moved to the install script, allowing any cluster changes to be done in one place in the master branch.

* xds_k8s tests: Fix xlang install script sourcing. (#27658)

This change sources the test driver install script correctly for the xlang tests.
This fixes a mistake in #27462 where this was missed.

* Fix Python Interop (#27620) (#27703)

* WIP. Attempt to fix interop

* Yapf

* Switch Python xDS Example Server to Listen on IPV4 Only (#27679)

* Switch to IPV4

* Update to all hosts

* Fix rvm ruby install failure (#27769)

Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>

* added perf_counters.cc manually since the script didn't work

Co-authored-by: Mark D. Roth <roth@google.com>
Co-authored-by: donnadionne <donnadionne@google.com>
Co-authored-by: Lidi Zheng <lidiz@google.com>
Co-authored-by: Jan Tattermusch <jtattermusch@users.noreply.github.com>
Co-authored-by: sanjaypujare <sanjaypujare@users.noreply.github.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
Co-authored-by: Terry Wilson <terrymwilson@gmail.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
sergiitk pushed a commit to sergiitk/grpc that referenced this pull request Dec 8, 2023
@lidizheng
Allow more advanced test case support semantic (grpc#27373)
b672fe6 
* Allow more advanced test case support semantic

* Fix comments and docs

* Correct the supported Golang version in retry test

* Make isort happy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@menghanl menghanl menghanl approved these changes

Assignees

@menghanl menghanl

Labels
area/test imported Specifies if the PR has been imported to the internal repository release notes: no Indicates if PR should not be in release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@lidizheng @menghanl