Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xDS interop: enable pod log collection in the buildscripts #30735

Merged
merged 6 commits into from
Aug 26, 2022
Merged

xDS interop: enable pod log collection in the buildscripts #30735

merged 6 commits into from
Aug 26, 2022

Conversation

sergiitk
Copy link
Member

@sergiitk sergiitk commented Aug 24, 2022 •
edited

  • Enables pod log collection in all PSM interop jobs implemented in xDS interop: collect pod logs #30594.
  • Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

All test runs with log collection enabled (feel free to ignore wip runs in the comments):

@sergiitk
Copy link
Member Author

Limited test invocation: grpc/core/master/linux/grpc_xds_k8s_lb

@sergiitk
Copy link
Member Author

sergiitk commented Aug 24, 2022 •
edited

Collect all *.log files in artifacts: grpc/core/master/linux/grpc_xds_k8s_lb

@sergiitk sergiitk mentioned this pull request Aug 25, 2022
Merged
@sergiitk
Copy link
Member Author

Latest run with no timestamps

@sergiitk
Copy link
Member Author

Latest branch run with timestamps brought back:

@sergiitk sergiitk force-pushed the xds-interop-buildscript-enable-logs branch from 1bf42da to c3840fa Compare August 26, 2022 01:24
@sergiitk
Copy link
Member Author

First set of full runs:

@sergiitk
Copy link
Member Author

Added missing \ to url_map buildscripts:

@sergiitk
Copy link
Member Author

Python jobs:

@sergiitk sergiitk added release notes: no Indicates if PR should not be in release notes area/psm interop labels Aug 26, 2022
@sergiitk
Copy link
Member Author

Fixed python security test:

@sergiitk
Copy link
Member Author

xlang:

@sergiitk
Copy link
Member Author

All passes!

@sergiitk sergiitk requested a review from gnossen August 26, 2022 17:08
sergiitk added a commit that referenced this pull request Aug 26, 2022
@sergiitk
xDS interop: collect pod logs (#30594)
1107617 
- Added support for pod log collection. To enable, set `--collect_app_logs` flag, and specify `--log_dir`.
- Added support and helpers for operating on the `--log_dir` (natively provided by absl)
- Added support for `--follow` to `bin/run_test_server.py` and `bin/run_test_client.py` to follow pod logs printed to stdout
- Moved `PortForwarder` from k8s.py to its own file

The collection itself will be enabled per-suite in #30735.
@sergiitk sergiitk marked this pull request as ready for review August 26, 2022 19:36
@sergiitk sergiitk merged commit 1e42fbe into grpc:master Aug 26, 2022
sergiitk added a commit to sergiitk/grpc that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (grpc#30735)
75f29c7 
- Enables pod log collection in all PSM interop jobs implemented in grpc#30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
@sergiitk sergiitk mentioned this pull request Sep 7, 2022
Merged
sergiitk added a commit to sergiitk/grpc that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (grpc#30735)
3f3aeda 
- Enables pod log collection in all PSM interop jobs implemented in grpc#30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
@sergiitk sergiitk mentioned this pull request Sep 7, 2022
Merged
sergiitk added a commit to sergiitk/grpc that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (grpc#30735)
1cbca9c 
- Enables pod log collection in all PSM interop jobs implemented in grpc#30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
@sergiitk sergiitk mentioned this pull request Sep 7, 2022
Merged
sergiitk added a commit to sergiitk/grpc that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (grpc#30735)
de20d83 
- Enables pod log collection in all PSM interop jobs implemented in grpc#30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
- Removed set command from the xlang script that doesn't exist in the
  following branches
@sergiitk sergiitk mentioned this pull request Sep 7, 2022
Merged
sergiitk added a commit to sergiitk/grpc that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (grpc#30735)
9a2cc07 
- Enables pod log collection in all PSM interop jobs implemented in grpc#30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
- Ignore xlang script as it hasn't been created
@sergiitk sergiitk mentioned this pull request Sep 7, 2022
Merged
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
fed6c04 
…30860)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
78907d0 
…30861)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
3a04a56 
…30862)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
32afaee 
…30863)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
ab526b0 
…30864)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
- Removed set command from the xlang script that doesn't exist in the
  following branches
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
afc3874 
…30865)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

Cherry-pick conflicts resolved:
- Removed authz test
- Ignore xlang script as it hasn't been created
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
8c04459 
…30857)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab
ejona86 pushed a commit that referenced this pull request Sep 7, 2022
@sergiitk
xDS interop: enable pod log collection in the buildscripts (#30735) (#…
ef66748 
…30856)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab
This was referenced Sep 7, 2022
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
veblush added a commit that referenced this pull request Sep 20, 2022
@veblush @apolcyn
@drfloob @gnossen @sergiitk @markdroth
Upmerge changes from 1.49.x (#31068)
fa65716 
* Bump 1.49.x branch to 1.49.0.pre1 (#30615)

* bump version to 1.49.0-pre1

* regenerate projects

* [backport][v1.49.x] forkable fixes (#30646)

* Fix forkable globals (#30608)

* Fix forkable repeated registration (#30642)

This fixes a bug that could occur on repeated grpc initialization (after a complete shutdown)

Fixes #30640

* Drop support for ruby 2.5 (#30699) (#30762)

* Drop ruby 2.5 support

* Backport: "stabilize the C2P resolver URI scheme" to v1.49.x (#30654)

* stabilize the C2P resolver URI scheme

* Bump 1.49 branch to 1.49.0.pre2 (#30786)

* bump version to 1.49.0-pre2

* regenerate projects

* Update protobuf on ancillary packages (#30795) (#30805)

* Bump release version on 1.49 to 1.49.0.pre3 (#30814)

* bump version to 1.49.0-pre3

* regenerate projects

* xDS interop: enable pod log collection in the buildscripts (#30735) (#30856)

- Enables pod log collection in all PSM interop jobs implemented in #30594.
- Associate test suite runs with their own log file, so it's displayed on "Target Log" tab

* xDS interop: buildscripts: fix run_test return status (#30768) (#30875)

To capture the return status of the test in run_test the last command must be the call to the test itself.
This removes `set +x`, which makes the run_test always return success, and not propagate the test status.

I can't find it, but this exact error bit us before. Looks like it leaked to other scripts.
The good thing is if the test was executed, it's failure would still be picked up from the result xml.

However, if the test framework didn't start in the first place, the result will be false positive.
Example: https://source.cloud.google.com/results/invocations/98d3e679-ec8a-40bd-9f36-88179747b0d6/targets

```
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3: Error while finding module specification for 'tests.authz_test' (ModuleNotFoundError: No module named 'tests')
+ set +x
Failed test suites: 0

[ID: 3548168] Command finished after 625 secs, exit value: 0
```

* client_channel: fix crash when cancelling a watch after SHUTDOWN (#30885) (#30928)

* Support Python 3.11 (#30818) (#30944)

* Support Python 3.11

* Update build images for 3.11

* Whoopsie

* The architecture of this thing is garbage

* Silence ownership warning

* Account for change in git behavior

* Fix directory

* I am in great pain

* Update Windows and arm linux

* Agh

* Clean up

* Bump 1.49 branch to 1.49.0 (#30974)

* bump version to 1.49.0

* regenerate projects

* Update protobuf to v21.6 on 1.49.x (#31028)

* Update third_party/protobuf

* run tools/distrib/python/make_grpcio_tools.py

* update build_handwritten.yaml

* regenerate projects

* Bump v1.49.x to v1.49.1 (#31037)

* bump version to 1.49.1

* regenerate projects

* Automated change: Fix sanity tests

Co-authored-by: gnossen <gnossen@users.noreply.github.com>

* Fix ruby windows ucrt build (#31053)

Co-authored-by: apolcyn <apolcyn@google.com>
Co-authored-by: AJ Heller <hork@google.com>
Co-authored-by: Richard Belleville <rbellevi@google.com>
Co-authored-by: Sergii Tkachenko <sergiitk@google.com>
Co-authored-by: Mark D. Roth <roth@google.com>
Co-authored-by: Richard Belleville <gnossen@gmail.com>
Co-authored-by: gnossen <gnossen@users.noreply.github.com>
sergiitk added a commit to grpc/psm-interop that referenced this pull request Nov 8, 2023
@sergiitk
xDS interop: collect pod logs (#30594)
e489068 
- Added support for pod log collection. To enable, set `--collect_app_logs` flag, and specify `--log_dir`.
- Added support and helpers for operating on the `--log_dir` (natively provided by absl)
- Added support for `--follow` to `bin/run_test_server.py` and `bin/run_test_client.py` to follow pod logs printed to stdout
- Moved `PortForwarder` from k8s.py to its own file

The collection itself will be enabled per-suite in grpc/grpc#30735.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gnossen gnossen gnossen approved these changes

Assignees

@ejona86 ejona86

Labels
area/psm interop bloat/none imported Specifies if the PR has been imported to the internal repository per-call-memory/neutral per-channel-memory/neutral release notes: no Indicates if PR should not be in release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@sergiitk @gnossen @ejona86