[CSM O11y test] Ping GMP endpoint during test for debugging purpose #33

stanley-cheung · 2024-02-07T08:16:48Z

Ping GMP endpoint in client and server pod <pod IP>:9464/metrics during the CSM Observability test
This will provide insight into what GMP sees from the OTel plugin, before sending metrics to Cloud Monitoring
This will help debug in case we get these extra metrics time series without a valid data point, in a future test flake.

- Ping GMP endpoint in client and server pod curl localhost:9464/metrics during test - This will provide insight into what GMP gets from the OTel plugin before sending metrics to Cloud Monitoring

sergiitk · 2024-02-07T17:40:42Z

I most definitely won't recommend this approach. Instead, we should consider making normal HTTP requests using requests or urllib3.

- But this doesn't work yet. Can't quite figure out what's the exact request URL to use. Tried quite a few combinations and none of them works.

stanley-cheung · 2024-02-07T20:58:45Z

I most definitely won't recommend this approach. Instead, we should consider making normal HTTP requests using requests or urllib3.

Do you know what the request URL should I be using? I tried quite a few combinations but none of them works

http://{server pod name}.{server namespace}.svc.cluster.local:9464/metrics
http://{server pod name}.{server namespace}.pod.cluster.local:9464/metrics
http://{frontend service name}.{server namespace}.svc.cluster.local:9464/metrics
http://{server pod IP replaced dots with dashes}.{server namespace}.pod.cluster.local:9464/metrics
http://{server pod IP}:9464/metrics

If I supply {server pod name}, {server namespace}, {container name} into pod_exec and then run curl -s localhost:9464/metrics, it does work. So it seems like it's a matter arranging {server pod name}, {server namespace}, {container name} into the correct FQDN?

I just realized I am running the requests.get() from the main test driver (csm_observability_test.py). That is outside of the cluster isn't it? So it won't understand the pod IP address or anything .svc.cluster.local isn't it.

tests/gamma/csm_observability_test.py

- `enable_csm_observability` is now a constructor parameter for the client runner and the server runner, instead of a `run()` parameter. - If `enable_csm_observability`, we want to do a port forwarding for the `pod_monitoring_port = 9464`, such that we can issue a GET request against it in the CSM test - Added `monitoring_port` and `monitoring_host` as class variable to the client runner and server runner. - Override `_xds_test_server_for_pod` in the `GammaServerRunner` class - Minimize the places where we need to hardcode the port `9464`

stanley-cheung · 2024-02-08T00:59:41Z

grpc/core/master/linux/psm-csm

stanley-cheung · 2024-02-08T03:55:59Z

grpc/core/master/linux/psm-csm

tests/gamma/csm_observability_test.py

requirements.txt

framework/test_app/runners/k8s/k8s_xds_client_runner.py

tests/gamma/csm_observability_test.py

framework/test_app/runners/k8s/gamma_server_runner.py

requirements.txt

- Added a PrometheusLogger class to write the prometheus endpoint log to a separate log file related to logs_subdir - monitoring_port is now stored in XdsTestClient and XdsTestServer rather than the runner - use rpc_host instead of a separate monitoring_host - put the constant DEFAULT_MONITORING_PORT in the KubernetesBaseRunner - fixed requirements.txt and requirements.lock - removed a previous hack is_legit_time_series() - renamed function to ping_prometheus_endpoint instead of referring it as GMP

stanley-cheung · 2024-02-21T03:57:47Z

grpc/core/master/linux/psm-csm

stanley-cheung · 2024-02-21T04:34:31Z

grpc/core/master/linux/psm-csm

stanley-cheung · 2024-03-05T21:33:29Z

grpc/core/master/linux/psm-csm

sergiitk

LGTM with

FIxing the log collection arg
Running full csm test suite + lb test suite

tests/gamma/csm_observability_test.py

…erop into ping-gmp-endpoint

stanley-cheung · 2024-05-17T00:24:57Z

stanley-cheung · 2024-05-17T02:04:38Z

@sergiitk Incorporated latest feedback. Using ClientDeploymentArgs now. Will do ServerDeploymentArgs in a separate PR.

Want to do a final check and merge? Thanks.

sergiitk

It appears that a bunch grpc/core/master/linux/psm-csm test run failed with

Traceback (most recent call last):
  File "/tmp/tmp.2ObQJ9ydNu/psm-interop/tests/app_net_ssa_test.py", line 111, in test_session_affinity_policy
    test_client: _XdsTestClient = self.startTestClient(
  File "/tmp/tmp.2ObQJ9ydNu/psm-interop/framework/xds_k8s_testcase.py", line 928, in startTestClient
    return self._start_test_client(test_server.xds_uri, **kwargs)
  File "/tmp/tmp.2ObQJ9ydNu/psm-interop/framework/xds_k8s_testcase.py", line 831, in _start_test_client
    test_client = self.client_runner.run(
  File "/tmp/tmp.2ObQJ9ydNu/psm-interop/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 188, in run
    **self.deployment_args.as_dict(),
AttributeError: 'NoneType' object has no attribute 'as_dict'

sergiitk · 2024-05-20T19:43:46Z

The lb test suite failed with the same error.

framework/test_app/runners/k8s/k8s_xds_client_runner.py

stanley-cheung · 2024-05-20T21:10:49Z

[CSM O11y test] Ping GMP endpoint regularly for debug purpose

81b567e

- Ping GMP endpoint in client and server pod curl localhost:9464/metrics during test - This will provide insight into what GMP gets from the OTel plugin before sending metrics to Cloud Monitoring

stanley-cheung mentioned this pull request Feb 7, 2024

[CSM O11y test] Filter out time series with no valid data point #32

Merged

stanley-cheung marked this pull request as draft February 7, 2024 18:44

Use requests instead of pod exec

d2c1e97

- But this doesn't work yet. Can't quite figure out what's the exact request URL to use. Tried quite a few combinations and none of them works.

sergiitk reviewed Feb 7, 2024

View reviewed changes

tests/gamma/csm_observability_test.py Show resolved Hide resolved

sergiitk reviewed Feb 7, 2024

View reviewed changes

tests/gamma/csm_observability_test.py Outdated Show resolved Hide resolved

stanley-cheung added 2 commits February 8, 2024 00:18

Add requests to requirements.txt

b9caa33

stanley-cheung changed the title ~~[CSM O11y test] [DO NOT MERGE] Ping GMP endpoint regularly for debug purpose~~ [CSM O11y test] Ping GMP endpoint regularly for debug purpose Feb 8, 2024

stanley-cheung changed the title ~~[CSM O11y test] Ping GMP endpoint regularly for debug purpose~~ [CSM O11y test] Ping GMP endpoint during test for debugging purpose Feb 8, 2024

try catch RequestException

3ab0f31

stanley-cheung marked this pull request as ready for review February 8, 2024 01:03

Merge branch 'main' into ping-gmp-endpoint

73765f0

sergiitk requested changes Feb 9, 2024

View reviewed changes

stanley-cheung added 5 commits February 20, 2024 10:50

Merge branch 'main' into ping-gmp-endpoint

dc6c854

Merge branch 'main' into ping-gmp-endpoint

f11a723

remove unnecessary variable which broke unit test

d025083

fix pylint for function signature

1c0f3dd

fixed variable initialization

90c533c

stanley-cheung added 4 commits February 21, 2024 18:31

add comment for temp debugging class

01f4c4d

Merge branch 'main' into ping-gmp-endpoint

a3e52dd

Add a marker in the prometheus log files

bcf43a0

Merge branch 'main' into ping-gmp-endpoint

02ea1cc

stanley-cheung added 5 commits March 4, 2024 15:32

Merge branch 'main' into ping-gmp-endpoint

d4fabf2

Merge branch 'main' into ping-gmp-endpoint

9d117f4

Merge branch 'main' into ping-gmp-endpoint

5a49e3a

Merge branch 'main' into ping-gmp-endpoint

bae1c81

Merge branch 'main' into ping-gmp-endpoint

e5d0270

Merge branch 'main' into ping-gmp-endpoint

c71a050

stanley-cheung requested a review from sergiitk March 27, 2024 23:21

stanley-cheung mentioned this pull request Apr 18, 2024

Add App Net based CSM Observability Test #72

Closed

sergiitk force-pushed the ping-gmp-endpoint branch 2 times, most recently from 2bb2bd3 to c71a050 Compare May 10, 2024 03:24

Merge branch 'main' into ping-gmp-endpoint

32bfb74

sergiitk approved these changes May 13, 2024

View reviewed changes

tests/gamma/csm_observability_test.py Outdated Show resolved Hide resolved

tests/gamma/csm_observability_test.py Outdated Show resolved Hide resolved

stanley-cheung added 5 commits May 16, 2024 23:26

Review feedback: check should_collect_logs flags first

3d0437f

Merge branch 'main' into ping-gmp-endpoint

3020499

Merge branch 'ping-gmp-endpoint' of github.com:stanley-cheung/psm-int…

19a9a2b

…erop into ping-gmp-endpoint

Fully use ClientDeploymentArgs

2a4c603

ran black.sh

95edebd

stanley-cheung requested a review from sergiitk May 17, 2024 02:04

sergiitk requested changes May 20, 2024

View reviewed changes

sergiitk reviewed May 20, 2024

View reviewed changes

framework/test_app/runners/k8s/k8s_xds_client_runner.py Outdated Show resolved Hide resolved

Fix ClientDeploymentArgs initialization

0518a7d

sergiitk approved these changes May 21, 2024

View reviewed changes

sergiitk merged commit d07a3ab into grpc:main May 21, 2024
7 checks passed

stanley-cheung deleted the ping-gmp-endpoint branch May 21, 2024 19:44

stanley-cheung mentioned this pull request May 23, 2024

Add App Net based CSM Observability Test #86

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CSM O11y test] Ping GMP endpoint during test for debugging purpose #33

[CSM O11y test] Ping GMP endpoint during test for debugging purpose #33

stanley-cheung commented Feb 7, 2024 •

edited by sergiitk

sergiitk commented Feb 7, 2024

stanley-cheung commented Feb 7, 2024 •

edited

stanley-cheung commented Feb 8, 2024 •

edited

stanley-cheung commented Feb 8, 2024 •

edited

stanley-cheung commented Feb 21, 2024 •

edited

stanley-cheung commented Feb 21, 2024 •

edited

stanley-cheung commented Mar 5, 2024 •

edited

sergiitk left a comment

stanley-cheung commented May 17, 2024 •

edited

stanley-cheung commented May 17, 2024

sergiitk left a comment

sergiitk commented May 20, 2024

stanley-cheung commented May 20, 2024 •

edited

[CSM O11y test] Ping GMP endpoint during test for debugging purpose #33

[CSM O11y test] Ping GMP endpoint during test for debugging purpose #33

Conversation

stanley-cheung commented Feb 7, 2024 • edited by sergiitk

sergiitk commented Feb 7, 2024

stanley-cheung commented Feb 7, 2024 • edited

stanley-cheung commented Feb 8, 2024 • edited

stanley-cheung commented Feb 8, 2024 • edited

stanley-cheung commented Feb 21, 2024 • edited

stanley-cheung commented Feb 21, 2024 • edited

stanley-cheung commented Mar 5, 2024 • edited

sergiitk left a comment

Choose a reason for hiding this comment

stanley-cheung commented May 17, 2024 • edited

stanley-cheung commented May 17, 2024

sergiitk left a comment

Choose a reason for hiding this comment

sergiitk commented May 20, 2024

stanley-cheung commented May 20, 2024 • edited

stanley-cheung commented Feb 7, 2024 •

edited by sergiitk

stanley-cheung commented Feb 7, 2024 •

edited

stanley-cheung commented Feb 8, 2024 •

edited

stanley-cheung commented Feb 8, 2024 •

edited

stanley-cheung commented Feb 21, 2024 •

edited

stanley-cheung commented Feb 21, 2024 •

edited

stanley-cheung commented Mar 5, 2024 •

edited

stanley-cheung commented May 17, 2024 •

edited

stanley-cheung commented May 20, 2024 •

edited