Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Intermittent warning/skips on Composer rules #72

Open
qgallet opened this issue Jun 30, 2023 · 1 comment
Open

Intermittent warning/skips on Composer rules #72

qgallet opened this issue Jun 30, 2023 · 1 comment

Comments

@qgallet
Copy link

qgallet commented Jun 30, 2023

This was working fine on 0.59, now there seems to be some lock issue on Composer API queries. Here's the command used:

gcpdiag lint --project=my-project --include=composer

At the start of the command, the following log will be temporarily visible on the console, before the results are printed.

... still fetching logs (project: my-project, resource type: cloud_composer_environment, max wait: 117s)

After two minutes, some of the checks are skipped because of (I guess) a timeout:

🔎  composer/BP/2023_001: Cloud Composer logging level is set to INFO
   - my-project/europe-west1/my-composer               [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_003 
🔎  composer/ERR/2022_001: Composer Service Agent permissions
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2023_001 
🔎  composer/WARN/2022_001: Composer Service Agent permissions for Composer 2.x
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_003 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_001 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_003 
🔎  composer/WARN/2023_004: Cloud Composer database CPU usage does not exceed 80%
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_005: Cloud Composer is consistently in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_006: Airflow schedulers are healthy for the last hour
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_007: Cloud Composer Scheduler CPU limit exceeded.
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_008: Cloud Composer Airflow database is in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

Rules summary: 9 skipped, 8 ok, 0 failed

If I launch the command with a more restrictive parameter, like --include=composer/WARN, this still happens although less frequently.

@ropeck
Copy link
Collaborator

ropeck commented Jul 31, 2023

Thanks for the issue report. Can you check if this still occurs with v0.63 recently released?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants