g-w multi on linux: improve detection of launched session to avoid races

[aerickson]

We're currently seeing races between autologin on Wayland and g-w running tasks on our test pool (gecko-t/t-linux-2204-wayland-exp-a2) on FirefoxCI (running g-w 60.4.2). It seems to be an issue around detecting GNOME Session state.

alissy on https://mozilla.slack.com/archives/C03DF2FG5QX/p1713444159025359 and in #wayland-support slack:

I think there's still a race in how we detect wayland session is ready.

I think the checking just gnome-session might still be racy?

pmoore in https://mozilla.slack.com/archives/C03DF2FG5QX/p1702400170517189:

some context for tomorrow …. on this particular worker, it looks like the gnome session at least started, because the code was able to query the interactive user and move on. later when it queried it again, the session was gone. so there are a few options here.

• gdm-wayland-session / gdm died e.g. during initialisation
• for some reason w stopped showing the session, after showing it for some time, e.g. because its internal database was being rotated or something
• generic-worker doesn’t correctly detect if gnome session is running
  note, we are getting two different failure cases - this particular one means that w was reporting the session at some point, and then stopped doing so - tomorrow i will grab logs for the other case, which might be when gdm doesn’t come up at all, i’m not sure - tomorrow we’ll know

note this is the rather hacky detection process:

taskcluster/workers/generic-worker/gdm3/gdm3_nonfreebsd.go

Line 16 in efd36f0

gnomeSessionUserList, err := host.CombinedOutput("/usr/bin/env", "bash", "-c", "PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d' '")

log output of g-w getting stuck detecting user:

Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Detected linux platform
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Detected multiuser engine
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 WORKER_METRICS {"eventType":"instanceBoot","instanceType":"projects/887720501152/machineTypes/n2-standard-2","region":"us-central1","timestamp":1713441812,"worker":"generic-worker","workerId":"4962968278517201153","workerPoolId":"gecko-t/t-linux-2204-wayland-exp-a2"}
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Working directory: "/"
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Tasks resolved count file: "/tasks-resolved-count.txt"
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature Live Log...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature Taskcluster Proxy...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature OS Groups...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature Mounts/Caches...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Loaded file file-caches.json
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Loaded file directory-caches.json
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature Interactive...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature LoopbackAudio...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature LoopbackVideo...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:05 Initialising task feature Chain of Trust...
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Running command: /usr/bin/id -u
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Running command: /usr/bin/id -g
Apr 18 12:04:09 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 All features initialised.
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Current task user file: "/current-task-user.json"
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Next task user file: "/next-task-user.json"
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Running command: /usr/bin/id -u
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Running command: /usr/bin/id -g
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Checking if user is logged in...
Apr 18 12:04:12 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:06 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:30 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:30 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:31 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:31 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:31 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:31 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:32 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:32 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:32 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:32 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:33 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:33 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:33 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:33 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:34 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:34 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:34 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:34 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:35 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:35 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:35 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:35 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:36 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:36 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:36 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:36 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:37 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:37 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:37 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:37 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:39 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:38 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:39 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:38 WARNING: Error checking for interactive user: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:39 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/env bash -c 'PROCPS_USERLEN=20 /usr/bin/w | /bin/grep gnome-[s]ession | /usr/bin/cut -f1 -d''' ''
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Saving file file-caches.json (absolute path: /file-caches.json)
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Saving file directory-caches.json (absolute path: /directory-caches.json)
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/id -u
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/id -g
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/id -u
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 Running command: /usr/bin/id -g
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 goroutine 1 [running]:
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: runtime/debug.Stack()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/go/go/src/runtime/debug/stack.go:24 +0x5e
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.HandleCrash({0xa246c0, 0xc0003ba410})
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:338 +0x2f
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.RunWorker.func1()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:360 +0x2f
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: panic({0xa246c0?, 0xc0003ba410?})
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/go/go/src/runtime/panic.go:770 +0x132
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.PlatformTaskEnvironmentSetup({0xc0000e68b8, 0x14})
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/multiuser.go:73 +0xb28
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.PrepareTaskEnvironment()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:1186 +0xc5
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.RotateTaskEnvironment()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:1252 +0x17
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.RunWorker()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:417 +0x78e
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: main.main()
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: /task_171148220171370/taskcluster/workers/generic-worker/main.go:159 +0x565
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 *********** PANIC occurred! ***********
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:39 could not determine interactive username: number of gnome session users is not exactly one - not sure which user is interactively logged on: []string{""}
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:40 WARNING: Could not get sentry DSN:
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: CALL SUMMARY
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: ============
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Method: GET
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Service: firefox-ci-tc.services.mozilla.com:
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Response Body:
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: {
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "code": "InsufficientScopes",
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "message": "Client ID worker/fxci-level1-gcp/gecko-t/t-linux-2204-wayland-exp-a2/us-central1-b/4962968278517201153 does not have sufficient scopes and is missing the following scopes:\n\n\nauth:sentry:generic-worker\n\n\nThis request requires the client to satisfy the following scope expression:\n\n\nauth:sentry:generic-worker\n\n\n---\n\n* method: sentryDSN\n* errorCode: InsufficientScopes\n* statusCode: 403\n* time: 2024-04-18T12:04:39.840Z",
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "requestInfo": {
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "method": "sentryDSN",
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "params": {
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "project": "generic-worker"
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: },
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "payload": {},
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "time": "2024-04-18T12:04:39.840Z"
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: }
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: }
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Attempts: 1
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: (Permanent) HTTP response code 403
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: HTTP/2.0 403 Forbidden
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Content-Length: 674
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Access-Control-Allow-Headers: X-Requested-With,Content-Type,Authorization,Accept,Origin,Cache-Control
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Access-Control-Allow-Methods: OPTIONS,GET,HEAD,POST,PUT,DELETE,TRACE,CONNECT
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Access-Control-Allow-Origin: *
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Access-Control-Max-Age: 900
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Access-Control-Request-Method: *
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Alt-Svc: clear
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Cache-Control: no-store no-cache must-revalidate
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Content-Security-Policy: report-uri /cspreport;default-src 'none';frame-ancestors 'none';
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Content-Type: application/json; charset=utf-8
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Date: Thu, 18 Apr 2024 12:04:39 GMT
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Etag: W/"2a2-l6Qcj+B+N4zK7CRci62jPhfESss"
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Server: openresty
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Strict-Transport-Security: max-age=31536000
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: Via: 1.1 google
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: X-Content-Type-Options: nosniff
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: X-For-Request-Id: 2e5f934e-2dff-442f-8c07-93eebd2b97bf
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: X-For-Trace-Id: 56cb30b3b4b54bd7e88c0f8fbed7e117
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]:
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: {
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "code": "InsufficientScopes",
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "message": "Client ID worker/fxci-level1-gcp/gecko-t/t-linux-2204-wayland-exp-a2/us-central1-b/4962968278517201153 does not have sufficient scopes and is missing the following scopes:\n\n\nauth:sentry:generic-worker\n\n\nThis request requires the client to satisfy the following scope expression:\n\n\nauth:sentry:generic-worker\n\n\n---\n\n* method: sentryDSN\n* errorCode: InsufficientScopes\n* statusCode: 403\n* time: 2024-04-18T12:04:39.840Z",
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "requestInfo": {
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "method": "sentryDSN",
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "params": {
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "project": "generic-worker"
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: },
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "payload": {},
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: "time": "2024-04-18T12:04:39.840Z"
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: }
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: }
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:40 Exiting worker with exit code 69
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:40 WORKER_METRICS {"eventType":"instanceShutdown","instanceType":"projects/887720501152/machineTypes/n2-standard-2","region":"us-central1","timestamp":1713441879,"worker":"generic-worker","workerId":"4962968278517201153","workerPoolId":"gecko-t/t-linux-2204-wayland-exp-a2"}
Apr 18 12:04:41 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg start-worker[565]: 2024/04/18 12:04:40 exit status 69
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg systemd[1]: worker.service: Main process exited, code=exited, status=1/FAILURE
Apr 18 12:04:40 gecko-t-t-linux-2204-wayland-exp-a2-c54ei9jltxslbc-q9pxkqg systemd[1]: worker.service: Failed with result 'exit-code'

[lissyx]

From discussing with GNOME folks there does not seems to be anything really standard besides dbus we could rely on. @petemoore Would it be possible we make use of dbus bindings and listen for some GNOME event there? Not sure yet which one.

[petemoore]

Sure, we can change to something else if we find a more reliable method.

[lissyx]

I'll spy on my system to see what dbus-level thing we might pick on then