g-w multi linux: improve user deletion speed

[aerickson]

User deletion after task completion takes a long time (sometimes minutes).

alissy on slack in #wayland-support (https://mozilla.slack.com/archives/C03DF2FG5QX/p1713428059345379):

sometimes it's 30s sometimes it's 6-7 mins

pmoore:

i wonder if it is slow because it scans the entire file system. perhaps there is a way to index the files to avoid the full scan, or perhaps we can limit the paths that get scanned. the worry is things getting missed and workers running out of disk space because of e.g. a global shared folder somewhere getting swallowed up...

i'm open to doing it differently. also my hypothesis about full scan could be wrong or maybe there is a real time indexer we could use to avoid scan if that is the cause

logs:

Apr 18 08:09:54 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta systemd[1]: Starting Extra 30s wait...
Apr 18 08:10:29 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta systemd[1]: Started Extra 30s wait.
Apr 18 08:10:29 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta gdm-autologin][1435]: gkr-pam: no password is available for user
Apr 18 08:10:29 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta gdm-autologin][1435]: pam_unix(gdm-autologin:session): session opened for user task_171342722967242(uid=1005) by (uid=0)
Apr 18 04:57:08 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[554]: 2024/04/18 04:57:08 Running command: /usr/sbin/deluser --force --remove-all-files task_171339858312887
Apr 18 04:57:51 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2568]: delete user 'task_171339858312887'
Apr 18 04:59:23 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 04:59:23 Running command: /usr/sbin/deluser --force --remove-all-files task_171339868062403
Apr 18 05:00:06 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2320]: delete user 'task_171339868062403'
Apr 18 05:01:29 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:01:29 Running command: /usr/sbin/deluser --force --remove-all-files task_171341619384138
Apr 18 05:02:12 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2271]: delete user 'task_171341619384138'
Apr 18 05:03:35 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:03:34 Running command: /usr/sbin/deluser --force --remove-all-files task_171341632748681
Apr 18 05:04:18 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2265]: delete user 'task_171341632748681'
Apr 18 05:05:40 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[552]: 2024/04/18 05:05:40 Running command: /usr/sbin/deluser --force --remove-all-files task_171341645438547
Apr 18 05:06:25 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2461]: delete user 'task_171341645438547'
Apr 18 05:07:50 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[552]: 2024/04/18 05:07:50 Running command: /usr/sbin/deluser --force --remove-all-files task_171341658090281
Apr 18 05:08:31 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2271]: delete user 'task_171341658090281'
Apr 18 05:11:21 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[556]: 2024/04/18 05:11:21 Running command: /usr/sbin/deluser --force --remove-all-files task_171341670472643
Apr 18 05:12:04 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2264]: delete user 'task_171341670472643'
Apr 18 05:13:28 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:13:28 Running command: /usr/sbin/deluser --force --remove-all-files task_171341683612104
Apr 18 05:14:11 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2290]: delete user 'task_171341683612104'
Apr 18 05:15:35 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[554]: 2024/04/18 05:15:35 Running command: /usr/sbin/deluser --force --remove-all-files task_171341704838720
Apr 18 05:16:18 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2276]: delete user 'task_171341704838720'
Apr 18 05:17:43 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[554]: 2024/04/18 05:17:43 Running command: /usr/sbin/deluser --force --remove-all-files task_171341717226004
Apr 18 05:18:25 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2312]: delete user 'task_171341717226004'
Apr 18 05:21:15 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:21:15 Running command: /usr/sbin/deluser --force --remove-all-files task_171341730103189
Apr 18 05:21:57 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2277]: delete user 'task_171341730103189'
Apr 18 05:23:38 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[553]: 2024/04/18 05:23:38 Running command: /usr/sbin/deluser --force --remove-all-files task_171341742857117
Apr 18 05:24:23 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2468]: delete user 'task_171341742857117'
Apr 18 05:26:02 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:26:02 Running command: /usr/sbin/deluser --force --remove-all-files task_171341763861161
Apr 18 05:26:45 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2276]: delete user 'task_171341763861161'
Apr 18 05:28:12 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[551]: 2024/04/18 05:28:12 Running command: /usr/sbin/deluser --force --remove-all-files task_171341778466986
Apr 18 05:28:54 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2311]: delete user 'task_171341778466986'
Apr 18 05:55:58 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[566]: 2024/04/18 05:55:58 Running command: /usr/sbin/deluser --force --remove-all-files task_171341792847946
Apr 18 05:56:41 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2368]: delete user 'task_171341792847946'
Apr 18 06:00:45 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[567]: 2024/04/18 06:00:45 Running command: /usr/sbin/deluser --force --remove-all-files task_171341805761614
Apr 18 06:01:28 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2317]: delete user 'task_171341805761614'
Apr 18 06:05:43 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[569]: 2024/04/18 06:05:43 Running command: /usr/sbin/deluser --force --remove-all-files task_171341972361695
Apr 18 06:06:26 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2344]: delete user 'task_171341972361695'
Apr 18 06:13:54 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[569]: 2024/04/18 06:13:54 Running command: /usr/sbin/deluser --force --remove-all-files task_171342000847805
Apr 18 06:14:39 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2310]: delete user 'task_171342000847805'
Apr 18 06:18:49 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[568]: 2024/04/18 06:18:49 Running command: /usr/sbin/deluser --force --remove-all-files task_171342030763784
Apr 18 06:19:36 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2335]: delete user 'task_171342030763784'
Apr 18 06:27:15 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[569]: 2024/04/18 06:27:15 Running command: /usr/sbin/deluser --force --remove-all-files task_171342079815376
Apr 18 06:28:00 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2359]: delete user 'task_171342079815376'
Apr 18 06:35:33 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[570]: 2024/04/18 06:35:33 Running command: /usr/sbin/deluser --force --remove-all-files task_171342109361588
Apr 18 06:36:19 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2344]: delete user 'task_171342109361588'
Apr 18 06:43:48 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[566]: 2024/04/18 06:43:48 Running command: /usr/sbin/deluser --force --remove-all-files task_171342160096394
Apr 18 06:44:32 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2332]: delete user 'task_171342160096394'
Apr 18 06:52:09 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[563]: 2024/04/18 06:52:09 Running command: /usr/sbin/deluser --force --remove-all-files task_171342209853695
Apr 18 06:52:53 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2330]: delete user 'task_171342209853695'
Apr 18 07:00:26 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[568]: 2024/04/18 07:00:26 Running command: /usr/sbin/deluser --force --remove-all-files task_171342259140083
Apr 18 07:01:10 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2354]: delete user 'task_171342259140083'
Apr 18 07:08:51 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[564]: 2024/04/18 07:08:51 Running command: /usr/sbin/deluser --force --remove-all-files task_171342309330045
Apr 18 07:09:35 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2318]: delete user 'task_171342309330045'
Apr 18 07:17:09 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[569]: 2024/04/18 07:17:09 Running command: /usr/sbin/ddebug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
eluser --force --remove-all-files task_171342358908597
Apr 18 07:17:54 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2331]: delete user 'task_171342358908597'
Apr 18 07:25:32 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[567]: 2024/04/18 07:25:32 Running command: /usr/sbin/deluser --force --remove-all-files task_171342409610970
Apr 18 07:26:17 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2349]: delete user 'task_171342409610970'
Apr 18 07:30:39 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[566]: 2024/04/18 07:30:39 Running command: /usr/sbin/deluser --force --remove-all-files task_171342459437826
Apr 18 07:31:23 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2352]: delete user 'task_171342459437826'
Apr 18 07:35:43 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[567]: 2024/04/18 07:35:43 Running command: /usr/sbin/deluser --force --remove-all-files task_171342509611057
Apr 18 07:36:28 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2354]: delete user 'task_171342509611057'
Apr 18 07:40:48 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[569]: 2024/04/18 07:40:48 Running command: /usr/sbin/deluser --force --remove-all-files task_171342540495583
Apr 18 07:41:34 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2328]: delete user 'task_171342540495583'
Apr 18 07:45:51 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[565]: 2024/04/18 07:45:51 Running command: /usr/sbin/deluser --force --remove-all-files task_171342570824785
Apr 18 07:46:36 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2359]: delete user 'task_171342570824785'
Apr 18 07:52:22 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[565]: 2024/04/18 07:52:22 Running command: /usr/sbin/deluser --force --remove-all-files task_171342601347403
Apr 18 07:54:16 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2399]: delete user 'task_171342601347403'
Apr 18 08:01:55 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[565]: 2024/04/18 08:01:55 Running command: /usr/sbin/deluser --force --remove-all-files task_171342631540992
Apr 18 08:03:50 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta userdel[2471]: delete user 'task_171342631540992'
Apr 18 08:10:36 gecko-t-t-linux-2204-wayland-exp-a2-aui1xalfrqolnnqofn8hta start-worker[565]: 2024/04/18 08:10:36 Running command: /usr/sbin/deluser --force --remove-all-files task_171342670418032