/keybase mount, on linux, behaves poorly - hangs ls /

[mcint]

Keybase reliably causes ls / to hang.

• ls / -> hang
• sudo ls / -> ok
• sudo ls -l / -> hang
• sudo strace -l / -> hang, at statx(..., "/keybase",...)

Additionally, when I "Quit Keybase", which is unreasonably hidden and discouraged, echoes of dark pattern fashion, it doesn't clean up the mount, and the same issue still occurs!

statx(AT_FDCWD, "/usr", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=14, ...}) = 0
lgetxattr("/usr", "security.selinux", 0x63ab931f4fe0, 255) = -1 ENODATA (No data available)
getxattr("/usr", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("/usr", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
statx(AT_FDCWD, "/etc", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=285, ...}) = 0
lgetxattr("/etc", "security.selinux", 0x63ab931f5000, 255) = -1 ENODATA (No data available)
getxattr("/etc", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("/etc", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
statx(AT_FDCWD, "/run", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, {stx_mask=STATX_BASIC_STATS|STATX_MNT_ID, stx_attributes=STATX_ATTR_MOUNT_ROOT, stx_mode=S_IFDIR|0755, stx_size=1160, ...}) = 0
lgetxattr("/run", "security.selinux", 0x63ab931f5020, 255) = -1 ENODATA (No data available)
getxattr("/run", "system.posix_acl_access", NULL, 0) = -1 ENODATA (No data available)
getxattr("/run", "system.posix_acl_default", NULL, 0) = -1 ENODATA (No data available)
statx(AT_FDCWD, "/keybase", AT_STATX_SYNC_AS_STAT|AT_SYMLINK_NOFOLLOW|AT_NO_AUTOMOUNT, STATX_MODE|STATX_NLINK|STATX_UID|STATX_GID|STATX_MTIME|STATX_SIZE, ^C

Start with debugging

[sudo] systemctl status keybase* -> keybase.mount is the only systemd service installed.

$ sudo systemctl status keybase.mount
 ● keybase.mount - /keybase
      Loaded: loaded (/proc/self/mountinfo)
      Active: active (mounted) since Wed 2024-05-15 13:31:38 PDT; 1h 22min ago
       Where: /keybase
        What: keybase-redirector
         CPU: 8ms

After quitting keybase, not merely closing it, the keybase.mount is still mounted, but now throws errors,

$ ls /
ls: cannot access '/keybase': Transport endpoint is not connected
...

Resolve this with systemctl stop keybase.mount, only after keybase has been quit.

System Information

$ lsb_release -a
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.4 LTS
Release:	22.04
Codename:	jammy

$ uname -rvs
Linux 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  4 14:39:20 UTC 2

[doqfgc]

Can confirm this is the case.

It also causes all Flatpaks to hang or not start at all while Keybase is running.

[jbvv1]

I also believe that this same issue causes df to hang; sudo strace df showed that df was hanging when enumerating the /keybase mount. After I unloaded the keybase mount using systemctl stop keybase.mount, I was able to run df without any issues.

[phouverneyuff]

Same issue on Ubuntu 24.04

I need to start keybase -> kill it -> umount /keybase -> run keybase again

[GwynethLlewelyn]

Ah. Interesting. I have one server still on Ubuntu 22.04.4. For a while now (a few weeks?), there has been an issue in mounting /keybase — it doesn't work at all. I haven't figured out why, but the error was similar to what you report as "hanging". Or it would refuse to mount. The culprit seems not to be the Keybase daemon at all, not even the FUSE subsystem, but rather the ingenious Redirector — that's the one that fails to work.

What happens when you do a systemctl --user status keybase-redirector.service (as the non-privileged user)?

In my case, I get the following:

× keybase-redirector.service - Keybase Root Redirector for KBFS
     Loaded: loaded (/usr/lib/systemd/user/keybase-redirector.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-06-05 13:00:14 WEST; 53s ago
    Process: 1199828 ExecStartPre=/usr/bin/keybase --use-root-config-file config get --direct --assert-false --assert-ok-on-nil disable-root-redirector (code=exited, status=2)
        CPU: 41ms

Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Scheduled restart job, restart counter is at 5.
Jun 05 13:00:14 myServer systemd[1093]: Stopped Keybase Root Redirector for KBFS.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Start request repeated too quickly.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Failed with result 'exit-code'.
Jun 05 13:00:14 myServer systemd[1093]: Failed to start Keybase Root Redirector for KBFS.

The journal is not very informative, either (journalctl --user -u keybase-redirector.service):

[...]
Jun 05 13:00:14 myServer keybase[1199817]: true
Jun 05 13:00:14 myServer keybase[1199817]: 2024-06-05T13:00:14.489592+01:00 ▶ [ERRO keybase main.go:86] 001 Assertion failed.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Failed with result 'exit-code'.
Jun 05 13:00:14 myServer systemd[1093]: Failed to start Keybase Root Redirector for KBFS.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Scheduled restart job, restart counter is at 4.
Jun 05 13:00:14 myServer systemd[1093]: Stopped Keybase Root Redirector for KBFS.
Jun 05 13:00:14 myServer systemd[1093]: Starting Keybase Root Redirector for KBFS...
Jun 05 13:00:14 myServer keybase[1199828]: true
Jun 05 13:00:14 myServer keybase[1199828]: 2024-06-05T13:00:14.739666+01:00 ▶ [ERRO keybase main.go:86] 001 Assertion failed.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Control process exited, code=exited, status=2/INVALIDARGUMENT
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Failed with result 'exit-code'.
Jun 05 13:00:14 myServer systemd[1093]: Failed to start Keybase Root Redirector for KBFS.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Scheduled restart job, restart counter is at 5.
Jun 05 13:00:14 myServer systemd[1093]: Stopped Keybase Root Redirector for KBFS.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Start request repeated too quickly.
Jun 05 13:00:14 myServer systemd[1093]: keybase-redirector.service: Failed with result 'exit-code'.
Jun 05 13:00:14 myServer systemd[1093]: Failed to start Keybase Root Redirector for KBFS.

My quick hack was to turn it off, and simply look for my files at /run/user/1000/keybase/kbfs/ (my UID is 1000, as you can guess). Since I'm the only user on that server which has Keybase enabled for their account, I'm seriously considering turning /keybase into just a symlink to /run/user/1000/keybase/kbfs/! 🤣