Support SandboxFS (currently experiences FS errors)

[joeljeske]

The goal of this request is to be compatible with SandboxFS. SandboxFS uses a FUSE mount to provide an arbitrary r/rw view of files on disk, as opposed to using other sandboxing/symlinking techniques. Theoretically, rules_nodejs might benefit the most from SandboxFS compared to any other ruleset due to the frequency of having enormous node_modules trees where each file needs to be symlinked into a sandbox.

Rulesets like this should likely not be concerned with how Bazel chooses to sandbox the processes, but it appears that rules_nodejs has a couple of places where it is writing to disk in undeclared output locations, causing file access errors due to the stricter nature of sandboxfs compared to linux-sandbox or darwin-sandbox.

It would seem that these places should either be disabled or refactored in order to not write to disk in undeclared locations, causing issues.

node_patches.js
https://github.com/bazelbuild/rules_nodejs/blob/stable/internal/node/node_patches.js#L545-L600
It would seem to me that this patching of fs is made unneeded if running in sandboxfs; a user cannot accidentally hop out of the sandbox to my understanding.

The error from this file looks like this

Use --sandbox_debug to see verbose messages from the sandbox
/mnt/raid/bazel/3352fc3e2110fe708526cee164a04507/sandbox/sandboxfs/1991/ns/bazel-out/host/bin/tool.sh.runfiles/build_bazel_rules_nodejs/internal/node/node_patches.js:566
            throw e;
            ^

Error: EACCES: permission denied, mkdir '/mnt/raid/bazel/3352fc3e2110fe708526cee164a04507/sandbox/sandboxfs/1991/ns/bazel-out/host/bin/tool.sh.runfiles/build_bazel_rules_nodejs/internal/node/_node_bin'
    at Object.mkdirSync (fs.js:921:3)
    at Object.exports.patcher [as subprocess] (/mnt/raid/bazel/3352fc3e2110fe708526cee164a04507/sandbox/sandboxfs/1991/ns/bazel-out/host/bin/tool.sh.runfiles/build_bazel_rules_nodejs/internal/node/node_patches.js:561:14)
    at Object.<anonymous> (/mnt/raid/bazel/3352fc3e2110fe708526cee164a04507/sandbox/sandboxfs/1991/ns/bazel-out/host/bin/tool.sh.runfiles/build_bazel_rules_nodejs/internal/node/node_patches.js:670:5)
    at Module._compile (internal/modules/cjs/loader.js:999:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)
    at Module.load (internal/modules/cjs/loader.js:863:32)
    at Function.Module._load (internal/modules/cjs/loader.js:708:14)
    at Module.require (internal/modules/cjs/loader.js:887:19)
    at Module._preloadModules (internal/modules/cjs/loader.js:1147:12)
    at loadPreloadModules (internal/bootstrap/pre_execution.js:446:5) {
  errno: -13,
  syscall: 'mkdir',
  code: 'EACCES',
  path: '/mnt/raid/bazel/3352fc3e2110fe708526cee164a04507/sandbox/sandboxfs/1991/ns/bazel-out/host/bin/tool.sh.runfiles/build_bazel_rules_nodejs/internal/node/_node_bin'
}

link_node_modules.ts
https://github.com/bazelbuild/rules_nodejs/blob/stable/internal/linker/link_node_modules.ts
This issues seems quite a bit more complicated as the goal of this file is to ensure that the node_modules tree is represented properly as desired by the rule's dependencies. I think that in the past rules_nodejs would use a require patch to rewrite require requests from the dev.

The error that comes from this file looks like this:

[link_node_modules.js] An error has been reported: [Error: EACCES: permission denied, mkdir 'apps'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: 'apps'
} Error: EACCES: permission denied, mkdir 'apps'

Other places?

Alternatives

It would be nice if we could configure a bazel sandbox layer on top of the presented view that could be written to and then thrown away at the end of an action. That would seem like it would easier than handling all of the cases where rules_nodejs might write. I am not aware of this capability in bazel today

note: please advise if my understanding of sandboxfs or rules_nodejs is lacking, and different approaches should be considered.

Related:

#1982 - not restricted to TSC only, rather all nodejs_binary invocations

[alexeagle]

Can sandboxfs allow writes into the execroot? That's fundamentally part of how our linker works.

Also do you know if there's any movement on sandboxfs? Is the project healthy and gaining adoption?

[joeljeske]

I honestly do not know what can be configured with sandbox fs. I had previously been under the impression that actions were supposed to only write to pre-declared output locations, and it was surprising to me that rules_nodejs did this, although it makes sense of course.

There does not seem to be a lot of documentation with sandbox-fs, and I cannot say if it is a healthy project.

I have determined that some of my rules using npm_package_bin seem to be very slow, and I am presuming that it is because the node_modules tree can be super heavy when including some particular packages. Theoretically, something like sandbox fs would be of significant benefit to npm_package_bin actions, especially those that are used in very many places in bazel, take a relatively short amount of time to actually execute the package, and those that have a heavy node_module tree (especially when most of the files symlinked into the tree are never actually read).

[pauldraper]

Also do you know if there's any movement on sandboxfs? Is the project healthy and gaining adoption?

It's used a fair amount, though there's no development on it.

Biggest issue I've run into is that it precludes using linux-sandbox for tests.