Std ops.mkStandardOCI compatibility problem with ops.mkOperable to wrap

[Pegasust]

Fixed in #331

---

I have a pretty standard poetry2nix web application that I wrap on its environment derivation using hypercorn. The built OCI fails to detect the correct binary location because it relies on lib.getExe. mkOperable works as expected, though.

Symptoms

I don't have the exact error that throws from Docker engine. It should be something along the line of Docker container not being to start up because /bin/entrypoint not found.

If you have access to ls (I use Python's os.listdir because I'm too lazy to build a dev layer), You would see that /bin/entrypoint exists, but it is a broken symlink.

$ std //ops/oci/p2n-env-oci:load
docker-daemon:<image>:<hash>
$ docker run -it --rm --entrypoint /bin/sh <image>:<hash>
$ python3

import os

def follow_symlinks(path):
    if os.path.islink(path):
        link_path = os.readlink(path)
        print(f'{path} -> {link_path}')
        return follow_symlinks(link_path)
    elif os.path.exists(path):
        print(f'Path is a real file or directory: {path}')
        return path
    else:
        print(f'Path not found: {path}')
        return None

follow_symlinks('/bin/entrypoint')

One example output is

/bin/entrypoint -> /nix/store/<hash>-oci-setup-links/bin/entrypoint
/nix/store/<hash>-oci-setup-links/bin/entrypoint -> /nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3
Path not found: /nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3
>>> os.listdir('/nix/store/<hash>-operable-python3-3.10.11-env/bin/')
["operable-python3-3.10.11-env"]

Workaround

  p2n-env-fix = workspace_app.dependencyEnv.overrideAttrs (prev: {
    # HACK: This allows mkOperable and mkStandardOCI to resolve symlink correctly
    name = "p2n-env-fix";
    meta.mainProgram = "python3";
  });

Nix flakes and its workaround variant

# nix/dev/packages/tools.nix
{
  inputs,
  cell,
}: let
  system = inputs.nixpkgs.system;
  poetry2nix = inputs.nix-boost.pkgs.${system}.mypkgs.poetry2nix;
  workspace_root = "${inputs.self}";
  workspace_app = poetry2nix.mkPoetryApplication {
    projectDir = workspace_root;
  };
in {
  p2n-env = workspace_app.dependencyEnv;
  p2n-env-fix = workspace_app.dependencyEnv.overrideAttrs (prev: {
    # HACK: This allows mkOperable and mkStandardOCI to resolve symlink correctly
    name = "p2n-env-fix";
    meta.mainProgram = "python3";
  });
  frontend = {}; # omitted
}

# nix/ops/exe.nix
{
  inputs,
  cell,
}: let
  inherit (inputs.std.lib) ops;
  runtimeScript = ''
      hypercorn my.backend:app --bind '0.0.0.0:10140' --worker-class uvloop
  '';
  runtimeEnv = {
    DIST_LOC = "${inputs.cells.dev.packages.frontend}/dist";
  };
in {
  p2n-env-ops = ops.mkOperable {
    package = inputs.cells.dev.packages.p2n-env;
    runtimeInputs = [
      inputs.cells.dev.packages.p2n-env
      inputs.cells.dev.packages.frontend
    ];
    inherit runtimeScript runtimeEnv;
  };

  p2n-env-fix-ops = ops.mkOperable {
    package = inputs.cells.dev.packages.p2n-env-fix;
    runtimeInputs = [
      inputs.cells.dev.packages.p2n-env-fix
      inputs.cells.dev.packages.frontend
    ];
    inherit runtimeScript runtimeEnv;
  };
}

# nix/ops/oci.nix
{
  inputs,
  cell,
}: let
  inherit (inputs.std.lib) ops;
  pyproject = builtins.fromTOML (builtins.readFile "${inputs.self}/pyproject.toml");
  ws-authors = pyproject.tool.poetry.authors;
in {
  p2n-env-oci = ops.mkStandardOCI {
    name = "p2n-env-oci";
    # NOTE: swap to p2n-env-fix-ops fixes it
    operable = inputs.cells.ops.exe.p2n-env-ops;
    meta = {
      # NOTE: unfair, std doesn't allow me to give more tags :(
      # including this and `nix` and `std` tells me to remove either `meta.tags` or `tag`
      # tags = ["ops-latest" "ops-${ws-version}" "${ws-name}-${ws-version}"];
    };
    config = {
      ExposedPorts."10140/tcp" = {};
      Volumes = {
        # For secrets and env
        "/var/run" = {};
      };
      Env = [
        "ENV_PATH=/var/run/.env"
        "GNMI_CONFIG_LOC=/var/run/.config.yml"
      ];
      Labels."org.opencontainers.image.title" = "p2n-env";
      Labels."org.opencontainers.image.authors" = builtins.concatStringsSep ", " ws-authors;
    };
  };

  # Currently, there is no good documented way to publish to arbitrary repositories at once
  # NOTE: untested
  p2n-env-gl = cell.oci.p2n-env-oci.overrideAttrs {
    meta.repo = "gitlab.<domain>.com/<group>/<repo>/<image>";
  };
}

Some more insights from build artifacts

nix-repl> packages.aarch64-darwin.p2n-env.name
"python3-3.10.11-env"

nix-repl> packages.aarch64-darwin.p2n-env.meta.name
"python3-3.10.11"

nix-repl> lib.getExe packages.aarch64-darwin.p2n-env-ops
"/nix/store/<hash>-operable-python3-3.10.11-env/bin/operable-python3"

Thoughts

• poetry2nix recommends using dependencyEnv when working with ASGI runtime like hypercorn, but I have yet to try this with mkPoetryEnv. mkPoetryEnv does provide a simple way to override onto a simpler name, whereas with dependencyEnv overwrites the declared name.
• Nevertheless, this case should be considered in general for packages that may have rough edges in its name.

Requests

• Resolve this issue along with Straightforward example implementing operables/containers? #299 by adding minimal build recipes along with troubleshoots
• There is a bit of rough edges mentioned in the provided workspace for Improve container actions #251 in the case of tags
• Add check phase for mkStandardOCI that checks for symlink integrity. Might be doable at the phase of writing /bin/entrypoint and before using nix2container to write .json manifest

[jboyens]

Ran into this same issue today as well, but not with Python.

It looks like if the package uses pname/version (vs. just name) and thus has a name of ${pname}-${version} the entrypoint generated script can't find it as it drops the version.

I think it's related to the fact that the operable wraps by using ${package.name} at one point, but I got a little lost in the weeds trying to trace it thru the various blockTypes and functions.

[blaggacao]

Sorry for the delay in looking into this and thanks for the report!

The issue, so to speak lies in this line and derives indeed from the implementation choices of getExe:

std/src/lib/ops/mkOCI.nix

Line 48 in 301a649

ln -s ${l.getExe entrypoint} $out/bin/entrypoint

---

# including this and nixandstdtells me to remove eithermeta.tagsortag``

This has fortunately just been fixed.

---

For my own better understanding:

  runtimeScript = ''
      hypercorn my.backend:app --bind '0.0.0.0:10140' --worker-class uvloop
  '';
  

Which of the code in the flake brings my.backend:app into the python path? (I'm not 100% familiar, atm).

Add check phase for mkStandardOCI that checks for symlink integrity. Might be doable at the phase of writing /bin/entrypoint and before using nix2container to write .json manifest

That is a good idea, however it also seems to me that this python case could have maybe a better designed interface?

Just ideas, but why not e.g. mkStandardOCIWithHypercorn? For my taste # nix/ops/oci.nix could be an even shorter file. Would creating a library of tool-specific mkStandardOCI-interfaces with all the hard learned expriences already codified be a good idea?

I'm musing about this because currently, the operable's package is not even put into the PATH environmet, but I can see how different runtime enviroenments would like to see that package made available in their relevant environment, whether that be PATH or PYTHONPATH or anything else. It would be somewhat scoped to scripted languages.

[Pegasust]

Context on hypercorn + poetry2nix

The part that brings in (I guess $PYTHONPATH, or allow for python -m my.backend) is workspace_app.dependencyEnv, which is

  workspace_app = poetry2nix.mkPoetryApplication {
    projectDir = workspace_root;
  }.dependencyEnv;

and pyproject.toml that uses poetry. Poetry2nix takes care of taking these dependencies and either build them, or acts as Nix .whl broker via poetry.lock for preferWheels = true builds.

# minimal pyproject.toml
[tool.poetry]
name = "acme-monolith"

[tool.poetry.dependencies]
python = "^3.10"
databases = "^0.7"
hypercorn = "^0.14.3"
uvloop = "^0.17.0"

# This requires the path for `./my/backend.py:app` or `./my/backend/__init__.py:app`
# Though, out-of-the-box top-level package detection from `poetry` should cover most 
# cases, I'm putting this here for explicitness
[[tool.poetry.packages]]
include = "my"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

For context/archival purposes, .dependencyEnv sort of just calls inputs.nixpkgs.python.mkEnv, which calls inputs.nixpkgs.buildEnv or something similar then calls it a day, so the original package name is not passed through to .dependencyEnv.

I'm unsure which specific part injects onto PYTHONPATH, by inspecting the built artifacts, it looks like the python executable is wrapped

Framework-specific recipes

As @jboyens pointed out, this issue is unfortunately not specific to ${poetryApplication}.dependencyEnv, but is arose from some common patterns when it comes to interpreted languages, so this issue could probably be addressed via common pattern like mkStandardOCIFromInterpretedEnvironment, though feel free to suggest shorter names.

If there are some framework-specific nitpicks, I do think there's novelty in keeping mkStandardOCI having a low-level interface with some basic default configuration that covers 80%+ cases, with integrations having their own modules or transformed like (lib.dev.mkNixago lib.cfg.lefthook) into something like (mkStandardOCIWith oci.interpretedEnv), and add some recipe docs, linking to this issue. My only concern is at some point, our API change, and the recipe docs effectively points to some out-dated way to solve this

Other

# including this and nix and std tells me to remove either meta.tags or tag

This has fortunately just been fixed.

Thanks for this!

Also taking a look on #331