Monorepo for server-side components (#417)

* We use Pants (www.pantsbuild.org) as our build toolchain.
  - Details will follow in subsequent document updates.
* Add a custom setup generator plugin for Pants
  - It single-sources the version number from VERSION.
  - It takes the description, license, name from kwargs.
  - It validates the package name to start with "backend.ai-".
  - It takes the long-description from package-specific README.
* Add a custom platform-specific dependency selector for Pants
  - It provides `platform_resources()` target with a per-platform
    dependency map. (Thanks to Andreas Stenius)
* Move mypy/pytest configs to root `pyproject.toml`
  - flake8 does not support pyproject.toml yet (PyCQA/flake8#234),
    so keep it at `.flake8`
  - Explicitly add `setuptools` as requirement of flake8 because flake8
    uses `pkg_resources` to detect its own plugin entrypoints.
* Implement entrypoint scan with BUILD files for CLI
* Our new plugin subsystem (`ai.backend.plugin`) uses
  `importlib`-based entrypoints.
  - This removes dependency to setuptools at runtime.
* Update gitignore
* Add `./py`, `./backend.ai` shortcuts to run commands in exported venv
  - Finally implement truly unified CLI via `./backend.ai`!
* Rewrite GitHub Actions workflows
* Notable non-trivial Pants configs
  - Set `[GLOBAL].local_execution_root_dir` to a non-tmp directory
    because Snap-based Docker cannot access it!
  - `aiosqlite` is not explicitly imported but specified as a SQLAlchemy
    server URL scheme, so declare a manual dependency in manager tests.
  - Add `[pytest].execution_slot_var` config for test parallelization
* Mark storage-proxy tests as "integration" that require external dependency
  - Integration tests are skipped!
* Update test fixtures
  - Spawn a single-node etcd container with OS-assigned port numbers
  - Self-bootstrap db containers for isolated and parallel testing
* Reorganize packages
  - ai.backend.helpers -> backend.ai-kernel-helper
  - ai.backend.kernel -> backend.ai-kernel
  - ai.backend.runner -> backend.ai-kernel-binary
* Let it build packages for only Python 3.10
  - Without the upper bound, it tries to use Python 3.11-dev if available
    but aiohttp fails to build there. (aio-libs/aiohttp#6600)
* Rewrite scripts/install-dev and scripts/delete-dev
  - Change the container volume path for halfstack containers to "./volumes"
  - When Docker is installed via Snap, it must be 20.10.15 or later
    to have a working `docker compose` (v2) plugin with `sudo`.
  - Remove the auto-install routine but just show the guides
  - Now we support and use docker-compose v2 only
* Import `backend.ai-common` source (c864ccbe1)
* Import `backend.ai-agent` source (98aeeb98)
* Import `backend.ai-manager` source (85d16f0)
* Import `backend.ai-client-py` source (b6d03cc)
* Import `backend.ai-webserver` source (81506cc)
* Import `backend.ai-storage-proxy` source (8019533)
* Import `backend.ai-tester` source (ab85fab5c)

Co-authored-by: Andreas Stenius <git@astekk.se>

.editorconfig

@@ -7,7 +7,7 @@ trim_trailing_whitespace = true
 charset = utf-8
 
 [*.{py,md}]
-max_line_length = 85
+max_line_length = 105
 indent_style = space
 indent_size = 4
 
@@ -19,3 +19,10 @@ indent_size = 3
 [.travis.yml]
 indent_style = space
 indent_size = 2
+
+[BUILD]
+indent_style = space
+indent_size = 4
+
+[VERSION]
+insert_final_newline = false

.flake8

@@ -0,0 +1,8 @@
+[flake8]
+# ref: http://pep8.readthedocs.io/en/latest/intro.html#error-codes
+ignore = E126,E127,E128,E129,E722,E731,E221,E241,E401,W503,W504,N801,N802
+max-line-length = 125
+builtins = _
+exclude = .git,.cache,.idea,.egg,__pycache__,venv,dist,build,docs,src/ai/backend/manager/models/alembic/**,*.pyi
+
+# vim: ft=dosini

.gitattributes

@@ -0,0 +1,66 @@
+src/ai/backend/runner/scp.alpine3.8.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu20.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.alpine3.8.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.centos7.6.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.centos7.6.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu16.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/tmux.glibc.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbear.musl.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/jail.alpine3.8.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.alpine3.8.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearconvert.musl.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbear.glibc.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbear.musl.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.centos7.6.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.centos7.6.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu18.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu16.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/tmux.musl.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearkey.glibc.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearkey.glibc.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearkey.musl.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu16.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu16.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.alpine3.8.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu20.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu20.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearconvert.glibc.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbear.glibc.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu18.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu18.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.alpine3.8.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu20.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu18.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/jail.ubuntu16.04.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu18.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.centos7.6.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.centos7.6.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu18.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/tmux.glibc.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearconvert.glibc.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearkey.musl.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/tmux.musl.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/dropbearconvert.musl.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.alpine3.8.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu16.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/scp.ubuntu20.04.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/sftp-server.ubuntu20.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/su-exec.ubuntu16.04.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/krunner-extractor.img.aarch64.tar.xz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/krunner-extractor.img.x86_64.tar.xz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/backendai-socket-relay.img.x86_64.tar.gz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/linuxkit-nsenter.img.aarch64.tar.gz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/linuxkit-nsenter.img.x86_64.tar.gz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/backendai-socket-relay.img.aarch64.tar.gz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/linuxkit-metadata-proxy-worker.aarch64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/docker/linuxkit-metadata-proxy-worker.x86_64.bin filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/agent/kubernetes/backendai-socket-relay.img.tar.gz filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.alpine3.8.aarch64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.alpine3.8.x86_64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.centos7.6.aarch64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.centos7.6.x86_64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.ubuntu18.04.x86_64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.ubuntu20.04.aarch64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/libbaihook.ubuntu20.04.x86_64.so filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/roboto-italic.ttf filter=lfs diff=lfs merge=lfs -text
+src/ai/backend/runner/roboto.ttf filter=lfs diff=lfs merge=lfs -text

.github/workflows/default.yml

@@ -6,116 +6,229 @@ jobs:
   lint:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+    - uses: actions/cache@v2
+      id: cache
+      with:
+        # pants-specific cache
+        path: |
+          ~/.cache/pants/setup
+          ~/.cache/pants/lmdb_store
+          ~/.cache/pants/named_caches
+        key: ${{ runner.os }}-
     - name: Set up Python
       uses: actions/setup-python@v2
       with:
-        python-version: "3.9"
-    - name: Cache pip packages
-      uses: actions/cache@v2
-      with:
-        path: ~/.cache/pip
-        key: lint-flake8-${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('setup.cfg') }}
-        restore-keys: |
-          lint-flake8-${{ runner.os }}-pip-${{ matrix.python-version }}
-          lint-flake8-${{ runner.os }}-pip-
-    - name: Install dependencies
-      env:
-        REQUIREMENTS_FILE: lint
+        python-version: "3.10"
+        cache: pip
+    - name: Bootstrap Pants
       run: |
-        python -m pip install -U pip setuptools wheel
-        python -m pip install -U -r requirements/${REQUIREMENTS_FILE}.txt
-    - name: Lint with flake8
+        mkdir .tmp
+        ./pants --no-verify-config version
+    - name: Check BUILD files
+      run: ./pants tailor --check update-build-files --check
+    - name: Lint
       run: |
         if [ "$GITHUB_EVENT_NAME" == "pull_request" -a -n "$GITHUB_HEAD_REF" ]; then
           echo "(skipping matchers for pull request from local branches)"
         else
           echo "::add-matcher::.github/workflows/flake8-matcher.json"
         fi
-        python -m flake8 src/ai/backend tests
-  
+        if [ -n "$GITHUB_BASE_REF" ]; then
+          BASE_REF="origin/${GITHUB_BASE_REF}"
+          git fetch --no-tags --depth=1 origin "$GITHUB_BASE_REF"
+        else
+          BASE_REF="HEAD~1"
+        fi
+        ./pants lint --changed-since=$BASE_REF --changed-dependees=transitive
+    - name: Upload pants log
+      uses: actions/upload-artifact@v2
+      with:
+        name: pants.lint.log
+        path: .pants.d/pants.log
+      if: always()  # We want the log even on failures.
+
+
   typecheck:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+    - uses: actions/cache@v2
+      id: cache
+      with:
+        # pants-specific cache
+        path: |
+          ~/.cache/pants/setup
+          ~/.cache/pants/lmdb_store
+          ~/.cache/pants/named_caches
+        key: ${{ runner.os }}-
     - name: Set up Python
       uses: actions/setup-python@v2
       with:
-        python-version: "3.9"
-    - name: Cache pip packages
-      uses: actions/cache@v2
-      with:
-        path: ~/.cache/pip
-        key: typecheck-mypy-${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('setup.cfg') }}
-        restore-keys: |
-          typecheck-mypy-${{ runner.os }}-pip-${{ matrix.python-version }}
-          typecheck-mypy-${{ runner.os }}-pip-
-    - name: Install dependencies
-      env:
-        REQUIREMENTS_FILE: typecheck
+        python-version: "3.10"
+        cache: pip
+    - name: Bootstrap Pants
       run: |
-        python -m pip install -U pip setuptools wheel
-        python -m pip install -U -r requirements/${REQUIREMENTS_FILE}.txt
-    - name: Type check with mypy
+        mkdir .tmp
+        ./pants --no-verify-config version
+    - name: Check BUILD files
+      run: ./pants tailor --check update-build-files --check
+    - name: Typecheck
       run: |
         if [ "$GITHUB_EVENT_NAME" == "pull_request" -a -n "$GITHUB_HEAD_REF" ]; then
           echo "(skipping matchers for pull request from local branches)"
         else
           echo "::add-matcher::.github/workflows/mypy-matcher.json"
         fi
-        python -m mypy --no-color-output src/ai/backend
+        if [ -n "$GITHUB_BASE_REF" ]; then
+          BASE_REF="origin/${GITHUB_BASE_REF}"
+          git fetch --no-tags --depth=1 origin "$GITHUB_BASE_REF"
+        else
+          BASE_REF="HEAD~1"
+        fi
+        ./pants check --changed-since=$BASE_REF --changed-dependees=transitive
+    - name: Upload pants log
+      uses: actions/upload-artifact@v2
+      with:
+        name: pants.check.log
+        path: .pants.d/pants.log
+      if: always()  # We want the log even on failures.
 
   test:
     runs-on: ubuntu-latest
     steps:
-    - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
+    - uses: actions/checkout@v3
+      with:
+        fetch-depth: 2
+        submodules: true
+    - uses: actions/cache@v2
+      id: cache
       with:
-        python-version: "3.9"
-    - name: Cache pip packages
+        # pants-specific cache
+        path: |
+          ~/.cache/pants/setup
+          ~/.cache/pants/lmdb_store
+          ~/.cache/pants/named_caches
+        key: ${{ runner.os }}-
+    - name: Create LFS file hash list
+      run: git lfs ls-files -l | cut -d ' ' -f1 | sort > .lfs-assets-id
+    - name: Restore LFS cache
       uses: actions/cache@v2
+      id: lfs-cache
       with:
-        path: ~/.cache/pip
-        key: typecheck-mypy-${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('setup.cfg') }}
-    - name: Install dependencies
-      env:
-        REQUIREMENTS_FILE: test
+          path: .git/lfs
+          key: lfs-${{ hashFiles('.lfs-assets-id') }}
+    - name: Git LFS Pull
+      run: git lfs pull
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: "3.10"
+        cache: pip
+    - name: Bootstrap Pants
       run: |
-        python -m pip install -U pip setuptools wheel
-        python -m pip install -U -r requirements/${REQUIREMENTS_FILE}.txt
-    - name: Test with pytest
+        mkdir .tmp
+        ./pants --no-verify-config version
+    - name: Check BUILD files
+      run: ./pants tailor --check update-build-files --check
+    - name: Test
       run: |
-        python -m pytest tests
+        if [ -n "$GITHUB_BASE_REF" ]; then
+          BASE_REF="origin/${GITHUB_BASE_REF}"
+          git fetch --no-tags --depth=1 origin "$GITHUB_BASE_REF"
+        else
+          BASE_REF="HEAD~1"
+        fi
+        ./pants test --changed-since=$BASE_REF --changed-dependees=transitive
+    - name: Upload pants log
+      uses: actions/upload-artifact@v2
+      with:
+        name: pants.test.log
+        path: .pants.d/pants.log
+      if: always()  # We want the log even on failures.
 
   deploy-to-pypi:
     needs: [lint, typecheck, test]
     if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
     runs-on: ubuntu-latest
     steps:
     - uses: actions/checkout@v2
-    - name: Set up Python
-      uses: actions/setup-python@v2
       with:
-        python-version: "3.9"
-    - name: Cache pip packages
+        submodules: true
+    - uses: actions/cache@v2
+      id: cache
+      with:
+        # pants-specific cache
+        path: |
+          ~/.cache/pants/setup
+          ~/.cache/pants/lmdb_store
+          ~/.cache/pants/named_caches
+        key: ${{ runner.os }}-
+    - name: Create LFS file hash list
+      run: git lfs ls-files -l | cut -d ' ' -f1 | sort > .lfs-assets-id
+    - name: Restore LFS cache
       uses: actions/cache@v2
+      id: lfs-cache
       with:
-        path: ~/.cache/pip
-        key: test-pytest-${{ runner.os }}-pip-${{ matrix.python-version }}-${{ hashFiles('setup.py') }}
-        restore-keys: |
-          test-pytest-${{ runner.os }}-pip-${{ matrix.python-version }}
-          test-pytest-${{ runner.os }}-pip-
-    - name: Install dependencies
-      env:
-        REQUIREMENTS_FILE: build
+          path: .git/lfs
+          key: lfs-${{ hashFiles('.lfs-assets-id') }}
+    - name: Git LFS Pull
+      run: git lfs pull
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: "3.10"
+        cache: pip
+    - name: Bootstrap Pants
+      run: |
+        mkdir .tmp
+        ./pants --no-verify-config version
+    - name: Install local dependencies for packaging and publishing
+      run: |
+        pip install -U 'twine~=4.0' 'packaging>=21.3'
+    - name: Build packages
       run: |
-        python -m pip install -U pip setuptools wheel
-        python -m pip install -U -r requirements/${REQUIREMENTS_FILE}.txt
-    - name: Build and publish
+        # Normalize the package version
+        PKGVER=$(python -c "import packaging.version,pathlib; print(str(packaging.version.Version(pathlib.Path('VERSION').read_text())))")
+        # Build non-platform-specific wheels
+        ./pants --platform-specific-resources-target=linux_x86_64 --tag="wheel" --tag="-platform-specific" package '::'
+        # Build x86_64 wheels
+        MANYLINUX_PTAG=manylinux2014_x86_64
+        MACOS_PTAG=macosx_11_0_x86_64
+        ./pants --platform-specific-resources-target=linux_x86_64 --tag="wheel" --tag="+platform-specific" package '::'
+        for pkgname in "kernel_binary"; do
+          mv "dist/backend.ai_${pkgname}-${PKGVER}-py3-none-any.whl" \
+             "dist/backend.ai_${pkgname}-${PKGVER}-py3-none-${MANYLINUX_PTAG}.${MACOS_PTAG}.whl"
+        done
+        # Build arm64 wheels
+        MANYLINUX_PTAG=manylinux2014_aarch64
+        MACOS_PTAG=macosx_11_0_arm64
+        ./pants --platform-specific-resources-target=linux_arm64 --tag="wheel" --tag="+platform-specific" package '::'
+        for pkgname in "kernel_binary"; do
+          mv "dist/backend.ai_${pkgname}-${PKGVER}-py3-none-any.whl" \
+             "dist/backend.ai_${pkgname}-${PKGVER}-py3-none-${MANYLINUX_PTAG}.${MACOS_PTAG}.whl"
+        done
+        ls -lh dist
+    - name: Publish
       env:
         TWINE_USERNAME: ${{ secrets.PYPI_USERNAME }}
         TWINE_PASSWORD: ${{ secrets.PYPI_PASSWORD }}
+      # We don't use `./pants publish ::` because we manually rename the
+      # wheels after buildling them to add arch-specific tags.
       run: |
-        python setup.py sdist bdist_wheel
-        twine upload dist/*
+        twine upload dist/*.whl
+    - name: Upload artifacts
+      uses: actions/upload-artifact@v2
+      with:
+        name: dist
+        path: dist
+    - name: Upload pants log
+      uses: actions/upload-artifact@v2
+      with:
+        name: pants.deploy.log
+        path: .pants.d/pants.log
+      if: always()  # We want the log even on failures.