diff --git a/docs/source/concepts/bento.rst b/docs/source/concepts/bento.rst index 583ffa4e0d5..343adc5c623 100644 --- a/docs/source/concepts/bento.rst +++ b/docs/source/concepts/bento.rst @@ -9,10 +9,10 @@ What is a Bento? code, models, data files and dependency configurations required for running a user-defined :ref:`reference/core:bentoml.Service`, packaged into a standardized format. -While :code:`bentoml.Service` standardizes the inference API definition, including the +While ``bentoml.Service`` standardizes the inference API definition, including the serving logic, runners initialization and API input, output types. -:code:`Bento` standardizes how to reproduce the required environment for running a -:code:`bentoml.Service` in production. +``Bento`` standardizes how to reproduce the required environment for running a +``bentoml.Service`` in production. .. note:: "Bento Build" is essentially the build process in traditional software development, @@ -26,10 +26,10 @@ The Build Command ----------------- A Bento can be created with the :ref:`bentoml build ` CLI command -with a :code:`bentofile.yaml` build file. Here's an example from the +with a ``bentofile.yaml`` build file. Here's an example from the :doc:`tutorial `: -.. code:: yaml +.. code-block:: yaml service: "service:svc" # Same as the argument passed to `bentoml serve` labels: @@ -42,22 +42,22 @@ with a :code:`bentofile.yaml` build file. Here's an example from the - scikit-learn - pandas -.. code:: bash +.. code-block:: bash - > bentoml build + » bentoml build - INFO [cli] Building BentoML service "iris_classifier:dpijemevl6nlhlg6" from build context "/home/user/gallery/quickstart" - INFO [cli] Packing model "iris_clf:7drxqvwsu6zq5uqj" from "/home/user/bentoml/models/iris_clf/7drxqvwsu6zq5uqj" - INFO [cli] Locking PyPI package versions.. - INFO [cli] - ██████╗░███████╗███╗░░██╗████████╗░█████╗░███╗░░░███╗██╗░░░░░ - ██╔══██╗██╔════╝████╗░██║╚══██╔══╝██╔══██╗████╗░████║██║░░░░░ - ██████╦╝█████╗░░██╔██╗██║░░░██║░░░██║░░██║██╔████╔██║██║░░░░░ - ██╔══██╗██╔══╝░░██║╚████║░░░██║░░░██║░░██║██║╚██╔╝██║██║░░░░░ - ██████╦╝███████╗██║░╚███║░░░██║░░░╚█████╔╝██║░╚═╝░██║███████╗ - ╚═════╝░╚══════╝╚═╝░░╚══╝░░░╚═╝░░░░╚════╝░╚═╝░░░░░╚═╝╚══════╝ + Building BentoML service "iris_classifier:dpijemevl6nlhlg6" from build context "/home/user/gallery/quickstart" + Packing model "iris_clf:zy3dfgxzqkjrlgxi" + Locking PyPI package versions.. + + ██████╗░███████╗███╗░░██╗████████╗░█████╗░███╗░░░███╗██╗░░░░░ + ██╔══██╗██╔════╝████╗░██║╚══██╔══╝██╔══██╗████╗░████║██║░░░░░ + ██████╦╝█████╗░░██╔██╗██║░░░██║░░░██║░░██║██╔████╔██║██║░░░░░ + ██╔══██╗██╔══╝░░██║╚████║░░░██║░░░██║░░██║██║╚██╔╝██║██║░░░░░ + ██████╦╝███████╗██║░╚███║░░░██║░░░╚█████╔╝██║░╚═╝░██║███████╗ + ╚═════╝░╚══════╝╚═╝░░╚══╝░░░╚═╝░░░░╚════╝░╚═╝░░░░░╚═╝╚══════╝ - INFO [cli] Successfully built Bento(tag="iris_classifier:dpijemevl6nlhlg6") at "~/bentoml/bentos/iris_classifier/dpijemevl6nlhlg6/" + Successfully built Bento(tag="iris_classifier:dpijemevl6nlhlg6") Similar to :doc:`saving a model `, a unique version tag will be automatically generated for the newly created Bento. @@ -66,13 +66,13 @@ It is also possible to customize the Bento version string by specifying it in th :code:`--version` CLI argument. However this is generally not recommended. Only use it if your team has a very specific naming convention for deployable artifacts, e.g.: -.. code:: bash +.. code-block:: bash - bentoml build --version 1.0.1 + » bentoml build --version 1.0.1 .. note:: - The Bento build process requires importing the :code:`bentoml.Service` object + The Bento build process requires importing the ``bentoml.Service`` object defined. This means, the build environment must have all its dependencies installed. Support for building from a docker environment is on the roadmap, see :issue:`2495`. @@ -81,29 +81,29 @@ Advanced Project Structure For projects that are part of a larger codebase and interacts with other local python modules; Or for projects containing multiple Bentos/Services, it may not be possible to -put all service definition code and :code:`bentofile.yaml` under the project's root +put all service definition code and ``bentofile.yaml`` under the project's root directory. BentoML allows placing the service definition file and bentofile anywhere in the project -directory. In this case, the user needs to provide the :code:`build_ctx` and -:code:`bentofile` argument to the :code:`bentoml build` CLI command. +directory. In this case, the user needs to provide the ``build_ctx`` and +``bentofile`` argument to the ``bentoml build`` CLI command. build_ctx Build context is your Python project's working directory. This is from where you start the Python interpreter during development so that your local python modules can be imported properly. Default to current directory where the - :code:`bentoml build` takes place. + ``bentoml build`` takes place. bentofile - :code:`bentofile` is a :code:`.yaml` file that specifies the - :ref:`concepts/bento:Bento Build Options`. Default to the :code:`bentofile.yaml` + ``bentofile`` is a ``.yaml`` file that specifies the + :ref:`concepts/bento:Bento Build Options`. Default to the ``bentofile.yaml`` file under the build context. They can also be customized via the CLI command, e.g.: -.. code:: bash +.. code-block:: bash - bentoml build -f ./src/my_project_a/bento_fraud_detect.yaml ./src/ + » bentoml build -f ./src/my_project_a/bento_fraud_detect.yaml ./src/ Managing Bentos @@ -122,9 +122,9 @@ Similar to Models, Bentos built locally can be managed via the .. tab-item:: List - .. code:: bash + .. code-block:: bash - > bentoml list + » bentoml list Tag Size Creation Time Path iris_classifier:nvjtj7wwfgsafuqj 16.99 KiB 2022-05-17 21:36:36 ~/bentoml/bentos/iris_classifier/nvjtj7wwfgsafuqj @@ -132,9 +132,9 @@ Similar to Models, Bentos built locally can be managed via the .. tab-item:: Get - .. code:: bash + .. code-block:: bash - > bentoml get iris_classifier:latest + » bentoml get iris_classifier:latest service: service:svc name: iris_classifier @@ -164,11 +164,11 @@ Similar to Models, Bentos built locally can be managed via the .. tab-item:: Delete - .. code:: bash + .. code-block:: bash - > bentoml delete iris_classifier:latest -y + » bentoml delete iris_classifier:latest -y - INFO [cli] Bento(tag="iris_classifier:nvjtj7wwfgsafuqj") deleted + Bento(tag="iris_classifier:nvjtj7wwfgsafuqj") deleted Import and Export @@ -209,20 +209,20 @@ that comes with flexible APIs and Web UI for managing all Bentos created by your It can be configured to store Bento files on cloud blob storage such as AWS S3, MinIO or GCS, and automatically build docker images when a new Bento was pushed. -.. code:: bash +.. code-block:: bash - > bentoml push iris_classifier:latest + » bentoml push iris_classifier:latest Successfully pushed Bento "iris_classifier:nvjtj7wwfgsafuqj" -.. code:: bash +.. code-block:: bash - > bentoml pull iris_classifier:nvjtj7wwfgsafuqj + » bentoml pull iris_classifier:nvjtj7wwfgsafuqj Successfully pulled Bento "iris_classifier:nvjtj7wwfgsafuqj" .. image:: /_static/img/yatai-bento-repos.png - :alt: Yatai Bento Repo UI + :alt: Yatai Bento Repo UI Bento Management API @@ -235,7 +235,7 @@ provided for managing Bentos: .. tab-item:: Get - .. code:: python + .. code-block:: python import bentoml bento = bentoml.get("iris_classifier:latest") @@ -246,19 +246,19 @@ provided for managing Bentos: .. tab-item:: List - .. code:: python + .. code-block:: python import bentoml bentos = bentoml.list() .. tab-item:: Import / Export - .. code:: python + .. code-block:: python import bentoml bentoml.export_bento('my_bento:latest', '/path/to/folder/my_bento.bento') - .. code:: bash + .. code-block:: bash bentoml.import_bento('/path/to/folder/my_bento.bento') @@ -274,18 +274,18 @@ provided for managing Bentos: created by your team, stores Bento files on cloud blob storage such as AWS S3, MinIO or GCS, and automatically builds docker images when a new Bento was pushed. - .. code:: bash + .. code-block:: bash import bentoml bentoml.push("iris_classifier:nvjtj7wwfgsafuqj") - .. code:: bash + .. code-block:: bash bentoml.pull("iris_classifier:nvjtj7wwfgsafuqj") .. tab-item:: Delete - .. code:: bash + .. code-block:: bash import bentoml bentoml.delete("iris_classifier:nvjtj7wwfgsafuqj") @@ -295,13 +295,13 @@ What's inside a Bento ^^^^^^^^^^^^^^^^^^^^^ It is possible to view the generated files in a specific Bento. Simply use the -:code:`-o/--output` option of the :code:`bentoml get` command to find the file path to +:code:`-o/--output` option of the ``bentoml get`` command to find the file path to the Bento archive directory. -.. code:: bash +.. code-block:: bash - > cd $(bentoml get iris_classifier:latest -o path) - > tree + » cd $(bentoml get iris_classifier:latest -o path) + » tree . ├── README.md ├── apis @@ -327,23 +327,31 @@ the Bento archive directory. └── train.py -The :code:`src/` directory here contains files specified under the -:ref:`include ` field in the :code:`bentofile.yaml`, -it will be set as user Python code's CWD(current working directory), so that importing -local module and reading relative file path work properly. +* ``src`` directory contains files specified under the :ref:`include ` field in the ``bentofile.yaml``. These + files are relative to user Python code's CWD (current working directory), which makes + importing relative modules and file path inside user code possible. -:code:`models/` directory contains all models required by the Service. This is -automatically determined from the :code:`bentoml.Service` object's runners list. +* ``models`` directory contains all models required by the Service. This is automatically determined from the ``bentoml.Service`` object's runners list. -:bdg-warning:`Warning:` users **should never** change files in the generated Bento -archive, unless it's for debugging purpose. +* ``apis`` directory contains all API definitions. This directory contains API specs + that are generated from the ``bentoml.Service`` object's API definitions. + +* ``env`` directory contains all environment-related files which will help boostrap the Bento 🍱. This directory contains files that are generated + from :ref:`concepts/bento:Bento Build Options` that is specified under ``bentofile.yaml``. + +.. note:: + + :bdg-warning:`Warning:` users **should never** change files in the generated Bento + archive, unless it's for debugging purpose. Bento Build Options ------------------- -Build options are specified in a :code:`.yaml` file, which customizes the final Bento -produced. By convention, this file is named :code:`bentofile.yaml`. +Build options are specified in a ``.yaml`` file, which customizes the final Bento +produced. + +By convention, this file is named ``bentofile.yaml``. In this section, we will go over all the build options, including defining dependencies, configuring files to include, and customize docker image settings. @@ -351,96 +359,134 @@ dependencies, configuring files to include, and customize docker image settings. Service ^^^^^^^ -The :code:`service` field is a required which specifies where the -:code:`bentoml.Service` object is defined. In the tutorial example, we have -:code:`service: "service:svc"`, which means: +``service`` is a **required** field which specifies where the +``bentoml.Service`` object is defined. -- :code:`service` refers to the python module (the :code:`service.py` file) -- :code:`svc` refers to the object created in :code:`service.py`, with :code:`svc = bentoml.Service(...)` +In the :doc:`tutorial `, we defined ``service: "service:svc"``, which can be +interpreted as: -This is exact same as how the :ref:`bentoml serve ` command -specifies a :code:`bentoml.Service` target. +- ``service`` refers to the Python module (the ``service.py`` file) +- ``svc`` refers to the ``bentoml.Service`` object created in ``service.py``, with ``svc = bentoml.Service(...)`` + +.. tip:: + + This is synonymous to how the :ref:`bentoml serve ` command specifies a ``bentoml.Service`` target. + + .. code-block:: zsh + + ┌──────────────┐ + ┌────────────────┤bentofile.yaml│ + │ └───────────┬──┘ + │ │ + │ service: "service:svc" │ + │ ─┬─ │ + │ │ │ + └─────────────────┼──────────┘ + │ + │ + │ ┌────┐ + ┌─────────────────────┼────┤bash│ + │ │ └──┬─┘ + │ ▼ │ + │ » bentoml serve service:svc │ + │ │ + │ │ + └─────────────────────────────┘ Description ^^^^^^^^^^^ -The :code:`description` filed allow user to provide custom documentation for a Bento. -The description contents must be plain text, optionally in the -`Markdown `_ format. Description -can be specified either inline in the :code:`bentofile.yaml`, or via a file path to an +``description`` field allows user to customize documentation for any given Bento. + +The description contents must be plain text, optionally in `Markdown `_ format. Description +can be specified either inline in the ``bentofile.yaml``, or via a file path to an existing text file: -.. code:: yaml +.. tab-set:: + + .. tab-item:: Inline + + .. code-block:: yaml + + service: "service.py:svc" + description: | + ## Description For My Bento 🍱 - service: "service.py:svc" - description: | - ## Description For My Bento + Use **any markdown syntax** here! - Use **any markdown syntax** here! + > BentoML is awesome! + include: + ... - > BentoML is awesome! - includes: - ... + .. tab-item:: File path -.. code:: yaml + .. code-block:: yaml - service: "service.py:svc" - description: "file: ./README.md" - includes: - ... + service: "service.py:svc" + description: "file: ./README.md" + include: + ... .. tip:: When pointing to a description file, it can be either an absolute path or a relative - path. The file must exist on the given path upon :code:`bentoml build` command run, - and for relative file path, the current path is set to the :code:`build_ctx`, which - default to the directory where :code:`bentoml build` was executed from. + path. The file must exist on the given path upon ``bentoml build`` command run, + and for relative file path, the current path is set to the ``build_ctx``, which + default to the directory where ``bentoml build`` was executed from. Labels ^^^^^^ -:code:`Labels` are key value pairs that are attached to an object. In BentoML, both -:code:`Bento` and :code:`Model` can have labels attached to them. Labels are intended to +``labels`` are key-value pairs that are attached to an object. + +In BentoML, both ``Bento`` and ``Model`` can have labels attached to them. Labels are intended to be used to specify identifying attributes of Bentos/Models that are meaningful and relevant to users, but do not directly imply semantics to the rest of the system. Labels can be used to organize models and Bentos in `Yatai `_, which also allow users to add or modify labels at any time. +.. code-block:: yaml + + labels: + owner: bentoml-team + stage: not-ready Files to include ^^^^^^^^^^^^^^^^ -In the example above, the :code:`*.py` is including every Python file from the -:code:`build_ctx`. You can also include other wildcard and directory matching. +In the example :ref:`above `, the :code:`*.py` includes every Python files under ``build_ctx``. +You can also include other wildcard and directory pattern matching. -.. code:: yaml +.. code-block:: yaml ... include: - - "data/" - - "**/*.py" - - "config/*.json" - - "path/to/a/file.csv" + - "data/" + - "**/*.py" + - "config/*.json" + - "path/to/a/file.csv" + +If the include field is not specified, BentoML will include all files under the ``build_ctx`` directory, besides the ones explicitly set to be excluded, as will be demonstrated in :ref:`concepts/bento:Files to exclude`. + +.. seealso:: -If the include field is not specified, BentoML will include all files under the -:code:`build_ctx` directory, besides the ones explicitly set to be excluded, as shown in -the section below. + Both ``include`` and ``exclude`` fields support `gitignore style pattern + matching. `_. Files to exclude ^^^^^^^^^^^^^^^^ -If the user needs to include a lot of files under a directory, another approach is to +If there are a lot of files under the working directory, another approach is to only specify which files to be ignored. -The :code:`exclude` field specifies the pathspecs (similar to the :code:`.gitignore` -files) of files to be excluded in the final Bento build. The pathspecs are relative to -the :code:`build_ctx` directory. +``exclude`` field specifies the pathspecs (similar to ``.gitignore`` files) of files to be excluded in the final Bento build. The pathspecs are relative to +the ``build_ctx`` directory. -.. code:: yaml +.. code-block:: yaml ... include: @@ -450,10 +496,11 @@ the :code:`build_ctx` directory. - "tests/" - "secrets.key" -Users can also opt to place a :code:`.bentoignore` file in the :code:`build_ctx` -directory. This is what a :code:`.bentoignore` file would look like: +Users can also opt to place a ``.bentoignore`` file in the ``build_ctx`` +directory. This is what a ``.bentoignore`` file would look like: -.. code:: bash +.. code-block:: bash + :caption: .bentoignore __pycache__/ *.py[cod] @@ -462,18 +509,20 @@ directory. This is what a :code:`.bentoignore` file would look like: training_data/ .. note:: - :code:`exclude` is always applied after :code:`include`. + + ``exclude`` is always applied after ``include``. Python Packages ^^^^^^^^^^^^^^^ -Required Python packages can be specified under the :code:`python.packages` field. When -a package name is left without a version, BentoML will by default lock the version to -the current environment when running :code:`bentoml build`. User can also specify the -desired version, install from a custom PyPI source, or install from a github repo: +Required Python packages for a given Bento can be specified under the ``python.packages`` field. + +When a package name is left without a version, BentoML will lock the package to the +version available under the current environment when running ``bentoml build``. User can also specify the +desired version, install from a custom PyPI source, or install from a GitHub repo: -.. code:: yaml +.. code-block:: yaml python: packages: @@ -492,21 +541,22 @@ desired version, install from a custom PyPI source, or install from a github rep If you already have a `requirements.txt `_ file that defines python packages for your project, you may also supply a path to the -:code:`requirements.txt` file directly: +``requirements.txt`` file directly: -.. code:: yaml +.. code-block:: yaml python: requirements_txt: "./project-a/ml-requirements.txt" -Pip install options +Pip Install Options """"""""""""""""""" -Additional :code:`pip install` arguments can also be provided, as shown below. Note that -these arguments will be applied to all packages defined in the :code:`packages` list, as -well as the :code:`requirements_txt` file if provided. +Additional ``pip install`` arguments can also be provided. -.. code:: yaml +Note that these arguments will be applied to all packages defined in ``python.packages``, as +well as the ``requirements_txt`` file, if provided. + +.. code-block:: yaml python: requirements_txt: "./requirements.txt" @@ -523,15 +573,16 @@ well as the :code:`requirements_txt` file if provided. pip_args: "--pre -U --force-reinstall" .. note:: + **BentoML by default will cache pip artifacts across all local image builds to speed - up the build process**. If you want to force a re-download instead of using the cache, - you can specify the :code:`pip_args: "--no-cache-dir"` option in your - :code:`bentofile.yaml`, or use the :code:`--no-cache` option in - :code:`bentoml containerize` command, e.g.: + up the build process**. - .. code:: + If you want to force a re-download instead of using the cache, you can specify the :code:`pip_args: "--no-cache-dir"` option in your + ``bentofile.yaml``, or use the :code:`--no-cache` option in ``bentoml containerize`` command, e.g.: - bentoml containerize my_bento:latest --no-cache + .. code-block:: + + » bentoml containerize my_bento:latest --no-cache PyPI Package Locking @@ -543,23 +594,23 @@ generates a :code:`requirements.lock.txt` file. This process uses `pip-compile `_ under the hood. If you have already specified a version for all packages, you can optionally disable -this behavior by setting the :code:`lock_packages` field to False: +this behavior by setting the ``lock_packages`` field to False: -.. code:: yaml +.. code-block:: yaml python: requirements_txt: "requirements.txt" - lock_packages: False + lock_packages: false Python Wheels """"""""""""" -Python :code:`.whl` files are also supported as a type of dependency to include in a -Bento. Simply provide a path to your :code:`.whl` files under the :code:`wheels`` field. +Python ``.whl`` files are also supported as a type of dependency to include in a +Bento. Simply provide a path to your ``.whl`` files under the ``wheels``` field. -.. code:: yaml +.. code-block:: yaml python: wheels: @@ -581,15 +632,15 @@ Python Options Table +-------------------+------------------------------------------------------------------------------------+ | lock_packages | Whether to lock the packages or not | +-------------------+------------------------------------------------------------------------------------+ -| index_url | Inputs for the `--index-url` pip argument | +| index_url | Inputs for the ``--index-url`` pip argument | +-------------------+------------------------------------------------------------------------------------+ -| no_index | Whether to include the `--no-index` pip argument | +| no_index | Whether to include the ``--no-index`` pip argument | +-------------------+------------------------------------------------------------------------------------+ -| trusted_host | List of trusted hosts used as inputs using the `--trusted-host` pip argument | +| trusted_host | List of trusted hosts used as inputs using the ``--trusted-host`` pip argument | +-------------------+------------------------------------------------------------------------------------+ -| find_links | List of links to find as inputs using the `--find-links` pip argument | +| find_links | List of links to find as inputs using the ``--find-links`` pip argument | +-------------------+------------------------------------------------------------------------------------+ -| extra_index_url | List of extra index urls as inputs using the `≈` pip argument | +| extra_index_url | List of extra index urls as inputs using the ``≈`` pip argument | +-------------------+------------------------------------------------------------------------------------+ | pip_args | Any additional pip arguments that you would like to add when installing a package | +-------------------+------------------------------------------------------------------------------------+ @@ -600,11 +651,10 @@ Python Options Table Conda Options ^^^^^^^^^^^^^ -Conda dependencies can be specified under the :code:`conda` field. For example: +Conda dependencies can be specified under ``conda`` field. For example: -.. code:: yaml +.. code-block:: yaml - ... conda: channels: - default @@ -613,35 +663,37 @@ Conda dependencies can be specified under the :code:`conda` field. For example: pip: - "scikit-learn==1.2.0" -When :code:`channels` filed is left unspecified, BentoML will use the community -maintained :code:`conda-forge` channel as the default. +When ``channels`` filed is left unspecified, BentoML will use the community +maintained ``conda-forge`` channel as the default. Optionally, you can export all dependencies from a preexisting conda environment to -an :code:`environment.yml` file, and provide this file in your :code:`bentofile.yaml` +an ``environment.yml`` file, and provide this file in your ``bentofile.yaml`` config: Export conda environment: -.. code:: bash +.. code-block:: bash - conda env export > environment.yml + » conda env export > environment.yml -In your :code:`bentofile.yaml`: +In your ``bentofile.yaml``: -.. code:: yaml +.. code-block:: yaml conda: environment_yml: "./environment.yml" .. note:: + Unlike Python packages, BentoML does not support locking conda packages versions automatically. It is recommended for users to specify a version in the config file. .. seealso:: - When :code:`conda` options are provided, BentoML will select a docker base image + + When ``conda`` options are provided, BentoML will select a docker base image that comes with Miniconda pre-installed in the generated Dockerfile. Note that only - the :code:`debian` and :code:`alpine` distro support :code:`conda`. Learn more at + the ``debian`` and ``alpine`` distro support ``conda``. Learn more at the :ref:`concepts/bento:Docker Options` section below. @@ -653,11 +705,11 @@ Conda Options Table +==================+==================================================================================================================================+ | environment_yml | Path to a conda environment file to copy into the bento. If specified, this file will overwrite any additional option specified | +------------------+----------------------------------------------------------------------------------------------------------------------------------+ -| channels | Custom conda channels to use. If not specified will use "defaults" | +| channels | Custom conda channels to use. If not specified will use ``conda-forge`` | +------------------+----------------------------------------------------------------------------------------------------------------------------------+ | dependencies | Custom conda dependencies to include in the environment | +------------------+----------------------------------------------------------------------------------------------------------------------------------+ -| pip | The specific "pip" conda dependencies to include | +| pip | The specific ``pip`` conda dependencies to include | +------------------+----------------------------------------------------------------------------------------------------------------------------------+ @@ -669,19 +721,19 @@ available options for customizing the docker image generated from a Bento. Here's a basic Docker options configuration: -.. code:: yaml +.. code-block:: yaml docker: distro: debian python_version: "3.8.12" cuda_version: "11.6.2" system_packages: - - libblas-dev - - liblapack-dev - - gfortran + - libblas-dev + - liblapack-dev + - gfortran env: - - FOO=value1 - - BAR=value2 + - FOO=value1 + - BAR=value2 .. note:: @@ -699,10 +751,10 @@ OS Distros The following OS distros are currently supported in BentoML: -- :code:`debian`: **default**, similar to Ubuntu -- :code:`alpine`: A minimal Docker image based on Alpine Linux -- :code:`ubi8`: Red Hat Universal Base Image -- :code:`amazonlinux`: Amazon Linux 2 +- ``debian``: **default**, similar to Ubuntu +- ``alpine``: A minimal Docker image based on Alpine Linux +- ``ubi8``: Red Hat Universal Base Image +- ``amazonlinux``: Amazon Linux 2 Some of the distros may not support using conda or specifying CUDA for GPU. Here is the support matrix for all distros: @@ -726,20 +778,23 @@ support matrix for all distros: GPU support """"""""""" -The :code:`cuda_version` field specifies the target cuda version to -install on the generated docker image. Currently, the only supported -cuda versions are :code:`"11.6.2"`, :code:`"11.4.3"` and -:code:`"11.2.2"`. BentoML also installs additional packages required -for the target cuda version. +The ``cuda_version`` field specifies the target CUDA version to install on the +the generated docker image. Currently, the following CUDA version are supported: -.. code:: yaml +* ``"11.6.2"`` +* ``"11.4.3"`` +* ``"11.2.2"`` + +BentoML will also install additional packages required for given target CUDA version. + +.. code-block:: yaml docker: cuda_version: "11.6.2" If you need a different cuda version that is not currently supported in BentoML, it is -possible to install it by specifying it in the :code:`system_packages` or via the -:code:`setup_script`. +possible to install it by specifying it in the ``system_packages`` or via the +``setup_script``. .. dropdown:: Installing custom CUDA version with conda :icon: code @@ -747,7 +802,7 @@ possible to install it by specifying it in the :code:`system_packages` or via th We will demonstrate how you can install custom cuda version via conda. - Add the following to your :code:`bentofile.yaml`: + Add the following to your ``bentofile.yaml``: .. code-block:: yaml @@ -766,36 +821,36 @@ possible to install it by specifying it in the :code:`system_packages` or via th - numba=0.48 - pandas=1.0 - Then proceed with :code:`bentoml build` and :code:`bentoml containerize` respectively: + Then proceed with ``bentoml build`` and ``bentoml containerize`` respectively: .. code-block:: bash - bentoml build + » bentoml build - bentoml containerize : + » bentoml containerize : Setup Script """""""""""" -For advanced Docker customization, you can also use a :code:`setup_script` to inject +For advanced Docker customization, you can also use a ``setup_script`` to inject arbitrary user provided script during the image build process. For example, with NLP projects you can pre-download NLTK data in the image with: -In your :code:`bentofile.yaml`: +In your ``bentofile.yaml``: -.. code:: yaml +.. code-block:: yaml ... python: - packages: - - nltk + packages: + - nltk docker: - setup_script: "./setup.sh" + setup_script: "./setup.sh" -In the :code:`setup.sh` file: +In the ``setup.sh`` file: -.. code:: bash +.. code-block:: bash #!/bin/bash set -euxo pipefail @@ -809,28 +864,28 @@ pre-downloaded NLTK dataset. .. tip:: - When working with bash scripts, it is recommended to add :code:`set -euxo pipefail` + When working with bash scripts, it is recommended to add ``set -euxo pipefail`` to the beginning. Especially when `set -e` is missing, the script will fail silently - without raising an exception during :code:`bentoml containerize`. Learn more about + without raising an exception during ``bentoml containerize``. Learn more about `Bash Set builtin `_. It is also possible to provide a Python script for initializing the docker image. Here's an example: -In :code:`bentofile.yaml`: +In ``bentofile.yaml``: -.. code:: yaml +.. code-block:: yaml ... python: - packages: - - nltk + packages: + - nltk docker: - setup_script: "./setup.py" + setup_script: "./setup.py" -In the :code:`setup.py` file: +In the ``setup.py`` file: -.. code:: python +.. code-block:: python #!/usr/bin/env python @@ -841,7 +896,7 @@ In the :code:`setup.py` file: .. note:: - Pay attention to :code:`#!/bin/bash` and :code:`#!/usr/bin/env python` in the + Pay attention to ``#!/bin/bash`` and ``#!/usr/bin/env python`` in the first line of the example scripts above. They are known as `Shebang `_ and they are required in a setup script provided to BentoML. @@ -864,7 +919,7 @@ Docker Options Table +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Description | +=====================+===========================================================================================================================================+ -| distro | The OS distribution on the Docker image, Default to :code:`debian`. | +| distro | The OS distribution on the Docker image, Default to ``debian``. | +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | python_version | Specify which python to include on the Docker image [`3.7`, `3.8`, `3.9`, `3.10`]. Default to the Python version in build environment. | +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ @@ -876,5 +931,5 @@ Docker Options Table +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ | base_image | A user-provided docker base image. This will override all other custom attributes of the image. | +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ -| dockerfile_template | Customize the generated dockerfile by providing a jinja2 template that extends the default dockerfile. | +| dockerfile_template | Customize the generated dockerfile by providing a Jinja2 template that extends the default dockerfile. | +---------------------+-------------------------------------------------------------------------------------------------------------------------------------------+ diff --git a/docs/source/concepts/service.rst b/docs/source/concepts/service.rst index afbb14951c8..f1ba8e8b83a 100644 --- a/docs/source/concepts/service.rst +++ b/docs/source/concepts/service.rst @@ -12,9 +12,9 @@ Creating a Service ------------------ A BentoML service is composed of Runners and APIs. Consider the following service -definition from the :doc:`tutorial `: +definition from the :doc:`tutorial `: -.. code:: python +.. code-block:: python import numpy as np import bentoml @@ -30,10 +30,10 @@ definition from the :doc:`tutorial `: return result -Services are initialized through `bentoml.Service()` call, with the service name and a +Services are initialized through ``bentoml.Service()`` call, with the service name and a list of :doc:`Runners ` required in the service: -.. code:: python +.. code-block:: python # Create the iris_classifier_service with the ScikitLearn runner svc = bentoml.Service("iris_classifier", runners=[iris_clf_runner]) @@ -41,10 +41,10 @@ list of :doc:`Runners ` required in the service: .. note:: The service name will become the name of the Bento. -The :code:`svc` object created provides a decorator method :code:`svc.api`for defining +The ``svc`` object created provides a decorator method ``svc.api`` for defining` APIs in this service: -.. code:: python +.. code-block:: python @svc.api(input=NumpyNdarray(), output=NumpyNdarray()) def classify(input_series: np.ndarray) -> np.ndarray: @@ -60,7 +60,7 @@ throughput and resource utilization. BentoML provides a convenient way of creating Runner instance from a saved model: -.. code:: python +.. code-block:: python runner = bentoml.sklearn.get("iris_clf:latest").to_runner() @@ -103,7 +103,7 @@ Service APIs Inference APIs define how the service functionality can be called remotely. A service can have one or more APIs. An API consists of its input/output specs and a callback function: -.. code:: python +.. code-block:: python # Create new API and add it to "svc" @svc.api(input=NumpyNdarray(), output=NumpyNdarray()) # define IO spec @@ -114,20 +114,19 @@ have one or more APIs. An API consists of its input/output specs and a callback # Define post-processing logic return result -By decorating a function with :code:`@svc.api`, we declare that the function shall be +By decorating a function with ``@svc.api`, we declare that the function shall be` invoked when this API is called. The API function is a great place for defining your serving logic, such as feature fetching, pre and post processing, and model inferences via Runners. -When running :code:`bentoml serve` with the example above, this API function is -transformed into an HTTP endpoint, :code:`/predict`, that takes in a ``np.ndarray`` as +When running ``bentoml serve`` with the example above, this API function is +transformed into an HTTP endpoint, ``/predict``, that takes in a ``np.ndarray`` as input, and returns a ``np.ndarray`` as output. The endpoint can be called with the following ``curl`` command: -.. code:: bash +.. code-block:: bash - > curl \ - -X POST \ + » curl -X POST \ -H "content-type: application/json" \ --data "[[5.9, 3, 5.1, 1.8]]" \ http://127.0.0.1:3000/predict @@ -142,9 +141,9 @@ Route ^^^^^ By default, the function name becomes the endpoint URL. Users can also customize -this URL via the :code:`route` option, e.g.: +this URL via the ``route`` option, e.g.: -.. code:: python +.. code-block:: python @svc.api( input=NumpyNdarray(), @@ -167,7 +166,7 @@ argument added to the service API function. Both the request and response contex accessed through the inference context for getting and setting the headers, cookies, and status codes. -.. code:: python +.. code-block:: python @svc.api( input=NumpyNdarray(), @@ -195,7 +194,7 @@ status codes. ) ] ctx.response.headers.append("X-Custom-Header", "value") - + return result @@ -205,12 +204,12 @@ IO Descriptors IO descriptors are used for defining an API's input and output specifications. It describes the expected data type, helps validate that the input and output conform to the expected format and schema and convert them from and to the native types. They are -specified through the :code:`input` and :code:`output` arguments in the :code:`@svc.api` +specified through the ``input`` and ``output`` arguments in the ``@svc.api`` decorator method. -Recall the API we created in the :doc:`tutorial`. The classify API both accepts +Recall the API we created in the :doc:`tutorial `. The ``classify`` API both accepts arguments and returns results in the type of -:ref:`bentoml.io.NumpyNdarray `: +:ref:`bentoml.io.NumpyNdarray `: .. code-block:: python @@ -222,9 +221,9 @@ arguments and returns results in the type of ... -Besides the :code:`NumpyNdarray` IO descriptor, BentoML supports a variety of IO -descriptors including :code:`PandasDataFrame`, :code:`JSON`, :code:`String`, -:code:`Image`, :code:`Text`, and :code:`File`. For detailed documentation on how to +Besides the ``NumpyNdarray`` IO descriptor, BentoML supports a variety of IO +descriptors including ``PandasDataFrame``, ``JSON``, ``String``, +``Image``, ``Text``, and ``File``. For detailed documentation on how to declare and invoke these descriptors please see the :doc:`IO Descriptors ` API reference page. @@ -234,16 +233,16 @@ Schema and Validation IO descriptors allow users to define the expected data types, shape, and schema, based on the type of the input and output descriptor specified. IO descriptors can also be defined -through examples with the :code:`from_sample` API to simplify the development of service +through examples with the ``from_sample`` API to simplify the development of service definitions. Numpy ~~~~~ -The data type and shape of the :code:`NumpyNdarray` can be specified with the :code:`dtype` -and :code:`shape` arguments. By setting the :code:`enforce_shape` and :code:`enforce_dtype` +The data type and shape of the ``NumpyNdarray`` can be specified with the ``dtype`` +and ``shape`` arguments. By setting the ``enforce_shape`` and ``enforce_dtype`` arguments to `True`, the IO descriptor will strictly validate the input and output data -based the specified data type and shape. To learn more, see IO descriptor reference for +based the specified data type and shape. To learn more, see IO descrptor reference for :ref:`reference/api_io_descriptors:NumPy ndarray`. .. code-block:: python @@ -272,8 +271,8 @@ based the specified data type and shape. To learn more, see IO descriptor refere Pandas DataFrame ~~~~~~~~~~~~~~~~ -The data type and shape of the :code:`PandasDataFrame` can be specified with the :code:`dtype` -and :code:`shape` arguments. By setting the :code:`enforce_shape` and :code:`enforce_dtype` +The data type and shape of the ``PandasDataFrame`` can be specified with the ``dtype`` +and ``shape`` arguments. By setting the ``enforce_shape`` and ``enforce_dtype`` arguments to `True`, the IO descriptor will strictly validate the input and output data based the specified data type and shape. To learn more, see IO descrptor reference for :ref:`reference/api_io_descriptors:Tabular Data with Pandas`. @@ -336,7 +335,7 @@ model and return. To learn more, see IO descrptor reference for Built-in Types ^^^^^^^^^^^^^^ -Beside :code:`NumpyNdarray`, BentoML supports a variety of other built-in IO descriptor +Beside ``NumpyNdarray``, BentoML supports a variety of other built-in IO descriptor types under the :doc:`bentoml.io ` module. Each type comes with support of type validation and OpenAPI specification generation. For example: @@ -361,7 +360,7 @@ Learn more about other built-in IO Descriptors :doc:`here ` was a synchronous API. BentoML will +The API we created in the :doc:`tutorial ` was a synchronous API. BentoML will intelligently create an optimally sized pool of workers to execute the synchronous logic. Synchronous APIs are simple and capable of getting the job done for most model serving scenarios. diff --git a/docs/source/frameworks/tensorflow.rst b/docs/source/frameworks/tensorflow.rst index 088f6b2b75d..f3cdecc8312 100644 --- a/docs/source/frameworks/tensorflow.rst +++ b/docs/source/frameworks/tensorflow.rst @@ -31,7 +31,7 @@ BentoML requires TensorFlow version 2.0 or higher. For TensorFlow version 1.0, c Saving a Trained Model ---------------------- -``bentoml.tensorflow`` supports saving ``tf.Module``s, ``keras.models.Sequential``s, and ``keras.Model``s. +``bentoml.tensorflow`` supports saving ``tf.Module``, ``keras.models.Sequential``, and ``keras.Model``. .. tab-set:: @@ -285,6 +285,6 @@ to .. note:: - You can find more examples for **TensorFlow** in our `bentoml/examples https://github.com/bentoml/BentoML/tree/main/examples`_ directory. + You can find more examples for **TensorFlow** in our `bentoml/examples `_ directory. .. currentmodule:: bentoml.tensorflow diff --git a/docs/source/reference/frameworks/index.rst b/docs/source/reference/frameworks/index.rst index 94a1a8bb6b6..2af1ec821d6 100644 --- a/docs/source/reference/frameworks/index.rst +++ b/docs/source/reference/frameworks/index.rst @@ -17,6 +17,7 @@ Framework APIs onnx sklearn transformers + tensorflow xgboost sklearn pickable_model