Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

L97: Python: Setuptools entrypoint for generating file #290

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
177 changes: 177 additions & 0 deletions L97-python-setuptools-plugin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
L97 - Python: Setuptools entrypoint for generating file
----
* Author(s): Patrick Lannigan
* Approver: lidizheng and gnossen
* Status: Draft
* Implemented in: Python
* Last updated: 2022-02-18
* Discussion at: https://groups.google.com/g/grpc-io/c/H617sSzoV-8

## Abstract

Implement a `setuptools` entrypoint for generating files when a Python distribution is being built.
This type of integration would be called automatically by `setuptools` and would not require and
explicit call by the project to generate the files. The entrypoint would be configured using an
established python project configuration file and would only require the `grpcio-tools` package to be
listed as a build time dependency.

## Background

Currently `grpc_tools` [provides a command][grpc_command] for integrating with `setuptools`. However,
this requires custom integration on the part of the package (example:
[grpc_health_checking][grpc_command_example_integration] and
[explicitly calling the command][grpc_command_example_call] when building the wheel).

With the introduction of [PEP 517][pep_517] and [PEP 518][pep_518], directly calling `setup.py` will
start becoming less common. Additionally, the use of custom commands will not work with projects that
use this mechanism and solely rely on the [declarative config in `setup.cfg`][declarative_config],
which was added to `setuptools` in [v30.3.0][release_declarative_config] (Dec. 2016). Though, use of
the functionality defined in this proposal does not require a project to use the declarative config.

Custom commands are not the only integration point that is available. `setuptools` also supports
entrypoints that allows for packages to act as a plugin and provide custom behavior. There are a few
different options for entrypoint hooks ([1][setuptools_customize], [2][setuptools_file_finder]) which
could be used to generate source files based on `.proto` files. (There is also an
[open proposal][setuptools_sub_commands] to allow for adding `build` sub-commands. Which more directly
represents what we are trying to achieve.)

[Original Feature Request][original_feature_request]

## Proposal

`grpcio-tools` will register a new entrypoint function to respond to the
`setuptools.finalize_distribution_options` hook. This hook will generate Python files based on the
`.proto` files.

### Configuration

This functionality will use a [tool table][tool_table] in `pyproject.toml` named `tool.grpcio-tools`
to accept configuration parameters.

__TODO__: List configuration parameters and how they would be used. If anything is required, add it to
the list in "How would a project use this feature".

### Functional Implementation

Add an entrypoint function to `grpcio-tools` that would register as a `file_finder` hook with
`setuptools`. To simplify the implementation without introducing an additional code path, the
entrypoint will target the same `main()` function used by the `grpc_tools.protoc` CLI.

In order to prevent unintended behavior, the first thing the function will do is check the
configuration file to see if the section has been configured (See [Configuration](#configuration)). If
the file does not exist or the section is not configured, no additional steps will be taken. The
function will return an empty list back to `setuptools` to indicate that there are no additional files
to include in the distribution.

Starting with Python 3.11, the configuration file will be readable using a
[standard library module][pep_680]. For earlier Python versions, `tomli` will be added as an optional
dependency.

### How would a project use this feature

A project that would like to use this `setuptools` integration would need to perform two actions to
enable the functionality.

* List `grpcio-tools` as a [build time dependency][build_dependency]. This will ensure that the library
is installed when the project's package is built.
* Add a `tool.grpcio-tools` table to the file `pyproject.toml`.

Minimal example for a project using [PEP 518][pep_518]:

pyproject.toml
```toml
[build-system]
requires = ["setuptools", "wheel", "grpcio-tools >= X.Y.Z"]
build-backend = "setuptools.build_meta"

[tool.grpcio-tools]
example_parameter = "CHANGE ME" # This is a place holder until the configuration parameters are defined
```

Minimal example for a project using `setup.py`:

setup.py
```python
from setuptools import setup

setup(
...,
setup_requires=[
'grpcio-tools >= X.Y.Z'
],
)
```

pyproject.toml
```toml
[tool.grpcio-tools]
example_parameter = "CHANGE ME" # This is a place holder until the configuration parameters are defined
```

## Rationale

A downside of this proposal is that it will introduce yet another way to get the Python modules that
correspond to the proto files.

The primary way generate the Python files is using the
[`grpc_tools.protoc` command line interface][protoc_cli]. The advantage of this mechanism of this
method is also its disadvantage. It is a completely standalone way to generate the source code for
the python modules. While projects are allowed to decide how to work this generated files in the way
that best suits the project, there is no integration point. This means the project **must** decide
how and when this command is executed, likely needing to write a script to execute this CLI.

For projects that use `setuptools` to publish a package, `grpcio-tools` includes a
[custom command][grpc_command]. However, it also requires a specific call like the `grpc_tools.protoc`
CLI and is not compatible with installing from a source distribution for projects that declare
`setuptools` as build system that should be used. Additionally, one of the top result for "grpc
setuptools command" is [this post][setuptools_gen_question], which suggests a custom command that
doesn't rely on the one provided by `grpcio-tools`. This suggest that there is a desire for functionality
that doesn't require a specific all and that the existing functionality is under-documented.

gRPC also provides a way to get the Python [classes at runtime][runtime_classes]
([Proposal][runtime_proposal]). However, it is still [marked as experimental][runtime_api] almost 2
years later and requires `grpcio-tools` to be available at runtime. Similar to the current `setuptools`
integration, it is not mentioned in the basic tutorial documentation.

## Implementation

[Pull Request][feature_pr]

- [x] Proof of Concept with hard coded values.
- [ ] Configuration file parsing & mapping to CLI arguments.
- [ ] Document feature and how projects would use it.

Could be done later:

- [ ] Migrate existing `grpc*` packages to use this functionality.

I am planning work on it. I will be able to spend a few hours of each work week on this effort.

## Open issues (if applicable)

- Looking at the implementation of the [custom command][grpc_command], it looks as though only a
sub-set of the CLI arguments supported by `grpc_tools.protoc` need to be supported. I believe this
will become more clear after starting on the implementation. However, there could always be an
"additional arguments" type parameter to cover this if a project has a specific need. This would
also allow for CLI arguments to be passed to any gRPC plugins that might be used by a project.

[grpc_command]: https://github.com/grpc/grpc/blob/05e17e92390d4685f1418f535604a201a7f8e1a3/tools/distrib/python/grpcio_tools/grpc_tools/command.py#L50
[grpc_command_example_integration]: https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/src/python/grpcio_health_checking/health_commands.py#L48
[grpc_command_example_call]: https://github.com/grpc/grpc/blob/2d4f3c56001cd1e1f85734b2f7c5ce5f2797c38a/tools/run_tests/artifacts/build_artifact_python.sh#L196-L197
[pep_517]: https://www.python.org/dev/peps/pep-0517/
[pep_518]: https://www.python.org/dev/peps/pep-0518/
[declarative_config]: https://setuptools.pypa.io/en/latest/userguide/declarative_config.html
[release_declarative_config]: https://setuptools.pypa.io/en/latest/history.html#v30-3-0
[setuptools_customize]: https://setuptools.pypa.io/en/latest/userguide/extension.html#customizing-distribution-options
[setuptools_file_finder]: https://setuptools.pypa.io/en/latest/userguide/extension.html#adding-support-for-revision-control-systems
[setuptools_sub_commands]: https://github.com/pypa/setuptools/issues/2591
[original_feature_request]: https://github.com/grpc/grpc/issues/28662
[build_dependency]: https://setuptools.pypa.io/en/latest/userguide/dependency_management.html?highlight=setup_requires#build-system-requirement
[tool_table]: https://www.python.org/dev/peps/pep-0518/#tool-table
[pep_680]: https://www.python.org/dev/peps/pep-0680/
[protoc_cli]: https://grpc.io/docs/languages/python/basics/#generating-client-and-server-code
[setuptools_gen_question]: https://stackoverflow.com/q/52994857
[runtime_classes]: https://github.com/grpc/grpc/blob/a72c8ebb7def13a317a1afc7c08455388d1fa2e4/src/python/grpcio/grpc/_runtime_protos.py#L1
[runtime_proposal]: ./L64-python-runtime-proto-parsing.md
[runtime_api]: https://grpc.github.io/grpc/python/grpc.html#runtime-protobuf-parsing
[feature_pr]: https://github.com/grpc/grpc/pull/29541