Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move default template to static pyproject.toml #2569

Closed
wants to merge 10 commits into from
2 changes: 1 addition & 1 deletion docs/source/deployment/airflow_astronomer.md
Expand Up @@ -84,7 +84,7 @@ To follow this tutorial, ensure you have the following:
pip install kedro-airflow~=0.4
```

5. Run `pip install -r src/requirements.txt` to install all dependencies.
5. Run `pip install -r requirements.txt` to install all dependencies.

### Deployment process

Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/aws_step_functions.md
Expand Up @@ -55,7 +55,7 @@ The rest of the tutorial will explain each step in the deployment process above

* Create a `conf/aws` directory in your Kedro project
* Put a `catalog.yml` file in this directory with the following content
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `src/requirements.txt` so the data can be read from S3.
* Ensure that you have `s3fs>=0.3.0,<0.5` defined in your `requirements.txt` so the data can be read from S3.

<details>
<summary><b>Click to expand</b></summary>
Expand Down
Expand Up @@ -184,7 +184,7 @@ Open your newly-created notebook and create **four new cells** inside it. You wi
1. Before you import and run your Python code, you'll need to install your project's dependencies on the cluster attached to your notebook. Your project has a `requirements.txt` file for this purpose. Add the following code to the first new cell to install the dependencies:

```ipython
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/src/requirements.txt"
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

2. To run your project in your notebook, you must load the Kedro IPython extension. Add the following code to the second new cell to load the IPython extension:
Expand Down
Expand Up @@ -213,7 +213,7 @@ Create **four new cells** inside your notebook. You will fill these cells with c
1. Before you import and run your Python code, you'll need to install your project's dependencies on the cluster attached to your notebook. Your project has a `requirements.txt` file for this purpose. Add the following code to the first new cell to install the dependencies:

```ipython
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/src/requirements.txt"
%pip install -r "/Workspace/Repos/<databricks_username>/iris-databricks/requirements.txt"
```

2. To run your project in your notebook, you must load the Kedro IPython extension. Add the following code to the second new cell to load the IPython extension:
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/distributed.md
Expand Up @@ -14,7 +14,7 @@ For better dependency management, we encourage you to containerise the entire pi
Firstly make sure your [project requirements are up-to-date](../kedro_project_setup/dependencies.md) by running:

```bash
pip-compile --output-file=<project_root>/src/requirements.txt --input-file=<project_root>/src/requirements.txt
pip-compile --output-file=<project_root>/requirements.txt --input-file=<project_root>/requirements.txt
```

We then recommend the [`Kedro-Docker`](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker) plugin to streamline the process of building the image. [Instructions for using this are in the plugin's README.md](https://github.com/kedro-org/kedro-plugins/blob/main/README.md).
Expand Down
2 changes: 1 addition & 1 deletion docs/source/deployment/single_machine.md
Expand Up @@ -114,7 +114,7 @@ conda install -c conda-forge kedro
Install the project’s dependencies, by running the following in the project's root directory:

```console
pip install -r src/requirements.txt
pip install -r requirements.txt
```

After having installed your project on the remote server you can run the Kedro project as follows from the root of the project:
Expand Down
8 changes: 4 additions & 4 deletions docs/source/development/commands_reference.md
Expand Up @@ -292,18 +292,18 @@ _This command will be deprecated from Kedro version 0.19.0._
kedro build-reqs
```

This command runs [`pip-compile`](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) on the project's `src/requirements.txt` file and will create `src/requirements.lock` with the compiled requirements.
This command runs [`pip-compile`](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) on the project's `requirements.txt` file and will create `requirements.lock` with the compiled requirements.

`kedro build-reqs` has two optional arguments to specify which file to compile the requirements from and where to save the compiled requirements to. These arguments are `--input-file` and `--output-file` respectively.

`kedro build-reqs` also accepts and passes through CLI options accepted by `pip-compile`. For example, `kedro build-reqs --generate-hashes` will call `pip-compile --output-file=src/requirements.lock --generate-hashes src/requirements.txt`.
`kedro build-reqs` also accepts and passes through CLI options accepted by `pip-compile`. For example, `kedro build-reqs --generate-hashes` will call `pip-compile --output-file=requirements.lock --generate-hashes requirements.txt`.

#### Install all package dependencies

The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `src/requirements.txt`:
The following runs [`pip`](https://github.com/pypa/pip) to install all package dependencies specified in `requirements.txt`:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

For further information, see the [documentation on installing project-specific dependencies](../kedro_project_setup/dependencies.md#install-project-specific-dependencies).
Expand Down
6 changes: 3 additions & 3 deletions docs/source/development/linting.md
Expand Up @@ -23,7 +23,7 @@ and check the [cyclomatic complexity](https://www.ibm.com/docs/en/raa/6.1?topic=
type. [You can read more in the `isort` documentation](https://pycqa.github.io/isort/).

### Install the tools
Install `black`, `flake8`, and `isort` by adding the following lines to your project's `src/requirements.txt`
Install `black`, `flake8`, and `isort` by adding the following lines to your project's `requirements.txt`
file:
```text
black # Used for formatting code
Expand All @@ -33,7 +33,7 @@ isort # Used for formatting code (sorting module imports)
To install all the project-specific dependencies, including the linting tools, navigate to the root directory of the
project and run:
```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```
Alternatively, you can individually install the linting tools using the following shell commands:
```bash
Expand Down Expand Up @@ -72,7 +72,7 @@ These hooks are run before committing your code to your repositories to automati
making code reviews easier and less time-consuming.

### Install `pre-commit`
You can install `pre-commit` along with other dependencies by including it in the `src/requirements.txt` file of your
You can install `pre-commit` along with other dependencies by including it in the `requirements.txt` file of your
Kedro project by adding the following line:
```text
pre-commit
Expand Down
13 changes: 8 additions & 5 deletions docs/source/experiment_tracking/index.md
Expand Up @@ -72,7 +72,7 @@ cd spaceflights
Install the project's dependencies:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Set up the session store
Expand Down Expand Up @@ -268,17 +268,20 @@ In this section, we illustrate how to compare Matplotlib plots across experiment

### Update the dependencies

Update the `src/requirements.txt` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:
Update the `pyproject.toml` file in your Kedro project by adding the following dataset to enable Matplotlib for your project:

```text
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
seaborn~=0.12.1
dependencies = [
# ...
"kedro-datasets[matplotlib.MatplotlibWriter]~=1.1",
"seaborn~=0.12.1",
]
```

And install the requirements with:

```bash
pip install -r src/requirements.txt
pip install --editable .
```

### Add a plotting node
Expand Down
4 changes: 2 additions & 2 deletions docs/source/get_started/new_project.md
Expand Up @@ -11,7 +11,7 @@ There are a few ways to create a new project once you have [set up Kedro](instal

Once you've created a project:

* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r src/requirements.txt`
* You need to **navigate to its project folder** and **install its dependencies**: `pip install -r requirements.txt`
* **To run the project**: `kedro run`
* **To visualise the project**: `kedro viz`

Expand Down Expand Up @@ -79,7 +79,7 @@ kedro new --starter=pandas-iris
However you create a Kedro project, once `kedro new` has completed, the next step is to navigate to the project folder (`cd <project-name>`) and install dependencies with `pip` as follows:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

Now run the project:
Expand Down
19 changes: 11 additions & 8 deletions docs/source/kedro_project_setup/dependencies.md
Expand Up @@ -5,38 +5,41 @@ Both `pip install kedro` and `conda install -c conda-forge kedro` install the co
When you create a project, you then introduce additional dependencies for the tasks it performs.

## Project-specific dependencies
You can specify a project's exact dependencies in the `src/requirements.txt` file to make it easier for you and others to run your project in the future,
and to avoid version conflicts downstream. This can be achieved with the help of [`pip-tools`](https://pypi.org/project/pip-tools/).
You can specify a project's exact dependencies in the `pyproject.toml` file,
as well as any development dependencies in `requirements.txt`,
to make it easier for you and others to run your project in the future and to avoid version conflicts downstream.
This can be achieved with the help of [`pip-tools`](https://pypi.org/project/pip-tools/).

To install `pip-tools` in your virtual environment, run the following command:
```bash
pip install pip-tools
```

To add or remove dependencies to a project, edit the `src/requirements.txt` file, then run the following:
To add or remove dependencies to a project, edit the `requirements.txt` file, then run the following:

```bash
pip-compile --output-file=<project_root>/src/requirements.txt --input-file=<project_root>/src/requirements.txt
pip-compile --output-file=<project_root>/requirements.txt --input-file=<project_root>/requirements.txt
```

This will [pip compile](https://github.com/jazzband/pip-tools#example-usage-for-pip-compile) the requirements listed in
the `src/requirements.txt` file into a `src/requirements.lock` that specifies a list of pinned project dependencies
the `requirements.txt` file into a `requirements.lock` that specifies a list of pinned project dependencies
(those with a strict version). You can also use this command with additional CLI arguments such as `--generate-hashes`
to use `pip`'s Hash Checking Mode or `--upgrade-package` to update specific packages to the latest or specific versions.
[Check out the `pip-tools` documentation](https://pypi.org/project/pip-tools/) for more information.

```{note}
The `src/requirements.txt` file contains "source" requirements, while `src/requirements.lock` contains the compiled version of those and requires no manual updates.
The `requirements.txt` and `pyproject.toml` files contain "source" requirements, while `requirements.lock` contains the compiled version of those and requires no manual updates.
```

To further update the project requirements, modify the `src/requirements.txt` file (not `src/requirements.lock`) and re-run the `pip-compile` command above.
To further update the project requirements, modify the `requirements.txt` file (not `requirements.lock`) and re-run the `pip-compile` command above.


## Install project-specific dependencies

To install the project-specific dependencies, navigate to the root directory of the project and run:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Workflow dependencies
Expand Down
8 changes: 4 additions & 4 deletions docs/source/kedro_project_setup/starters.md
Expand Up @@ -155,17 +155,17 @@ Here is the layout of the project as a Cookiecutter template:
├── docs # Project documentation
├── notebooks # Project related Jupyter notebooks (can be used for experimental code before moving the code to src)
├── README.md # Project README
├── setup.cfg # Configuration options for tools e.g. `pytest` or `flake8`
├── .flake8 # Configuration options for `flake8`
├── requirements.txt # Development dependencies
├── pyproject.py # Package metadata and configuration options for tools
├── tests # Tests
└── src # Project source code
└── {{ cookiecutter.python_package }}
├── __init.py__
├── pipelines
├── pipeline_registry.py
├── __main__.py
└── settings.py
├── requirements.txt
├── setup.py
└── tests
```

```{note}
Expand Down
2 changes: 1 addition & 1 deletion docs/source/tutorial/spaceflights_tutorial_faqs.md
Expand Up @@ -46,7 +46,7 @@ documentation on how to install relevant dependencies for kedro_datasets.pandas.
https://kedro.readthedocs.io/en/stable/kedro_project_setup/dependencies.html
```

The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r src/requirements.txt` to install them.
The Kedro Data Catalog is missing [dependencies needed to parse the data](../kedro_project_setup/dependencies.md#install-dependencies-related-to-the-data-catalog). Check that you have [all the project dependencies to `requirements.txt`](./tutorial_template.md#install-project-dependencies) and then call `pip install -r requirements.txt` to install them.

### Pipeline run

Expand Down
6 changes: 3 additions & 3 deletions docs/source/tutorial/tutorial_template.md
@@ -1,6 +1,6 @@
# Set up the spaceflights project

This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r src/requirements.txt`).
This section shows how to create a new project (with `kedro new` using the [Kedro spaceflights starter](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights)) and install project dependencies (with `pip install -r requirements.txt`).

## Create a new project

Expand Down Expand Up @@ -28,7 +28,7 @@ cd spaceflights

Kedro projects have a `requirements.txt` file to specify their dependencies and enable sharable projects by ensuring consistency across Python packages and versions.

The spaceflights project dependencies are stored in `src/requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):
The spaceflights project dependencies are stored in `requirements.txt`(you may find that the versions differ slightly depending on the version of Kedro):

```text
# code quality packages
Expand Down Expand Up @@ -63,7 +63,7 @@ scikit-learn~=1.0
To install all the project-specific dependencies, run the following from the project root directory:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

## Optional: logging and configuration
Expand Down
2 changes: 1 addition & 1 deletion docs/source/visualisation/kedro-viz_visualisation.md
Expand Up @@ -18,7 +18,7 @@ When prompted for a project name, you can enter anything, but we will assume `Sp
When your project is ready, navigate to the root directory of the project and install the dependencies for the project, which include Kedro-Viz:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

The next step is optional, but useful to check that all is working. Run the full set of pipelines for the tutorial project:
Expand Down
12 changes: 7 additions & 5 deletions docs/source/visualisation/visualise_charts_with_plotly.md
Expand Up @@ -27,7 +27,7 @@ There are two types of Plotly datasets supported by Kedro:
* `plotly.PlotlyDataSet` which only supports [Plotly Express](https://plotly.com/python/plotly-express)
* `plotly.JSONDataSet` which supports Plotly Express and [Plotly Graph Objects](https://plotly.com/python/graph-objects/)

To use the Plotly datasets, you must update the `requirements.txt` file in the `src` folder of the Kedro project to add the following dependencies:
To use the Plotly datasets, you must update the `requirements.txt` file in the root folder of the Kedro project to add the following dependencies:


```text
Expand All @@ -38,7 +38,7 @@ kedro-datasets[plotly.PlotlyDataSet, plotly.JSONDataSet]~=1.1
Navigate to the root directory of the project in your terminal and install the dependencies for the tutorial project:

```bash
pip install -r src/requirements.txt
pip install -r requirements.txt
```

### Configure the Data Catalog
Expand Down Expand Up @@ -177,11 +177,13 @@ You can view Matplotlib charts in Kedro-Viz when you use the [Kedro MatplotLibWr

### Update the dependencies

You must update the `src/requirements.txt` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:
You must update the `pyproject.toml` file in the Kedro project by adding the following dataset to enable Matplotlib for the project:

```bash
kedro-datasets[matplotlib.MatplotlibWriter]~=1.1
seaborn~=0.12.1
dependencies = [
"kedro-datasets[matplotlib.MatplotlibWriter]~=1.1",
"seaborn~=0.12.1",
]
```

### Configure the Data Catalog
Expand Down
15 changes: 10 additions & 5 deletions features/environment.py
Expand Up @@ -116,13 +116,18 @@ def _setup_minimal_env(context):

def _install_project_requirements(context):
install_reqs = (
Path(
"kedro/templates/project/{{ cookiecutter.repo_name }}/src/requirements.txt"
)
Path("kedro/templates/project/{{ cookiecutter.repo_name }}/requirements.txt")
.read_text(encoding="utf-8")
.splitlines()
)
install_reqs = [req for req in install_reqs if "{" not in req]
install_reqs.append(".[pandas.CSVDataSet]")
install_reqs = [
req
for req in install_reqs
if (req.strip())
and ("{" not in req)
and (not req.startswith("-e"))
and (not req.startswith("#"))
]
install_reqs.append("kedro-datasets[pandas.CSVDataSet]")
call([context.pip, "install", *install_reqs], env=context.env)
return context
10 changes: 5 additions & 5 deletions features/steps/cli_steps.py
Expand Up @@ -162,7 +162,7 @@ def create_config_file(context):
@given("I have installed the project dependencies")
def pip_install_dependencies(context):
"""Install project dependencies using pip."""
reqs_path = "src/requirements.txt"
reqs_path = "requirements.txt"
res = run(
[context.pip, "install", "-r", reqs_path],
env=context.env,
Expand Down Expand Up @@ -410,7 +410,7 @@ def update_kedro_req(context: behave.runner.Context):
"""Replace kedro as a standalone requirement with a line
that includes all of kedro's dependencies (-r kedro/requirements.txt)
"""
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"
kedro_reqs = f"-r {context.requirements_path.as_posix()}"

if reqs_path.is_file():
Expand All @@ -428,7 +428,7 @@ def update_kedro_req(context: behave.runner.Context):

@when("I add {dependency} to the requirements")
def add_req(context: behave.runner.Context, dependency: str):
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"
if reqs_path.is_file():
reqs_path.write_text(reqs_path.read_text() + "\n" + str(dependency) + "\n")

Expand Down Expand Up @@ -610,14 +610,14 @@ def check_docs_generated(context: behave.runner.Context):
@then("requirements should be generated")
def check_reqs_generated(context: behave.runner.Context):
"""Check that new project requirements are generated."""
reqs_path = context.root_project_dir / "src" / "requirements.lock"
reqs_path = context.root_project_dir / "requirements.lock"
assert reqs_path.is_file()
assert "This file is autogenerated by pip-compile" in reqs_path.read_text()


@then("{dependency} should be in the requirements")
def check_dependency_in_reqs(context: behave.runner.Context, dependency: str):
reqs_path = context.root_project_dir / "src" / "requirements.txt"
reqs_path = context.root_project_dir / "requirements.txt"
assert dependency in reqs_path.read_text()


Expand Down