From a4b447345c9145f12120c772f7e84e6f4821c5fb Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 09:52:32 +0100 Subject: [PATCH 01/13] revamp build instructions --- CONTRIBUTING.md | 54 ++++++++++++++++++++++--------------------------- 1 file changed, 24 insertions(+), 30 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cef99f9fa8..27dd23bc86 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -306,7 +306,20 @@ before you insert the macro call into the markdown document. We are using mkdocs to render our specification. Please follow these instructions if you would like to build the specification locally. -#### 1. Install mkdocs, the material theme and the required extensions +#### 1. Download the BIDS specification [repository](https://github.com/bids-standard/bids-specification/tree/master) onto your computer + +This can be done by clicking the green button on the right titled "Clone or +download" +or using [this link](https://github.com/bids-standard/bids-specification/archive/master.zip). + +#### 2. In the terminal (command line) navigate to your local version of the specification + +This location will have the same files you see on our +[main specification page](https://github.com/bids-standard/bids-specification). +Note that a file browser window may not show the hidden files +(those that start with a period, like `.remarkrc`). + +#### 3. Install mkdocs, the material theme and the required extensions In the following links, you can find more information about @@ -318,40 +331,21 @@ You will also need several other mkdocs plugins, like `branchcustomization` and To install all of this make sure you have a recent version of Python on your computer. The [DataLad Handbook](http://handbook.datalad.org/en/latest/intro/installation.html#python-3-all-operating-systems) provides helpful instructions for setting up Python. -An easy way to install the correct version of mkdocs and all the other required extensions -is to use the `requirements.txt` file contained in this repository, -by using the following command: - -```bash -pip install -r requirements.txt -``` +In general, we strongly recommend that you install all dependencies in an isolated Python environment. +For example using `conda`, as described in the section on Python in the DataLad Handbook, linked above. +Or alternatively using `venv`, as described in this [Real Python tutorial](https://realpython.com/python-virtual-environments-a-primer/). -However this will also install some other packages you might not want to have (like `numpy`). -So if you only want to install what you need to build the specification, -use the following command: +Once you have activated your isolated Python environment, +an easy way to install the correct version of mkdocs and all the other required extensions +is to use the `requirements.txt` file contained in this repository as follows: ```bash -pip install \ - mkdocs \ - mkdocs-material \ - pymdown-extensions \ - mkdocs-branchcustomization-plugin \ - mkdocs-macros-plugin \ - tabulate +pip install -U pip +pip install -r requirements.txt ``` -#### 2. Download the BIDS specification [repository](https://github.com/bids-standard/bids-specification/tree/master) onto your computer - -This can be done by clicking the green button on the right titled "Clone or -download" -or using [this link](https://github.com/bids-standard/bids-specification/archive/master.zip). - -#### 3. In the terminal (command line) navigate to your local version of the specification - -This location will have the same files you see on our -[main specification page](https://github.com/bids-standard/bids-specification). -Note: A finder window may not show the hidden files (those that start with a -period, like `.remarkrc`) +The first command ensures you are using an up to date version of `pip`, +and the second command installs all dependencies. #### 4. Ready to build! From c012abcec5eb999247d10cb8ff636f6f7f48e90f Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 09:55:09 +0100 Subject: [PATCH 02/13] add a note --- CONTRIBUTING.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 27dd23bc86..29b3b7b898 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -319,6 +319,8 @@ This location will have the same files you see on our Note that a file browser window may not show the hidden files (those that start with a period, like `.remarkrc`). +Enter all commands below from the command line prompt located at the root of the local version of the specification. + #### 3. Install mkdocs, the material theme and the required extensions In the following links, you can find more information about From b20e120a618e44a729eac1a55f0ee89c0ba684d9 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 10:17:15 +0100 Subject: [PATCH 03/13] Apply suggestions from code review Co-authored-by: Remi Gau --- CONTRIBUTING.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 29b3b7b898..f2409f62b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -312,6 +312,8 @@ This can be done by clicking the green button on the right titled "Clone or download" or using [this link](https://github.com/bids-standard/bids-specification/archive/master.zip). +Or you can use the following git command in a terminal: `git clone https://github.com/bids-standard/bids-specification.git`. + #### 2. In the terminal (command line) navigate to your local version of the specification This location will have the same files you see on our @@ -319,6 +321,8 @@ This location will have the same files you see on our Note that a file browser window may not show the hidden files (those that start with a period, like `.remarkrc`). +If you cloned the repository using the git command above, you can then just do: `cd bids-specification`. + Enter all commands below from the command line prompt located at the root of the local version of the specification. #### 3. Install mkdocs, the material theme and the required extensions @@ -335,8 +339,22 @@ The [DataLad Handbook](http://handbook.datalad.org/en/latest/intro/installation. In general, we strongly recommend that you install all dependencies in an isolated Python environment. For example using `conda`, as described in the section on Python in the DataLad Handbook, linked above. + +```bash +conda create --name bids-spec +conda activate bids-spec +.``` + Or alternatively using `venv`, as described in this [Real Python tutorial](https://realpython.com/python-virtual-environments-a-primer/). +A short version of the commands needed to create and activate your `venv` virtual environment would look like: + +```bash +python3 -m venv env +source env/bin/activate +.``` + + Once you have activated your isolated Python environment, an easy way to install the correct version of mkdocs and all the other required extensions is to use the `requirements.txt` file contained in this repository as follows: From 5151d04fe4092e14ed454ef4468e86534acd8a86 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 10:20:29 +0100 Subject: [PATCH 04/13] streamline --- CONTRIBUTING.md | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index f2409f62b4..cae4e4971b 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -312,7 +312,11 @@ This can be done by clicking the green button on the right titled "Clone or download" or using [this link](https://github.com/bids-standard/bids-specification/archive/master.zip). -Or you can use the following git command in a terminal: `git clone https://github.com/bids-standard/bids-specification.git`. +Or you can use the following `git` command in a terminal: + +```bash +git clone https://github.com/bids-standard/bids-specification.git +``` #### 2. In the terminal (command line) navigate to your local version of the specification @@ -321,7 +325,11 @@ This location will have the same files you see on our Note that a file browser window may not show the hidden files (those that start with a period, like `.remarkrc`). -If you cloned the repository using the git command above, you can then just do: `cd bids-specification`. +If you cloned the repository using the `git` command above, you can then just do: + +```bash +cd bids-specification +``` Enter all commands below from the command line prompt located at the root of the local version of the specification. @@ -343,7 +351,7 @@ For example using `conda`, as described in the section on Python in the DataLad ```bash conda create --name bids-spec conda activate bids-spec -.``` +``` Or alternatively using `venv`, as described in this [Real Python tutorial](https://realpython.com/python-virtual-environments-a-primer/). @@ -352,8 +360,11 @@ A short version of the commands needed to create and activate your `venv` virtua ```bash python3 -m venv env source env/bin/activate -.``` +``` +Note however, that this will create a local directory called `env` within the bids-specification directory. +Make sure not to commit that directory (your environment) when adding and committing your changes via `git` +(this is not a problem when using `conda`, because `conda` saves the environment in a different place). Once you have activated your isolated Python environment, an easy way to install the correct version of mkdocs and all the other required extensions From fa177d77a4e11b3d2a6e45a82fca2d15f7c3cd46 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 10:24:00 +0100 Subject: [PATCH 05/13] Update CONTRIBUTING.md Co-authored-by: Remi Gau --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index cae4e4971b..493e916099 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -363,7 +363,7 @@ source env/bin/activate ``` Note however, that this will create a local directory called `env` within the bids-specification directory. -Make sure not to commit that directory (your environment) when adding and committing your changes via `git` +Make sure not to commit that directory (your environment) when adding and committing your changes via `git` (this is not a problem when using `conda`, because `conda` saves the environment in a different place). Once you have activated your isolated Python environment, From 86f58bc2ac79b25aa59feb9fb97a304735e155f5 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 10:37:21 +0100 Subject: [PATCH 06/13] add remis comment on gitignore --- CONTRIBUTING.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 493e916099..3cd45b2749 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -362,9 +362,8 @@ python3 -m venv env source env/bin/activate ``` -Note however, that this will create a local directory called `env` within the bids-specification directory. -Make sure not to commit that directory (your environment) when adding and committing your changes via `git` -(this is not a problem when using `conda`, because `conda` saves the environment in a different place). +Note that this will create a local directory called `env` within the bids-specification directory +but that its content will not be tracked by `git` because it is listed in the `.gitignore` file. Once you have activated your isolated Python environment, an easy way to install the correct version of mkdocs and all the other required extensions From 0d78e09b195bdb9b76e747802d95c33e72b80564 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 14:17:13 +0100 Subject: [PATCH 07/13] Update CONTRIBUTING.md Co-authored-by: Mateusz Pawlik --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3cd45b2749..33d85950e3 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -358,7 +358,7 @@ Or alternatively using `venv`, as described in this [Real Python tutorial](https A short version of the commands needed to create and activate your `venv` virtual environment would look like: ```bash -python3 -m venv env +python -m venv env source env/bin/activate ``` From ac6319897a71dfe09a166423c5dabffc25569ad8 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Mon, 21 Mar 2022 14:21:11 +0100 Subject: [PATCH 08/13] add Geohackweek resource on conda --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 33d85950e3..2ad746bab0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -346,7 +346,7 @@ To install all of this make sure you have a recent version of Python on your com The [DataLad Handbook](http://handbook.datalad.org/en/latest/intro/installation.html#python-3-all-operating-systems) provides helpful instructions for setting up Python. In general, we strongly recommend that you install all dependencies in an isolated Python environment. -For example using `conda`, as described in the section on Python in the DataLad Handbook, linked above. +For example using `conda`, as described in this [Geohackweek tutorial](https://geohackweek.github.io/Introductory/01-conda-tutorial/). ```bash conda create --name bids-spec From 387a56f981d9538ae70fc7c61ab15a3c79f18fe4 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Thu, 24 Mar 2022 15:35:24 +0100 Subject: [PATCH 09/13] add info on editable install --- CONTRIBUTING.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2ad746bab0..e00567f523 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -372,10 +372,14 @@ is to use the `requirements.txt` file contained in this repository as follows: ```bash pip install -U pip pip install -r requirements.txt +pip install -e tools/schemacode/ ``` The first command ensures you are using an up to date version of `pip`, and the second command installs all dependencies. +The third command ensures to install the BIDS schema code as an "editable" install, +so that if you make changes to the schema files, +these are automatically reflected in the sourcecode. #### 4. Ready to build! From 70a7ae456d845317d6af9b513641854d231330aa Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Thu, 24 Mar 2022 15:50:50 +0100 Subject: [PATCH 10/13] CI: migrate from legacy convenience images --- .circleci/config.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index db88083e9a..1f8d6075ac 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -2,7 +2,7 @@ version: 2 jobs: build_docs: docker: - - image: circleci/python:3.8 + - image: cimg/python:3.8 steps: # checkout code to default ~/project - checkout @@ -25,7 +25,7 @@ jobs: linkchecker: docker: - - image: circleci/python:3.8 + - image: cimg/python:3.8 steps: # checkout code to default ~/project - checkout @@ -123,7 +123,7 @@ jobs: # Run remark on the auto generated changes.md file remark: docker: - - image: node:latest + - image: cimg/node:lts steps: # checkout code to default ~/project - checkout @@ -156,7 +156,7 @@ jobs: # Push built changelog to repo Changelog-bot: docker: - - image: circleci/openjdk:8-jdk + - image: cimg/node:lts steps: - setup_remote_docker: version: 17.11.0-ce From 1b7282c1f727d92b603ae955517438b6884bd07b Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Thu, 24 Mar 2022 15:54:43 +0100 Subject: [PATCH 11/13] try pin jinja <3.1 --- requirements.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/requirements.txt b/requirements.txt index edc1538e9f..68f6e99757 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,3 +5,6 @@ mkdocs-branchcustomization-plugin~=0.1.3 mkdocs-macros-plugin numpy tools/schemacode/ + +# see https://github.com/bids-standard/bids-specification/pull/1032#issuecomment-1077707688 +jinja2<3.1 From 813b377977a96ed966d3430e4df6c273dac327fa Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Thu, 24 Mar 2022 16:12:15 +0100 Subject: [PATCH 12/13] use venv in new circleci images --- .circleci/config.yml | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 1f8d6075ac..2ac96d2fcb 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -9,12 +9,16 @@ jobs: - run: name: install dependencies command: | + python -m venv env + source env/bin/activate python -m pip install --upgrade pip pip install -r requirements.txt - pip install ~/project/tools/schemacode/ + pip install -e ~/project/tools/schemacode/ - run: name: generate docs - command: mkdocs build --clean --strict --verbose + command: | + source env/bin/activate + mkdocs build --clean --strict --verbose - persist_to_workspace: # the mkdocs build outputs are in ~/project/site root: ~/project @@ -35,11 +39,14 @@ jobs: - run: name: install linkchecker command: | - python -m ensurepip + python -m venv env + source env/bin/activate + python -m pip install --upgrade pip python -m pip install linkchecker - run: name: check links command: | + source env/bin/activate git status if (! git log -1 --pretty=oneline | grep REL:) ; then chmod a+rX -R ~ @@ -156,7 +163,7 @@ jobs: # Push built changelog to repo Changelog-bot: docker: - - image: cimg/node:lts + - image: cimg/base:stable steps: - setup_remote_docker: version: 17.11.0-ce From 19b3d6751f9aeebb9120818e71e1499db991de15 Mon Sep 17 00:00:00 2001 From: Stefan Appelhoff Date: Tue, 29 Mar 2022 10:59:12 +0200 Subject: [PATCH 13/13] remove jinja pin, this has been fixed upstream --- requirements.txt | 3 --- 1 file changed, 3 deletions(-) diff --git a/requirements.txt b/requirements.txt index 68f6e99757..edc1538e9f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -5,6 +5,3 @@ mkdocs-branchcustomization-plugin~=0.1.3 mkdocs-macros-plugin numpy tools/schemacode/ - -# see https://github.com/bids-standard/bids-specification/pull/1032#issuecomment-1077707688 -jinja2<3.1