Opinionated Python Library Project Template

• GitHub repo: https://github.com/opinionated-digital-center/python-library-project-generator .
• Free software: MIT license.

The "Opinionated Python Library Project Template" is a fully tested, feature rich Cookiecutter scaffold to kick start your Python library and command line interface development.

It is "opinionated" as it reflects strong structural and tooling choices, which we believe help developers being more effective within Digital Centers. It can however seamlessly be used for Open Source development (works also out-of-the-box with gitlab.com, github.com, pypi.org and readthedocs.org, as a standalone project).

In addition to the intrinsic features offered by the template (see Features list below), the specific features that make it suitable and effective for Digital Centers are the abilities to:

• Use self-hosted GitLab and GitHub.
• Publish to self-hosted package repositories.
• Reuse centralised and optimised "toolbox" items, such as:
  • GitLab-CI pipeline templates.
  • Centralised (and expandable) Make targets. (available soon)

The argumentation behind the Opinionated Digital Center's choices (for this and other projects) are/will be gradually documented as Architecture Decision Records, either on ODC's ADR repository for transversal decisions, or on each individual repository's docs/adr directory (here for this project) for project related decisions.

Fully tested features

All the features of this template are fully tested through CI/CD pipelines, including:

• The Test and make release pipeline, which does:
  • Generate projects for each options of the template.
  • Run the project's built-in tests and checks for each option specifically.
• The Test GitLab Pipeline pipeline (GitHub available soon), which does:
  • Generate a project with the default options.
  • Push the result to the pipeline test project on gitlab.com.
  • Create a Merge Request.
  • Verify the Merge Request's pipeline runs successfully, validating the project's tests and checks.
  • Merge the Merge Request.
  • Verify that the post-merge pipeline successfully creates and publishes a release, which includes:
    • Automatically bump the package's version following ADR-0003: Use Semantic Versioning, based on commit messages following ADR-0005: Use Conventional Commits.
    • Generate the release notes.
    • Publish the release on GitLab.
    • Publish the released package to the TestPyPI package repository.
    • (Also published but not tested: the project's generated doc on Read The Docs).
• The Test GitHub Actions workflow (available soon).

Features list

Projects initialised with the template will benefit straight away from the following out-of-the-box and ready to use features (Note: external environments like GitLab-CI, the Python package repository and the documentation host will require to be set up separately):

• Packaging and dependency management with Poetry.
• Local, multi-version testing automation with Tox.
• Unit testing with Pytest.
• BDD/Functional testing with Behave (includes functional cli testing with Behave4cli).
• Linting/Style guide enforcement with Flake8.
• "Uncompromising" formatting with Black
• Automated import sorting with isort.
• Static typing with Mypy.
• Code coverage with Coverage.
• Task management centralisation with Make, for all your tasks:
  • For setup (development host, project environment, CI/CD pipeline jobs).
  • For test and CI/CD job tasks.
  • For the whole development and release lifecycle.
• Pre-commit hooks for enforcing specific checks before committing with pre-commit.
• CI-CD pipeline with GitLab-CI and GitHub Actions (available soon), including:
  • A test stage running all tests in parallel jobs.
  • A release stage (see Release cycle immediately below).
  • Caching (available soon).
• Release cycle with Semantic-release, including:
  • Automatically bump the package's version following ADR-0003: Use Semantic Versioning, based on commit messages following ADR-0005: Use Conventional Commits).
  • Generate the release notes.
  • Publish the release on the chosen hosting (GitLab or GitHub, cloud or self-hosted).
  • Publish the released package to the chosen Python package repository.
• Command line interface (optional), with testing, using Cleo (default) or Click.
• Documentation generation with Sphinx.
• Documentation publishing to Read the Docs (publishing to GitLab and GitHub Pages planned).

On our development machines, we use Pyenv to install and manage multiple versions of Python (make target available to facilitate installation).

Quickstart

Prerequisites

Cookiecutter installation

Install the latest Cookiecutter if you haven't installed it yet (this requires Cookiecutter 1.4.0 or higher; see the installation doc for more details):

# On macOS
$ brew install cookiecutter

# On linux and other systems
$ pip install -U cookiecutter

PyEnv and Poetry installation

• If you have not installed Pyenv and Poetry on your machine yet, you can use the following make target to do so:

  $ make setup-dev-host

  Don't forget to restart your shell or source your shell configuration file. For example for Bash:

  $ source ~/.bashrc

• You might want to tell Poetry to create virtual environments in the project directories (makes it easier to add the virtualenv to your IDE):

  $ poetry config virtualenvs.in-project true

Project initialisation and development setup

• Generate a project scaffold from the template:

  $ cd your/projects/root/dir
  $ cookiecutter https://github.com/opinionated-digital-center/python-library-project-generator
  # Follow the prompts
  [..]

• Move to your newly created project's directory, initialise its git repo and commit the generated code:

  $ cd <your-project>
  $ git init .
  $ git add --all .
  $ git commit -m 'chore: initial commit'

• Create an initial release tag, which will be used as a basis to bump upcoming releases. By convention, we use v0.0.0:

  $ git tag v0.0.0

• Set up your project's environment:

  # Full setup (installs ``tox`` and all testing and checking libraries)
  $ make setup-dev-env-full
  
  
  # Or alternatively, minimal setup (installs ``tox`` and formatting libraries only)
  $ make setup-dev-env-minimal

• Set up pre-commit hooks to enforce minimal formatting checks before committing (which will otherwise cause the CI/CD pipeline to fail):

  $ make setup-pre-commit-hooks

Hosting and pipeline setup

Hosting setup

• Create a repo on GitHub or GitLab (cloud or self-hosted).

• Push your local repo to it:

  $ git remote add origin https://<hosting-domain>/<your-namespace>/<your-project>.git
  $ git push -u origin master
  
  # Also push the previously created tag
  $ git push --tags

GitLab CI specific setup

• Configure your GitLab project environment variables with the following variables:
  • For release publishing to GitLab, follow @semantic-release/gitlab's doc, and set:
    • GITLAB_TOKEN (don't forget to mask it).
    • GITLAB_URL (optional - see doc).
    • GITLAB_PREFIX (optional - see doc).
  • For Python package publishing to your designated repository, set:

    • PYPI_REPOSITORY_NAME (only needed if you are using a repository other than pypi): name for your Python package repository.

      name can only contain alphanumerical characters, ".", "-" and "_" (valid: my.foo-bar_repository).

      In the remaining environment variables, <NAME> is to be replaced by this repository's name, in UPPERCASE, with "." and "-" replaced by "_" (for instance my.foo-bar_repository becomes MY_FOO_BAR_REPOSITORY).

    • POETRY_REPOSITORIES_<NAME>_URL (required if repository is not pypi): URL of the repository.
    • One of the following credential mechanism has to be set (http basic will take precedence if set):
      • Http basic credential:
        • POETRY_HTTP_BASIC_<NAME>_USERNAME: username credential for repository name.
        • POETRY_HTTP_BASIC_<NAME>_PASSWORD: password credential for repository name.
      • API token credential :
        • POETRY_PYPI_TOKEN_<NAME>: API token credential for repository name. => there is currently an issue with setting API tokens through environment variables. As a workaround, use:
          • POETRY_HTTP_BASIC_<NAME>_USERNAME=__token__.
          • POETRY_HTTP_BASIC_<NAME>_PASSWORD=<your_api_token>.
• Release your first package by running a manual pipeline on your master branch.

GitHub Actions specific setup

(available soon)

Read The Docs setup

• Follow the Webhooks setup doc.

Usage

Once you are all set up, you can use make targets to test and check your work before pushing and opening a pull/merge request.

Here are a few useful, day-to-day targets:

# Display help for targets
$ make

# Full project setup (installs ``tox`` and all testing and checking libraries)
$ make setup-dev-env-full

# Minimal project setup (installs ``tox`` and formatting libraries only)
$ make setup-dev-env-minimal

# Setup pre-commit hooks
$ make setup-pre-commit-hooks

# Run unit tests
$ make test

# Run bdd tests
$ make bdd

# Enforce correct format with black and isort
$ make format

# Check style with flake8
$ make lint

# Check Python typing
$ make type

# Run all tests and checks with tox
$ make tox

# Run tox target in parallel mode
$ make tox-p

# Generate Sphinx HTML documentation
$ make docs

Contributing

We accept pull requests on this, if they're small, atomic, and if they make the packaging experience better (in our opinionated way, which can be discussed and argued... :) ).