[DEPR]: Paver #34467

kdmccormick · 2024-04-03T15:33:45Z

ℹ️ Context: the Paver assets module was accepted for deprecation back in May 2023, but we haven't removed it yet. Now, in May 2024, we are ready to deprecate the entire Paver suite. For simplicity, we are deprecating all Paver commands in this single ticket, assets included.

Timeline

Communicated

2024-05-01 (thread)

Target Acceptance

2024-05-21

Earliest Open edX Named Release Without This Functionality

Sumac

Latest Open edX Named Release With the Replacement Available

Internal CI commands: Sumac

All other commands, including Asset commands: Redwood

(See table below for details)

Rationale

edx-platform historically handled its build scripts with paver:

Paver is a Python-based software project scripting tool along the lines of Make or Rake. It is not designed to handle the dependency tracking requirements of, for example, a C program. It is designed to help out with all of your other repetitive tasks (run documentation generators, moving files about, downloading things), all with the convenience of Python’s syntax and massive library of code.

Using paver has a few problems:

Paver adds dependencies to edx-platform: Paver follows the Python packaging model, so anything that's a dependency of edx-platform's pavelib suite becomes a dependency of edx-platform itself. In addition to the paver package, this means that we install libsass-python in production, which takes 60+ seconds (!) to compile and install.
Paver makes Python a requirement for all build actions: When your build tooling runs on Python, then you need a Python environment in order to do anything, even things that don't involve Python (including building static assets, running JS tests, etc.). This harms the cachability of edx-platform Dockerfiles, because it means that any change to the Python requirements list will invalidate any build steps used by Paver.
Paver is indirect: Paver basically just wraps shell commands that you could have run directly, adding a layer of complexity and potential bugs. Build scripts are ideally very simple--unlike application code, repetition is preferable to abstraction; they should be easy to write, easy to read, and easy to throw away and replace if necessary. Standard tools like Makefiles, Bash scripts, and Dockerfiles all encourage and excel at this sort of simplicity; Paver does not.
Paver is idiosyncratic: Unlike Make and Bash, Paver is not at all a standard part of a programmer's toolset, so it makes edx-platform less approachable for new contributors. Even for programmers who want to script in Python, Click is significantly more popular, well-documented, and user-friendly.

For the Paver Asset commands in particular, more depth is provided in the 'Reimplement edx-platform static asset processing' ADR.

Replacement

Module	Replacement
assets	New static assets guide
quality	Directly invoking tools (`pylint`, etc.) when possible, Makefile targets otherwise
test_js	Makefile target
prereqs	`pip`
docs	Makefile target
i18n	Makefile targets for common tasks; i18n_tool for advanced tasks
servers	Tutor (or the deprecated Docker-based Devstack, or bare-metal development

Migration

To ease migration for Paver users, in Redwood, each Paver Asset command will simply proxy to its replacement command, and will raise a deprecation warning explaining the new command that it is running.

Requirements

In Redwood and earlier, Paver and its dependencies were included in requirements/edx/base.txt

Starting in Sumac, these dependencies will be removed from requirements/edx/base.txt. Instead, operators will need to install:

requirements/edx/assets.txt to build static assets
requirements/edx/testing.txt to run Python tests and linting
requirements/edx/base.txt download translations and collect static assets

Commands

Module	Known Users	Replacement Status	Before	After
assets	Ansible, Devstack	Ready to use	`paver update_assets`	`npm run build && ./manage.py lms collecstatic --noinput && ./manage.py cms collectstatic`ns
assets	None	Ready to use	`paver process_xmodule_assets`	No longer needed - no replacement
assets	Tutor, pavelib	Ready to use	`paver compile_sass`	`npm run compile-sass`
assets	pavelib	Ready to use	`./manage.py [lms/cms] compile_sass`	`npm run compile-sass`
assets	pavelib	Ready to use	`paver webpack`	`npm run webpack`
assets	Devstack	Ready to use	`paver watch_assets`	`npm run watch`
quality	edx-platform CI	This is a no-op	`paver find_fixme`	Remove
quality	edx-platform CI	Need to implement	`paver run_eslint`	`eslint`
quality	edx-platform CI	Need to implement	`paver run_stylelint`	`make stylelint_js`
quality	edx-platform CI	Need to implement	`paver run_xsslint`	`make xsslint`
quality	edx-platform CI	Need to implement	`paver run_pii_check`	`make pii_check`
quality	edx-platform CI	Need to implement	`paver check_keywords`	`make check_keywords`
quality	None	Ready to use	`paver run_quality`	`pylint`
quality	None	Ready to use	`paver run_pylint`	`pylint`
quality	edx-platform CI	Need to implement	`paver run_pep8`	`pycodestyle`
js_test	edx-platform CI	Need to implement	`paver diff_coverage`	`make diff_coverage_js`
js_test	edx-platform CI	Need to implement	`paver test_js`	`make test_js`
js_test	edx-platform CI	Need to implement	`paver test_js_run`	`make test_js`
js_test	edx-platform CI	Need to implement	`paver test_js_dev`	`make test_js MODE=browser`
prereqs	Ansible, Devstack, pavelib	Ready to use	`paver install_prereqs`	`pip install -r requirements/edx/base.txt -r requirements/edx/assets.txt && npm clean-install`
prereqs	pavelib	Ready to use	`paver install_node_prereqs`	`npm clean-install`
prereqs	pavelib	Will not be replacedl	`paver uninstall_python_packages`	Will not be replaced
prereqs	pavelib	Ready to use	`paver install_coverage_prereqs`	`pip install -r requirements/edx/coverage.txt`
prereqs	pavelib	Ready to use	`paver install_python_prereqs`	`pip install -r requirements/edx/base.txt`
docs	None	Ready to use	`paver build_docs`	`make docs`
i18n	None	Ready to use	`paver i18n_validate_gettext`	`which xgettext`
i18n	None	Ready to use	`paver i18n_extract`	`make extract_translations`
i18n	None	Ready to use	`paver i18n_dummy`	`i18n_tool dummy && i18n_tool generate`
i18n	None	Ready to use	`paver i18n_generate`	`i18n_tool generate`
i18n	None	Ready to use	`paver i18n_generate_strict`	`i18n_tool generate --strict`
i18n	None	Ready to use	`paver i18n_clean`	`make clean_translations`
servers	Vagrant Devstack	Ready to use	`paver check_settings`	None
servers	Vagrant Devstack	Ready to use	`paver lms`	`./manage.py lms runserver`
servers	Vagrant Devstack	Ready to use	`paver studio`	`./manage.py cms runserver`
servers	Vagrant Devstack	Ready to use	`paver run_all_servers`	`tutor local ...`
servers	Vagrant Devstack	Ready to use	`paver devstack`	`tutor dev ...`

Notes:

'Ansible' is the deprecated configuration repo.
'Devstack' refers to the Docker-based devstack which was deprecated recently but still in use by some community members.
'Vagrant Devstack' refers to the old VM-based devstack which was replaced by Docker-based devstack eight years ago. We do not retain any support for this.
'pavelib' is the pavelib folder in edx-platform that defines Paver tasks. In other words, these commands are only used by other Paver tasks.
The paver uninstall_python_packages command is being removed without replacement. It is not used by Ansible or Devstack as far as I can tell. Operators who need to uninstall specific Python packages can do so by running pip uninstall as needed.

Django settings (for Asset commands)

In order to reimplement the Paver Asset commands without Python/Django, we are changing how several settings are configured.

The following new environment variables are available (set them in your system, not in Django):

STATIC_ROOT_LMS (path)
STATIC_ROOT_CMS (path)
COMPREHENSIVE_THEME_DIRS (colon-separated paths)
WEBPACK_CONFIG_PATH (path)
JS_ENV_EXTRA_CONFIG (json)

The following Django settings are becoming read-only mirrors (do not override them):

LMS:
- STATIC_ROOT (a string, loaded from STATIC_ROOT_LMS env var)
- COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)
CMS:
- STATIC_ROOT (a string, loaded from STATIC_ROOT_CMS env var)
- COMPREHENSIVE_THEME_DIRS (loaded from the env var, parsed into a list of strings)

The following Django settings are being removed (overriding them will have no effect):

LMS/CMS:

STATIC_ROOT_BASE
WEBPACK_CONFIG_PATH
JS_ENV_EXTRA_CONFIG

If you previously set STATIC_ROOT_BASE to /blah, then you should now set environment variables:

STATIC_ROOT_LMS=/blah
STATIC_ROOT_LMS=/blah/studio

Deprecation

In time for Redwood, deprecation warnings will be added to all edx-platform Paver Asset commands.

Removal

In edx-platform:
- rm -rf ./pavelib/
- rm ./requirements/edx/paver.in
- rm ./requirements/edx/paver.txt
- Search docs for 'paver' and replace with new commands
Elsewhere in the openedx GitHub org:
- Search docs for 'paver' and replace with new commands
In Tutor:
- Replace paver Asset references with new commands

Tasks/PRs

Give feedback

Remove special XModule asset handling #31624

7 of 7

code health epic
refactor!: remove unused update-assets-dev.sh #31810
build: move collectstatic ignore patterns into configuration #31934
docs: add ADR: "Reimplement edx-platform static asset processing" #31790
build: copy from node_modules using NPM postinstall hook, not Paver #32717
build: copy from node_modules using NPM postinstall hook, not Paver (RE-MERGE) #32767
docs: update static assets ADR with the new plan #32804
build: npm run build (experimental) #32823
build: npm run watch (experimental) #32866
build: compile/watch sass with new npm scripts #34318

create-sandbox
test: remove some unused paver references from CI scripts #34468
build: finish replacing paver assets #34554
feat!: remove dependency on Paver scripts overhangio/tutor#1042
Paver Removal 1/3 #34829
Paver Removal 2/3 #34830
Replace Paver quality and js_test commands #34845

0 of 7

epic
Paver Removal 3/3 #34832
TODO update all edx-platform documentation
TODO update external documentation
Options

The text was updated successfully, but these errors were encountered:

All CI used to go through scripts/generic-ci-tests.sh, which is a wrapper around various `paver` test/linting/check invocations. These days, most edx-platform CI checks just invoke their tools (pylint, pycodestyle, pytest, etc.) directly. In anticipation of the proposed Paver deprecation [1], let's remove the parts of this script that aren't used any more, including several `paver` command invocations. This should have no impact on CI. [1] openedx#34467

All CI used to go through scripts/generic-ci-tests.sh, which is a wrapper around various `paver` test/linting/check invocations. These days, most edx-platform CI checks just invoke their tools (pylint, pycodestyle, pytest, etc.) directly. In anticipation of the proposed Paver deprecation [1], let's remove the parts of this script that aren't used any more, including several `paver` command invocations. This should have no impact on CI. Furthermore, we are able to remove the SHARD environment variable, which was formely used to split unit and quality checks up into smaller pieces. Unit tests and pylint checks now have their own separate sharding logic, so there is only one "quality" shard remaining (SHARD=4, ie generic quality checks), thus we don't need a SHARD variable at all. [1] openedx#34467

All CI used to go through scripts/generic-ci-tests.sh, which is a wrapper around various `paver` test/linting/check invocations. These days, most edx-platform CI checks just invoke their tools (pylint, pycodestyle, pytest, etc.) directly. In anticipation of the proposed Paver deprecation [1], let's remove the parts of this script that aren't used any more, including several `paver` command invocations. This should have no impact on CI. Furthermore, we are able to remove the SHARD environment variable, which was formely used to split unit and quality checks up into smaller pieces. Unit tests and pylint checks now have their own separate sharding logic, so there is only one "quality" shard remaining (SHARD=4, ie generic quality checks), thus we don't need a SHARD variable at all. [1] #34467

) All CI used to go through scripts/generic-ci-tests.sh, which is a wrapper around various `paver` test/linting/check invocations. These days, most edx-platform CI checks just invoke their tools (pylint, pycodestyle, pytest, etc.) directly. In anticipation of the proposed Paver deprecation [1], let's remove the parts of this script that aren't used any more, including several `paver` command invocations. This should have no impact on CI. Furthermore, we are able to remove the SHARD environment variable, which was formely used to split unit and quality checks up into smaller pieces. Unit tests and pylint checks now have their own separate sharding logic, so there is only one "quality" shard remaining (SHARD=4, ie generic quality checks), thus we don't need a SHARD variable at all. [1] openedx#34467

Together, these changes make it so that all features of the Paver-based asset compilation system are supported with drop-in Paver-free replacements. The remaining Paver asset functions are trivial wrappers, which can be comfortably deleted before Sumac * Turn `./manage.py ... compile_sass` into a simple wrapper around `npm run compile-sass` * Turn `paver webpack` into a simple wrapper around `npm run webpack` * Turn `pavelib.assets:collect_assets` into a simple wrapper around `./manage.py ... collectstatic` * Add/improve deprecation warnings for all Paver asset commands. * Load defaults for asset-related Django settings from environment variables. This allows the build to work without Python. For the settings which will be removed in Sumac, I've added deprecation warnings. * Change EDX_PLATFORM_THEME_DIRS env var to COMPREHENSIVE_THEME_DIRS. This simplifies the migration instructions, because all the new env vars now match their corresponding Django settings. This amends an ADR, but it should not be a breaking change because the env var was recently added (since Quince) and nobody should be using it yet. * Future-proof the static assets ADR with links. The linked pages will be kept up-to-date even if the ADR isn't. Part of: openedx#34467

Together, these changes make it so that all features of the Paver-based asset compilation system are supported with drop-in Paver-free replacements. The remaining Paver asset functions are trivial wrappers, which can be comfortably deleted before Sumac. * Turn `./manage.py ... compile_sass` into a simple wrapper around `npm run compile-sass` * Turn `paver webpack` into a simple wrapper around `npm run webpack` * Turn `pavelib.assets:collect_assets` into a simple wrapper around `./manage.py ... collectstatic` * Add/improve deprecation warnings for all Paver asset commands. * Load defaults for asset-related Django settings from environment variables. This allows the build to work without Python. For the settings which will be removed in Sumac, I've added deprecation warnings. * Change EDX_PLATFORM_THEME_DIRS env var to COMPREHENSIVE_THEME_DIRS. This simplifies the migration instructions, because all the new env vars now match their corresponding Django settings. This amends an ADR, but it should not be a breaking change because the env var was recently added (since Quince) and nobody should be using it yet. * Future-proof the static assets ADR with links. The linked pages will be kept up-to-date even if the ADR isn't. Part of: #34467

kdmccormick · 2024-05-07T17:52:10Z

I have updated the Django settings to clarify that we will also be removing the STATIC_ROOT_BASE Django setting.

This is the first step of Pave removal. These are the parts that we are confident are unused. Part of: openedx#34467

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: openedx#34467

TODO describe merge timing concerns Part of: openedx#34467

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: openedx#34467

WIP Closes: openedx#34467

kdmccormick · 2024-05-21T18:02:58Z

Now that this is accepted, I'll split the removal into three steps:

Step 1 is free of downstream contingencies. It is safe to happen at any time.

For Step 2, we'll coordinate with early deployers (2U) so that they have time to update their deployment scripts to avoid the assets and prereqs commands. This step will deliver a speed-up to the production Tutor build.

For Step 3, we'll need to move the remaining quality/js checks out of Paver, uninstall Paver, and a do bulk sweep through the docs and update all instructions that still specify "paver".

feanil · 2024-05-21T18:48:16Z

FYI, I created #34834 to handle a specific github workflow around paver assets that also needs an enhancement.

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: #34467

TODO describe merge timing concerns Part of: openedx#34467

These packages were installed transitively through paver.in, but they are used as direct dependencies in edx-platform application code: * psutil * pymemcache * wrapt Since we are demoting paver.in to be a dev-only dependency (with plans to remove paver.in entirely), we need to make those three packages explicit dependencies in kernel.in Part of: openedx#34467

TODO describe merge timing concerns Part of: openedx#34467

kdmccormick · 2024-05-22T15:45:10Z

That workflow update ^ is now handled.

Here is an epic to handle the remaining upstream CI updates:

Replace Paver quality and js_test commands #34845

TODO details Part of: openedx#34467

github-actions bot added the depr Proposal for deprecation & removal per OEP-21 label Apr 3, 2024

kdmccormick mentioned this issue Apr 3, 2024

[DEPR]: Asset processing in Paver #31895

Closed

kdmccormick self-assigned this Apr 3, 2024

kdmccormick mentioned this issue Apr 3, 2024

test: remove some unused paver references from CI scripts #34468

Merged

robrap mentioned this issue Apr 29, 2024

Create arch-bom tickets for 2U specific DEPR follow-up edx/edx-arch-experiments#612

Open

kdmccormick mentioned this issue May 1, 2024

build: finish replacing paver assets #34554

Merged

kdmccormick mentioned this issue May 7, 2024

feat!: remove dependency on Paver scripts overhangio/tutor#1042

Merged

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 21, 2024

feat!: remove paver docs, i18n, and servers

79f2b8a

This is the first step of Pave removal. These are the parts that we are confident are unused. Part of: openedx#34467

kdmccormick mentioned this issue May 21, 2024

Paver Removal 1/3 #34829

Merged

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 21, 2024

feat!: remove paver docs, i18n, servers, and pylint

87e4943

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: openedx#34467

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 21, 2024

feat!: remove paver assets, prereqs

f3bb192

TODO describe merge timing concerns Part of: openedx#34467

kdmccormick mentioned this issue May 21, 2024

Paver Removal 2/3 #34830

Draft

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 21, 2024

feat!: remove paver docs, i18n, servers, and pylint

e6ebfcd

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: openedx#34467

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 21, 2024

feat!: remove final Pvaer quality commands

c7b4d36

WIP Closes: openedx#34467

kdmccormick mentioned this issue May 21, 2024

Paver Removal 3/3 #34832

Draft

feanil mentioned this issue May 21, 2024

Build and Test both the Dev and Production Webpack Asset Process #34834

Closed

kdmccormick added a commit that referenced this issue May 21, 2024

feat!: remove paver docs, i18n, servers, and pylint

c4a1f80

This is the first step of Paver removal. These are the parts that we are confident are unused. Part of: #34467

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 22, 2024

feat!: remove paver assets, prereqs

3772314

TODO describe merge timing concerns Part of: openedx#34467

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 22, 2024

feat!: remove paver assets, prereqs

f91bf1c

TODO describe merge timing concerns Part of: openedx#34467

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 22, 2024

feat!: remove paver assets, prereqs

e2873c3

TODO describe merge timing concerns Part of: openedx#34467

kdmccormick mentioned this issue May 22, 2024

Replace Paver quality and js_test commands #34845

Open

kdmccormick added a commit to kdmccormick/edx-platform that referenced this issue May 22, 2024

feat!: remove paver assets, prereqs

bc77e38

TODO details Part of: openedx#34467

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DEPR]: Paver #34467

[DEPR]: Paver #34467

kdmccormick commented Apr 3, 2024 •

edited

Tasks/PRs

kdmccormick commented May 7, 2024

kdmccormick commented May 21, 2024 •

edited

feanil commented May 21, 2024

kdmccormick commented May 22, 2024

[DEPR]: Paver #34467

[DEPR]: Paver #34467

Comments

kdmccormick commented Apr 3, 2024 • edited

Timeline

Communicated

Target Acceptance

Earliest Open edX Named Release Without This Functionality

Latest Open edX Named Release With the Replacement Available

Rationale

Replacement

Migration

Requirements

Commands

Django settings (for Asset commands)

Deprecation

Removal

Tasks/PRs

kdmccormick commented May 7, 2024

kdmccormick commented May 21, 2024 • edited

feanil commented May 21, 2024

kdmccormick commented May 22, 2024

kdmccormick commented Apr 3, 2024 •

edited

kdmccormick commented May 21, 2024 •

edited