py38: AttributeError: Can't pickle local object 'ParallelTasks.add_task.<locals>.<lambda>'

[ssbarnea]

Describe the bug
Attempt to build docs with python3.8 throws the exception below. It does not reproduce with other versions of python, like 2.7, 3.6 or 3.7

Exception occurred:
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

To Reproduce
Steps to reproduce the behavior:

<Paste your command-line here which cause the problem>
git clone https://github.com/ansible/ansible-lint
cd ansible-lint
# assuming that python points to python3.8 ! (use pyenv if needed)
tox -e docs

Expected behavior
Building docs should pass, just like with other python versions.

Your project
https://github.com/ansible/ansible-lint/pull/621/checks?check_run_id=289392168

Screenshots
If applicable, add screenshots to help explain your problem.

Environment info

• OS: [e.g. Unix/Linux/Mac/Win/other with version]
• Python version: [e.g. 3.7.1]
• Sphinx version: [e.g. 1.8.2]
• Sphinx extensions: [e.g. sphinx.ext.autodoc, recommonmark]
• Extra tools: [e.g. Browser, tex or something else]

Additional context

$ tox -re docs                                                                                                                                                            [17:26:21]
.tox create: /Users/ssbarnea/os/ansible-lint/.tox/.tox
.tox installdeps: setuptools >= 41.4.0, pip >= 19.3.0, tox >= 3.5.3
.package recreate: /Users/ssbarnea/os/ansible-lint/.tox/.package
.package installdeps: setuptools >= 41.4.0, setuptools_scm >= 1.15.0, setuptools_scm_git_archive >= 1.0, wheel, ansible, pathlib2; python_version < '3.2', ruamel.yaml, six
docs recreate: /Users/ssbarnea/os/ansible-lint/.tox/docs
docs installdeps: sphinx
docs inst: /Users/ssbarnea/os/ansible-lint/.tox/.tmp/package/1/ansible-lint-4.1.1a5.dev2+gabfd2a9.d20191108.tar.gz
docs installed: alabaster==0.7.12,ansible==2.9.0,ansible-lint==4.1.1a5.dev2+gabfd2a9.d20191108,Babel==2.7.0,certifi==2019.9.11,cffi==1.13.2,chardet==3.0.4,cryptography==2.8,docutils==0.15.2,idna==2.8,imagesize==1.1.0,Jinja2==2.10.3,MarkupSafe==1.1.1,packaging==19.2,pycparser==2.19,Pygments==2.4.2,pyparsing==2.4.4,pytz==2019.3,PyYAML==5.1.2,requests==2.22.0,ruamel.yaml==0.16.5,six==1.13.0,snowballstemmer==2.0.0,Sphinx==2.2.1,sphinxcontrib-applehelp==1.0.1,sphinxcontrib-devhelp==1.0.1,sphinxcontrib-htmlhelp==1.0.2,sphinxcontrib-jsmath==1.0.1,sphinxcontrib-qthelp==1.0.2,sphinxcontrib-serializinghtml==1.1.3,urllib3==1.25.6
docs run-test-pre: PYTHONHASHSEED='2023353715'
docs run-test: commands[0] | make -C docs/docsite htmldocs
cat _themes/srtd/static/css/theme.css | sed -e 's/^[ 	]*//g; s/[ 	]*$//g; s/\([:{;,]\) /\1/g; s/ {/{/g; s/\/\*.*\*\///g; /^$/d' | sed -e :a -e '$!N; s/\n\(.\)/\1/; ta' > _themes/srtd/static/css/theme.min.css
cd ../../lib/ansiblelint; python ./generate_docs.py
../../docs/docsite/rst/rules/default_rules.rst file written
CPUS=8 /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.sphinx html
sphinx-build -M html "rst" "_build" -j 8 -n -w rst_warnings
Running Sphinx v2.2.1
loading pickled environment... done
loading intersphinx inventory from ../python2-2.7.13.inv...
loading intersphinx inventory from ../python3-3.6.2.inv...
loading intersphinx inventory from ../jinja2-2.9.7.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 10 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] rules/default_rules

looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [ 11%] configuring/index

Exception occurred:
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ParallelTasks.add_task.<locals>.<lambda>'
The full traceback has been saved in /var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/sphinx-err-pyj1vvlk.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make[1]: *** [html] Error 2
make: *** [htmldocs] Error 2
ERROR: InvocationError for command /usr/bin/make -C docs/docsite htmldocs (exited with code 2)
______________________________________________________________________________________ summary _________

# Sphinx version: 2.2.1
# Python version: 3.8.0 (CPython)
# Docutils version: 0.15.2 release
# Jinja2 version: 2.10.3
# Last messages:
#   
#   looking for now-outdated files...
#   none found
#   pickling environment...
#   done
#   checking consistency...
#   done
#   preparing documents...
#   done
#   writing output... [ 11%] configuring/index
# Loaded extensions:
#   sphinx.ext.mathjax (2.2.1) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.1) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.1) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.2) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.3) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.2) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc (2.2.1) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.intersphinx (2.2.1) from /Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/ext/intersphinx.py
Traceback (most recent call last):
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/application.py", line 346, in build
    self.builder.build_update()
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 297, in build_update
    self.build(to_build,
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 361, in build
    self.write(docnames, list(updated_docnames), method)
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 532, in write
    self._write_parallel(sorted(docnames),
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 572, in _write_parallel
    tasks.add_task(write_process, arg)
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/util/parallel.py", line 96, in add_task
    self._join_one()
  File "/Users/ssbarnea/os/ansible-lint/.tox/docs/lib/python3.8/site-packages/sphinx/util/parallel.py", line 120, in _join_one
    self._procs[newtid].start()
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/context.py", line 283, in _Popen
    return Popen(process_obj)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/Users/ssbarnea/.pyenv/versions/3.8.0/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ParallelTasks.add_task.<locals>.<lambda>'

[tk0miya]

It seems start method of multiprocessing has been changed in macOS. I guess this would be related with this issue. @ssbarnea Is your environment is macOS, right?
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods

Current design of Sphinx expects to use fork-model for the start method. We need to change its architecture...

[tk0miya]

I determined to disable the feature. And post an issue to reimplement it again as #6881. I can't say it how long time to do that. But current implementation is not recommended by python team.

[tk0miya]

Fixed by #6879
Thank you for reporting!