Papermill fails when injecting parameters

[VoLKyyyOG]

I have tried running papermill over a test notebook on my newly refreshed WSL2 environment and seem to have come across an issue. (I really did triple make sure it wasn't a typo at the start)

Here's the test notebook:

OS: Windows 10 WSL2 (Ubuntu 20.04)
Python Version: 3.8.5
Papermill Version: 2.2.2

The thing is, I've worked with papermill previously on the same device but with WSL1 (Ubuntu 18.04) and also for work (Windows 10, Python 3.7.3) with no issues whatsoever. However, I seem to be getting an issue whenever I pass through parameters today.

Here's the code I used to run it:

import papermill as pm

pm.execute_notebook(
    './print_msg.ipynb',
    './output/print_msg.ipynb',
    parameters={"p1": "please work :("}
)

and here's the error output:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-51ea58f8c568> in <module>
      1 import papermill as pm
      2 
----> 3 pm.execute_notebook(
      4     './print_msg.ipynb',
      5     './output/print_msg.ipynb',

~/.local/lib/python3.8/site-packages/papermill/execute.py in execute_notebook(input_path, output_path, parameters, engine_name, request_save_on_cell_execute, prepare_only, kernel_name, progress_bar, log_output, stdout_file, stderr_file, start_timeout, report_mode, cwd, **engine_kwargs)
     81         # Parameterize the Notebook.
     82         if parameters:
---> 83             nb = parameterize_notebook(nb, parameters, report_mode)
     84 
     85         nb = prepare_notebook_metadata(nb, input_path, output_path, report_mode)

~/.local/lib/python3.8/site-packages/papermill/parameterize.py in parameterize_notebook(nb, parameters, report_mode, comment)
     78 
     79     # Generate parameter content based on the kernel_name
---> 80     param_content = translate_parameters(kernel_name, language, parameters, comment)
     81 
     82     newcell = nbformat.v4.new_code_cell(source=param_content)

~/.local/lib/python3.8/site-packages/papermill/translators.py in translate_parameters(kernel_name, language, parameters, comment)
    552 
    553 def translate_parameters(kernel_name, language, parameters, comment='Parameters'):
--> 554     return papermill_translators.find_translator(kernel_name, language).codify(parameters, comment)

~/.local/lib/python3.8/site-packages/papermill/translators.py in codify(cls, parameters, comment)
    187         if sys.version_info >= (3, 6):
    188             # Put content through the Black Python code formatter
--> 189             import black
    190 
    191             fm = black.FileMode(string_normalization=False)

~/.local/lib/python3.8/site-packages/black/__init__.py in <module>
    486 )
    487 @click.pass_context
--> 488 def main(
    489     ctx: click.Context,
    490     code: Optional[str],

~/.local/lib/python3.8/site-packages/click/decorators.py in decorator(f)
    168             attrs['help'] = inspect.cleandoc(attrs['help'])
    169         OptionClass = attrs.pop('cls', Option)
--> 170         _param_memo(f, OptionClass(param_decls, **attrs))
    171         return f
    172     return decorator

~/.local/lib/python3.8/site-packages/click/core.py in __init__(self, param_decls, show_default, prompt, confirmation_prompt, hide_input, is_flag, flag_value, multiple, count, allow_from_autoenv, type, help, **attrs)
   1458                  type=None, help=None, **attrs):
   1459         default_is_missing = attrs.get('default', _missing) is _missing
-> 1460         Parameter.__init__(self, param_decls, type=type, **attrs)
   1461 
   1462         if prompt is True:

TypeError: __init__() got an unexpected keyword argument 'hidden'

The CLI call also results in the same error:

papermill /mnt/c/users/akira/documents/github/airflow-test/print_msg.ipynb /mnt/c/users/akira/documents/github/airflow-test/output/output.ipynb -p p1 test

Input Notebook:  /mnt/c/users/akira/documents/github/airflow-test/print_msg.ipynb
Output Notebook: /mnt/c/users/akira/documents/github/airflow-test/output/airflow.ipynb
Generating grammar tables from /home/akira/.local/lib/python3.8/site-packages/blib2to3/Grammar.txt
Writing grammar tables to /home/akira/.cache/black/20.8b1/Grammar3.8.5.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/akira/.cache/black/20.8b1/tmpauwl4xr3'
Generating grammar tables from /home/akira/.local/lib/python3.8/site-packages/blib2to3/PatternGrammar.txt
Writing grammar tables to /home/akira/.cache/black/20.8b1/PatternGrammar3.8.5.final.0.pickle
Writing failed: [Errno 2] No such file or directory: '/home/akira/.cache/black/20.8b1/tmph8la99cw'
Traceback (most recent call last):
  File "/home/akira/.local/bin/papermill", line 8, in <module>
    sys.exit(papermill())
  File "/home/akira/.local/lib/python3.8/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/akira/.local/lib/python3.8/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/akira/.local/lib/python3.8/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/akira/.local/lib/python3.8/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/akira/.local/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/akira/.local/lib/python3.8/site-packages/papermill/cli.py", line 240, in papermill
    execute_notebook(
  File "/home/akira/.local/lib/python3.8/site-packages/papermill/execute.py", line 91, in execute_notebook
    nb = parameterize_notebook(nb, parameters, report_mode)
  File "/home/akira/.local/lib/python3.8/site-packages/papermill/parameterize.py", line 80, in parameterize_notebook
    param_content = translate_parameters(kernel_name, language, parameters, comment)
  File "/home/akira/.local/lib/python3.8/site-packages/papermill/translators.py", line 554, in translate_parameters
    return papermill_translators.find_translator(kernel_name, language).codify(parameters, comment)
  File "/home/akira/.local/lib/python3.8/site-packages/papermill/translators.py", line 189, in codify
    import black
  File "/home/akira/.local/lib/python3.8/site-packages/black/__init__.py", line 488, in <module>
    def main(
  File "/home/akira/.local/lib/python3.8/site-packages/click/decorators.py", line 170, in decorator
    _param_memo(f, OptionClass(param_decls, **attrs))
  File "/home/akira/.local/lib/python3.8/site-packages/click/core.py", line 1460, in __init__
    Parameter.__init__(self, param_decls, type=type, **attrs)
TypeError: __init__() got an unexpected keyword argument 'hidden'

Any advice on how I can debug this?

(It's my first time ever opening an issue, so I hope this was okay)

[rgbkrk]

Can you share your version of click and papermill? The (literal) hidden variable name doesn't show up in papermill source (at least on main), so I'm wondering if there's a version mismatch. You can run pip freeze to find out.

[VoLKyyyOG]

Thanks for the reply! Here's the list of packages atm:

absl-py==0.11.0
alembic==1.4.2
ansiwrap==0.8.4
apache-airflow==1.10.12
apispec==1.3.3
appdirs==1.4.4
argcomplete==1.12.0
argon2-cffi==20.1.0
astroid==2.4.2
astunparse==1.6.3
async-generator==1.10
attrs==19.3.0
Automat==0.8.0
Babel==2.8.0
backcall==0.2.0
black==20.8b1
bleach==3.2.1
blinker==1.4
bokeh==2.2.3
branca==0.4.1
cached-property==1.5.1
cachetools==4.1.1
cattrs==1.0.0
certifi==2020.6.20
cffi==1.14.3
chardet==3.0.4
click==6.7
click-plugins==1.1.1
cligj==0.7.0
cloud-init==20.3
colorama==0.4.3
colorlog==4.0.2
command-not-found==0.3
configobj==5.0.6
configparser==3.5.3
constantly==15.1.0
croniter==0.3.34
cryptography==2.8
cycler==0.10.0
dbus-python==1.2.16
decorator==4.4.2
defusedxml==0.6.0
dill==0.3.2
distro==1.4.0
distro-info===0.23ubuntu1
dnspython==1.16.0
docutils==0.16
email-validator==1.1.1
entrypoints==0.3
Fiona==1.8.17
Flask==1.1.2
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.4
Flask-Babel==1.0.0
Flask-Caching==1.3.3
Flask-JWT-Extended==3.24.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.4
flask-swagger==0.2.14
Flask-WTF==0.14.3
folium==0.11.0
funcsigs==1.0.2
future==0.18.2
fuzzywuzzy==0.18.0
gast==0.3.3
geopandas==0.8.1
google-auth==1.23.0
google-auth-oauthlib==0.4.2
google-pasta==0.2.0
graphviz==0.14.1
grpcio==1.33.2
gunicorn==20.0.4
h5py==2.10.0
httplib2==0.14.0
hyperlink==19.0.0
idna==2.10
importlib-metadata==1.5.0
incremental==16.10.1
ipykernel==5.3.4
ipython==7.19.0
ipython-genutils==0.2.0
ipywidgets==7.5.1
iso8601==0.1.12
isort==5.6.4
itsdangerous==1.1.0
jedi==0.17.2
Jinja2==2.11.2
joblib==0.17.0
json-merge-patch==0.2
jsonpatch==1.22
jsonpointer==2.0
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==6.1.7
jupyter-console==6.2.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.6.3
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1
jupyterlab-pygments==0.1.2
Keras-Preprocessing==1.1.2
keyring==18.0.1
kiwisolver==1.3.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lazy-object-proxy==1.4.3
lockfile==0.12.2
lxml==4.6.1
Mako==1.1.3
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==2.21.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.23.1
matplotlib==3.3.3
mccabe==0.6.1
mistune==0.8.4
more-itertools==4.2.0
munch==2.5.0
mypy-extensions==0.4.3
natsort==7.0.1
nbclient==0.5.1
nbconvert==6.0.7
nbformat==5.0.8
nbgrader==0.6.1
nest-asyncio==1.4.3
netifaces==0.10.4
notebook==6.1.5
numpy==1.19.1
oauthlib==3.1.0
opt-einsum==3.3.0
packaging==20.4
pandas==1.1.0
pandocfilters==1.4.3
papermill==2.2.2
parso==0.7.1
pathspec==0.8.1
patsy==0.5.1
pendulum==1.4.4
pexpect==4.6.0
pickleshare==0.7.5
Pillow==8.0.1
prison==0.1.3
prometheus-client==0.9.0
prompt-toolkit==3.0.8
protobuf==3.14.0
psutil==5.7.2
ptyprocess==0.6.0
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.20
Pygments==2.6.1
PyGObject==3.36.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pylint==2.6.0
pymacaroons==0.13.0
PyNaCl==1.3.0
pyodbc==4.0.30
pyOpenSSL==19.0.0
pyparsing==2.4.7
pyproj==3.0.0.post1
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.1
python-daemon==2.2.4
python-dateutil==2.8.1
python-debian===0.1.36ubuntu1
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2020.1
pytzdata==2020.1
PyYAML==5.3.1
pyzmq==20.0.0
qtconsole==4.7.7
QtPy==1.9.0
regex==2020.11.13
requests==2.24.0
requests-oauthlib==1.3.0
requests-unixsocket==0.2.0
rsa==4.6
scikit-learn==0.23.2
scipy==1.5.4
seaborn==0.11.0
SecretStorage==2.3.1
Send2Trash==1.5.0
service-identity==18.1.0
setproctitle==1.1.10
Shapely==1.7.1
simplejson==3.16.0
six==1.15.0
sklearn==0.0
sos==4.0
SQLAlchemy==1.3.18
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.36.8
ssh-import-id==5.10
statsmodels==0.12.1
systemd-python==234
tabulate==0.8.7
tenacity==4.12.0
tensorboard==2.4.0
tensorboard-plugin-wit==1.7.0
tensorflow==2.3.1
tensorflow-estimator==2.3.0
termcolor==1.1.0
terminado==0.9.1
testpath==0.4.4
text-unidecode==1.3
textwrap3==0.9.2
threadpoolctl==2.1.0
thrift==0.13.0
toml==0.10.2
tornado==6.1
tqdm==4.52.0
traitlets==5.0.5
Twisted==18.9.0
typed-ast==1.4.1
typing-extensions==3.7.4.3
tzlocal==1.5.1
ubuntu-advantage-tools==20.3
ufw==0.36
unattended-upgrades==0.1
unicodecsv==0.14.1
urllib3==1.25.10
wadllib==1.3.3
wcwidth==0.2.5
webencodings==0.5.1
Werkzeug==0.16.1
widgetsnbextension==3.5.1
wrapt==1.12.1
WTForms==2.3.3
zipp==1.0.0
zope.deprecation==4.4.0
zope.interface==4.7.1

[MSeal]

Looks like it's this issue in black: psf/black#1628
The root problem is your click version is too old. Try updating that to click>=7.1.2.

[VoLKyyyOG]

That solved it! Many thanks 👍