Offline mode

[andreoliwa]

[@sobolevn wrote this originally.
I used hub pull-request --draft -i 129 -b andreoliwa:master -h andreoliwa:offline-mode, and then it replaced the description... 😐 ]

Hi!

I faced an issue with running nitpick while offline.
That's what it shows:

» flake8 .
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/util/connection.py", line 61, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/Users/sobolev/.pyenv/versions/3.7.4/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x10d03cc10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /wemake-services/wemake-python-styleguide/master/styles/nitpick-style-wemake.toml (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10d03cc10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/sobolev/.pyenv/versions/3.7.4/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/Users/sobolev/.pyenv/versions/3.7.4/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/checker.py", line 666, in _run_checks
    return checker.run_checks()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/checker.py", line 598, in run_checks
    self.run_ast_checks()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/checker.py", line 502, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/plugin.py", line 50, in run
    Nitpick.current_app().config.merge_styles(), self.check_files(True), self.check_files(False)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/config.py", line 65, in merge_styles
    style.find_initial_styles(configured_styles)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/style.py", line 61, in find_initial_styles
    self.include_multiple_styles(chosen_styles)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/style.py", line 76, in include_multiple_styles
    style_path = self.get_style_path(style_uri)  # type: Optional[Path]
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/style.py", line 105, in get_style_path
    style_path = self.fetch_style_from_url(clean_style_uri)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/nitpick/style.py", line 134, in fetch_style_from_url
    response = requests.get(new_url)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /wemake-services/wemake-python-styleguide/master/styles/nitpick-style-wemake.toml (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10d03cc10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/sobolev/Documents/github/returns/.venv/bin/flake8", line 10, in <module>
    sys.exit(main())
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/main/cli.py", line 18, in main
    app.run(argv)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/main/application.py", line 393, in run
    self._run(argv)
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/main/application.py", line 381, in _run
    self.run_checks()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/main/application.py", line 300, in run_checks
    self.file_checker_manager.run()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/checker.py", line 329, in run
    self.run_parallel()
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/flake8/checker.py", line 293, in run_parallel
    for ret in pool_map:
  File "/Users/sobolev/.pyenv/versions/3.7.4/lib/python3.7/multiprocessing/pool.py", line 354, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/Users/sobolev/.pyenv/versions/3.7.4/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
requests.exceptions.ConnectionError: None: Max retries exceeded with url: /wemake-services/wemake-python-styleguide/master/styles/nitpick-style-wemake.toml (Caused by None)

I guess that there two things to be improved:

1. --offline flag to skip nitpick check
2. Better exception message that would say: "you probably should add --offline flag because it seems that your network is unreachable. Or fix your connection."

What do you think?

[andreoliwa]

Hello again!

I don't know about the --offline flag...
You can use flake8 --ignore=NIP to ignore all nitpick errors.

But I agree with a better error message when a ConnectionError happens.
I sometimes get this message myself.

I've been thinking of a solution for a while:

1. Caching the style files locally for a certain time (1 hour), to avoid HTTP requests all the time;
2. Maybe make this period configurable in pyproject.toml under [tool.nitpick];
3. Add a CLI option to force cache refreshing.

Would a caching solution like this work for you?

[sobolevn]

It would, only if I had a connection, but don't have right now. Imagine, that you are writing a new project and copy-pasting an existing nitpick configuration.

Considering --nitpick-offline and --ignore=NIP for me it is not the same things. Because --nitpick-offline is semantic, documented, and hopefully won't raise any exceptions. And --ignore=NIP is a hack that will raise ConnectionErrors.

I like clean solutions more 🙂

[bibz]

This would be a neat feature!
Agreed that --nitpick-offline carries more meaning.

@sobolevn How do you invoke nitpick?

We embed it in our pre-commit hook and so passing arguments to it implies changing the hook configuration. Not acceptable for us.

Though when run as a pre-commit hook, one can simply use SKIP=nitpick git commit instead of using a new flag.

---

Regarding caching, I created a new feature (#130) request as I believe it is an orthogonal idea.

[sobolevn]

I use wemake-python-styleguide and run it with flake8 .
So, that's pretty useful to me. Env variable also works fine!

[andreoliwa]

Thank you guys for sharing your ideas and how you invoke Nitpick.
I also use it as a pre-commit hook.

And --ignore=NIP is a hack that will raise ConnectionErrors.

You're right, I tried using it and it raises errors.
I thought that the styles would not be downloaded on --ignore=NIP, but I was wrong.
Sorry about that.

Here are my suggested specs:

1. "Offline mode" means that no style will be downloaded (no HTTP requests at all).
2. Hence, no connection errors should be raised.
3. If no local style file if found, no checks will be performed. But Nitpick should still quit with exit code 0 (success).
4. Use --nitpick-offline flag to activate offline mode.
5. Alternatively use an env var NITPICK_OFFLINE to activate offline mode.
6. If a connection error happens when in online mode, display a message similar to "you probably should add --nitpick-offline flag because it seems that your network is unreachable. Or fix your connection" and fail with some exit code greater than 0.

Does that work for you?
Any other suggestions?

[sobolevn]

Sounds like a plan! 👍

P.S. nitpick is so cool, thanks for building it! 🚀

[andreoliwa]

@sobolevn @bibz it's ready.
Thanks for your valuable help (and for the patience waiting for this).

Would you like to review it officially, or can I just merge and release?

and fail with some exit code greater than 0

I chose to display a message on stderr if there is a connection error instead.
I didn't want to use sys.exit(1), because I would interrupt flake8 execution, and Nitpick is just a plugin.

[sobolevn]

@andreoliwa thanks a lot!

Two questions (I am not very familiar with the codebase to figure this out):

1. Will nitpick still work offline for local path-based configs?
2. Will it cache configs from the network? (It looks like there's some kind of caching and I just want to be 100% sure)

[andreoliwa]

Hi @sobolevn,

1. Will nitpick still work offline for local path-based configs?

Yes. There was a bug 😬, but I added tests for this situation and now it is working.

2. Will it cache configs from the network? (It looks like there's some kind of caching and I just want to be 100% sure)

There is a "merged style file", rebuilt on every execution: all the style files are gathered and joined into a single TOML file inside the .cache/nitpick dir.
A proper cache will be made on #130.

[sobolevn]

Awesome work 🎉

Thanks a lot for this project.

P.S. We also building an awesome --baseline feature for wemake-python-styleguide: wemake-services/wemake-python-styleguide#1276 It allows to integrate our linter into any existing codebase. Check it out!

[andreoliwa]

🎉 This PR is included in version 0.22.0 🎉

The release is available on:

• v0.22.0
• GitHub release

Your semantic-release bot 📦🚀