Weekly Digest (26 July, 2020 - 2 August, 2020) #4315

Co-authored-by: Saugat Pachhai suagatchhetri@outlook.com](ed1c3a7) by efiop
🛠️ [tree: get rid of explicit tree arguments (#4310) * cache: don't make save() compute the hash

This is incorrect and leads to weird bugs like #4305

tree: get rid of explicit tree arguments](fcf2e35) by efiop
🛠️ dvc: bump to 1.3.0 by efiop
🛠️ Use pytest>=6.0.1 (#4308) pytest 6.0.0 had issue with pylint, complaining for pytest.mark.* being not-callable. by skshetry
🛠️ commit: make sure deps/outs exist (#4305) Fixes #4266 by efiop
🛠️ [dvc: fix --quiet mode (dvc: fix --quiet mode #4304) Part of the issue is a bug from logger: add trace, count --verbose/--quiet #3806

Kudos to @karajan1001 for finding this bug while working on #4282](815444f) by efiop
🛠️ [tree,remote: add support for WebDAV (#4256) * tree,remote: add support for webdav

Webdav support is based on https://pypi.org/project/webdavclient3/ and
supports basic download/upload operation, directory creation as well as
existence, file hash and isdir query. Copy, move and remove are also
implemented, though probably not used yet.

WebdavURLInfo is taken from https://github.com/shizacat/dvc/tree/remote-webdav

Fixes #1153

tree,remote: add further webdavclient3 options

Webdav token auth, certificate and key path and connection timeout are
configurable. Webdav username might be specified or extracted from URL.

Refs #1153

tree,remote: validate webdav client configuration

Refs #1153

tree,remote: WebDAV is written with capitalized 'DAV'...

Refs #1153

tree,remote: terminate WebDAV makedirs at self.path_info.path

This enables the WebDAV api location (e.g. '/public.php/webdav') to be
part of the remote 'url' configuration instead of beeing specified
separately via the 'root' option. The 'root' option may then be used to
specify real directories at the WebDAV storage, although using it to
set the api location is still possible.

Refs #1153

tree,remote: use >=3.14.5 for WebDAV dependency webdavclient3

Context: #4256 (comment)

Refs #1153

tree,remote: get rid of WebDAV 'root' option and add connection check

The WebDAV 'root' option was rather confusing and should be handled by
the initial 'path_info' from the config 'url' option.

Context: #4256 (comment)

While stripping the path/root from the hostname the port got lost, which
is fixed now by simply using the URLInfo 'replace' method as suggested.

Context: #4256 (comment)

The WebDAV client connection is tested by probing the existence of the
root (self.path_info.path).

Refs #1153

tree,remote: implement walk_files for WebDAV

Context: #4256 (comment)

Refs: #1153

tree,remote: let WebDAV client list query file info in walk_files

Context: #4256 (comment)

Refs #1153

tree,remote: add some unit tests for WebDAVTree

Refs #1153

tree,remote: use ConfigError for WebDAVTree and move Error to webdav.py

Context: #4256 (comment)

Refs #1153

tree,remote: remove/change some (unnecessary) comments

Context: #4256 (comment)

tree,remote: uploading to WebDAV only create directories if not exist
tree,remote: add BaseTree parameter use_dvcignore to WebDAVTree exists

Refs iterative/#1153

tree,remote: remove WebDAVTree copy method as proposed

Context: #4256 (comment)

Refs #1153

tree,remote: add progress bar to WebDAV _download and _upload method

Context: #4256 (comment)

Refs #1153

Co-authored-by: Christoph Berganski christoph.berganski@uni-bielefeld.de](2f48bb4) by iksnagreb
🛠️ [experiments: support local parallel execution in temp directories (#4257) * experiments: add initial local tmpdir executor

fix detached head pull
experiments: include unchanged (unrepro'd) stages in experiment hash
experiments: repro single experiment using tmpdir executor by default
stage experiments as stash commits before running
include repro args/kwargs in stashed experiments
support running arbitrary experiment commits (including stash commits)
experiments: use ProcessPoolExecutor
experiments: add dvc exp alias for dvc experiments
experiments: add dvc repro --experiment --queue
--queue can be used to stage an experiment for future execution
experiments: show queued (unexecuted) experiments in dvc exp show
revert ProcessPoolExecutor change
experiments: add dvc repro --run-all [--jobs]
--run-all can be used to run all queued experiments in parallel
cleanup output
experiments: use ProcessPoolExecutor
fix returning unpicklable objects error
update tests
fix windows cleanup issue
on windows tempdir cannot be removed if we are chdir'd into that
directory](c8d2f4f) by pmrowla
🛠️ [Exclude pylint==6.0 (Exclude pylint==6.0 #4300) https://github.com/pytest 6: pylint still shows not-callable error for builtin pytest marks even after #7473 and #7476 pytest-dev/pytest#7558
pylint throws error for pytest.mark.* functions in pytest6.

tests/unit/repo/test_repo.py:15:1: E1102: pytest.mark.parametrize is not callable (not-callable)
tests/unit/repo/test_repo.py:33:1: E1102: pytest.mark.parametrize is not callable (not-callable)
************* Module tests.unit.remote.ssh.test_connection
tests/unit/remote/ssh/test_connection.py:87:1: E1102: pytest.mark.skipif is not callable (not-callable)
tests/unit/remote/ssh/test_connection.py:100:1: E1102: pytest.mark.skipif is not callable (not-callable)

https://github.com/iterative/dvc/pull/4257#issuecomment-665474467](https://github.com/iterative/dvc/commit/9ca3c05f6dc8d53d11af80c1f2b965e93fb28cdf) by skshetry
🛠️ dvc: bump to 1.2.3 by efiop
🛠️ [gdrive: add workaround for downloading empty files (#4287) * [WIP] gdrive: add workaround for downloading empty files

Better to solve it on PyDrive's side:
#4286

Fixes #4286

deps: limit flatten_json

pylint...................................................................Failed
- hook id: pylint
- exit code: 2

Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.command.experiments
dvc/command/experiments.py:14:4: E0401: Unable to import 'flatten_json'
(import-error)
dvc/command/experiments.py:39:4: E0401: Unable to import 'flatten_json'
(import-error)

-----------------------------------
Your code has been rated at 9.99/10

Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.utils.diff
dvc/utils/diff.py:38:8: E0401: Unable to import 'flatten_json'
(import-error)
************* Module dvc.command.metrics
dvc/command/metrics.py:17:4: E0401: Unable to import 'flatten_json'
(import-error)

-----------------------------------
```](https://github.com/iterative/dvc/commit/5d1c1c418c75677379108980970c6546fba8fe18) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [version: redesign output format for the dvc version command #3499 (#4114)  * Version: Redesign output format #3499

Redesigned the output format for the command: dvc version

* version: reformat output format #3499

Done some suggested fixes in code for the dvc command output format issue.

* version: redesign output format #3499

* version: redesign output format for dvc command #3499

* version: redesign output format #3499

* version: redesign output format #3499

* Redesign output format for the dvc version command #3499

* Redesign output format for the dvc version command #3499

* Update dvc/command/version.py

Co-authored-by: Saugat Pachhai <suagatchhetri@outlook.com>

Co-authored-by: Ruslan Kuprieiev <kupruser@gmail.com>
Co-authored-by: Saugat Pachhai <suagatchhetri@outlook.com>](https://github.com/iterative/dvc/commit/f4595a02b20be9f0ed7f391f9fd485071b3ab17d) by [sahilbhosale63](https://github.com/sahilbhosale63)
:hammer_and_wrench: [dvc: bump to 1.2.2](https://github.com/iterative/dvc/commit/f4b820aa4c6e47ea50be291acb3e75b3fd2ef915) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [daemon: use Popen on Mac (#4296)  Turns out macs are not happy with double-forks without exec.

Fixes #4294](https://github.com/iterative/dvc/commit/0b111d2a1052cdffcd3ddbc117097ae7148519f7) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [dvc: bump to 1.2.1](https://github.com/iterative/dvc/commit/b14ad4f8ccd38a8c80fa61c75d14f2afa12d7d69) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [experiments: temporarily disable styling in `show` table (#4290)](https://github.com/iterative/dvc/commit/864bc999d1b538ee6972fba5ffb13cdbc8555357) by [pmrowla](https://github.com/pmrowla)
:hammer_and_wrench: [deps: limit flatten_json  ```
pylint...................................................................Failed
- hook id: pylint
- exit code: 2

Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.command.experiments
dvc/command/experiments.py:14:4: E0401: Unable to import 'flatten_json'
(import-error)
dvc/command/experiments.py:39:4: E0401: Unable to import 'flatten_json'
(import-error)

-----------------------------------
Your code has been rated at 9.99/10

Registered custom plugin. Some checks will be disabled for tests.
************* Module dvc.utils.diff
dvc/utils/diff.py:38:8: E0401: Unable to import 'flatten_json'
(import-error)
************* Module dvc.command.metrics
dvc/command/metrics.py:17:4: E0401: Unable to import 'flatten_json'
(import-error)

-----------------------------------
```](https://github.com/iterative/dvc/commit/2bf6eef53b06c720c1b96db7879517e2525187ee) by [efiop](https://github.com/efiop)
:hammer_and_wrench: [ignore: dynamically collect dvcignore (#4284)  This allows us avoid collecting dvcignore for the whole repo if we only
care about particular paths. As a result, in a repo with 2 datasets
(2M + 0.5M files), creating a defunct stage takes ~4sec on 1.2.0, but
~1sec(most of it is actually dvc module initialization) with this PR.

This is also a pre-requisite for dynamic dvcignore and subrepo
collection (https://github.com/iterative/dvc/pull/4247) while walking
the tree.

Also, it is important to clarify that regular `dvc status`(without
arguments) has the same performance after this PR, because when we check
dataset for changes, we call things like `tree.exists()`, which call
dvcignore and make it collect dvcignore in the dataset itself, so we
still endup collecting dvcignore for the whole repo (including walking
into the datasets). This should be solved soon by telling dvcignore that
it shouldn't walk into the datasets searching for `.dvcignore`s.](https://github.com/iterative/dvc/commit/fe9ae2c5d1ab94e54ef5620fd2cb1ead16e2d082) by [efiop](https://github.com/efiop)

- - - 
# CONTRIBUTORS
Last week there were 6 contributors.
:bust_in_silhouette: [efiop](https://github.com/efiop)
:bust_in_silhouette: [arthurcgusmao](https://github.com/arthurcgusmao)
:bust_in_silhouette: [skshetry](https://github.com/skshetry)
:bust_in_silhouette: [iksnagreb](https://github.com/iksnagreb)
:bust_in_silhouette: [pmrowla](https://github.com/pmrowla)
:bust_in_silhouette: [sahilbhosale63](https://github.com/sahilbhosale63)

- - - 
# STARGAZERS
Last week there were 102 stagazers.
:star: [dafinley](https://github.com/dafinley)
:star: [petrbela](https://github.com/petrbela)
:star: [skillachie](https://github.com/skillachie)
:star: [Dovyski](https://github.com/Dovyski)
:star: [monoidcat](https://github.com/monoidcat)
:star: [alxbrd](https://github.com/alxbrd)
:star: [zmiftah](https://github.com/zmiftah)
:star: [bear0330](https://github.com/bear0330)
:star: [lcbasu](https://github.com/lcbasu)
:star: [MHM5000](https://github.com/MHM5000)
:star: [elsbrock](https://github.com/elsbrock)
:star: [andreistirb](https://github.com/andreistirb)
:star: [mohammadnaraghi](https://github.com/mohammadnaraghi)
:star: [mediovia](https://github.com/mediovia)
:star: [deenadayalancs](https://github.com/deenadayalancs)
:star: [1e100](https://github.com/1e100)
:star: [Ahacad](https://github.com/Ahacad)
:star: [omarryhan](https://github.com/omarryhan)
:star: [TalhaKarabiyik](https://github.com/TalhaKarabiyik)
:star: [gmenegatti](https://github.com/gmenegatti)
:star: [AleksaC](https://github.com/AleksaC)
:star: [honood](https://github.com/honood)
:star: [ankane](https://github.com/ankane)
:star: [ahmgeek](https://github.com/ahmgeek)
:star: [jayaramcloud](https://github.com/jayaramcloud)
:star: [Sija](https://github.com/Sija)
:star: [Azatik1000](https://github.com/Azatik1000)
:star: [jawwad3](https://github.com/jawwad3)
:star: [4oo4](https://github.com/4oo4)
:star: [phillbaker](https://github.com/phillbaker)
:star: [Emiller88](https://github.com/Emiller88)
:star: [bdittmer](https://github.com/bdittmer)
:star: [harshavamsi](https://github.com/harshavamsi)
:star: [simonlin20](https://github.com/simonlin20)
:star: [bobzhen](https://github.com/bobzhen)
:star: [gEndelf](https://github.com/gEndelf)
:star: [peeush-agarwal](https://github.com/peeush-agarwal)
:star: [berniechiu](https://github.com/berniechiu)
:star: [greenspray9](https://github.com/greenspray9)
:star: [admariner](https://github.com/admariner)
:star: [ferencberes](https://github.com/ferencberes)
:star: [LilJoshua](https://github.com/LilJoshua)
:star: [lipovsek](https://github.com/lipovsek)
:star: [JoaoVasques](https://github.com/JoaoVasques)
:star: [kasimkobakci](https://github.com/kasimkobakci)
:star: [AlmogBaku](https://github.com/AlmogBaku)
:star: [endymion64](https://github.com/endymion64)
:star: [bnel1201](https://github.com/bnel1201)
:star: [dnakov](https://github.com/dnakov)
:star: [melkiga](https://github.com/melkiga)
:star: [jeffreyguenther](https://github.com/jeffreyguenther)
:star: [vai](https://github.com/vai)
:star: [cyrillay](https://github.com/cyrillay)
:star: [ssami](https://github.com/ssami)
:star: [qdriven](https://github.com/qdriven)
:star: [slaughtering](https://github.com/slaughtering)
:star: [ThomasK33](https://github.com/ThomasK33)
:star: [yoziru](https://github.com/yoziru)
:star: [AhmedAzzu25](https://github.com/AhmedAzzu25)
:star: [izurietajr](https://github.com/izurietajr)
:star: [tzvsi](https://github.com/tzvsi)
:star: [JessicaSousa](https://github.com/JessicaSousa)
:star: [aninoy](https://github.com/aninoy)
:star: [martinbjeldbak](https://github.com/martinbjeldbak)
:star: [Kogam22](https://github.com/Kogam22)
:star: [Prev](https://github.com/Prev)
:star: [blawok](https://github.com/blawok)
:star: [naihetxi](https://github.com/naihetxi)
:star: [pasqualesalza](https://github.com/pasqualesalza)
:star: [ari-hacks](https://github.com/ari-hacks)
:star: [adamaliang](https://github.com/adamaliang)
:star: [singal95](https://github.com/singal95)
:star: [equ1n0x93](https://github.com/equ1n0x93)
:star: [lrq3000](https://github.com/lrq3000)
:star: [h3ndrk](https://github.com/h3ndrk)
:star: [apoorv4wadhwani](https://github.com/apoorv4wadhwani)
:star: [abunuwas](https://github.com/abunuwas)
:star: [Jeket](https://github.com/Jeket)
:star: [shibuiwilliam](https://github.com/shibuiwilliam)
:star: [ericvoorhis](https://github.com/ericvoorhis)
:star: [vikramkalabi](https://github.com/vikramkalabi)
:star: [leon90dm](https://github.com/leon90dm)
:star: [lianaing](https://github.com/lianaing)
:star: [avinashkoyyana](https://github.com/avinashkoyyana)
:star: [rsnitsch](https://github.com/rsnitsch)
:star: [noorvir](https://github.com/noorvir)
:star: [aoelvp94](https://github.com/aoelvp94)
:star: [duffrind](https://github.com/duffrind)
:star: [md5](https://github.com/md5)
:star: [deuxtres](https://github.com/deuxtres)
:star: [yxxhero](https://github.com/yxxhero)
:star: [dvbvr](https://github.com/dvbvr)
:star: [FerdiHr](https://github.com/FerdiHr)
:star: [sootlasten](https://github.com/sootlasten)
:star: [nicohlr](https://github.com/nicohlr)
:star: [johnowennixon](https://github.com/johnowennixon)
:star: [syfreed](https://github.com/syfreed)
:star: [amandujano](https://github.com/amandujano)
:star: [yihengli](https://github.com/yihengli)
:star: [leepeterson](https://github.com/leepeterson)
:star: [AntoineToubhans](https://github.com/AntoineToubhans)
:star: [eyaltrabelsi](https://github.com/eyaltrabelsi)
You all are the stars! :star2:

- - - 
# RELEASES
Last week there were 5 releases.
:rocket: [1.3.1 1.3.1](https://github.com/iterative/dvc/releases/tag/1.3.1)
:rocket: [1.3.0 1.3.0](https://github.com/iterative/dvc/releases/tag/1.3.0)
:rocket: [1.2.3 1.2.3](https://github.com/iterative/dvc/releases/tag/1.2.3)
:rocket: [1.2.2 1.2.2](https://github.com/iterative/dvc/releases/tag/1.2.2)
:rocket: [1.2.1 1.2.1](https://github.com/iterative/dvc/releases/tag/1.2.1)

- - - 

That's all for last week, please <kbd>:eyes: **Watch**</kbd> and <kbd>:star: **Star**</kbd> the repository [*iterative/dvc*](https://github.com/iterative/dvc) to receive next weekly updates. :smiley:

*You can also [view all Weekly Digests by clicking here](https://github.com/iterative/dvc/issues?q=is:open+is:issue+label:weekly-digest).* 

> Your [**Weekly Digest**](https://github.com/apps/weekly-digest) bot. :calendar:

The text was updated successfully, but these errors were encountered:

weekly-digest bot added the weekly-digest label Aug 2, 2020

efiop closed this as completed Aug 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Weekly Digest (26 July, 2020 - 2 August, 2020) #4315

Weekly Digest (26 July, 2020 - 2 August, 2020) #4315

weekly-digest bot commented Aug 2, 2020

Weekly Digest (26 July, 2020 - 2 August, 2020) #4315

Weekly Digest (26 July, 2020 - 2 August, 2020) #4315

Comments

weekly-digest bot commented Aug 2, 2020

ISSUES

OPEN ISSUES

CLOSED ISSUES

LIKED ISSUE

NOISY ISSUE

PULL REQUESTS

UPDATED PULL REQUEST

MERGED PULL REQUEST

COMMITS