list: fails if a DVC-tracked directory is empty but present

[shcheklein]

Bug Report

Description

(.env) √ Projects/get-started-experiments % dvc list . data/fashion-mnist/prepared
ERROR: failed to list '.' - The path 'data/fashion-mnist/prepared' does not exist in the target repository
'/Users/ivan/Projects/get-started-experiments' neither as a DVC output nor as a Git-tracked file.

even though, this path does exist, and it is being tracked by DVC:

schema: '2.0'
stages:
  prepare:
    cmd: python3 src/prepare.py
    deps:
    - path: data/fashion-mnist/raw/
      md5: 8a31b1338b609cf1133d1df639de606c.dir
      size: 30878645
      nfiles: 4
    - path: src/prepare.py
      md5: 2c9ddd60ad065771628c58c618663050
      size: 3881
    params:
      params.yaml:
        prepare.remix: false
        prepare.remix_split: 0.2
        prepare.seed: 20210428
    outs:
    - path: data/fashion-mnist/prepared
      md5: c5095cd3e98121bc8d2bebe27d702ab5.dir
      size: 54951020
      nfiles: 2
...

Reproduce

git clone git@github.com:iterative/get-started-experiments.git
cd get-started-experiments
mkdir data/fashion-mnist/prepared
dvc list . data/fashion-mnist/prepared

gives:

ERROR: failed to list '.' - The path 'data/fashion-mnist/prepared' does not exist in the target repository '/Users/ivan/Projects/test-bug' neither as a DVC output nor as a Git-tracked file.

Expected

To see prepared.

Environment information

(.env) √ Projects/test-bug % dvc version
DVC version: 2.3.0+eb46c0
---------------------------------
Platform: Python 3.8.9 on macOS-10.15.6-x86_64-i386-64bit
Supports: All remotes
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: apfs on /dev/disk1s1
Repo: dvc, git

[dberenbaum]

@shcheklein could you give some context on priority?

[shcheklein]

We can discuss this on the next VS Code <> DVC meetings what would be the best way for us to communicate - regular issues vs those that we have in VS Code. Broadly speaking, VS Code depends on dvc list, dvc diff, dvc status, dvc pull, dvc checkout in its basic (files/datasets) feature. And it uses those commands to the full extent - different options, targets, etc - and we see a lot of rough edges now. In a lot of cases it can wait, but it will block the release (not date yet).

My take. Before the release, we'll need to stabilize those commands - their output (exits codes, etc), performance, their behavior with different options/targets.

[shcheklein]

VS Code alpha users hit this issue: iterative/vscode-dvc#529

[karajan1001]

duplicated with #5841 ?

[dberenbaum]

#5841 doesn't look like it's a dvc-tracked directory, but otherwise they look the same, and I don't think that part should matter here? In either case, dvc list should show the directory and not throw an error.

[shcheklein]

Quick note: I know that get-started-experiments has changed underneath us. Please let me know if you need a better reproduction script. The current one is not exactly correct. I'll try to update it today. And sorry about that, I'll be using mkdir style going forward.

[karajan1001]

Quick note: I know that get-started-experiments has changed underneath us. Please let me know if you need a better reproduction script. The current one is not exactly correct. I'll try to update it today. And sorry about that, I'll be using mkdir style going forward.

#6120 might fix it, and we have a reproduction script in it.

[dberenbaum]

@karajan1001 @efiop When will get back to #6120?

[karajan1001]

@mattlbeck could you please try if it works on the latest main branch?

[efiop]

Closing for now, should be fixed by #6120