Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ui: fix missing progress & tidy #3335

Merged
merged 3 commits into from Feb 19, 2020
+4 −4
Merged

ui: fix missing progress & tidy #3335

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0724e7e
ui: show missing progress
casperdcl Feb 14, 2020
37850cb
ui: more verbose progress
casperdcl Feb 14, 2020
780f33d
remote: base: back to no_progres_bar=True
casperdcl Feb 18, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion dvc/remote/base.py
View file
Open in desktop
Expand Up @@ -234,7 +234,7 @@ def _get_dir_info_checksum(self, dir_info):

from_info = PathInfo(tmp)
to_info = self.cache.path_info / tmp_fname("")
self.cache.upload(from_info, to_info, no_progress_bar=True)
self.cache.upload(from_info, to_info, no_progress_bar=False)
Copy link
Contributor Author

@casperdcl casperdcl Feb 14, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efiop wanted to ask if this change is OK

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to add to this - if this change is OK then there's a lot of nice tidying/refactoring possible since upload() would never require a no_progress_bar argument

Copy link
Member

@efiop efiop Feb 15, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperdcl Btw, how does the pbar look for this operation? I'm worried about it having some random looking paths in it.

Copy link
Contributor Author

@casperdcl casperdcl Feb 16, 2020 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this would occur on dvc add <dir>. It gets short cache names but is cleared away upon completion. Padding it with some persistent logging messages (and padding fs.copyfile with some logging & sleep(5)) gives this:

asciicast

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperdcl Yeah, not sure about that. That random name is not very helpful. How about we put some desc instead? There is a name param for that upload method, so should be pretty simple. Though, not sure how to describe this in a user-friendly way. Btw, did this particular part of code cause any issues in your research? It works with tiny files and usually uploading them is very quick.

As to the name here, maybe something like "Computing effective hash for directory '{}'" would be good enough.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperdcl Got it. So the plan is to only refactor the defaults in those remotes, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we allow no_progress_bar=False then we can refactor away. But perhaps for now we should keep no_progress_bar=True and not refactor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@casperdcl Sorry, looks like I'm not following. So gdrive/gs had no_progress_bar=True, which contradicts the other remotes. So we will fix those, right? Or are we going to do something else as well? 🙂 Just trying to understand the plan for this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was (and have always been) only talking about this specific cache.upload() line. The PR is now fine to merge.

gdrive/gs things elsewhere in this PR are no-brainers I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to clarify further - if we change the cache.upload() line to say no_progress_bar=False, then it would mean every single instance of upload() in the code base would only ever have no_progress_bar=False and thus we can drop that argument from all upload() functions.


checksum = self.get_file_checksum(to_info) + self.CHECKSUM_DIR_SUFFIX
return checksum, to_info
Expand Down
2 changes: 1 addition & 1 deletion dvc/remote/gdrive.py
View file
Open in desktop
Expand Up @@ -246,7 +246,7 @@ def gdrive_upload_file(
self,
parent_id,
title,
no_progress_bar=True,
no_progress_bar=False,
casperdcl marked this conversation as resolved.
Show resolved Hide resolved
from_file="",
progress_name="",
):
Expand Down
6 changes: 3 additions & 3 deletions dvc/remote/gs.py
View file
Open in desktop
Expand Up @@ -51,7 +51,7 @@ def _upload_to_bucket(
to_info,
chunk_size=None,
name=None,
no_progress_bar=True,
no_progress_bar=False,
):
blob = bucket.blob(to_info.path, chunk_size=chunk_size)
with io.open(from_file, mode="rb") as fobj:
Expand Down Expand Up @@ -166,7 +166,7 @@ def exists(self, path_info):
"""
return self.isfile(path_info) or self.isdir(path_info)

def _upload(self, from_file, to_info, name=None, no_progress_bar=True):
def _upload(self, from_file, to_info, name=None, no_progress_bar=False):
bucket = self.gs.bucket(to_info.bucket)
_upload_to_bucket(
bucket,
Expand All @@ -176,7 +176,7 @@ def _upload(self, from_file, to_info, name=None, no_progress_bar=True):
no_progress_bar=no_progress_bar,
)

def _download(self, from_info, to_file, name=None, no_progress_bar=True):
def _download(self, from_info, to_file, name=None, no_progress_bar=False):
bucket = self.gs.bucket(from_info.bucket)
blob = bucket.get_blob(from_info.path)
with io.open(to_file, mode="wb") as fobj:
Expand Down