prefetch mvp #986

Scitator · 2020-11-04T13:27:20Z

Before submitting

Was this discussed/approved via a Github issue? (no need for typos and docs improvements)
Did you read the contribution guide?
Did you check the code style? catalyst-make-codestyle && catalyst-check-codestyle (pip install -U catalyst-codestyle).
Did you make sure to update the docs? We use Google format for all the methods and classes.
Did you check the docs with make check-docs?
Did you write any new necessary tests?
Did you check that your code passes the unit tests pytest . ?
Did you add your new functionality to the docs?
Did you update the CHANGELOG?

Description

Related Issue

Type of Change

Examples / docs / tutorials / contributors update
Bug fix (non-breaking change which fixes an issue)
Improvement (non-breaking change which improves an existing feature)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

PS

I know, that I could join slack for pull request discussion.

github-actions · 2020-11-04T13:36:54Z

catalyst/data/loader.py

+        for x in iterable:
+            result = func(x)
+            result_queue.put(result)
+    except BaseException:


[pep8] _{reported by reviewdog 🐶}
WPS424 Found except BaseException

github-actions · 2020-11-04T13:36:54Z

catalyst/data/loader.py

+    return loader
+
+
+class BatchPrefetchLoaderWrapper(ILoaderWrapper):


[pep8] _{reported by reviewdog 🐶}
D101 Missing docstring in public class

github-actions · 2020-11-04T13:36:54Z

catalyst/data/loader.py

+
+
+class BatchPrefetchLoaderWrapper(ILoaderWrapper):
+    def __init__(self, loader: DataLoader, num_prefetches: int = None):


[pep8] _{reported by reviewdog 🐶}
D107 Missing docstring in init

github-actions · 2020-11-04T13:36:54Z

catalyst/data/loader.py

+        super().__init__(loader)
+        self.num_prefetches = num_prefetches or loader.batch_size
+
+    def __iter__(self):


[pep8] _{reported by reviewdog 🐶}
D105 Missing docstring in magic method

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+
+            def predict_batch(self, batch):
+                # model inference step
+                return self.model(batch[0].to(self.device).view(batch[0].size(0), -1))


[pep8] _{reported by reviewdog 🐶}
E501 line too long (86 > 79 characters)

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+
+            def predict_batch(self, batch):
+                # model inference step
+                return self.model(batch[0].to(self.device).view(batch[0].size(0), -1))


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (86 > 79 characters)

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+                y_hat = self.model(x.view(x.size(0), -1))
+
+                loss = F.cross_entropy(y_hat, y)
+                accuracy01, accuracy03 = metrics.accuracy(y_hat, y, topk=(1, 3))


[pep8] _{reported by reviewdog 🐶}
E501 line too long (80 > 79 characters)

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+                y_hat = self.model(x.view(x.size(0), -1))
+
+                loss = F.cross_entropy(y_hat, y)
+                accuracy01, accuracy03 = metrics.accuracy(y_hat, y, topk=(1, 3))


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (80 > 79 characters)

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+                loss = F.cross_entropy(y_hat, y)
+                accuracy01, accuracy03 = metrics.accuracy(y_hat, y, topk=(1, 3))
+                self.batch_metrics.update(
+                    {"loss": loss, "accuracy01": accuracy01, "accuracy03": accuracy03}


[pep8] _{reported by reviewdog 🐶}
E501 line too long (86 > 79 characters)

github-actions · 2020-11-08T08:07:34Z

catalyst/data/loader.py

+
+        batch_size=32
+        loaders = {
+            "train": DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=batch_size),


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (124 > 79 characters)

github-actions · 2020-11-08T08:07:35Z

catalyst/data/loader.py

+        batch_size=32
+        loaders = {
+            "train": DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=batch_size),
+            "valid": DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=ToTensor()), batch_size=batch_size),


[pep8] _{reported by reviewdog 🐶}
E501 line too long (125 > 79 characters)

github-actions · 2020-11-08T08:07:35Z

catalyst/data/loader.py

+        batch_size=32
+        loaders = {
+            "train": DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=batch_size),
+            "valid": DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=ToTensor()), batch_size=batch_size),


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (125 > 79 characters)

github-actions · 2020-11-08T08:07:35Z

catalyst/data/loader.py

+            "train": DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=batch_size),
+            "valid": DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=ToTensor()), batch_size=batch_size),
+        }
+        loaders = {k: BatchPrefetchLoaderWrapper(v) for k, v in loaders.items()}


[pep8] _{reported by reviewdog 🐶}
E501 line too long (80 > 79 characters)

github-actions · 2020-11-08T08:07:35Z

catalyst/data/loader.py

+            "train": DataLoader(MNIST(os.getcwd(), train=True, download=True, transform=ToTensor()), batch_size=batch_size),
+            "valid": DataLoader(MNIST(os.getcwd(), train=False, download=True, transform=ToTensor()), batch_size=batch_size),
+        }
+        loaders = {k: BatchPrefetchLoaderWrapper(v) for k, v in loaders.items()}


[pep8] _{reported by reviewdog 🐶}
W505 doc line too long (80 > 79 characters)

…prefetcher

pep8speaks · 2020-11-10T05:12:23Z

Hello @Scitator! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-11-10 06:41:51 UTC

Pull request has been modified.

prefetch mvp

e68610a

Scitator added enhancement WIP labels Nov 4, 2020

Scitator requested review from ditwoo, bagxi, AlekseySh, julia-shenshina and elephantmipt November 4, 2020 13:27

github-actions bot reviewed Nov 4, 2020

View reviewed changes

docs draft

3cda19c

github-actions bot reviewed Nov 8, 2020

View reviewed changes

bagxi previously approved these changes Nov 8, 2020

View reviewed changes

Merge branch 'master' of github.com:catalyst-team/catalyst into data/…

ba8db30

…prefetcher

Scitator added 3 commits November 10, 2020 09:02

cleanup

7035bc1

logs

9f66c34

docs

fd03a8b

Scitator removed the WIP label Nov 10, 2020

Scitator changed the title ~~[wip] prefetch mvp~~ prefetch mvp Nov 10, 2020

Scitator merged commit b02faa2 into master Nov 10, 2020

mergify bot deleted the data/prefetcher branch November 10, 2020 07:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prefetch mvp #986

prefetch mvp #986

Scitator commented Nov 4, 2020

github-actions bot Nov 4, 2020

github-actions bot Nov 4, 2020

github-actions bot Nov 4, 2020

github-actions bot Nov 4, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

github-actions bot Nov 8, 2020

pep8speaks commented Nov 10, 2020 •

edited

Loading

		return loader


		class BatchPrefetchLoaderWrapper(ILoaderWrapper):



		class BatchPrefetchLoaderWrapper(ILoaderWrapper):
		def __init__(self, loader: DataLoader, num_prefetches: int = None):

prefetch mvp #986

prefetch mvp #986

Conversation

Scitator commented Nov 4, 2020

Before submitting

Description

Related Issue

Type of Change

PR review

github-actions bot Nov 4, 2020

Choose a reason for hiding this comment

github-actions bot Nov 4, 2020

Choose a reason for hiding this comment

github-actions bot Nov 4, 2020

Choose a reason for hiding this comment

github-actions bot Nov 4, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

github-actions bot Nov 8, 2020

Choose a reason for hiding this comment

pep8speaks commented Nov 10, 2020 • edited Loading

Comment last updated at 2020-11-10 06:41:51 UTC

pep8speaks commented Nov 10, 2020 •

edited

Loading