Fixing GPU for token-classification in a better way. #13856
Conversation
Co-authored-by: Pierre Snell <pierre.snell@botpress.com>
| @@ -216,7 +217,7 @@ def _forward(self, model_inputs): | |||
| } | |||
|
|
|||
| def postprocess(self, model_outputs, aggregation_strategy=AggregationStrategy.NONE): | |||
| outputs = model_outputs["outputs"] | |||
| outputs = model_outputs["outputs"].numpy() | |||
There was a problem hiding this comment.
I'm not as familiar as you with the pipeline but will the model_outputs["outputs"] can still be on GPU ?
Maybe all preprocessing is firstly casted to CPU but I would rather double check.
Else should we use model_outputs["outputs"].cpu().numpy() instead ?
There was a problem hiding this comment.
Hi @ierezell,
Just like this file does not ever mention .to(device) it shouldn't mention .cpu() as it's not the task of the pipeline to encapsulate that logic. Before it was done this way meaning many pipelines wouldn't actually support some features.
The logic is now in the Pipeline.forward method.
Rougly:
preprocess: (generic python objects) -> Tensors
_forward: Tensors -> Tensors
postprocess: Tensors -> Generic python objects
forward encapsulate classic logic for inference (inference mode, GPU tensor moving and other if needed, like batching)
There was a problem hiding this comment.
Ok, then everything will be at the good place at the good moment so that's perfect! Thanks for the clarification.
Co-authored-by: Pierre Snell <pierre.snell@botpress.com> Co-authored-by: Pierre Snell <pierre.snell@botpress.com>
Co-authored-by: Pierre Snell <pierre.snell@botpress.com> Co-authored-by: Pierre Snell <pierre.snell@botpress.com>
What does this PR do?
We probably need a way to ensure this test is ran !
Superseeds #13819 (different fix for same issue)
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@LysandreJik