Add ort export in exporters for encoder-decoder models #497

mht-sharma · 2022-11-21T13:53:00Z

What does this PR do?

Adds support to export the encoder and decoder separately for encoder-decoder models using exporters cli based on cmd line arguments.

Fixes #496

Example usage:

python -m optimum.exporters.onnx --model="openai/whisper-tiny.en" --task=speech2seq-lm-with-past output_whisper  --for-ort

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2022-11-22T07:42:53Z

The documentation is not available anymore as the PR was closed or merged.

mht-sharma · 2022-11-22T08:14:18Z

optimum/exporters/onnx/base.py

@@ -303,6 +303,56 @@ def flatten_output_collection_property(cls, name: str, field: Iterable[Any]) ->
        """
        return {f"{name}.{idx}": item for idx, item in enumerate(itertools.chain.from_iterable(field))}

+    def generate_dummy_inputs_onnxruntime(self, reference_model_inputs: Mapping[str, Any]) -> Mapping[str, Any]:


Discussion with lewtun regarding the use of the function. huggingface/transformers#19525 (comment)

It is needed to generate the inputs for the separate encoder and decoder models?
Is it only used for validation?

It is needed only for validation using onnxruntime. Since the onnx model and torch model will have different input signatures when using encoder_outputs for exporting the model.

What about calling it generate_dummy_inputs_for_validation?
I think it can be misleading otherwise (more so now that we use a for-ort argument that does not mean the same thing)

Agree. Updated!

mht-sharma · 2022-11-22T08:18:48Z

Currently only Whisper is exported for ORT in this PR. Should I add the support for all the models or atleast a few more (example T5) in this PR. Or they could have their own PRs? @michaelbenayoun WDYT.

optimum/exporters/onnx/__main__.py

michaelbenayoun · 2022-11-22T08:39:09Z

optimum/exporters/onnx/base.py

@@ -303,6 +303,56 @@ def flatten_output_collection_property(cls, name: str, field: Iterable[Any]) ->
        """
        return {f"{name}.{idx}": item for idx, item in enumerate(itertools.chain.from_iterable(field))}

+    def generate_dummy_inputs_onnxruntime(self, reference_model_inputs: Mapping[str, Any]) -> Mapping[str, Any]:


It is needed to generate the inputs for the separate encoder and decoder models?
Is it only used for validation?

optimum/exporters/onnx/base.py

tests/exporters/test_onnx_export.py

michaelbenayoun · 2022-11-22T09:16:14Z

To answer your question: you can also add support for other models if you feel like it!

mht-sharma · 2022-11-29T07:55:34Z

To answer your question: you can also add support for other models if you feel like it!

Added support for exporting the Seq2Seq-lm models also. AFAIK I covered the models with existing support let me know if I missed something.

mht-sharma · 2022-11-29T08:00:03Z

optimum/exporters/onnx/__main__.py

+    parser.add_argument(
+        "--for-ort",
+        action="store_true",
+        help=(
+            "This exports models ready to be run with optimum.onnxruntime ORTModelXXX. Useful for encoder-decoder models for"
+            "conditional generation. If enabled the encoder and decoder of the model are exported separately."
+        ),
+    )


Could the name for-ort be misleading as the models can have other tasks apart from the conditional generation?
I have mentioned Useful for encoder-decoder models for conditional generation in help. But not sure if this would be enough.

Probably updating ORTModelXXX -> ORTModelForConditionalGeneration?

I would just say to run with optimum.onnxruntime. I think for-ort is good enough, or at least I do not have a better naming in mind.

michaelbenayoun

That's really great!
Left a few comments and questions, but huge work @mht-sharma !

michaelbenayoun · 2022-11-29T09:59:39Z

optimum/exporters/onnx/__main__.py

+    parser.add_argument(
+        "--for-ort",
+        action="store_true",
+        help=(
+            "This exports models ready to be run with optimum.onnxruntime ORTModelXXX. Useful for encoder-decoder models for"
+            "conditional generation. If enabled the encoder and decoder of the model are exported separately."
+        ),
+    )


I would just say to run with optimum.onnxruntime. I think for-ort is good enough, or at least I do not have a better naming in mind.

michaelbenayoun · 2022-11-29T10:02:52Z

optimum/exporters/onnx/__main__.py

-        args.opset,
-        args.output,
-    )
+    use_past = True if "-with-past" in task else False


I think you do not need that since it is already in the onnx config no?

Yes, was probably thinking of how the ORTModelForConditionalGeneration is currently implemented which create onnx config always without past. But that can be modified later.

Currently, updated to use use_past from the onnx config.

michaelbenayoun · 2022-11-29T10:03:48Z

optimum/exporters/onnx/base.py

@@ -303,6 +303,56 @@ def flatten_output_collection_property(cls, name: str, field: Iterable[Any]) ->
        """
        return {f"{name}.{idx}": item for idx, item in enumerate(itertools.chain.from_iterable(field))}

+    def generate_dummy_inputs_onnxruntime(self, reference_model_inputs: Mapping[str, Any]) -> Mapping[str, Any]:


What about calling it generate_dummy_inputs_for_validation?
I think it can be misleading otherwise (more so now that we use a for-ort argument that does not mean the same thing)

optimum/exporters/onnx/base.py

michaelbenayoun · 2022-11-29T10:05:40Z

optimum/exporters/onnx/base.py

+            f"{config.model_type} encoder export is not supported yet. ",
+            f"If you want to support {config.model_type} please propose a PR or open up an issue.",
+        )
+


Suggested change

@abstractmethod

optimum/exporters/onnx/convert.py

optimum/exporters/onnx/model_configs.py

optimum/exporters/onnx/utils.py

optimum/utils/input_generators.py

Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>

michaelbenayoun

LGTM!

JingyaHuang

LGTM, love the consistency of new ONNX configs.

mht-sharma requested a review from michaelbenayoun November 22, 2022 07:58

mht-sharma commented Nov 22, 2022

View reviewed changes

mht-sharma marked this pull request as ready for review November 22, 2022 08:14

michaelbenayoun reviewed Nov 22, 2022

View reviewed changes

mht-sharma added 7 commits November 28, 2022 10:10

Add ort export in exporters for encoder-decoder models

5126f53

Updated error docstring

efcdeb0

Update encoder decoder config location

2dd0339

Added tests

90b9271

Update arguments help

90153cf

Updated docstring and removed redundant code

bbb89e1

Updated config location

fc97e78

mht-sharma force-pushed the export-encoder-decoder-separately-seq2seq-models branch from 28dc7d0 to fc97e78 Compare November 28, 2022 09:12

mht-sharma added 4 commits November 28, 2022 14:17

Added methods for ncoder/decoder onnx export and validation

7ec493d

Added Seq2Seq-lm encoder-decoder configs

634c682

Uncommented tests

db97f8d

Fixed test

bc81a41

Updated argument help

bbd201c

mht-sharma commented Nov 29, 2022

View reviewed changes

michaelbenayoun reviewed Nov 29, 2022

View reviewed changes

Removed use-past and updated docstring

f9713dc

JingyaHuang mentioned this pull request Nov 29, 2022

MBart model cannot be loaded #377

Closed

4 tasks

mht-sharma and others added 3 commits November 29, 2022 13:42

Updated input generator Seq2SeqDecoderConfig

936b114

Update docstrings to use Optional

56db9c5

Co-authored-by: Michael Benayoun <mickbenayoun@gmail.com>

Added optional import

224f79c

michaelbenayoun approved these changes Nov 30, 2022

View reviewed changes

Remove reduntant task from the export function

eef106c

JingyaHuang approved these changes Nov 30, 2022

View reviewed changes

mht-sharma merged commit 6ee424b into huggingface:main Nov 30, 2022

fxmarty mentioned this pull request Dec 6, 2022

Issue to use GPT2 ONNX export with past key values #552

Closed

4 tasks

fxmarty mentioned this pull request Dec 19, 2022

Validating ONNX model fails for GPT-J #607

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ort export in exporters for encoder-decoder models #497

Add ort export in exporters for encoder-decoder models #497

mht-sharma commented Nov 21, 2022 •

edited

HuggingFaceDocBuilderDev commented Nov 22, 2022 •

edited

mht-sharma Nov 22, 2022 •

edited

michaelbenayoun Nov 22, 2022

mht-sharma Nov 22, 2022

michaelbenayoun Nov 29, 2022

mht-sharma Nov 29, 2022

mht-sharma commented Nov 22, 2022 •

edited

michaelbenayoun Nov 22, 2022

michaelbenayoun commented Nov 22, 2022

mht-sharma commented Nov 29, 2022

mht-sharma Nov 29, 2022

michaelbenayoun Nov 29, 2022

michaelbenayoun left a comment

michaelbenayoun Nov 29, 2022

michaelbenayoun Nov 29, 2022

mht-sharma Nov 29, 2022

michaelbenayoun Nov 29, 2022

michaelbenayoun Nov 29, 2022

michaelbenayoun left a comment

JingyaHuang left a comment

Add ort export in exporters for encoder-decoder models #497

Add ort export in exporters for encoder-decoder models #497

Conversation

mht-sharma commented Nov 21, 2022 • edited

What does this PR do?

Before submitting

HuggingFaceDocBuilderDev commented Nov 22, 2022 • edited

mht-sharma Nov 22, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mht-sharma commented Nov 22, 2022 • edited

Choose a reason for hiding this comment

michaelbenayoun commented Nov 22, 2022

mht-sharma commented Nov 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelbenayoun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

michaelbenayoun left a comment

Choose a reason for hiding this comment

JingyaHuang left a comment

Choose a reason for hiding this comment

mht-sharma commented Nov 21, 2022 •

edited

HuggingFaceDocBuilderDev commented Nov 22, 2022 •

edited

mht-sharma Nov 22, 2022 •

edited

mht-sharma commented Nov 22, 2022 •

edited