[ONNX] Run old unittests with fx exporter #96479

BowenBao · 2023-03-10T02:15:16Z

Stack from ghstack (oldest at bottom):

Tracker #96538

[ghstack-poisoned]

pytorch-bot · 2023-03-10T02:15:19Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96479

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 465452b:
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: 43dd979fa8f318498a056e66b3305d80bdd39f04 Pull Request resolved: #96479

Tracker #96538 [ghstack-poisoned]

ghstack-source-id: 8142e854df36e3be6f56683dc1428722d5580f3e Pull Request resolved: #96479

justinchuby · 2023-03-10T19:16:19Z

Love this!

test/onnx/test_pytorch_onnx_onnxruntime.py

titaiwangms · 2023-03-10T19:25:53Z

This is great!!!

Tracker #96538 [ghstack-poisoned]

torch/onnx/_constants.py

Tracker #96538 [ghstack-poisoned]

ghstack-source-id: 91869fa81df3600ea20d60d36fde119dabc2676a Pull Request resolved: pytorch#96479 fix lint for unittest amend unittest

test/onnx/onnx_test_common.py

titaiwangms · 2023-03-20T19:41:46Z

test/onnx/onnx_test_common.py

            script_model = model if is_model_script else torch.jit.script(model)
            _run_test(
                script_model,
                scripting_remained_onnx_input_idx,
                flatten=False,
                ignore_none=False,
            )
-        if not is_model_script and not self.is_script:
+        elif not is_model_script and not self.is_script and not self.is_fx:


Can we move the elif self.is_fx on top of this and make this else? This is the tracing cases, right? I guess you think this is more explicit, but it actually seems more complicated. But not blocking though.

Could be just me, but adding a comment would be better.

Cleared the logic a bit.

titaiwangms · 2023-03-20T20:09:54Z

For those tests with dynamic_axes, I think we didn't test them properly. At least we should give additional test sets with different shape to prove it work. So it would be great if we can set additional test sets into fx test like in https://github.com/pytorch/pytorch/pull/96350/files. It's not blocking, if it's out of this PR scope, I can do it too. Also, I think it's worth to put the auto testing code up here? I figure we might need to use it a lot.

Tracker #96538 [ghstack-poisoned]

BowenBao · 2023-03-21T00:31:10Z

@titaiwangms I'll leave most for future PRs. I didn't include the auto script as it still requires a bit of manual process. I will consider adding as follow up when it's more mature.

…mbolic shape to ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…mbolic shape to ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…mbolic shape to ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…mbolic shape to ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

…ONNX" ~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py [ghstack-poisoned]

~~Need microsoft/onnxscript#484 Support dynamic export on fx-ONNX exporter. Essentially, we set inputs size and nodes all dynamic in torchscript, and leverage on `aten::sym_size` to catch dynamic size between each Op. 1. Add `dynamic_axes` switch between symbolic tracing (dynamic sizes) and fake mode (static). Set it to default True, as most of our tests are happy with sumbolic tracing. Except GPT2 stays with fake mode with error: microsoft/onnxscript#523 2. Add test_fx_dynamic_onnruntime.py to test on some addhoc we have from old exporter. This can be removed once tests are integrated with #96479 3. Since `aten::sym_size` are operated with built-in function, a built-in function mapping is added to support SymFloat/SymInt. (FIXME: #97201). sym_size output value is also fx.Node, and can be found in `fx_name_to_onnxscipt_value`, so it's operation stays the same as other ONNX ops in ONNX graph. 4. Fully deprecated FakeTensorProp as make_fx() should provide all node meta info. 5. Put complicated fx.Node related ArgumentType into _type_utils.py Pull Request resolved: #96350 Approved by: https://github.com/wschin, https://github.com/justinchuby

kit1980 · 2023-04-26T23:46:04Z

Should we merge this?

BowenBao · 2023-04-27T00:37:09Z

@kit1980 not in a hurry but will eventually do it. I'll convert it to draft as things must have been outdated for a while.

github-actions · 2023-06-26T00:49:27Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

[ONNX] Run old unittests with fx exporter

51838e1

[ghstack-poisoned]

pytorch-bot bot added the topic: not user facing topic category label Mar 10, 2023

BowenBao added a commit that referenced this pull request Mar 10, 2023

[ONNX] Run old unittests with fx exporter

0380e6c

ghstack-source-id: 43dd979fa8f318498a056e66b3305d80bdd39f04 Pull Request resolved: #96479

pytorchbot added the open source label Mar 10, 2023

Update on "[ONNX] Run old unittests with fx exporter"

71dd31f

Tracker #96538 [ghstack-poisoned]

BowenBao added a commit that referenced this pull request Mar 10, 2023

[ONNX] Run old unittests with fx exporter

0b5e842

ghstack-source-id: 8142e854df36e3be6f56683dc1428722d5580f3e Pull Request resolved: #96479

justinchuby requested review from titaiwangms and justinchuby March 10, 2023 19:16

BowenBao commented Mar 10, 2023

View reviewed changes

test/onnx/test_pytorch_onnx_onnxruntime.py Outdated Show resolved Hide resolved

Update on "[ONNX] Run old unittests with fx exporter"

b2ce01e

Tracker #96538 [ghstack-poisoned]

BowenBao mentioned this pull request Mar 10, 2023

[ONNX] Run old modeltests with fx exporter #96556

Closed

justinchuby reviewed Mar 10, 2023

View reviewed changes

torch/onnx/_constants.py Outdated Show resolved Hide resolved

Update on "[ONNX] Run old unittests with fx exporter"

1019835

Tracker #96538 [ghstack-poisoned]

BowenBao mentioned this pull request Mar 10, 2023

[ONNX] Bump fx export opset version #96567

Closed

Update on "[ONNX] Run old unittests with fx exporter"

3493672

Tracker #96538 [ghstack-poisoned]

BowenBao mentioned this pull request Mar 11, 2023

Bump black version to 23.1.0 #96578

Closed

Update on "[ONNX] Run old unittests with fx exporter"

eff2afd

Tracker #96538 [ghstack-poisoned]

BowenBao mentioned this pull request Mar 14, 2023

[ONNX] Analysis: unsupported call functions #96781

Closed

Update on "[ONNX] Run old unittests with fx exporter"

b13abed

Tracker #96538 [ghstack-poisoned]

BowenBao marked this pull request as ready for review March 15, 2023 00:46

BowenBao requested a review from abock as a code owner March 15, 2023 00:46

Update on "[ONNX] Run old unittests with fx exporter"

df80756

Tracker #96538 [ghstack-poisoned]

BowenBao added a commit to BowenBao/pytorch that referenced this pull request Mar 15, 2023

[ONNX] Run old unittests with fx exporter

f371bc5

ghstack-source-id: 91869fa81df3600ea20d60d36fde119dabc2676a Pull Request resolved: pytorch#96479 fix lint for unittest amend unittest

titaiwangms reviewed Mar 20, 2023

View reviewed changes

test/onnx/onnx_test_common.py Outdated Show resolved Hide resolved

titaiwangms reviewed Mar 20, 2023

View reviewed changes

Update on "[ONNX] Run old unittests with fx exporter"

465452b

Tracker #96538 [ghstack-poisoned]

titaiwangms mentioned this pull request Mar 22, 2023

[ONNX] Support converting fx graph with symbolic shape to ONNX #96350

Closed

BowenBao marked this pull request as draft April 27, 2023 00:37

github-actions bot added the Stale label Jun 26, 2023

github-actions bot closed this Jul 26, 2023

facebook-github-bot deleted the gh/BowenBao/215/head branch August 25, 2023 14:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ONNX] Run old unittests with fx exporter #96479

[ONNX] Run old unittests with fx exporter #96479

BowenBao commented Mar 10, 2023 •

edited

pytorch-bot bot commented Mar 10, 2023 •

edited

justinchuby commented Mar 10, 2023

titaiwangms commented Mar 10, 2023

titaiwangms Mar 20, 2023

titaiwangms Mar 20, 2023

BowenBao Mar 21, 2023

titaiwangms commented Mar 20, 2023

BowenBao commented Mar 21, 2023

kit1980 commented Apr 26, 2023

BowenBao commented Apr 27, 2023

github-actions bot commented Jun 26, 2023

[ONNX] Run old unittests with fx exporter #96479

[ONNX] Run old unittests with fx exporter #96479

Conversation

BowenBao commented Mar 10, 2023 • edited

pytorch-bot bot commented Mar 10, 2023 • edited

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/96479

✅ No Failures

justinchuby commented Mar 10, 2023

titaiwangms commented Mar 10, 2023

titaiwangms Mar 20, 2023

Choose a reason for hiding this comment

titaiwangms Mar 20, 2023

Choose a reason for hiding this comment

BowenBao Mar 21, 2023

Choose a reason for hiding this comment

titaiwangms commented Mar 20, 2023

BowenBao commented Mar 21, 2023

kit1980 commented Apr 26, 2023

BowenBao commented Apr 27, 2023

github-actions bot commented Jun 26, 2023

BowenBao commented Mar 10, 2023 •

edited

pytorch-bot bot commented Mar 10, 2023 •

edited