Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Implement FP8 functionality #2763

Open
wants to merge 125 commits into
base: release/2.3
Choose a base branch
from
Open

feat: Implement FP8 functionality #2763

wants to merge 125 commits into from
+488 −42

Conversation

peri044
Copy link
Collaborator

@peri044 peri044 commented Apr 18, 2024

Description

This PR adds FP8 & BF16 datatype support. It also implements converter for FP8 quantized ops.

Type of change

Please delete options that are not relevant and/or add your own.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@peri044
chore: Upgrade to TRT 10.0
9ad87ac
@peri044
chore: updates to trt api
a655c9a
@peri044
feat: Add save API for torch-trt compiled models
cd86660
@peri044
feat: Add FP8 support including dtype and converters
31285e5
@peri044
chore: minor fixes
7c9c646
@peri044
Merge branch 'main' into trt_10
4eabeb0
@peri044
Merge branch 'trt_10' into fp8_trt10
a320e56
@peri044
chore: resolve merge conflicts
3ece71b
@peri044
chore: Fix save failures
eab0dba
@peri044
chore: update to 2.3 rc build
b191d62
@peri044
chore: rebase with release/2.3 branch
ce606fe
@peri044
chore: minor fixes
8674a3c
@peri044
chore: remove duplicate bert test case
f4e8fe9
@peri044
chore: remove comments
4ae6ab9
@peri044
chore: Upgrade to TRT 10.0
fff1b80 
chore: updates to trt api

chore: trt 10 fixes

chore: more fixes
@peri044
chore: more fixes
39ca77d
@peri044
chore: update trt version
5431ee3
@peri044
chore: more updates
0c03de5
@peri044
parent f39e89e
982dbd2 
author Dheeraj Peri <peri.dheeraj@gmail.com> 1711393059 -0700
committer Dheeraj Peri <peri.dheeraj@gmail.com> 1711393072 -0700

chore: minor updates

chore: Fix save failures

chore: minor fixes

chore: remove duplicate bert test case

chore: remove comments

chore: add load api

chore: minor updates

chore: minor updates

chore: minor updates

chore: more updates
@peri044
chore: more updates
1ae46e9
@peri044
chore: rebase with save
ae87fba
@peri044
chore: Update versions
beb5920
@peri044
chore: update tensorrt version in CI
f0068c6
@peri044
chore: more updates
39261b9
@peri044
chore: more fixes
3753150
@peri044
Merge branch 'release/2.3' into trt_10
16a191c
@peri044
chore: remove NvUtils.h
c355766
@peri044
chore: more updates
2d237dc
@peri044
chore: change lib64 to lib in rhel BUILD file
e4b4429
@peri044
chore: more updates
fa4fb9c
@zewenli98
Copy link
Collaborator

@peri044 I remember we've already removed cudnn dependency on the release/2.3, but it still stays here:

TensorRT/py/torch_tensorrt/__init__.py

Line 9 in 3f6999d

__cudnn_version__,

This causes an error when I import torch-trt:

>>> import torch_tensorrt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/scratch.zewenl_sw/docker_workspace/TRT10/TensorRT/py/torch_tensorrt/__init__.py", line 7, in <module>
    from torch_tensorrt._version import (  # noqa: F401
ImportError: cannot import name '__cudnn_version__' from 'torch_tensorrt._version' (/home/scratch.zewenl_sw/docker_workspace/TRT10/TensorRT/py/torch_tensorrt/_version.py)

Can you take a look?

@peri044
Copy link
Collaborator Author

peri044 commented May 17, 2024 •
edited

@peri044 I remember we've already removed cudnn dependency on the release/2.3, but it still stays here:

TensorRT/py/torch_tensorrt/__init__.py

Line 9 in 3f6999d

__cudnn_version__,

This causes an error when I import torch-trt:

>>> import torch_tensorrt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/scratch.zewenl_sw/docker_workspace/TRT10/TensorRT/py/torch_tensorrt/__init__.py", line 7, in <module>
    from torch_tensorrt._version import (  # noqa: F401
ImportError: cannot import name '__cudnn_version__' from 'torch_tensorrt._version' (/home/scratch.zewenl_sw/docker_workspace/TRT10/TensorRT/py/torch_tensorrt/_version.py)

Can you take a look?

fixed it now

@zewenli98
Copy link
Collaborator

Cool thanks! And did you implement the unit test for torch.ops.trt.quantize_fp8.default?

@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 21, 2024
@zewenli98
Copy link
Collaborator

@peri044 Thanks for the comments. I have refactored based on your suggestions.

peri044
peri044 commented May 22, 2024
View reviewed changes
examples/dynamo/vgg16_fp8_ptq.py Outdated Show resolved Hide resolved
examples/dynamo/vgg16_fp8_ptq.py Outdated Show resolved Hide resolved
examples/dynamo/vgg16_fp8_ptq.py Outdated Show resolved Hide resolved
gs-olive
gs-olive reviewed May 23, 2024
View reviewed changes
Copy link
Collaborator

@gs-olive gs-olive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good! Added a few comments

py/torch_tensorrt/dynamo/conversion/impl/quantize.py Outdated Show resolved Hide resolved
py/torch_tensorrt/dynamo/conversion/impl/quantize.py Outdated Show resolved Hide resolved
py/torch_tensorrt/dynamo/lowering/_remove_sym_nodes.py Show resolved Hide resolved
py/torch_tensorrt/dynamo/lowering/passes/remove_detach.py Show resolved Hide resolved
narendasan
narendasan reviewed May 23, 2024
View reviewed changes
py/torch_tensorrt/_enums.py Show resolved Hide resolved
py/torch_tensorrt/dynamo/_defaults.py Outdated Show resolved Hide resolved
py/torch_tensorrt/dynamo/conversion/aten_ops_converters.py Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@narendasan narendasan narendasan left review comments

@gs-olive gs-olive gs-olive left review comments

@zewenli98 zewenli98 Awaiting requested review from zewenli98

Assignees
No one assigned
Labels
cla signed component: api [Python] Issues re: Python API component: conversion Issues re: Conversion stage component: converters Issues re: Specific op converters component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: lowering Issues re: The lowering / preprocessing passes component: tests Issues re: Tests component: torch_compile documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@peri044 @zewenli98 @narendasan @gs-olive @facebook-github-bot