Integrate triton row and blockwise fp8 gemm to llm inference. #2547
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary: Add fp8 row/block-wise GEMM kernels with tests and benchmarks. Will register benchmark with TritonBench in separate pr. H100 500W ```bf16: shape (8192, 8192, 8192) tflops 585.23 ms 1.879 fp8 scale + row gemm: shape (8192, 8192, 8192) tflops 931.80 ms 1.180 fp8 scale + block gemm: shape (8192, 8192, 8192) tflops 594.84 ms 1.848 fp8 row gemm only: shape (8192, 8192, 8192) tflops 1125.51 ms 0.977 fp8 block gemm only: shape (8192, 8192, 8192) tflops 870.40 ms 1.263 bf16: shape (65536, 8192, 7168) tflops 575.12 ms 13.383 fp8 scale + row gemm: shape (65536, 8192, 7168) tflops 1024.09 ms 7.516 fp8 scale + block gemm: shape (65536, 8192, 7168) tflops 762.04 ms 10.100 fp8 row gemm only: shape (65536, 8192, 7168) tflops 1082.75 ms 7.108 fp8 block gemm only: shape (65536, 8192, 7168) tflops 828.34 ms 9.292 bf16: shape (65536, 3584, 8192) tflops 546.31 ms 7.044 fp8 scale + row gemm: shape (65536, 3584, 8192) tflops 876.66 ms 4.390 fp8 scale + block gemm: shape (65536, 3584, 8192) tflops 547.62 ms 7.027 fp8 row gemm only: shape (65536, 3584, 8192) tflops 1141.38 ms 3.372 fp8 block gemm only: shape (65536, 3584, 8192) tflops 828.31 ms 4.646 ``` Differential Revision: https://www.internalfb.com/diff/D56337896?entry_point=27
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
Summary: Integrate triton row/block-wise fp8gemm to llm inference Next commits will add activation_scale_ub and fused rowwise quant. Differential Revision: D56503139
|
This pull request was exported from Phabricator. Differential Revision: D56503139 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Integrate triton row/block-wise fp8gemm to llm inference
Next commits will add activation_scale_ub and fused rowwise quant.
Differential Revision: D56503139