Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add int8 support in fused_multi_transformer_pass and fuse_multi_transformer_layer_pass #48209

Merged
merged 20 commits into from Nov 30, 2022
Merged

Add int8 support in fused_multi_transformer_pass and fuse_multi_transformer_layer_pass #48209

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
4f068e8
delete unnecessary shape and slice op
RichardWooSJTU Nov 17, 2022
8e3efa2
add delete_weight_dequant_linear_op pass
RichardWooSJTU Nov 18, 2022
6bb4611
use multi inputs for out_scales in fused_multi_transformer
RichardWooSJTU Nov 18, 2022
73fbd9d
add int8 support for fuse_mt pass
RichardWooSJTU Nov 18, 2022
bfeefc9
optimiz int8 gemm
RichardWooSJTU Nov 21, 2022
e6e9bf6
optimize gemm int8
Nov 18, 2022
c90aa16
Revert "delete unnecessary shape and slice op"
RichardWooSJTU Nov 21, 2022
aa16466
merge develop
RichardWooSJTU Nov 21, 2022
8082bfb
match quant/dequant scale
RichardWooSJTU Nov 22, 2022
047d965
resolve conflict of auto parallel
RichardWooSJTU Nov 22, 2022
10b533e
delete debug codes
RichardWooSJTU Nov 22, 2022
837e72f
fix unittest
RichardWooSJTU Nov 22, 2022
d319449
fix ci build error
RichardWooSJTU Nov 23, 2022
202a7fe
fix scale calc
RichardWooSJTU Nov 28, 2022
f3e245e
fit auto parallel
RichardWooSJTU Nov 28, 2022
22d450e
code clean
RichardWooSJTU Nov 28, 2022
3bfa068
merge develop
RichardWooSJTU Nov 28, 2022
d2a9b0d
fix ci-inference build error
RichardWooSJTU Nov 28, 2022
7b3307c
fix unittest
RichardWooSJTU Nov 29, 2022
59c1182
clean unnecessary notes
RichardWooSJTU Nov 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions paddle/fluid/framework/ir/CMakeLists.txt
View file
Open in desktop
Expand Up @@ -96,6 +96,8 @@ pass_library(shuffle_channel_detect_pass inference)
pass_library(delete_quant_dequant_op_pass inference)
pass_library(delete_quant_dequant_filter_op_pass inference)
pass_library(delete_weight_dequant_linear_op_pass inference)
pass_library(delete_weight_dequant_linear_op_encoder_pass inference)
pass_library(delete_weight_dequant_linear_op_decoder_pass inference)
pass_library(delete_quant_dequant_linear_op_pass inference)
pass_library(delete_dropout_op_pass inference)
pass_library(delete_c_identity_op_pass inference)
Expand Down
17 changes: 15 additions & 2 deletions paddle/fluid/framework/ir/delete_quant_dequant_linear_op_pass.cc
View file
Open in desktop
Expand Up @@ -121,14 +121,27 @@ void DeleteQuantDequantLinearOpPass::ApplyImpl(ir::Graph* graph) const {
true,
platform::errors::InvalidArgument(
"Input scale tensor's place should be CPU."));
const float* input_scale_data = input_scale_tensor.data<float>();
float input_scale = input_scale_data[0];

float input_scale;
if (input_scale_tensor.dtype() == paddle::experimental::DataType::FLOAT32) {
const float* input_scale_data = input_scale_tensor.data<float>();
input_scale = input_scale_data[0];
} else if (input_scale_tensor.dtype() ==
paddle::experimental::DataType::FLOAT16) {
const phi::dtype::float16* input_scale_data =
input_scale_tensor.data<phi::dtype::float16>();
input_scale = static_cast<float>(input_scale_data[0]);
} else {
PADDLE_THROW(platform::errors::Unimplemented("%d is not supported.",
input_scale_tensor.dtype()));
}

int nums_any_ops = dequantize_linear_op_out->outputs.size();
for (int i = 0; i < nums_any_ops; ++i) {
auto* any_op_desc = dequantize_linear_op_out->outputs[i]->Op();
any_op_desc->SetAttr("Input_scale_" + quantize_linear_op_x->Var()->Name(),
input_scale);

// link x to any_op2
any_op_desc->RenameInput(dequantize_linear_op_out->Var()->Name(),
quantize_linear_op_x->Var()->Name());
Expand Down