You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
RuntimeError: Unable to find data type for weight_name='/encoder/layer.0/attention/output/dense/MatMul_output_0'. shape_inference failed to return a type probably this node is from a different domain or using an input produced by such an operator. This may happen if you quantize a model already quantized. You may use extra_options DefaultTensorType to indicate the default weight type, usually onnx.TensorProto.FLOAT.
#2598
Open
ARES3366 opened this issue
Apr 17, 2024
· 1 comment
RuntimeError: Unable to find data type for weight_name='/encoder/layer.0/attention/output/dense/MatMul_output_0'. shape_inference failed to return a type probably this node is from a different domain or using an input produced by such an operator. This may happen if you quantize a model already quantized. You may use extra_options DefaultTensorType to indicate the default weight type, usually onnx.TensorProto.FLOAT.
The text was updated successfully, but these errors were encountered:
from optimum.onnxruntime import ORTQuantizer
from optimum.onnxruntime.configuration import AutoQuantizationConfig
dynamic_quantizer = ORTQuantizer.from_pretrained(
output_model_path, 'model_optimized.onnx')
extra_options = {'DefaultTensorType': onnx.TensorProto.FLOAT}
dqconfig = AutoQuantizationConfig.avx512_vnni(is_static=False, per_channel=False)
dynamic_quantizer.quantize(save_dir=output_model_path,quantization_config=dqconfig)
tokenizer.save_pretrained(output_model_path) How should I change it
RuntimeError: Unable to find data type for weight_name='/encoder/layer.0/attention/output/dense/MatMul_output_0'. shape_inference failed to return a type probably this node is from a different domain or using an input produced by such an operator. This may happen if you quantize a model already quantized. You may use extra_options
DefaultTensorType
to indicate the default weight type, usuallyonnx.TensorProto.FLOAT
.The text was updated successfully, but these errors were encountered: