【Sparse】add some kernels(csrdense->csr, densedense->csr) of SparseTensor matmul #42935

zhwesky2010 · 2022-05-23T08:39:37Z

PR types

New features

PR changes

APIs

Describe

稀疏矩阵乘系列一共7个API，14个计算Kernel，本PR增加其中2个：

序号	API	输入输出类型1	输入输出类型2	输入输出类型3(csr)	输入输出类型4(csr)
1	paddle.incubate.sparse.mv		coo dense -> dense		csr dense -> dense
2	paddle.incubate.sparse.matmul	coo coo -> coo	coo dense -> dense	csr csr -> csr	✅ csr dense -> dense
3	paddle.incubate.sparse.matmul_sp		coo dense -> coo
4	paddle.incubate.sparse.matmul_hsp		coo dense -> hybrid coo
5	paddle.incubate.sparse.addmm	coo + coo * coo -> coo	dense + coo * dense -> dense	csr + csr * csr -> csr	dense + csr * dense -> dense
6	paddle.incubate.sparse.addmm_sp		coo + coo * dense -> coo
7	paddle.incubate.sparse.masked_matmul			✅ dense dense csr_mask-> csr

paddle.incubate.sparse.matmul

通用稀疏矩阵乘，支持 sparse与sparse运算、sparse与dense交互运算 等常规运算

import paddle
from paddle.fluid.framework import _test_eager_guard 
paddle.seed(100)

# csr @ dense -> dense

with _test_eager_guard():         
    crows = [0, 2, 3, 5]
    cols = [1, 3, 2, 0, 1]
    values = [1., 2., 3., 4., 5.]
    dense_shape = [3, 4]
    csr = paddle.incubate.sparse.sparse_csr_tensor(crows, cols, values, dense_shape)
    # Tensor(shape=[3, 4], dtype=paddle.float32, place=Place(gpu:0), stop_gradient=True,
    #       crows=[0, 2, 3, 5],
    #       cols=[1, 3, 2, 0, 1],
    #       values=[1., 2., 3., 4., 5.])
    dense = paddle.randn([4, 3])
    
    out = paddle.incubate.sparse.matmul(csr, dense)
    # Tensor(shape=[3, 3], dtype=float32, place=Place(gpu:0), stop_gradient=True,
    #        [[-1.94294846 , -3.33990622 ,  0.62359387 ],
    #         [-4.12815523 ,  3.46535444 , -3.27413893 ],
    #         [-0.15209436 , -19.23207283, -3.35593438 ]])

paddle.incubate.sparse.masked_matmul

特殊矩阵乘，相当于 dense矩阵乘、mask稀疏采样 两个操作的叠加，在某些模型使用

import paddle
from paddle.fluid.framework import _test_eager_guard
paddle.seed(100)

# dense @ dense * csr_mask -> csr

with _test_eager_guard():
    crows = [0, 2, 3, 5]
    cols = [1, 3, 2, 0, 1]
    values = [1., 2., 3., 4., 5.]
    dense_shape = [3, 4]
    mask = paddle.incubate.sparse.sparse_csr_tensor(crows, cols, values, dense_shape)
    # Tensor(shape=[3, 4], dtype=paddle.float32, place=Place(gpu:0), stop_gradient=True,
    #       crows=[0, 2, 3, 5],
    #       cols=[1, 3, 2, 0, 1],
    #       values=[1., 2., 3., 4., 5.])

    x = paddle.rand([3, 5])
    y = paddle.rand([5, 3])

    out = paddle.incubate.sparse.masked_matmul(x, y, mask)
    # Tensor(shape=[3, 4], dtype=paddle.float32, place=Place(gpu:0), stop_gradient=True, 
    #        crows=[0, 2, 3, 5], 
    #        cols=[1, 3, 2, 0, 1], 
    #        values=[0.98986477, 0.97800624, 1.14591956, 0.68561077, 0.94714981])

paddle-bot-old · 2022-05-23T08:39:56Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

chenwhql · 2022-06-13T11:51:11Z

paddle/fluid/pybind/eager_method.cc

@@ -1777,6 +1777,8 @@ PyMethodDef variable_methods[] = {
     METH_VARARGS | METH_KEYWORDS, NULL},
    {"cols", (PyCFunction)(void (*)(void))tensor_method_get_non_zero_cols,
     METH_VARARGS | METH_KEYWORDS, NULL},
+    {"is_dense", (PyCFunction)(void (*)(void))tensor_method_is_dense,


这个已经有了，在前面，可以看下，这里重复了

chenwhql · 2022-06-13T11:53:46Z

paddle/phi/kernels/sparse/cpu/sparse_full_kernel.cc

@@ -0,0 +1,113 @@
+/* Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.


这些文件名，前缀能去掉吗？sparse/cpu/sparse...，目录已经有sparse信息了，文件名不需要再重复一遍，略显冗余

去掉了，后续统一与dense kernel保持一致文件命名风格，不加前缀

chenwhql · 2022-06-13T11:54:27Z

paddle/phi/kernels/sparse/gpu/sparse_mm_kernel.cu

+#include "paddle/phi/kernels/copy_kernel.h"
+#include "paddle/phi/kernels/empty_kernel.h"
+#include "paddle/phi/kernels/funcs/sparse/sparse_blas.h"
+#include "paddle/phi/kernels/sparse/sparse_mm_kernel.h"


头文件要在最上面，这里头文件顺序不太对

cu文件会被pre-commit自动按字母序修改，所以cu没办法保持full_kernel.h在首行了

加空行隔开也不行吗，之前是可以的

试了下不行，看了下其他cu，应该都没成功

chenwhql · 2022-06-13T11:55:12Z

paddle/phi/kernels/sparse/gpu/sparse_mm_grad_kernel.cu

+#include "paddle/phi/kernels/empty_kernel.h"
+#include "paddle/phi/kernels/funcs/sparse/sparse_blas.h"
+#include "paddle/phi/kernels/sparse/sparse_mm_grad_kernel.h"
+#include "paddle/phi/kernels/transpose_kernel.h"


多处有这个问题，建议check下，https://zh-google-styleguide.readthedocs.io/en/latest/google-cpp-styleguide/headers/#include

已修改头文件规范，thx~

XiaoguangHu01

API 名称建议修改一下mm -> matmul

zhwesky2010 · 2022-06-14T07:41:13Z

API 名称建议修改一下mm -> matmul

已修改，thx~

XiaoguangHu01

LGTM

wanghuancoder

LGTM

… develop

wanghuancoder

LGTM

XiaoguangHu01

LGTM

…tmul (PaddlePaddle#42935) * add some kernel(csr*dense->csr, dense*dense->csr) of SparseTensor matmul * fix CI * fix CI * fix comment * fix comment

zhwesky2010 force-pushed the sparse_mm branch 3 times, most recently from b9ca201 to 1c86c1c Compare May 23, 2022 08:48

zhwesky2010 force-pushed the sparse_mm branch from 1c86c1c to c1e502a Compare May 23, 2022 09:12

zhwesky2010 force-pushed the sparse_mm branch from c1e502a to 8d3fcd4 Compare May 31, 2022 08:38

zhwesky2010 force-pushed the sparse_mm branch 2 times, most recently from e73c7c6 to f2aa2f0 Compare June 6, 2022 11:49

zhwesky2010 changed the title ~~add csr dense matmul of API:paddle.sparse.mm~~ add some kernel(csr*dense->csr, dense*dense->csr) of SparseTensor matmul Jun 6, 2022

zhwesky2010 force-pushed the sparse_mm branch 4 times, most recently from cf56517 to dc128d8 Compare June 7, 2022 02:58

zhwesky2010 force-pushed the sparse_mm branch from dc128d8 to 27858dd Compare June 7, 2022 03:16

zhwesky2010 force-pushed the sparse_mm branch from 27858dd to 7a6919c Compare June 7, 2022 03:27

zhwesky2010 force-pushed the sparse_mm branch from 7a6919c to 23675c7 Compare June 7, 2022 04:17

zhwesky2010 force-pushed the sparse_mm branch 7 times, most recently from 100c8fa to eb2d869 Compare June 7, 2022 14:39

fix CI

f61234d

zhwesky2010 force-pushed the sparse_mm branch from 48f8fa3 to f61234d Compare June 13, 2022 06:50

zkh2016 previously approved these changes Jun 13, 2022

View reviewed changes

kolinwei previously approved these changes Jun 13, 2022

View reviewed changes

chenwhql reviewed Jun 13, 2022

View reviewed changes

fix comment

60857b0

zhwesky2010 dismissed stale reviews from kolinwei and zkh2016 via 60857b0 June 13, 2022 15:06

zhwesky2010 requested a review from XiaoguangHu01 June 14, 2022 02:49

XiaoguangHu01 reviewed Jun 14, 2022

View reviewed changes

fix comment

bcf3bfb

XiaoguangHu01 previously approved these changes Jun 14, 2022

View reviewed changes

chenwhql previously approved these changes Jun 14, 2022

View reviewed changes

zkh2016 previously approved these changes Jun 14, 2022

View reviewed changes

kolinwei previously approved these changes Jun 14, 2022

View reviewed changes

wanghuancoder previously approved these changes Jun 14, 2022

View reviewed changes

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e964b42

… develop

zhwesky2010 dismissed stale reviews from wanghuancoder, kolinwei, zkh2016, chenwhql, and XiaoguangHu01 via e964b42 June 14, 2022 14:48

chenwhql approved these changes Jun 15, 2022

View reviewed changes

wanghuancoder approved these changes Jun 15, 2022

View reviewed changes

XiaoguangHu01 approved these changes Jun 15, 2022

View reviewed changes

zhwesky2010 merged commit 346efe9 into PaddlePaddle:develop Jun 15, 2022

zhwesky2010 changed the title ~~add some kernels(csr*dense->csr, dense*dense->csr) of SparseTensor matmul~~ 【Sparse】add some kernels(csr*dense->csr, dense*dense->csr) of SparseTensor matmul Jun 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

【Sparse】add some kernels(csrdense->csr, densedense->csr) of SparseTensor matmul #42935

【Sparse】add some kernels(csrdense->csr, densedense->csr) of SparseTensor matmul #42935

zhwesky2010 commented May 23, 2022 •

edited

paddle-bot-old bot commented May 23, 2022

chenwhql Jun 13, 2022

zhwesky2010 Jun 13, 2022

chenwhql Jun 13, 2022

zhwesky2010 Jun 13, 2022 •

edited

chenwhql Jun 13, 2022

zhwesky2010 Jun 13, 2022

zhwesky2010 Jun 13, 2022 •

edited

chenwhql Jun 14, 2022

zhwesky2010 Jun 14, 2022 •

edited

chenwhql Jun 13, 2022

zhwesky2010 Jun 13, 2022

XiaoguangHu01 left a comment

zhwesky2010 commented Jun 14, 2022

XiaoguangHu01 left a comment

wanghuancoder left a comment

wanghuancoder left a comment

XiaoguangHu01 left a comment

		@@ -0,0 +1,113 @@
		/* Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.

【Sparse】add some kernels(csr*dense->csr, dense*dense->csr) of SparseTensor matmul #42935

【Sparse】add some kernels(csr*dense->csr, dense*dense->csr) of SparseTensor matmul #42935

Conversation

zhwesky2010 commented May 23, 2022 • edited

PR types

PR changes

Describe

稀疏矩阵乘系列一共7个API，14个计算Kernel，本PR增加其中2个：

paddle.incubate.sparse.matmul

paddle.incubate.sparse.masked_matmul

paddle-bot-old bot commented May 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 Jun 13, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 Jun 13, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 Jun 14, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

zhwesky2010 commented Jun 14, 2022

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

wanghuancoder left a comment

Choose a reason for hiding this comment

wanghuancoder left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

【Sparse】add some kernels(csrdense->csr, densedense->csr) of SparseTensor matmul #42935

【Sparse】add some kernels(csrdense->csr, densedense->csr) of SparseTensor matmul #42935

zhwesky2010 commented May 23, 2022 •

edited

zhwesky2010 Jun 13, 2022 •

edited

zhwesky2010 Jun 13, 2022 •

edited

zhwesky2010 Jun 14, 2022 •

edited