【PaddlePaddle Hackathon 2】16 新增 API RRelu #41823

thunder95 · 2022-04-14T13:13:57Z

PR types

New features

PR changes

APIs

Describe

完成第二期第16项目开发任务: #40317
RRELU激活函数是从Empirical Evaluation of Rectified Activations in Convolutional Network中提出的，它是在Leaky ReLU基础上，对每一个位置的负值输入线性项做了随机采样，来加强一定范围内的泛化能力。
RFC设计文档: PaddlePaddle/community#71
中文文档: PaddlePaddle/docs#4725

paddle-bot-old · 2022-04-14T13:14:02Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

dingjiaweiww · 2022-04-15T01:49:25Z

请先通过CI噢~

… rrelu

zhiboniu · 2022-04-29T05:13:25Z

paddle/fluid/operators/rrelu_op.cc

@@ -0,0 +1,153 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved.


2016->2022，其他文件自查一下吧

zhiboniu · 2022-04-29T06:19:04Z

python/paddle/nn/layer/activation.py

+                            [ 6.0,  7.0,  8.0,  9.0]]]], 'float64')
+            x = paddle.to_tensor(data)
+            m = paddle.nn.RReLU(0.1, 0.3)
+            out = m(x)


示例代码需要正式一点，不能用m这种随意的名字，最好名字能看出来表示的功能

zhiboniu · 2022-04-29T06:37:41Z

paddle/fluid/operators/rrelu_op.cc

+ public:
+  using framework::OperatorWithKernel::OperatorWithKernel;
+
+  void InferShape(framework::InferShapeContext* ctx) const override {


正向的infershape已经放到phi里了，反向的infershape也一起放过去吧

zhiboniu · 2022-04-29T06:39:59Z

paddle/phi/kernels/cpu/rrelu_kernel.cc

+
+  if (is_test) {
+    for (i = 0; i < numel; i++) {
+      T mid_val = static_cast<T>((lower + upper) / 2.0);


这个mid_val计算可以放到for循环外边吧

zhiboniu · 2022-04-29T07:00:44Z

paddle/phi/kernels/gpu/rrelu_kernel.cu

+                   ALL_LAYOUT,
+                   phi::RReluKernel,
+                   float,
+                   phi::dtype::float16,


这个float16 cpu kernel也注册上吧

zhiboniu · 2022-04-29T07:10:46Z

python/paddle/fluid/tests/unittests/test_rrelu_op.py

+        self.x_shape = [2, 3, 4, 5]
+
+    def init_attr(self):
+        self.attrs = {'lower': self.lower, "upper": self.upper, "is_test": True}


is_test: False的情况也需要测试

完成，因opt_test的输出校验在1e-7范围内而noise是随机的，所以将lower和upper差距调到尽可能小来测试。

zhiboniu · 2022-04-29T07:11:05Z

python/paddle/fluid/tests/unittests/test_rrelu_op.py

+                name="x", shape=self.x_np.shape, dtype="float64")
+            x_2 = paddle.fluid.data(
+                name="x2", shape=self.x_np.shape, dtype="float64")
+            out_1 = F.rrelu(x_1, self.lower_0, self.upper_0, training=False)


training=True的情况也需要测试

@zhiboniu training的时候noise和输出都是随机的，然而在optest中的输出都是固定的，请问老师有什么建议？可以将lower和upper设置成一样的值么？

可以试一下固定seed结果能不能固定。如果还是不行我看你现在检测范围是否在[lowerx, upperx]也可以。

… rrelu

zhwesky2010

1.fix_seed这个参数应该不需要，seed 也不需要，目前的2.0后的写法是直接从generator中获取seed，不需要单独去给OP设置一个seed。

auto gen_cuda = ctx.GetGenerator();
uint64_t seed = seed_offset.first;
uint64_t offset = seed_offset.second;

2.GPU kernel 参考下 https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/exponential_op.cu#L29-L31 中的简洁写法：

using MT = typename kps::details::MPTypeTrait<T>::Type;
funcs::uniform_distribution<MT> dist;
funcs::uniform_real_transform<MT> trans(min, max);
funcs::distribution_and_transform<T>(dev_ctx, out, dist, trans);

调用https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/funcs/distribution_helper.h 已经封装好的函数即可。不要用thrust库，这个性能不太好。可以参考下https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/gpu/poisson_kernel.cu 比较新的写法

… rrelu

thunder95 · 2022-05-05T16:48:43Z

请注意补充中文文档啊

@zhiboniu 已补充，但未通过CI，不知为什么会参数名称校验失败

thunder95 · 2022-05-05T16:51:17Z

随机数改成调封装好的函数吧（具体可以参考现有调用的代码）： funcs::uniform_distribution dist; funcs::uniform_real_transform trans(min, max); funcs::distribution_and_transform(dev_ctx, out, dist, trans);

已修改 @zhiboniu

thunder95 · 2022-05-05T16:52:33Z

1.fix_seed这个参数应该不需要，seed 也不需要，目前的2.0后的写法是直接从generator中获取seed，不需要单独去给OP设置一个seed。
auto gen_cuda = ctx.GetGenerator();
uint64_t seed = seed_offset.first;
uint64_t offset = seed_offset.second;
2.GPU kernel 参考下 https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/exponential_op.cu#L29-L31 中的简洁写法：
using MT = typename kps::details::MPTypeTrait<T>::Type;
funcs::uniform_distribution<MT> dist;
funcs::uniform_real_transform<MT> trans(min, max);
funcs::distribution_and_transform<T>(dev_ctx, out, dist, trans);
调用https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/funcs/distribution_helper.h 已经封装好的函数即可。不要用thrust库，这个性能不太好。可以参考下https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/gpu/poisson_kernel.cu 比较新的写法

@zhouwei25 谢谢老师指导，已完成修改意见，麻烦看看这样改是否可以。

Ligoml · 2022-05-06T08:40:32Z

python/paddle/nn/layer/activation.py

@@ -436,6 +436,75 @@ def extra_repr(self):
            name_str)


+class RReLU(Layer):
+    """


格式混乱，请参考API文档规范修改

Ligoml · 2022-05-06T08:44:55Z

python/paddle/nn/functional/activation.py

@@ -548,6 +548,102 @@ def prelu(x, weight, data_format="NCHW", name=None):
    return out


+def rrelu(x, lower=1. / 8., upper=1. / 3., training=True, name=None):
+    """


一样的，格式混乱

尝试着修改了一下，但不知道怎么预览，如果可以就先自己检查了。

中英文预览都在docs的pr里面，可以参考文档

英文的公式还是稍微有点显示错误

Ligoml · 2022-05-06T08:47:09Z

python/paddle/nn/layer/activation.py

+            :name: RReLU-example
+
+            import paddle
+            import numpy as np


代码示例中不需要import numpy，可以使用paddle.to_tensor来生成输入数据

… rrelu

Ligoml · 2022-05-10T06:34:00Z

python/paddle/nn/layer/activation.py

+    """
+    rrelu activation.
+
+    `Empirical Evaluation of Rectified Activations in Convolutional Network`: https://arxiv.org/abs/1505.00853


把论文的超链接加在论文标题上

描述里可以多加一些介绍性内容，对用户友好一些

英文公式显示方面还是有些问题

@Ligoml 已修改这三点，但是预览文档页面只找到该API的中文文档，没有这个api的英文文档，是遗漏什么操作或是需要等？

需要paddle的PR-CI-Build跑完才会有英文文档预览~

zhwesky2010 · 2022-05-10T09:27:39Z

1.fix_seed这个参数应该不需要，seed 也不需要，目前的2.0后的写法是直接从generator中获取seed，不需要单独去给OP设置一个seed。
auto gen_cuda = ctx.GetGenerator();
uint64_t seed = seed_offset.first;
uint64_t offset = seed_offset.second;
2.GPU kernel 参考下 https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/fluid/operators/exponential_op.cu#L29-L31 中的简洁写法：
using MT = typename kps::details::MPTypeTrait<T>::Type;
funcs::uniform_distribution<MT> dist;
funcs::uniform_real_transform<MT> trans(min, max);
funcs::distribution_and_transform<T>(dev_ctx, out, dist, trans);
调用https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/funcs/distribution_helper.h 已经封装好的函数即可。不要用thrust库，这个性能不太好。可以参考下https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/phi/kernels/gpu/poisson_kernel.cu 比较新的写法
@zhouwei25 谢谢老师指导，已完成修改意见，麻烦看看这样改是否可以。

这样实现可以的，直接调funcs里的随机数公共组件就可以了

zhwesky2010 · 2022-05-13T11:23:05Z

LGTM

… rrelu

Ligoml · 2022-05-18T07:53:28Z

python/paddle/nn/layer/activation.py

+            \right.
+
+    where :math:`x` is the input tensor,
+    :math:`lower` and :math:`upper` are the bounds of uniform distribution.


比之前好多了，但还是有些不太对

中文是正常显示的，等下我重新构建一个预览看看

公式是从代码里直接拷贝到中文文档的，二者是一样的，不清楚为什么英文的还是有问题 @Ligoml

可以试试在文档最前方加一个 “r”

Ligoml

LGTM for docs

jeff41404 · 2022-05-24T07:08:29Z

the design of paddle.nn.functional.rrelu should also be add in RFC

thunder95 · 2022-05-24T09:24:38Z

the design of paddle.nn.functional.rrelu should also be add in RFC

@jeff41404 done. pr link: PaddlePaddle/community#137

XiaoguangHu01

LGTM

chenwhql · 2022-05-26T02:54:49Z

python/paddle/fluid/tests/unittests/test_rrelu_op.py

@@ -0,0 +1,326 @@
+#   Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.


2018->2022，后面可以再追加个PR更新下

* rrelu逻辑部分 * unregistered op kernel (unresolved) * commit before merge * 丰富测试用例 * 修复rrelu-sig的bug * 修复cpu环境测试 * 修改拼写错误 * 修改code format * 尝试优化测试用例timeout的问题 * 优化测试用例 * 移除seed, 优化随机函数 * update en doc for rrelu * fix rrelu en docs, test=document_fix * add paper link for en docs, test=document_fix * udpate en doc * add r,test=document_fix

thunder95 added 2 commits March 28, 2022 16:56

rrelu逻辑部分

b4022f3

unregistered op kernel (unresolved)

b03c8d1

paddle-bot-old bot added contributor External developers status: proposed labels Apr 14, 2022

thunder95 mentioned this pull request Apr 14, 2022

【PFCC】op开发提示没有注册kernel #41621

Closed

dingjiaweiww mentioned this pull request Apr 15, 2022

【PaddlePaddle Hackathon 第二期】任务总览 #40234

Closed

thunder95 added 11 commits April 16, 2022 02:29

commit before merge

a83e2bb

merge develop

b34da04

丰富测试用例

e5f3910

合并develop解决冲突

a5388cb

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

1ee342e

… rrelu

修复rrelu-sig的bug

f74eb8a

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

bb3b47e

… rrelu

修复cpu环境测试

71fdbab

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

c51cc32

… rrelu

修改拼写错误

f7cf53b

修改code format

a0cd822

zhiboniu reviewed Apr 29, 2022

View reviewed changes

thunder95 added 7 commits April 29, 2022 07:22

尝试优化测试用例timeout的问题

e5cf547

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a6f7c04

… rrelu

merge develop

fc85f79

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

b50216e

… rrelu

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

db09397

… rrelu

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

90102c0

… rrelu

优化测试用例

28cd511

dingjiaweiww added status: open review and removed status: proposed labels May 5, 2022

dingjiaweiww assigned zhiboniu May 5, 2022

zhwesky2010 reviewed May 5, 2022

View reviewed changes

thunder95 mentioned this pull request May 5, 2022

【PaddlePaddle Hackathon 2】16、doc for rrelu PaddlePaddle/docs#4725

Merged

移除seed, 优化随机函数

a5fed7f

thunder95 dismissed zhiboniu’s stale review via a5fed7f May 5, 2022 16:46

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

f2a9834

… rrelu

Ligoml reviewed May 6, 2022

View reviewed changes

thunder95 added 2 commits May 6, 2022 14:06

update en doc for rrelu

916b5b8

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

51493a3

… rrelu

Ligoml reviewed May 10, 2022

View reviewed changes

fix rrelu en docs, test=document_fix

726a5d5

add paper link for en docs, test=document_fix

4502a01

thunder95 added 2 commits May 14, 2022 08:31

udpate en doc

2132bf5

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

116873b

… rrelu

Ligoml reviewed May 18, 2022

View reviewed changes

add r,test=document_fix

8f626c2

Ligoml approved these changes May 24, 2022

View reviewed changes

jeff41404 approved these changes May 24, 2022

View reviewed changes

XiaoguangHu01 approved these changes May 25, 2022

View reviewed changes

chenwhql approved these changes May 26, 2022

View reviewed changes

jeff41404 merged commit 21e1d10 into PaddlePaddle:develop May 31, 2022

		@@ -0,0 +1,153 @@
		/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved.

		@@ -0,0 +1,326 @@
		# Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved.

【PaddlePaddle Hackathon 2】16 新增 API RRelu #41823

【PaddlePaddle Hackathon 2】16 新增 API RRelu #41823

Conversation

thunder95 commented Apr 14, 2022 • edited

PR types

PR changes

Describe

paddle-bot-old bot commented Apr 14, 2022

dingjiaweiww commented Apr 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 left a comment • edited

Choose a reason for hiding this comment

thunder95 commented May 5, 2022 • edited

thunder95 commented May 5, 2022

thunder95 commented May 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ligoml May 10, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhwesky2010 commented May 10, 2022

zhwesky2010 commented May 13, 2022

Choose a reason for hiding this comment

Ligoml May 18, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TCChenlong May 20, 2022 • edited

Choose a reason for hiding this comment

Ligoml left a comment

Choose a reason for hiding this comment

jeff41404 commented May 24, 2022

thunder95 commented May 24, 2022

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thunder95 commented Apr 14, 2022 •

edited

zhwesky2010 left a comment •

edited

thunder95 commented May 5, 2022 •

edited

Ligoml May 10, 2022 •

edited

Ligoml May 18, 2022 •

edited

TCChenlong May 20, 2022 •

edited