Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'develop' into logspace_2
- Loading branch information
Showing
584 changed files
with
21,846 additions
and
5,439 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # Using PaddlePaddle Securely | ||
|
|
||
| This document describes model security and code security in PaddlePaddle. It also provides guidelines on how to report vulnerabilities in PaddlePaddle. | ||
|
|
||
| ## PaddlePaddle Model Security | ||
|
|
||
| PaddlePaddle attaches great importance to security and privacy of model. This includes how to prevent the model from outputting wrong decision results under the interference when it is used in security-related and safety-critical scenarios, and how to avoid leaking data and privacy information from the model itself, the model gradient or the model inference results. | ||
|
|
||
|
|
||
|
|
||
| [PaddleSleeve](https://github.com/PaddlePaddle/PaddleSleeve) provides a series of security and privacy tools, which can help model developers and users systematically evaluate and improve the model security and privacy in both development and deployment stages. | ||
|
|
||
|
|
||
|
|
||
| These tools include adversarial example evaluation test, pseudo-natural environment robustness evaluation test, model reversing evaluation test, member inference evaluation test, sample denoising, adversarial training, privacy enhancement optimizer, etc. | ||
|
|
||
| ## PaddlePaddle Code Security | ||
|
|
||
| PaddlePaddle always take code security seriously. However, due to the complexity of the framework and its dependence on other thirdparty open source libraries, there may still be some security issues undetected. Therefore, we hope that more security researchers and PaddlePaddle developers can participate in the code security program. We encourage responsible disclosure of security issues, as well as contributing code to improve our vulnerability finding tools to make PaddlePaddle safer. | ||
|
|
||
| ### Code security tools | ||
|
|
||
| PaddlePaddle security team attaches great importance to the security of the framework. In order to find and fix security issues as soon as possible, we are continuously conducting code security audit and developing automatic vunerability discovery tools. We have already open sourced some of them to the community, hoping this could encourage people to contribute and improve the safety and robustness of PaddlePaddle. [This tool](https://github.com/PaddlePaddle/PaddleSleeve/tree/main/CodeSecurity) includes two parts. The dynamic part includes some op fuzzer samples. And the static part includes some CodeQL samples. Both of them are aim to find vulnerabilities in PaddlePaddle framework codebase. By referring the samples, security researchers can write their own fuzzers or QLs to test more PaddlePaddle modules, and find more code security issues. | ||
|
|
||
| ### Reporting vulnerabilities | ||
|
|
||
| We encourage responsible disclosure of security issues to PaddlePaddle and please email reports about any security issues you find to security@paddlepaddle.org. | ||
|
|
||
|
|
||
|
|
||
| After the security team receives your email, they will communicate with you in time. The security team will work to keep you informed of an issue fix. | ||
|
|
||
|
|
||
|
|
||
| In order to reproduce and identify the issue, please include the following information along with your email: | ||
|
|
||
| - The details of the vulnerability including how to reproduce it. Try to attach a PoC. | ||
| - The attack scenario and what an attacker might be able to achieve with this issue. | ||
| - Whether this vulnerability has been made public. If it is, please attach details. | ||
| - Your name and affiliation. | ||
|
|
||
| We will indicate the bug fix in the release of PaddlePaddle, and publish the vulnerability detail and the reporter in the security advisories (Your name will not be published if you choose to remain anonymous). | ||
|
|
||
| ### What is a vulnerability? | ||
|
|
||
| In the process of computation graphs in PaddlePaddle, models can perform arbitrary computations , including reading and writing files, communicating with the network, etc. It may cause memory exhaustion, deadlock, etc., which will lead to unexpected behavior of PaddlePaddle. We consider these behavior to be security vulnerabilities only if they are out of the intention of the operation involved. | ||
|
|
||
|
|
||
|
|
||
| Some unexpected parameters and behaviors have been checked in PaddlePaddle by throwing exceptions in Python or return error states in C++. In these cases, denial of service is still possible, but the exit of the PaddlePaddle is clean. Since the error handling of PaddlePaddle is expected and correct, these cases are not security vulnerabilities. | ||
|
|
||
|
|
||
|
|
||
| If malicious input can trigger memory corruption or non-clean exit, such bug is considered a security problem. | ||
|
|
||
|
|
||
|
|
||
| [security advisories](https://github.com/PaddlePaddle/Paddle/blob/develop/security/README.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # 安全使用飞桨 | ||
|
|
||
|
|
||
|
|
||
| 本文将对飞桨模型及代码安全进行介绍，并介绍如何向飞桨提报漏洞。 | ||
|
|
||
| ## 飞桨模型安全 | ||
|
|
||
| 飞桨关注模型的安全性和隐私性。其中包括当模型被用于安全攸关场景时，如何避免模型在干扰下输出错误的决策结果，以及如何避免从模型本身、模型梯度或模型推理结果中泄露数据和隐私信息。 | ||
|
|
||
| 飞桨的安全和隐私套件[PaddleSleeve](https://github.com/PaddlePaddle/PaddleSleeve)提供了一系列工具，可帮助模型开发者及使用者在模型的开发或部署阶段，系统性地评估并提升模型的安全性和隐私性。这些工具包括对抗样本评估测试、拟自然环境鲁棒性评估测试、模型逆向评估测试、成员推断评估测试、样本去噪、对抗训练、隐私增强优化器等。 | ||
|
|
||
| ## 飞桨代码安全 | ||
|
|
||
| 飞桨团队一向非常重视代码安全，但鉴于飞桨框架的实现非常复杂，并且依赖了多个第三方开源库，其中仍可能会存在未被发现的问题。因此，我们希望有更多安全研究人员、飞桨开发者能参与到飞桨代码安全保障项目中来，我们鼓励向飞桨负责任的披露(Responsible Disclosure)安全问题，也鼓励向飞桨贡献代码完善动静态漏洞挖掘工具，让飞桨变得更安全。 | ||
|
|
||
| ### 安全工具 | ||
|
|
||
| 飞桨安全团队对于飞桨框架自身的安全高度重视，为了尽快地发现和修复安全问题，我们内部在持续地进行代码安全审计和研发自动化漏洞挖掘工具。我们将一些工具和方法开源给社区，希望能抛砖引玉，大家一起来贡献提高飞桨的安全性和鲁棒性。工具开源见[CodeSecurity](https://github.com/PaddlePaddle/PaddleSleeve/tree/main/CodeSecurity)。该开源工具包含两部分内容，分别从动态（模糊测试）和静态（CodeQL）两个角度对飞桨代码进行安全审计和漏洞挖掘。通过参照和添加新的测试模块，可以帮助覆盖更多飞桨代码模块，发现更多的代码安全问题。 | ||
|
|
||
| ### 报告安全问题 | ||
|
|
||
| 我们鼓励向飞桨负责任地披露安全问题，请将所发现的安全问题发送电子邮件到 security@paddlepaddle.org。 | ||
|
|
||
| 在安全团队收到邮件后将会及时与您沟通并反馈问题修复进度。 | ||
|
|
||
| 为了更好地复现和认定问题情况，请在邮件中： | ||
|
|
||
| - 详细描述漏洞细节，如何复现，并尽量附上PoC。 | ||
| - 描述攻击场景，介绍攻击者可能由此问题所能达到的效果。 | ||
| - 该问题是否已公开并描述情况。 | ||
| - 署名您的姓名和从属关系。 | ||
|
|
||
| 我们会将漏洞修复情况注明在飞桨的发布当中，并在致谢公告中发布漏洞情况和提报人（如果您选择不公开署名将不会发布提报人信息）。 | ||
|
|
||
| ### 安全问题认定说明 | ||
|
|
||
| 飞桨在计算图的过程中，由于模型可以执行任何计算，操作文件，进行网络通信等功能，可能造成内存耗尽，死锁等情况发生，这将导致飞桨产生一些非预期的行为。我们认为只有当这些行为超出了所涉及的操作意图时才算作是安全问题。 | ||
|
|
||
| 飞桨框架代码中对于一些非预期的参数和行为会进行检查，Python代码中以抛出异常为形式，C++代码中以返回错误状态为形式。这些情况下，飞桨代码的退出是干净的，但仍可能会因此造成拒绝服务，然而由于飞桨的处理是预期且正确的，所以造成这些情况并不算作是安全问题。 | ||
|
|
||
| 如果输入非预期的参数后，对飞桨代码造成了内存破坏，或者非干净退出，这类行为被认定为存在安全问题。 | ||
|
|
||
| ### [安全公告](https://github.com/PaddlePaddle/Paddle/blob/develop/security/README_cn.md) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # Copyright (c) 2018 PaddlePaddle Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| INCLUDE(ExternalProject) | ||
|
|
||
| #find_package(OpenSSL REQUIRED) | ||
|
|
||
| #message(STATUS "ssl:" ${OPENSSL_SSL_LIBRARY}) | ||
| #message(STATUS "crypto:" ${OPENSSL_CRYPTO_LIBRARY}) | ||
|
|
||
| #ADD_LIBRARY(ssl SHARED IMPORTED GLOBAL) | ||
| #SET_PROPERTY(TARGET ssl PROPERTY IMPORTED_LOCATION ${OPENSSL_SSL_LIBRARY}) | ||
|
|
||
| #ADD_LIBRARY(crypto SHARED IMPORTED GLOBAL) | ||
| #SET_PROPERTY(TARGET crypto PROPERTY IMPORTED_LOCATION ${OPENSSL_CRYPTO_LIBRARY}) | ||
|
|
||
| IF((NOT DEFINED ARM_BRPC_NAME) OR (NOT DEFINED ARM_BRPC_URL)) | ||
| SET(ARM_BRPC_VER "1.1.0" CACHE STRING "" FORCE) | ||
| SET(ARM_BRPC_NAME "arm_brpc" CACHE STRING "" FORCE) | ||
| ENDIF() | ||
|
|
||
| MESSAGE(STATUS "ARM_BRPC_NAME: ${ARM_BRPC_NAME}, ARM_BRPC_URL: ${ARM_BRPC_URL}") | ||
| SET(ARM_BRPC_PREFIX_DIR "${THIRD_PARTY_PATH}/arm_brpc") | ||
| SET(ARM_BRPC_PROJECT "extern_arm_brpc") | ||
| SET(ARM_BRPC_DOWNLOAD_DIR "${ARM_BRPC_PREFIX_DIR}/src/${ARM_BRPC_PROJECT}") | ||
| SET(ARM_BRPC_DST_DIR "output") | ||
| SET(ARM_BRPC_INSTALL_ROOT "${THIRD_PARTY_PATH}/install") | ||
| SET(ARM_BRPC_INSTALL_DIR ${ARM_BRPC_INSTALL_ROOT}/arm_brpc/output) | ||
| SET(ARM_BRPC_ROOT ${ARM_BRPC_INSTALL_DIR}) | ||
| SET(ARM_BRPC_INC_DIR ${ARM_BRPC_ROOT}/include) | ||
| SET(ARM_BRPC_LIB_DIR ${ARM_BRPC_ROOT}/lib) | ||
| SET(ARM_BRPC_LIB ${ARM_BRPC_LIB_DIR}/libbrpc.a) | ||
| SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_RPATH}" "${ARM_BRPC_ROOT}/lib") | ||
|
|
||
| INCLUDE_DIRECTORIES(${ARM_BRPC_INSTALL_ROOT}/${ARM_BRPC_NAME}/output/include) | ||
|
|
||
| FILE(WRITE ${ARM_BRPC_DOWNLOAD_DIR}/CMakeLists.txt | ||
| "PROJECT(ARM_BRPC)\n" | ||
| "cmake_minimum_required(VERSION 3.0)\n" | ||
| "install(DIRECTORY ${ARM_BRPC_DST_DIR} ${ARM_BRPC_DST_DIR} \n" | ||
| " DESTINATION ${ARM_BRPC_NAME})\n") | ||
|
|
||
| SET(ARM_BRPC_URL "https://paddlerec.bj.bcebos.com/online_infer/arm_brpc_ubuntu18/output.tar.gz" CACHE STRING "" FORCE) | ||
| ExternalProject_Add( | ||
| ${ARM_BRPC_PROJECT} | ||
| ${EXTERNAL_PROJECT_LOG_ARGS} | ||
| PREFIX ${ARM_BRPC_PREFIX_DIR} | ||
| DOWNLOAD_DIR ${ARM_BRPC_DOWNLOAD_DIR} | ||
| DOWNLOAD_COMMAND rm -rf output.tar.gz | ||
| && wget --no-check-certificate ${ARM_BRPC_URL} | ||
| && tar zxvf output.tar.gz | ||
| #DOWNLOAD_COMMAND cp /home/wangbin44/Paddle/build/output.tar.gz . | ||
| # && tar zxvf output.tar.gz | ||
| DOWNLOAD_NO_PROGRESS 1 | ||
| UPDATE_COMMAND "" | ||
| CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${ARM_BRPC_INSTALL_ROOT} | ||
| -DCMAKE_BUILD_TYPE=${THIRD_PARTY_BUILD_TYPE} | ||
| CMAKE_CACHE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=${ARM_BRPC_INSTALL_ROOT} | ||
| -DCMAKE_BUILD_TYPE=${THIRD_PARTY_BUILD_TYPE} | ||
| BUILD_BYPRODUCTS ${ARM_BRPC_LIB} | ||
| ) | ||
|
|
||
| ADD_LIBRARY(arm_brpc STATIC IMPORTED GLOBAL) # 直接导入已经生成的库 | ||
| SET_PROPERTY(TARGET arm_brpc PROPERTY IMPORTED_LOCATION ${ARM_BRPC_LIB}) | ||
| ADD_DEPENDENCIES(arm_brpc ${ARM_BRPC_PROJECT}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
8d04f59There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✅Congratulation! Your pull request passed all required CI. You could ask reviewer(s) to approve and merge. 🎉