Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

torch.jit.annotations.parse_type_line is not safe (command injection) even it seems already deprecated. #88868

Closed
Lyutoon opened this issue Nov 11, 2022 · 16 comments
Assignees
Labels
high priority oncall: jit Add this issue/PR to JIT oncall triage queue topic: security triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Milestone

Comments

@Lyutoon
Copy link

Lyutoon commented Nov 11, 2022

🐛 Describe the bug

In torch.jit.annotations, it looks like there are some functions that are deprecated, but still retain code, which may lead to some backdoors, especially since some of these functions still use eval while implementing.
But now I'm not sure if there are some features (jit decorator) in some version of pytorch are still using this function parse_type_line or get_signature which calls parse_type_line, if so, it can cause RCE, if not, maybe someone can also leave a backdoor by calling this function while writing code and share it to the people.

import torch

torch.jit.annotations.parse_type_line('# type: __import__("os").system("ls") -> 234', None, 1)

Versions

master

cc @ezyang @gchanan @zou3519 @EikanWang @jgong5 @wenzhe-nrv @sanchitintel

@malfet
Copy link
Contributor

malfet commented Nov 14, 2022

Marking for triage review(and not assigning oncall: jit yet otherwise it will disappear to the void) to discuss what to do with those kinds of security issues, which, in my opinion, is pretty minor: if one have an access to local Python runtime they can do anything they want.

@Lyutoon
Copy link
Author

Lyutoon commented Nov 14, 2022

Marking for triage review(and not assigning oncall: jit yet otherwise it will disappear to the void) to discuss what to do with those kinds of security issues, which is in my opinion is pretty minor: if one have an access to local Python runtime they can do anything they want.

Yes, this seems not a very urgent bug, but we still need to pay attention to these dangerouse functions such as eval. And in CVE-2022-0845, this bug is also caused by using eval to parse the args so it leads to code injection, and it seems also must have an access to local python. To be honest, I don't know how people think about these kind of problems, but we need to pay more attention :p. So we need a discuss about it whether it can be considered as a big security problem.

@malfet malfet self-assigned this Nov 14, 2022
@malfet
Copy link
Contributor

malfet commented Nov 14, 2022

We should have a doc marking unsafe function and safe versions of the same.
And also, probably should not use eval, if possible.

@albanD albanD added oncall: jit Add this issue/PR to JIT oncall triage queue and removed triage review labels Nov 14, 2022
@Lyutoon
Copy link
Author

Lyutoon commented Nov 14, 2022

We should have a doc marking unsafe function and safe versions of the same. And also, probably should not use eval, if possible.

That's right, there was also a cve (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/security/advisory/tfsa-2022-060.md) in tensorflow that used eval in saved_model_cli and caused code injection. Also I've found that PaddlePaddle has also eval problems (https://github.com/PaddlePaddle/Paddle/blob/develop/security/advisory/pdsa-2022-002.md). But sometimes, if we do not use eval, the code will become much more complex. Maybe developers can just add a critical check about the parameters of the function to avoid this problem easily (if possible).

@malfet malfet added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 17, 2022
@malfet
Copy link
Contributor

malfet commented Nov 17, 2022

It seems reasonable, that valid type annotations should not have any funciton/method calls, so splitting it into compile (which is safe)->error on function calls->eval would secure the deprecated codebase without overcomplicating deprecated code too much

@roywei
Copy link

roywei commented Nov 29, 2022

Hi guys, is there any plan to release this patch to torch 1.12.x and 1.13.x? any ETAs? thanks!

@seemethere
Copy link
Member

Hi guys, is there any plan to release this patch to torch 1.12.x and 1.13.x? any ETAs? thanks!

We don't currently have any plans to do continued releases for 1.12.x but given the security implications of this this patch will be included in the next patch release for 1.13.x

@seemethere seemethere added this to the 1.13.1 milestone Nov 30, 2022
atalman pushed a commit to atalman/pytorch that referenced this issue Nov 30, 2022
Introduce `_eval_no_call` method, that evaluates statement only if it
does not contain any calls(done by examining the bytecode), thus preventing command injection exploit

Added simple unit test to check for that
`torch.jit.annotations.get_signature` would not result in calling random
code.

Although, this code path exists for Python-2 compatibility, and perhaps
should be simply removed.

Fixes pytorch#88868

Pull Request resolved: pytorch#89189
Approved by: https://github.com/suo
malfet added a commit that referenced this issue Nov 30, 2022
Introduce `_eval_no_call` method, that evaluates statement only if it
does not contain any calls(done by examining the bytecode), thus preventing command injection exploit

Added simple unit test to check for that
`torch.jit.annotations.get_signature` would not result in calling random
code.

Although, this code path exists for Python-2 compatibility, and perhaps
should be simply removed.

Fixes #88868

Pull Request resolved: #89189
Approved by: https://github.com/suo

Co-authored-by: Nikita Shulga <nshulga@meta.com>
@bendeaton
Copy link

Can someone explain in more detail what the actual vulnerability is and how to determine whether a given code base actually uses the offending code?

The Pytorch/jit docs (link) note that “Python 3 type hints can be used in place of torch.jit.annotate” ; does this imply that it’s possible to ameliorate the vulnerability by fully type-hinting all user functions compiled by jit, and if so how can these functions be identified?

Does this vulnerability apply only to compiled functions using Pytorch, or does it extend to things that could be injected into an inference request to trigger the privilege escalation issue?

@ezyang
Copy link
Contributor

ezyang commented Dec 5, 2022

I believe the vulnerability is, if someone crafts a malicious Python file, and then you compile it TorchScript, it can trigger arbitrary code execution. That being said, I'm not really sure what your threat model is, since you probably already have problems if you're compiling arbitrary malicious Python code with TorchScript?

@ardell
Copy link

ardell commented Dec 5, 2022

That was my assessment as well, based on a cursory read of the issue and the docs I could find.

It does seem confusing that the CVSS score lists the attack vector as "Network", attack complexity as "Low", and privileges required as "None" if this vulnerability requires compiling the malicious python code to TorchScript.

The CVSS makes it seem like someone could trigger the vulnerability via (for example) a maliciously-crafted inference request (without any arbitrary code compilation) -- which would certainly be a critical security vulnerability. I'm interested in ruling that out if possible, especially while we wait for a patched version to be released...

@ezyang
Copy link
Contributor

ezyang commented Dec 6, 2022

Definite not via user controlled tensor inputs. There is no vector for type annotations there

@malfet
Copy link
Contributor

malfet commented Dec 6, 2022

I guess the problem here is that torch.jit.annotations.parse_type_line looks like a public API, which means that dev can use it in their code to extract function annotation and arbitrary code execution feels like a very unexpected side-effect of calling such function.

@abustamante-coveo
Copy link

hello, is there an ETA on 1.13.1?

@malfet
Copy link
Contributor

malfet commented Dec 6, 2022

@abustamante-coveo tentatively Dec 15th, see #89855

kulinseth pushed a commit to kulinseth/pytorch that referenced this issue Dec 10, 2022
Introduce `_eval_no_call` method, that evaluates statement only if it
does not contain any calls(done by examining the bytecode), thus preventing command injection exploit

Added simple unit test to check for that
`torch.jit.annotations.get_signature` would not result in calling random
code.

Although, this code path exists for Python-2 compatibility, and perhaps
should be simply removed.

Fixes pytorch#88868

Pull Request resolved: pytorch#89189
Approved by: https://github.com/suo
@abustamante-coveo
Copy link

hello, Snyk still marks torch as having a critical vulnerability even after the 1.13.1 patch. Was this vulnerability supposed to be patched on that version?

@vikcher
Copy link

vikcher commented Dec 16, 2022

hello, Snyk still marks torch as having a critical vulnerability even after the 1.13.1 patch. Was this vulnerability supposed to be patched on that version?

Looks like their vulnerability database is not updated after the fix. It shows that the fix is pushed but not published:
https://security.snyk.io/vuln/SNYK-PYTHON-TORCH-3149871

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high priority oncall: jit Add this issue/PR to JIT oncall triage queue topic: security triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Development

Successfully merging a pull request may close this issue.

10 participants