Patch Arbitrary Code Execution

[aaronclong]

My company's internal deps auditing system is beginning to flag loguru because of this potential exploit. I don't know if it has come to your attention.

I don't know exactly how to prevent it, but am willing to help out if you need support. I will study the issue more.

418sec/huntr#1592

[aaronclong]

It looks like this method is the culprit: _from_pickled_value

I wonder if making it an instance method with mangling would solve the problem. A better solution would be to find another way to do this without pickling.

[aaronclong]

As far as I can tell, this pickingling is for multiprocessing that makes this so much more complex.

It was introduce with the 0.3.0 release in response to this issue: #102

[Delgan]

Hi @aaronclong, thanks for reporting this issue.

The Huntr team contacted me some time ago about the vulnerability you're mentioning: 418sec#1

I'm copy-pasting the comment I had left:

Hi.

Thanks for the security report. However, I'm not sure protecting __reduce__() in this case is really relevant. Sure, I understand that pickle.loads() is dangerous, but only when used with an untrusted source.

The RecordException is only meant to serialize Python errors. It will not be used again arbitrary data coming from network for example.

From what I see, the PoC is not different than calling os.system() in __str__:

import os

class MyClass:
    def __str__(self):
        os.system("xcalc")

logger.info(MyClass())

I am still quite confused by this report and I do not understand why Loguru is to blame.

The user receiving untrusted data should be responsible for sanitizing it before processing it. We can't expect this job to be done by the third-library, otherwise there might be infinite way to execute arbitrary code as demonstrated by the above example. This has little to do with the pickle module. Loguru isn't fetching and executing arbitrary code by itself, that's why it is difficult for me to recognize that there is a security problem in this library.

It is of course a pity that Loguru is reported as a dangerous library, and I wish it could be avoided. Perhaps you disagree with my analysis and can explain to me how my reasoning is wrong?

Of course, I'm in favor of improving Loguru safety and thanks for offering your help. However I want to understand the problem first and foremost to justify changes that may have multiple impacts (on features, performances and code complexity).

[aaronclong]

I wonder if dill would be viable to work around this, since this library controls its own serialization.

[aaronclong]

@Delgan Since you are using this for exception handeling, it is possible that an exception could trigger by a third-party library or whatever that would trigger it. Loguru would be just a catalyst in that workflow. Likewise, because it is in exceptions, it is extremely hard to sanitize that input.

My personal opinion that it is a logging library's responsibility to cover rogue execution. Using pickling is in general kind of dangerous.

[aaronclong]

@Delgan if you don't have any plans on fixing this or remedying it, I understand. I would like to know though because I will need to remove it from my applications unfortunately.

[Delgan]

Since you are using this for exception handeling, it is possible that an exception could trigger by a third-party library or whatever that would trigger it. Loguru would be just a catalyst in that workflow. Likewise, because it is in exceptions, it is extremely hard to sanitize that input.

Shouldn't the third-library with malicious __reduce__ method be flagged instead?

My personal opinion that it is a logging library's responsibility to cover rogue execution. Using pickling is in general kind of dangerous.

How is it different from a third-library with malicious __str__ method?

[aaronclong]

@Delgan I agree with you that the third party should be flagged as well, but I don't agree that absolves your library in this case. It is on all sides, especially as zero trust principles are concerned.

[aaronclong]

@Delgan likewise this code could in theory cause the same issue as the log4j exploit that happened this last month. It is quite similar to their arbitrary execution exploit.

Again, it's your library. No stress if this isn't the the path you want to go, I just need to know to make the appropriate changes on my end.

[aaronclong]

@Delgan likewise this code could in theory cause the same issue as the log4j exploit that happened this last month. It is quite similar to their arbitrary execution exploit.

Again, it's your library. No stress if this isn't the the path you want to go, I just need to know to make the appropriate changes on my end.

[Delgan]

As I said, I'm all in favor or improving Loguru's security, but I don't know how to do it in this case without compromising the features.

The exception needs to be somehow serialized. The exception comes from user's code. I can't restrain the allowed attributes during serialization because it it would unfairly limit usability for users with custom and trusted exceptions.

I'm genuinely trying to understand. Do you think Loguru should be responsible for analyzing __str__ method of every argument before displaying it?

Here is another example:

import os
from multiprocessing import Pool


class Malicious:
    def __reduce__(self):
        os.system("firefox")
        return (Malicious, ())

def f(x):
    return 0

if __name__ == "__main__":
    with Pool(processes=4) as pool:
        pool.map(f, [Malicious()])

Should multiprocessing be flagged because it can execute arbitrary code?

[aaronclong]

@Delgan I think __str__ is different that __reduce__. __reduce__ being automatically triggered by the interpreter causes problems. These are just long standing design flaws within python that unfortunately we have to patch over.

I don't know necessarily how to solve this problem, maybe dill or another format would make this less problematic. It could also be a custom internal format that is simple and easy to serialize/deserialize. We could even look into how the built in logger handles this scenario. I am certain we can find other resources for recommendation.

As for multiprocessing, it is a library with some flaws in it's design. However, I understand the need to support it. In general, there are people that choose to use alternatives to the built in multiprocessor like pathos. I am neither for nor against multiprocessing, but just want to acknowledge those issues.

[Delgan]

Sorry for maybe sounding too defensing , by the way. It's just that I truly don't understand how pickle would be different that any other method in this very specific case. I feel every library is flagged as unsafe just for using pickle. 😕

For example, __str__ can also be "automatically triggered by the interpreter":

import os

class Malicious:
    def __str__(self):
        os.system("firefox")
        return "M()"

m = Malicious()
string = f"M: {m}"

Anyway, initially you were only trying to solve a problem reported by a third party, so I don't want to bother you further. Thanks for answering my questions. :)

I had already looked for safe alternatives to pickle but I didn't find anything that suited me. The dill library offers more features than pickle but suffers from the same vulnerability issues. The standard logging library also uses pickle similarly to share the record to another process (which is also why Loguru needs to serialize the exception). Again, sorry to insist, but Loguru is just using pickle to serialize an user's class, so it's not different if the malicious code lived in the __reduce__ of another user's class.

I'll try to get in touch with Huntr team for discussing this issue further.

[Delgan]

Also I don't think it's fair to compare this issue with the Log4Shell exploit. To clarify, there is no way for Loguru to execute arbitrary code from user string input alone.

[aaronclong]

@Delgan it is being flagged for pickle and how it exposes pickle. To be clear, pickle isn't a safe method of serialization. It is a problematic legacy format.

Unfortunately, you can convert a pickle response to string and back again. Not saying it is the most likely scenario, but it is possible. Yes, it is quite similar to the log4j issue because it allowed for Java Objects to be loaded dynamically by the logger from another source. The string expansion was only one half of the equation. For more context, read here.

[aaronclong]

@Delgan bare minimum we can ask 418sec for a recommendation and point out the default logger doing something similar?

On looking at that PR again, I think that's a good middle ground approach to this.

[Delgan]

it is being flagged for pickle and how it exposes pickle. To be clear, pickle isn't a safe method of serialization. It is a problematic legacy format.

I agree. However, the pickle serialization is safe as long as the serialized object is itself safe. Regarding what is done by Loguru, there is no more security implication in using pickle than in formatting an object. Loguru assumes the parameters it receives are trusted because if a malicious class exists in the user code, it is already game over!

Unfortunately, you can convert a pickle response to string and back again. Not saying it is the most likely scenario, but it is possible. Yes, it is quite similar to the log4j issue because it allowed for Java Objects to be loaded dynamically by the logger from another source. The string expansion was only one half of the equation. For more context, read here.

I still fail to see how it relates to Log4Shell exploit. The exploit relies on JNDI to retrieve data from external source which is then possibly run locally, leading to arbitrary code execution vulnerability. Nothing like that is possible in Python formatting and Loguru does not permit it neither.

The pickle.loads() is not used to execute string coming from network or user input. It can only load already existing Exception object, if it's malicious that means it has been loaded carelessly by someone else.

@Delgan bare minimum we can ask 418sec for a recommendation and point out the default logger doing something similar?

On looking at that PR again, I think that's a good middle ground approach to this.

Sure, I would love to hear other opinions and possible clarifications.

[aaronclong]

@Delgan pickle serialization isn't safe, controlling what and who is passed to it s the only way to prevent issues. Currently, Loguru makes no attempt at controlling its inputs to pickel. The patch from 418 sec basically solves this problem. It chokes this off at the source and prevents rogue modules and types form being serialized.

You keep emphasizing this false dilemma that user is the fault and sole responsibility. They must sanitize everything before sending it to Loguru. While yes, the user has some job to bear here. They will not know all the things done with their logs and what to watch out for. You and other contributors will know what to look out for.

Again, the same issue with Log4j can affect Loguru if the right circumstances arise. The underlying technology is different but the same general exploit would be possible. Instead of JNDI, it would be some rogue input with pickel. While you keep saying *"Nothing like that is possible in Python formatting and Loguru does not permit it neither", there is evidence from 418 sec. Personally, I can think of scenarios where this could accidentally happen as well. You can choose to ignore that fact, but the issue will still persist in this code base.

While you keep saying you are open to discussion and ideas, I have yet to find that in any part of this conversation. I know you are well intentioned, but there seems no leeway in the approach you are choosing to take. I generally appreciate this library and think it is great. I thank you for putting it together. However, since this issue will not be resolved, I think I will just have to start the process to remove it from my apps.

[Delgan]

Personally, I can think of scenarios where this could accidentally happen as well.

May I ask you to please elaborate your concerns with some concrete example you have in mind? Are you able to share some kind of "minimum reproducible example" demonstrating how Loguru could cause both introduction and execution of malicious code? That would help me a lot. All I want is to understand precisely the problem raised so that I can eventually solve it while minimizing its negative impact on Loguru functionalities and performances.

I'm very open to discussion but none of the arguments I mentioned have been fully addressed, which leads me to feel that the campaign against pickle is unjustified in this case. I'm still very confused for the following reasons:

• How is the 418' POC different from crafting a malicious __str__ method and passing it to Loguru's logger?
• Why Loguru should be considered unsafe for internally using pickle while multiprocessing.Pool does the same and is considered safe?

I know I already asked you about this, but it still puzzles me. Can you please highlight how pickle is fundamentally flawed in the way Loguru uses it compared to any other malicious method ?

[aaronclong]

@Delgan please reread all the above. I have already explained all of this prior. We are just going in circles at this point. For my sanity and time, I am stopping. I may not have explained in the most clear and precise way, but I have laid it out the best I can. I wish I was a better communicator, but I am not regurgitating the same information for a third or fourth time.

Please reread the above, review the PR, and read up on the pickle module.

[Delgan]

Sorry for bringing it up again, I was just trying to summarize my concerns. I understand that you may not have the capacity to answer me precisely. For this reason, I asked the reporter directly: 418sec/huntr#1592 (comment)

The problem being you claim this issue is similar to Log4Shell exploit without providing any proof. That's a big claim that sounds quite irresponsible to be honest. I understand that you don't want to waste additional time here, but in the absence of proof, I hope that the discerning reader will realize that this statement is false and that the two issues are not comparable.

This is probably the first time since I've been maintaining Loguru that I've noticed tension while discussing with an user, and I'm sorry for that. You came here with a lot of good will. I did indeed stick to my guns and I understand how frustrating that is. My intention was not to waste your time. We have two different views, yours that pickle is undoubtedly insecure, and mine for which this module has legitimate use cases. Our reasoning were fundamentally incompatible.

I hope we can both take a step back. I will continue to investigate this issue, my goal is for users to be able to use Loguru without concern.

[aaronclong]

@Delgan again the proof is in this 418 pr, which you conveniently ignore.

I think the more irresponsible thing is to ignore the pitfalls here and not learn from others. Avoiding problems doesn't make them go away.

For the record, things can be insecure and have legitimate use cases. The internet is insecure but guard rails are put in to protect servers from exploits because it has legitimate use cases.

My frustration resolves from you making no effort to understand the issue, ignoring comments even though I answer your questions in them. You decided on your approach from the get-go with no intention to address the problem, and I am fine with that. However, I wish you had just told me that some 19 comments prior to avoiding this one-sided conversation. Stick to your guns as much as you need to, but you're deciding to ignore that this library is an attack vector. More and more security scans will mark this as such, effectively limiting its use.

[Delgan]

Hi @aaronclong. A few weeks have passed, I hope it gave time for minds to cool down. I would like to conclude this ticket.

It seems I gave you the impression that my choice regarding this issue was settled from the very beginning, but I can assure you that this is not true. My goal was to understand and prove the absence or presence of a security threat. Again, my intention wasn't to waste your time: I thanked you for your answers and invited you to pause the discussion earlier in this comment, but then it escalated in an unpleasant way.

I do care about security. I was just trying to expose my point of view according to which the reported vulnerability was questionable. The Huntr website provide a way to mark a report as "valid" or "invalid", I thought that was the right approach.

Please, understand that addressing such concerns isn't without consequences. There are multiple implications that I need to take into account (performances, complexity, functionalities). I can't just blindly apply a patch without careful consideration. For example, one of the suggested solution (establishing a "white list" of allowed modules) is very restrictive for custom user classes, and that's what I was worried about. It would have resulted in new bug reports about deserialization issues.

I didn't "conveniently ignore" the 418's PR, I even reached the author for clarifications. From my understanding, it only demonstrates that it's possible to indirectly call pickle.loads() through Loguru private API (where it's used in a safe way). I found this criticism debatable, which led me to continually try to find a convincing explanation of the security implications. In any case, even if it is tempting to compare it to Log4Shell by seeing "logging" associated with "arbitrary code execution", it's not even close (logger.info("Malicious input: {}", input()) is safe).

Your point about responsibility and importance of mitigating risks is very valid. However, removing pickle.loads() from Loguru's code base makes technically no difference. That's why I was in search of convincing argument proving me wrong. I couldn't figure out one compelling attack example.

To illustrate why this looks a bit silly to me: the pickle.loads() lives in a __reduce__ method, which itself is called by modules needing to unpickle the object (like multiprocessing.Queue). That implies a malicious Exception will still be able to run arbitrary code when unpickled, it's just it won't be triggered by the pickle.loads() inside Loguru.

Anyway, you are right when you mention that other security tools might automatically flag Loguru by detecting a use of pickle.loads(). I came to the same conclusion. I therefore decided to remove it (see 4b0070a) to avoid repeated discussions about this subject.

There is no "easy fix" otherwise I would have applied with without much debate. There are two downsides with this removal:

• exception is serialized twice which is a waste of resources
• invalid deserialization isn't caught which is an usability concern

I am not very satisfied, and I will try to find an alternative solution later, but I wanted to be able to close this ticket due to how it turned out. Thanks anyway for opening a ticket about security concerns, in the end it leads to a positive outcome as developers should be able to use Loguru with confidence from now on.

[aaronclong]

@Delgan when you called me irresponsible that greatly escalated the matter. It took a technical matter, and made it personal to me. Especially since I was attempting to do the responsible thing to help you fix the issue. I am still offended by the assumption. I do not believe you acted in good faith in any part of this conversation.

I do not agree with you technically on any matter on this thread, and especially on how this library uses the pickle module. However, I tried to not attack you at all or even make insinuations about you until that point.

While I am glad, you made moves on the patch. I do generally regret even broaching the issue because it wasn't worth it.

I wish you the best with this library. You won't have to worry about me bothering you more about this issue or anything else with this library.

[lynshi]

Hi @Delgan, appreciate your measured response to the issue at hand and eventual resolution. However, it doesn't look like the patch has made it into a release yet - are you planning on creating a new version soon?

As my company is also flagging loguru for this (officially named CVE-2022-0329) the easiest and best long-term fix for us is to consume an update. I will also look for a temporary internal workaround like getting an exception, so no rush in case you are holding back a release to investigate a cleaner solution.

[Delgan]

Hi @lynshi.

There are several breaking-changes I would like to implement before releasing a new minor version. Some of them are already present on the master branch so I can't do a patch-release. This implies I won't publish a new release before several weeks probably.

If this is a possibility for you, maybe you can install a specific hash of Loguru containing the patch:

pip install git+git://github.com/Delgan/loguru.git@4b0070a4f30cbf6d5e12e6274b242b62ea11c81b

[lynshi]

Hi @Delgan, thanks for the information! No problem at all; turns out our exception process is pretty painless, so this is no longer a significant blocker. Appreciate your work on this!

[orf]

This issue thread is quite bizarre. The maintainer is quite right in asking how something can be exploited and upon reading the discussions here it does seem to boil down to “you use pickle”, which is rather silly. The example exploit code in the linked PR boils down to:

os.system(“echo hacked”)

If someone could post a snippet showing how a user-inputted string makes it’s way into pickle then we have an actual exploit on our hands.

@Delgan your replies here are fantastic and I’m sorry this kind of issue has come up. But as it currently stands Dependabot (and other systems that use the same vuln feeds) are reporting to tens of thousands of repositories that there is a “critical remote code vulnerability in loguru”. While it seems to be complete crap, you may want to consider cherry-picking the commit and putting out a patch release without the breaking changes, just so people can update with the minimum amount of effort?

[Delgan]

Oh, it seems like there's been a lot of traffic around here lately.

@orf I didn't realize the consequences of having this vulnerability officially disclosed.
Thanks for alerting me about this, thanks for your feedback regarding the vulnerability, and thanks for your wise advice!

Under these circumstances, it's indeed not reasonable to wait several weeks or months before publishing a new version. Consequently, I just release 0.6.0 (breaking changes were minor anyway) which contains the patch for CVE-2022-0329.

I hope it will be helpful. 👍

[layday]

I'm confused as well. All that CVE is showing is that someone who can run Python code can call a Python method. Where is the actual exploit?

[capactiyvirus]

Yea if possible can anyone explain why the CVE is so high ?

[ms7m]

FWIW -- It seems the original reporter/source (huntr?) classified this as a 3.8?

No idea why this was classified so high.

[orf]

I summarized my thoughts on this in a blog post, and I hope some interesting discussions on this might appear on hacker news.

It's disconcerting to me that such an issue can make it's way into a CVE, then into a github advisory.

[SharonBrizinov]

@aaronclong aaronclong
It appears that you have a strong opinion about this topic. Could you help us identify at least one vulnerable application?

[aaronclong]

For context, I didn't create the CVE. I merely started the github issue around it.

[talkeren]

Sorry I had to sum it...

[jhutchings1]

Hello, I run the product team for supply chain security at GitHub, including our Advisory Database (where you can see this advisory) and Dependabot alerts (which we sent for this advisory).

First up, thank you everyone for the above. Clearly things didn't go to plan here, but having read through every comment I can see that everyone here had good intentions.

Having reviewed the above, the GitHub Advisory Database team are going to stop alerting on this advisory. We don't currently have a way to retract the Dependabot alerts we've already sent, but I've asked the team to look into functionality to do that in future. I've also asked them to look into adding a clear way to display CVEs in the GitHub Advisory Database that we have chosen not to alert on (even if they have not been withdrawn from the National Vulnerability Database).

I'd like us (GitHub) to learn from ☝🏻 and improve our security-related features and processes to help more when a maintainer receives a security report. We have a team of security researchers on staff dedicated to open source security (the GitHub Security Lab) who can be part of the solution. @Delgan would you be up for chatting with two of my colleagues, @KateCatlin and @xcorail?

[Delgan]

Hey @jhutchings1, thanks for this update from the Github team! Sure, I'm open to discussion if it can help. How should we proceed?

[jhutchings1]

Hey @jhutchings1, thanks for this update from the Github team! Sure, I'm open to discussion if it can help. How should we proceed?

Thanks! You can email me at my github handle @github.com and I'll connect you with folks directly.

[wmealing]

It's disconcerting to me that such an issue can make it's way into a CVE, then into a github advisory.

There is a method to 'dispute' a CVE, please let me know if you want this followed up as I am very familiar with this process.

[orf]

There is a method to 'dispute' a CVE, please let me know if you want this followed up as I am very familiar with this process.

I think that would be up to @Delgan, no? If not then I’d be interested in knowing more about the process

[wmealing]

Mitre has it documented here:

https://cve.mitre.org/cve/list_rules_and_guidance/correcting_counting_issues.html#dispute

Specifically the part:

Not everyone shares the same definition of a vulnerability. One person’s vulnerability is another person’s security hardening opportunity, and another person’s intended functionality. How does CVE deal with these differing opinions?

When an authoritative source disputes the validity of the vulnerability, “** DISPUTED **” is added to the beginning of the description, and a short NOTE is added to the end explaining why the vulnerability is disputed. Ideally, the disputing party provides a link that can be added to the CVE as a reference, and a quote that can be used as the explanation in the NOTE. However, neither are required.

You can lodge a dispute here: https://cveform.mitre.org/

Choose "Request a update to e CVE Entry"
Type of update: "Other"

My advice is to make a github issue or wiki page separate to this outlining the specifics of why is not an issue, and lock it. (People will almost definitely have 'input' on this, and you want the message to be your definitive voice, nobody else)

State that you'd like it to be disputed (you can't reject it, only the original CVE submitter can do that, sadly) and provide a link to your description on why this is not a flaw. Also, you can provide them with a small blurb that explains why it is not a flaw for their page.

This 'small blurb' will become visible on the CVE pages and tooling (and of course, vulnerability scanners) showing the correct state of the flaw and the current DISPUTED status. This can take up to two weeks, but Mitre is pretty good at it, they -may- refer you to github as the CNA and put you in contact with the github CNA ( https://www.cve.org/PartnerInformation/ListofPartners/partner/GitHub_M )

I do not know the original reporter of the flaw but contacting github on the top address may be able to put you in contact with the original reporter who -may- request REJECTION of the CVE.. but I've never had a reporter ever reject a CVE in multiple attempts over 10 years of working with flaws.

If you have any more questions wmealing @ < #ff0000 > hat.com . Thanks.

[carnil]

The CVE was rejected apparently now.

[wmealing]

Posted to wrong github issue. Removing.