Memory Leak With Async File Target on Mono 4.2

[eleonel1]

We found that when we updated to the latest Mono Version (4.2) our services began to show an increase in memory usage, after some profiling and testing and some research we found out it was the Logging section the one to be blamed, this section uses NLog async writting to save different purpose logs. We removed the async part and the problem was fixed. You can find some test code and more details here https://github.com/eleonel1/Mono4.2NLogMemoryLeak

[304NotModified]

Hey! Can you create a PR for this?

[304NotModified]

I have read your post wrong. First of all, there is a config two (common) mistakes with the following config:

  <targets async="true">
    <target name="asyncFileDebug" xsi:type="AsyncWrapper">
      <target xsi:type="File" name="fileLogDebug" ...  />
    </target>
  </targets>
  <rules>
    <logger name="MemoryLeakTestLog" level="Debug" writeTo="fileLogDebug" />
  </rules>

1. We have now two nested async wrappers
2. You won't write to the async wrapper.

It should be (no AsyncWrapper)

  <targets async="true">
    <target xsi:type="File" name="fileLogDebug" ...  />
  </targets>
  <rules>
    <logger name="MemoryLeakTestLog" level="Debug" writeTo="fileLogDebug" />
  </rules>

OR (no async attribute, write to AsyncWrapper)

  <targets >
    <target name="asyncFileDebug" xsi:type="AsyncWrapper">
      <target xsi:type="File" name="fileLogDebug"       ...        />
    </target>
  </targets>
  <rules>
    <logger name="MemoryLeakTestLog" level="Debug" writeTo="asyncFileDebug" />
  </rules>

Can you test with one of the two configs?

Also, can you give me:

• The exact version number (nuget package version number)
• If this is new, and if so, which NLog version worked well
• Is this only under Mono?
• Is there a lot of log messages? How much?

Thanks in Advance!

[eleonel1]

Hello, glad to help, i tested both configurations and the result is the same, memory consumption increases over time.

This is the output when installing NLog from nuget

PM> Install-Package NLog
Attempting to gather dependencies information for package 'NLog.4.2.3' with respect to project 'MemoryLeakTest', targeting '.NETFramework,Version=v4.5'
Attempting to resolve dependencies for package 'NLog.4.2.3' with DependencyBehavior 'Lowest'
Resolving actions to install package 'NLog.4.2.3'
Resolved actions to install package 'NLog.4.2.3'
Adding package 'NLog.4.2.3' to folder 'C:\Users\Edgar\Documents\Visual Studio 2015\MemoryLeakTest\packages'
Added package 'NLog.4.2.3' to folder 'C:\Users\Edgar\Documents\Visual Studio 2015\MemoryLeakTest\packages'
Added package 'NLog.4.2.3' to 'packages.config'
Successfully installed 'NLog 4.2.3' to MemoryLeakTest

If i see the propertys of NLog.dll i get the following

This only hapens when running under mono

Mono JIT compiler version 4.2.1 (Stable 4.2.1.102/6dd2d0d Thu Nov 12 09:52:44 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS: __thread
SIGSEGV: altstack
Notifications: epoll
Architecture: amd64
Disabled: none
Misc: softdebug
LLVM: supported, not enabled.
GC: sgen

This didn't happen with the previous mono version, and this behaviour is shown even if the service isn't processing any requests. If you send 1 request and leave the service running idle the memory starts rising even if you don't send any other request.

[304NotModified]

This didn't happen with the previous mono version

Whats the previous version? 4.2.2? 4.1.0?

[304NotModified]

PS the changes of 4.2.3 (limited)

4.2.2...4.2.3

[eleonel1]

In Mono 3.12.1 NLog runs async without a problem.

[304NotModified]

I'm sorry, but what version do you mean? There is no NLog 3.12.1?

[eleonel1]

No, that's the mono version where the same code works without any problem.

[304NotModified]

Ow, so it's only a problem in Mono 4 and not in Mono 3.1.2?

But is there an NLog version (you know) which works well on Mono 4?

We found that when we updated to the latest Mono Version (4.2)

I did read NLog 4.2 for 4.2 here. Sorry about the confusion

[eleonel1]

Negative, only thing i did was update the mono version from 3.12.1 to 4.2.1, the NLog version is 4.2.3 and the process started to consume more memory over time.

I have 6 identical servers, i took one and removed the Async configurations of NLog and i got this results

[304NotModified]

you're using the nuget package right?

There is some mono-optimized code in NLog, but it won't work with the NuGet packages :( (there is no mono platform)

[eleonel1]

Yeah, i'm using the nuget package

[304NotModified]

I think this is related or this issue: http://lists.ximian.com/pipermail/mono-devel-list/2005-March/010883.html

edit: that's a too old post...

related code: https://github.com/NLog/NLog/blob/master/src/NLog/Targets/Wrappers/AsyncTargetWrapper.cs

[304NotModified]

There was also a PR for Mono 4 a while ago,

but AFAIK it did only copied the code #958

[304NotModified]

Has this been fixed in newer Mono builds?

[304NotModified]

I don't know if we can fix this. At least we need more info to look into this.

[304NotModified]

Is this still an issue?

[nyxiscoo1]

Yes, I'm expiriencing exact same issues when using NLog 4.4 with sync file target. Mono 4.6 and 4.8, but I believe leak is due mono bug. Donwgrade to Mono 4.0.4 solves the issue.
Platform Raspberry Pi 1 and 2.

[snakefoot]

I'm expiriencing exact same issues when using NLog 4.4 with sync file target

Sounds like a different problem, as the original poster doesn't have the problem when using sync-target (disappear when removing async).

Maybe the problem is reduced for original poster if the async-target uses timeToSleepBetweenBatches=0 (Requires NLog ver. 4.3.11 or newer). This will make the async-behavior more aggressive and decrease the chance that logevents are "captured" by the garbage collector for later cleanup (but instead will get immediate GC0 cleanup).

Also if using keepFilesOpen=false (the default, but slowest), then it can be a very good idea to lower bufferSize for the file-targets. Use bufferSize=4096 (Instead of default bufferSize=32768, that only makes sense when keepFilesOpen=true. No longer needed after #3444 in NLog 4.6.5)

[snakefoot]

An example of MONO_GC_PARAMS for keeping memory as low as possible:

export MONO_GC_PARAMS="nursery-size=1m,soft-heap-limit=128m,evacuation-threshold=90,save-target-ratio=0.1,default-allowance-ratio=1.0"
mono --desktop ASF.exe

If the device has a limited physical memory (Like Pi), then one can also consider to configure max-heap-size=256m for MONO_GC_PARAMS

[Sarsue]

Is anyone having this issue when deploying with docker??

[snakefoot]

Is anyone having this issue when deploying with docker??

Not sure what your issue is, but apparently there can be dragons when using console-out:

• http://stackoverflow.com/questions/27628276/docker-daemon-memory-leak-due-to-logs-from-long-running-process
• memory leak in buffer grow moby/moby#9139

[snakefoot]

Btw. original poster also posted a bug report to Xamarin/MONO, but their testers failed to reproduce the leak. When major collection occurred then memory dropped correctly:

https://bugzilla.xamarin.com/show_bug.cgi?id=37486

[Sarsue]

I dropped my mono version to v3.8.0 and the memory leak disappeared which works for me at the moment when I am less busy. I'll update my project .NET framework to 4.6 and test with mono version 4.2 again. Its just a hunch but I think there is some sort of correlation between the mono version and .NET framework.

[304NotModified]

Will close this as this is a known issue as it's polluting the issue list - there is no action for us here.