Add sigdump func to fluent-ctl

[daipom]

Which issue(s) this PR fixes:
Fixes #3600

What this PR does / why we need it:
Currently, Sigdump is available by sending the CONT signal, without Windows.
The main purpose of this PR is to make this function available for Windows too.

To do so, add the following features to fluent-ctl and rpc.

We can use the following methods to use Sigdump.

• command: $ fluent-ctl dump {pid or svcname}
• http request to Fluentd: /api/processes.dump

Windows

• Fluentd outputs dump-files about the supervisor process and all worker processes to the system temp dir C:\\Windows\\Temp.
• We can specify SIGDUMP_PATH environment variable to change the path.
  • Automatically add each process-id to the path to avoid a writing conflict.

Examples of dump files on Windows.

C:\\Windows\\Temp\\fluentd-sigdump-{supervisor-pid}.log
C:\\Windows\\Temp\\fluentd-sigdump-{worker0-pid}.log
C:\\Windows\\Temp\\fluentd-sigdump-{worker1-pid}.log
...

Examples of specifying SIGDUMP_PATH by powershell on Windows.

$ $env:SIGDUMP_PATH="/sigdump/sigdump.log"

... [info]: fluent/log.rb:330:info: dump to /sigdump/sigdump-41544.log.
... [info]: #0 fluent/log.rb:330:info: dump to /sigdump/sigdump-21152.log.
... [info]: #1 fluent/log.rb:330:info: dump to /sigdump/sigdump-15656.log.
...

UNIX-like

• command: $ fluent-ctl dump {pid}

  • This is the same behavior as sending SIGCONT to the process as before. (Output the process's dump-file.)

• http request to Fluentd: /api/processes.dump

  • This is the same as Windows and outputs all processes' dump-files.
  • However, we cannot use SIGDUMP_PATH to specify the path for this rpc command.
    • Since each process-id is not added in this case and it causes a writing conflict.
    • We need to fix Sigdump library.

Docs Changes:
fluent/fluentd-docs-gitbook#397

Release Note:
TODO

[daipom]

This is still in draft status. I will continue to work on it.

[daipom]

I have confirmed the following actions correctly working on Windows.
(Output all processes' dump files (including the supervisor process) under C:\tmp\.)

• fluent-ctl dump {pid} and fluent-ctl dump fluentdwinsvc
  • As for fluentdwinsvc, I used following commands to run Fluentd as a Windows Service.

Powershell

$ fluentd --reg-winsvc I
$ fluentd --reg-winsvc-fluentdopt "-v -c {conf-path} -o {log-path}"
$ Start-Service fluentdwinsvc
$ fluent-ctl dump fluentdwinsvc
$ Stop-Service fluentdwinsvc
$ fluentd --reg-winsvc u

• curl http://127.0.0.1:24444/api/processes.dump
  • set rpc_endpoint 127.0.0.1:24444 to the system directive

TODO

• Add test code
• Test on UNIX-like
• Consider about supporting an error which occurs without /tmp dir
• About reopen_log: https://github.com/fluent/fluentd/pull/3680/files#diff-6042a07157b33b2d729ab3d10ed6bba89ad5f71aaed2be1aeb22a179a45ebbaeR330
• About threading: https://github.com/fluent/fluentd/pull/3680/files#diff-6042a07157b33b2d729ab3d10ed6bba89ad5f71aaed2be1aeb22a179a45ebbaeR992

[daipom]

I found the problem with UNIX-like system.
Currently kill CONT signal is sent infinitely.
I will fix this.

[daipom]

I finished the implementation and tests on Windows and Ubuntu but I found a problem with this specification.

We can set SIGDUMP_PATH environment variable to specify the dump-filepath. (https://github.com/fluent/sigdump#usage)

However, this specified path is not be separated by each pid.
So multiple dump-contents are written into the same file at the same time and it causes conflicts of writing.

I see two possible solutions.

• Fix Sigdump so that it adds pid to the path even with the environmental variable specified.
  • If any user uses the variable and the complete path systematically, this change may break the system...
• Limit this feature to Windows only.

[daipom]

I have tested on macOS too.
The failed tests are about ChildProcessTest and seem to have nothing to do with this fix.

[daipom]

I finished the implementation and tests on Windows and Ubuntu but I found a problem with this specification.

We can set SIGDUMP_PATH environment variable to specify the dump-filepath. (https://github.com/fluent/sigdump#usage)

However, this specified path is not be separated by each pid. So multiple dump-contents are written into the same file at the same time and it causes conflicts of writing.

I see two possible solutions.

* Fix [Sigdump](https://github.com/fluent/sigdump) so that it adds pid to the path even with the environmental variable specified.
  
  * If any user uses the variable and the complete path systematically, this change may break the system...

* Limit this feature to Windows only.

I would like to ask for opinion on this point.

[daipom]

On my Windows envrionment, test_supervisor_event_dump_windows fails under certain conditions.

bundle exec rake test TESTOPTS="-t'SupervisorTest'"

Failure: test_supervisor_event_dump_windows(SupervisorTest):
  class()
  Called 0 times.
  Expected 1 times.
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:74:in `block in dump_object_count'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:73:in `each_object'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:73:in `dump_object_count'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:18:in `block in dump'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:144:in `open'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:144:in `_open_dump_path'
C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:14:in `dump'
.../fluentd/lib/fluent/supervisor.rb:1128:in `dump_windows'
.../fluentd/lib/fluent/supervisor.rb:331:in `block in supervisor_dump_handler_for_windows'

C:/Ruby26-x64/lib/ruby/gems/2.6.0/gems/sigdump-0.2.4/lib/sigdump.rb:74

73 ObjectSpace.each_object {|o|
74   c = o.class

This test succeeds when I execute only this test.

Also, with difficulty, the following line can be commented out so that the test succeeds.

test_rpc_server_windows

@@ -347,7 +347,7 @@ class SupervisorTest < ::Test::Unit::TestCase

     server.run_rpc_server

-    mock(server).restart(true) { nil }
+    # mock(server).restart(true) { nil }
     response = Net::HTTP.get(URI.parse("http://#{localhost}:24447/api/plugins.flushBuffers"))

     server.stop_rpc_server

[ashie]

On my Windows envrionment, test_supervisor_event_dump_windows fails under certain conditions.

Hmm, it's also occurred on CI:
https://github.com/fluent/fluentd/runs/5656988063?check_suite_focus=true

[daipom]

On my Windows envrionment, test_supervisor_event_dump_windows fails under certain conditions.

Hmm, it's also occurred on CI: https://github.com/fluent/fluentd/runs/5656988063?check_suite_focus=true

Maybe I should not use Sigdump library itself, but the mock of it.

[daipom]

Completes the whole process except for the documentation.
If there are no problems with the specification, I will re-test the whole and create a PR for the document.

[ashie]

/api/processes.dump should be disabled by default to prevent dumping sensitive information.
If a user update Fluentd from old versions, it might be enabled unexpectedly.

[daipom]

Thank you for the fix.
I fixed the document: fluent/fluentd-docs-gitbook#397