Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Guard launching duplicated fluentd instance with same configuration #617

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from
Draft

Guard launching duplicated fluentd instance with same configuration #617

wants to merge 1 commit into from

Conversation

kenhys
Copy link
Contributor

@kenhys kenhys commented Feb 15, 2024 •
edited

Guard launching duplicated fluentd instance with same configuration

Concept;

Inject blocking code to /opt/fluent/bin/fluentd.

Before:

If you launch multiple Fluentd instance with same
configuration file, it causes a disaster with inconsistent
processed buffer or pos file.

After:

Detect fluentd service's main process and fetch FLUENT_CONF.
if configuration is same as spawned process, abort it.

It can block the following conditions are met:

  • fluentd is launched as a systemd service.
    configuration file is specified via FLUENT_CONF in fluentd.service.

  • manually try to launch fluentd with same configuration file like this:

    sudo /usr/sbin/fluentd -c /etc/fluent/fluentd.conf
    sudo /usr/sbin/fluentd
    sudo /opt/fluent/bin/fluentd -c /etc/fluent/fluentd.conf

Note that following case doen't resolved.

  • sudo /opt/fluent/bin/fluentd
    because default configuration path is /etc/fluent/fluent.conf.
  • /usr/sbin/fluentd -c /etc/fluent/fluentd.conf
  • /usr/sbin/fluentd
  • /opt/fluent/bin/fluentd -c /etc/fluent/fluentd.conf
  • /opt/fluent/bin/fluentd

This is because there is no appropriate privilege to retrieve
command line parameters.

NOTE: Windows is out of scope in this PR.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 15, 2024

NOTE: /opt/fluent/bin/fluentd is auto-generated file, so it should be patched.

@daipom daipom self-requested a review February 16, 2024 05:41
@kenhys kenhys force-pushed the guard-duplicated-instance branch 5 times, most recently from 0dcba92 to ab8f2bd Compare February 19, 2024 07:13
@daipom daipom mentioned this pull request Feb 21, 2024
Open
@kenhys kenhys force-pushed the guard-duplicated-instance branch 2 times, most recently from b1fc5b9 to d341283 Compare February 22, 2024 08:00
@kenhys
Copy link
Contributor Author

kenhys commented Feb 22, 2024

Observed service restarting in short times.
Need to investigating further more.

@kenhys kenhys force-pushed the guard-duplicated-instance branch 2 times, most recently from f860f53 to 3e89a51 Compare February 26, 2024 05:04
@kenhys
Copy link
Contributor Author

kenhys commented Feb 26, 2024

Update comment in this PR.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 26, 2024

Unexpectedly, download-artifact fails.

actions/download-artifact@master
Downloading single artifact
Preparing to download the following artifacts:
- packages-rockylinux-8 (ID: 1274111102, Size: 108455247)
Redirecting to blob download url: https://productionresultssa8.blob.core.windows.net/actions-results/b9412b87-4106-46a2-8fb0-86c757be7f85/workflow-job-run-ff7a8386-1a15-5f2b-97b2-c907b43b2f17/artifacts/6d0d09c616a6cee6f67320471aa47616aa04bd135f06b78e56dad6b651114b8f.zip
Starting download of artifact to: /home/runner/work/fluent-package-builder/fluent-package-builder
(node:1729) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Error: Unable to download artifact(s): Unable to download and extract artifact: Artifact download failed after 5 retries.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 26, 2024

It seems that there is no need to stick to download-artifact@master, created PR to use stable download-artifact@v4.
#624

@kenhys
Copy link
Contributor Author

kenhys commented Feb 26, 2024 •
edited

NOTE:
download warning itself is not related to whether using download-artifact@master.

[bug] Node incompatibility: [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues
actions/download-artifact#283

@kenhys kenhys force-pushed the guard-duplicated-instance branch 3 times, most recently from f0960bd to 96c47f9 Compare February 28, 2024 03:08
@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024

got it, it may be missing runtime package dependency.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024

checking with timeout.

@kenhys kenhys force-pushed the guard-duplicated-instance branch 5 times, most recently from 27fd5fd to 5320ed8 Compare February 28, 2024 08:37
@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024

As v5.0.3 is not released yet, test case in fresh install v5, lts should be skipped.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024

With debug commit, it has passed on bookworm

https://github.com/fluent/fluent-package-builder/actions/runs/8077515347/job/22068761536?pr=617

image

On Rocky

https://github.com/fluent/fluent-package-builder/actions/runs/8077515348/job/22068663355

image

@kenhys kenhys marked this pull request as ready for review February 28, 2024 09:11
@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024

It should pass CI. waiting results.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 28, 2024 •
edited

CI has passed except windows.

Failing on windows, but not related with this PR.

https://github.com/fluent/fluent-package-builder/actions/runs/8077907378/job/22069129797?pr=617
something wrong with ffi gem dependency.

It should be fixed in another PR.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 29, 2024

https://github.com/fluent/fluent-package-builder/actions/runs/8077907378/job/22069129797?pr=617
something wrong with ffi gem dependency.

It may be not accurate.

https://github.com/fluent/fluent-package-builder/actions/runs/8076317129/job/22064585573
The version of ffi related gems are same as before.

@kenhys
Copy link
Contributor Author

kenhys commented Feb 29, 2024

Failing on windows, but not related with this PR.

It is a bit strange, but it is not reproducible now...

@kenhys
Copy link
Contributor Author

kenhys commented Feb 29, 2024

All checks has passed now!

@kenhys kenhys added this to the 5.0.3 milestone Feb 29, 2024
@daipom daipom linked an issue Feb 29, 2024 that may be closed by this pull request
Prevent accidental duplicate launching #611
Open
@daipom daipom added the enhancement New feature or request label Feb 29, 2024
@kenhys kenhys modified the milestones: 5.0.3, 5.0.4 (T.B.D.) Mar 1, 2024
@kenhys kenhys force-pushed the guard-duplicated-instance branch from 3819a0e to 5ba3ff4 Compare March 5, 2024 05:27
@kenhys kenhys marked this pull request as draft March 5, 2024 09:42
@kenhys
Copy link
Contributor Author

kenhys commented Mar 5, 2024 •
edited

Now considering appropriate solution to make it simple.

A) guard /usr/sbin/fluentd and /opt/fluent/bin/fluentd

  • need to inject specific code to block duplicated instance.
    This means that implementation becomes complicated one (e.g. previous PoC )

B) guard only /usr/sbin/fluentd

  • it may be simpler, but can't block via /opt/fluent/bin/fluentd.

@kenhys kenhys force-pushed the guard-duplicated-instance branch from 5ba3ff4 to b3e30ac Compare March 6, 2024 08:13
@kenhys
deb rpm: guard multiple Fluentd instance
9b9ec9e 
Before:

  Even though fluentd service is running, you can execute duplicated
  Fluentd instance.
  It may cause inconsistency lost of buffer or pos file.

After:

  Guard multiple Fluentd instance when fluentd service is
  running by default.

  Note that --force-running-multiple-instance option is specified, you
  can run additional Fluentd instance explicitly. (it is same as
  previous behavior)

  NOTE: This guard trick is effective when /usr/sbin/fluentd is
  executed via sudo. It doesn't work when /opt/fluent/bin/fluentd is
  executed directly. This is intentinal because usually
  /opt/fluent/bin/fluentd is not searched from PATH.

Signed-off-by: Kentaro Hayashi <hayashi@clear-code.com>
@kenhys kenhys force-pushed the guard-duplicated-instance branch from b3e30ac to 9b9ec9e Compare April 4, 2024 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@daipom daipom Awaiting requested review from daipom

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
5.0.4 (T.B.D.)
Development

Successfully merging this pull request may close these issues.

Prevent accidental duplicate launching
2 participants
@kenhys @daipom