Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for concurrent append in out_file #3808

Closed
wants to merge 10 commits into from
Closed

Add support for concurrent append in out_file #3808

wants to merge 10 commits into from

Commits on Jul 1, 2022

  1. Add support for simple inter-worker locking 

    This is the first cut at a simple locking mechanism that allows
Fluentd workers to coordinate each other. Here is how it works:

 * ServerModule prepares a lockfile directory and share that directory
   path with its workers (through ENV).

 * Now, Workers can define critical sessions by creating a lockfile
   in that directory.

 * The actual lock was held by flock, so it's automatically freed by
   OS even when Worker gets killed in the middle of critical session.

I'm going to use this mechanism to support concurrent appends in
out_file, but should be useful for other plugins as well.

Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    dfd4fc3 View commit details
    Browse the repository at this point in the history

  2. out_file: Add a simple lock to support concurrent appends 

    Previously, out_file did not support any locking on append mode.

With this, out_file will acquire a worker lock when appending to
a file, which makes the following configuration just works:

  <system>
    workers 2
  </system>
  <match **>
    @type file
    append true
    ...
  </match>

Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    6a18e4d View commit details
    Browse the repository at this point in the history

  3. Apply feedback from kou on GitHub#3808 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    a538b9b View commit details
    Browse the repository at this point in the history

  4. Use automatic cleanup with FLUENTD_LOCKDIR 

    This is a lot safer than doing a manual cleanup, and it also
allows to clean up directories in a more rubust manner.

Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    75ef92f View commit details
    Browse the repository at this point in the history

  5. Raise error on configuration if lockdir is unavailable 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    2ebce28 View commit details
    Browse the repository at this point in the history

  6. Apply feedback from ashie on GitHub#3808 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    0a7a590 View commit details
    Browse the repository at this point in the history

  7. Add testcase for worker lock functions 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    d31631e View commit details
    Browse the repository at this point in the history

  8. Apply feeback from kou on GitHub#3808 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    d9c793e View commit details
    Browse the repository at this point in the history

  9. Fix variable name 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    f3838cb View commit details
    Browse the repository at this point in the history

  10. Add more tests for inter-worker locking 

    Signed-off-by: Fujimoto Seiji <fujimoto@ceptord.net>
    @fujimotos
    fujimotos committed Jul 1, 2022
    Copy the full SHA
    2c9f3f3 View commit details
    Browse the repository at this point in the history