Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log throttling in files based on group rules #3535

Merged
merged 23 commits into from May 17, 2022

Conversation

Pranjal-Gupta2
Copy link
Contributor

@Pranjal-Gupta2 Pranjal-Gupta2 commented Oct 20, 2021

Which issue(s) this PR fixes:
None

What this PR does / why we need it:
Fluentd running in a big cluster with a high volume of logs has to deal with a lot of applications logging at different rates. High log generation rates for low priority applications can lead to log loss for high priority applications. We need to have a data flow control in the log collector so that we can limit the log collection rate of applications based on the priority of application workloads in the cluster.

Tail Plugin is watching a file and reads logs till it reaches the end of the file. Earlier accepted PR #3185 introduced throttling at the source level which applies a limit to the number of bytes collected by the in_tail plugin every second. However, it uniformly applies the same limit to every log file.

This PR enables us to form different groups based on configuration rules provided by the user. Within each group, one can set the rate limits for log collection. The set rate limit gets uniformly applied to each member (log file) of the group. If the group line limit of the log is reached within the rate_period time interval, Fluentd stops reading logs from that group. Once, rate_period (time) is over, Fluentd resets its group counters and restarts the reading process.

I strongly believe group level rate limiting can really help one to control low priority applications vs high priority applications log flow. Currently based on the namespace and app name (application name) extracted from file names, one can form groups for rate limiting.

Docs Changes:
fluent/fluentd-docs-gitbook#376

Release Note:
Same as Title

@ashie
Copy link
Member

ashie commented Oct 22, 2021

The concept would be nice, but implementing it as another plugin isn't good idea.
Please imagine several tail plugins that are focused on each features.

  • in_tail_feature1
  • in_tail_feature2
  • in_tail_feature3
  • ...

If a user wants feature1 and feature3, which plugin the user should select?
We should integrate it into existing tail plugin.

On the other hand, current in_tail’s code is complicated, I think refactoring might be needed before adding new major features.
Until doing it, providing it as an external plugin would be nice.

@Pranjal-Gupta2
Copy link
Contributor Author

The concept would be nice, but implementing it as another plugin isn't good idea. Please imagine several tail plugins that are focused on each features.

  • in_tail_feature1
  • in_tail_feature2
  • in_tail_feature3
  • ...

If a user wants feature1 and feature3, which plugin the user should select? We should integrate it into existing tail plugin.

Thank you for all your valuable feedback. In the in_tail plugin, one can rate limit using read_bytes_limit_per_second, but in this PR, in_tail_with_throttle provides a grouping mechanism for rate limiting files. These two features might seem mutually exclusive from an end-user point of view. That’s why we prepared a separate plugin that extends the functionality of the in_tail plugin. So, if one is interested in limiting each file by bytes per second, then one can use the in_tail plugin. Otherwise, you can add group rules in in_tail_with_throttle.

On the other hand, current in_tail’s code is complicated, I think refactoring might be needed before adding new major features. Until doing it, providing it as an external plugin would be nice.

Do you want to keep a separate plugin in this repository until in_tail is refactored and then merge it later?

@pmoogi-redhat
Copy link

@ashie can you further guide on this PR merging ? please note both in_tail and in_tail_with_throttle can co-exists and one can use both of them as set of . we didn't want to disturb the base line of in_tail hence we created a new plugin with this group based rate limiting functionality. let us know what step we need to take next.

@ashie
Copy link
Member

ashie commented Nov 4, 2021

My additional impressions:

  • Equivalent feature might be able to realized by multiple <source> and adding regexp based path parameter to in_tail. Could you describe the advantage of your patch compared with other ideas?
    • I think one of the strength of your idea is fallback mechanism, it makes easy to configure unmatched paths.
  • Again, implementing as an additional plugin isn't good idea, please consider integrating it into in_tail if you want to merge it into fluentd.
    • If you can integrate it cleanly, we might not need waiting refactoring.
  • The config format seems too rely on k8s's concept. It would be nice if you can generalize it.
  • We don't want maintaining Copy & Pasted codes. For example detach_watcher_after_rotate_wait and TailWatcher::IOHandler#rate_limit_handle_notify are almost same with parent's one. When parent one is modified, we have to modify both codes. Please consider integrating them.
  • Although in_tail_with_throttle might change the behaviour of in_tail, in_tail's tests aren't executed against in_tail_with_throttle.

@ashie ashie added the pending label Nov 4, 2021
@Pranjal-Gupta2
Copy link
Contributor Author

My additional impressions:

  • Equivalent feature might be able to realized by multiple <source> and adding regexp based path parameter to in_tail. Could you describe the advantage of your patch compared with other ideas?

    • I think one of the strength of your idea is fallback mechanism, it makes easy to configure unmatched paths.

@ashie Our plugin has the following advantages :

  1. Using multiple threads -> context switching and cpu usage

Adding regexp based path parameter to in_tail will add multiple source plugins which will create additional threads for reading from files. Hence cpu cycles will be wasted in context switching.

  1. Paths can be overlapped when using multiple source plugins

in_tail_with_throttle allows you to have generic and context-specifc rules for log rate limit. It will be difficult to manage with in_tail's path parameter based grouping and can lead to files falling In more than one group.

For example, consider the following requirement:
1. Rate limit all files with a particular namespace (Generic rule).
2. Give priority to specific app name in that namespace with a different limit (Context Specific Rule)

Configuration for such requirement will be like: 
<group_rule>
	...
	<rule>
		## rule1
		namespace ABC
		limit 1000 ## Generic rules for all files in a namespace
	</rule>	
	<rule>
		## rule2
		namespace ABC
		appname DEF  ## Specific rules with adjustable limit
		limit 1000
	</rule>	
	...
</group_rule>

in_tail_with_throttle will add a file with DEF app name and ABC namespace to rule 2 (Higher precedence over rule1). Using in_tail's path parameter will add the file in two groups and hence repeated log lines will be collected.

  1. Our plugin will take care of unexpected files.

In cases where we forget to add rules for some files, in_tail_with_throttle will add them to the default group and still collect logs. in_tail plugin with regex based path parameter grouping will fail in these situations.

  1. Config will get pretty cluttered.

Adding multiple source configs for in_tail's path based grouping will make your configuration file big in size. It will be difficult to manage and update configurations in case of very large number of groups.

@Pranjal-Gupta2
Copy link
Contributor Author

My additional impressions:

  • The config format seems too rely on k8s's concept. It would be nice if you can generalize it.

@ashie can you please elaborate on what this generalisation means? Should we replace our namespace and app-name fields with something else?

For example,

<rule> 
    ## level1 instead of namespace
    level1 ABC

    ## level2 instead of appname
    level2 DEF

    limit 1000
</rule>

@Pranjal-Gupta2
Copy link
Contributor Author

@ashie Could you please comment?

@agup006
Copy link
Member

agup006 commented Jan 31, 2022

@cosmo0920 would you also be able to take a look?

Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed and added a few concerns. Could you take a look about them?

lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
@ashie ashie removed the pending label Feb 1, 2022
Copy link
Member

@ashie ashie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for revising the patch, it's much better than before!
I'm sorry for my late response, I was thinking how to resolve remaining issues.
I commented about it above, please check it.

lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implementation idea is good. But, I found a nitpick issue for a constant name.

test/plugin/test_in_tail.rb Outdated Show resolved Hide resolved
lib/fluent/plugin/in_tail.rb Outdated Show resolved Hide resolved
@ashie ashie added this to the v1.15 milestone Feb 7, 2022
@ashie
Copy link
Member

ashie commented Feb 7, 2022

The big picture seems good, I'll check further more details, please wait for a few days...

@ashie
Copy link
Member

ashie commented Feb 7, 2022

A test always fails:

2022-02-07T07:27:29.7369178Z Error: test_shorter_than_rotate_wait(TailInputTest::singleline::log throttling per file::EOF with reads_bytes_per_second): Fluent::Test::Driver::TestTimedOut: Test case timed out with hard limit.
2022-02-07T07:27:29.7398448Z /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base.rb:201:in `rescue in run_actual'
2022-02-07T07:27:29.7398936Z /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base.rb:196:in `run_actual'
2022-02-07T07:27:29.7399358Z /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base.rb:95:in `run'
2022-02-07T07:27:29.7399753Z /home/runner/work/fluentd/fluentd/lib/fluent/test/driver/base_owner.rb:130:in `run'
2022-02-07T07:27:29.7400214Z /home/runner/work/fluentd/fluentd/test/plugin/test_in_tail.rb:714:in `test_shorter_than_rotate_wait'
2022-02-07T07:27:29.7400553Z      711:             tw
2022-02-07T07:27:29.7400872Z      712:           end.twice
2022-02-07T07:27:29.7401240Z      713: 
2022-02-07T07:27:29.7401498Z   => 714:           d.run(timeout: 10) do
2022-02-07T07:27:29.7403168Z      715:             until detached do
2022-02-07T07:27:29.7403680Z      716:               if d.events.size > 0 && !rotated
2022-02-07T07:27:29.7405460Z      717:                 cleanup_file("#{TMP_DIR}/tail.txt")
2022-02-07T07:27:29.7406123Z ===============================================================================
2022-02-07T07:29:08.6385152Z ...............................................................................
2022-02-07T07:29:08.6500370Z ...............................................................................
2022-02-07T07:29:08.6725292Z ...............................................................................

Signed-off-by: Pranjal Gupta <pranjal.gupta2@ibm.com>
Signed-off-by: Pranjal Gupta <pranjal.gupta2@ibm.com>
…r than string

Signed-off-by: Pranjal Gupta <pranjal.gupta2@ibm.com>
Signed-off-by: Pranjal Gupta <pranjal.gupta2@ibm.com>
Signed-off-by: Pranjal Gupta <pranjal.gupta2@ibm.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Copy link
Contributor

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good with a nitpick comment.

end

def remove_path_from_group_watcher(path)
return if @group.nil?
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we return nil on remove_path_from_group_watcher as well?
This is just nitpick and not important.

ashie added 2 commits May 9, 2022 01:00
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_in_tail.rb:183: warning: assigned but unused variable - d
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_in_tail.rb:205: warning: assigned but unused variable - d
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_in_tail.rb:220: warning: assigned but unused variable - d
/home/aho/Projects/Fluentd/fluentd/test/plugin/test_in_tail.rb:385: warning: ambiguous first argument; put parentheses or a space even after `-' operator
/home/aho/Projects/Fluentd/fluentd/lib/fluent/plugin/in_tail/group_watch.rb:123: warning: assigned but unused variable - e

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie
Copy link
Member

ashie commented May 9, 2022

Hmm, it takes too long time to run tests, more than 1 minutes longer than before:

master branch:

$ time bundle exec rake test TEST=test/plugin/test_in_tail.rb
...
real	2m33.910s
user	0m3.410s
sys	0m0.322s

this branch:

$ time bundle exec rake test TEST=test/plugin/test_in_tail.rb
...
real	3m36.044s
user	0m3.807s
sys	0m0.335s

We should shorten it if it's possible since it's affect to whole debugging efficiency.

ashie added 4 commits May 15, 2022 01:29
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie
Copy link
Member

ashie commented May 16, 2022

Hmm, it takes too long time to run tests, more than 1 minutes longer than before:

Trying to improve it by efa7d2d
It seems fine on my local environment but not sure on CI yet.

@ashie
Copy link
Member

ashie commented May 16, 2022

Hmm, it takes too long time to run tests, more than 1 minutes longer than before:

Trying to improve it by efa7d2d It seems fine on my local environment but not sure on CI yet.

It's also fine on CI.

@ashie
Copy link
Member

ashie commented May 16, 2022

I'll merge this after checking group_watch.rb once more.

@ashie
Copy link
Member

ashie commented May 16, 2022

Hmm, it takes too long time to run tests, more than 1 minutes longer than before:

Trying to improve it by efa7d2d It seems fine on my local environment but not sure on CI yet.

$ time bundle exec rake test TEST=test/plugin/test_in_tail.rb
...
real	2m45.993s
user	0m3.642s
sys	0m0.266s

ashie added 3 commits May 17, 2022 00:59
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
safety_mergin ->
safety_ratio

Signed-off-by: Takuro Ashie <ashie@clear-code.com>
Signed-off-by: Takuro Ashie <ashie@clear-code.com>
@ashie ashie merged commit e55d409 into fluent:master May 17, 2022
@ashie
Copy link
Member

ashie commented May 17, 2022

Merged.
I'm sorry for taking long time to merge this.
And thank you for contributing a nice feature 😃

@PranjalGupta2199
Copy link

Thanks @ashie @cosmo0920 @agup006 for your help and support! I am glad that it got merged! 🎉

@yuzs2
Copy link

yuzs2 commented Aug 8, 2022

looks this is a very useful feature, it would be even greater if you could provide more detailed user-guide for this. I'm new to fluentd, and the current example looks too simple for me: https://docs.fluentd.org/input/tail#less-than-group-greater-than-section, thank you

@yuzs2
Copy link

yuzs2 commented Sep 7, 2022

Hi folk, I tried to add the group section in my fluent.conf, but it was giving parse error:

/usr/local/rvm/rubies/ruby-2.7.6/lib/ruby/gems/2.7.0/gems/fluentd-1.15.0/lib/fluent/config/basic_parser.rb:92:in `parse_error!': got incomplete JSON hash configuration at kubernetes.conf line 230,10 (Fluent::ConfigParseError)

below is what my fluent.conf be like, removing group section it will be working fine:

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/{{.ID}}-fluentd-containers.log.pos
  pos_file_compaction_interval 1h
  tag kubernetes.*
  read_from_head true
  read_bytes_limit_per_second 8192
  <parse>
    @type multiline
    # cri-o
    format1 /^(?<partials>([^\n]+ (stdout|stderr) P [^\n]+\n)*)/
    format2 /(?<time>[^\n]+) (?<stream>stdout|stderr) F (?<log>[^\n]*)/
    # docker
    format3 /|^(?<json>{.*})/
    time_format %Y-%m-%dT%H:%M:%S.%N%:z
  </parse>
  <group>
    rate_period 600

    <rule>
      match {
          namespace: [/gulel-core/],
          podname: [/redis-master-0/]
      }
      limit 2
    </rule>
  </group>
</source>

could you please help? thanks @ashie @Pranjal-Gupta2 @cosmo0920 @agup006 @pmoogi-redhat

@ashie
Copy link
Member

ashie commented Sep 7, 2022

match value should be like this:

  match {
-      namespace: [/gulel-core/],
-      podname: [/redis-master-0/]
+      "namespace": "/gulel-core/",
+      "podname": "/redis-master-0/"
  }

The usage is describe in the document but it's also wrong, sorry for that. I'll fix it.

@ashie
Copy link
Member

ashie commented Sep 7, 2022

The usage is describe in the document but it's also wrong, sorry for that. I'll fix it.

fluent/fluentd-docs-gitbook#426

return true if @limit == 0

return false if @limit < 0
return false if @current_paths[path].number_lines_read < @limit / size
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this great feature!
I have a question about this code line.

The @limit controls the total number of lines collected for a group, right? (https://docs.fluentd.org/input/tail#less-than-group-greater-than-section)

That judgment seems to be done by this code line, but this line judges it by the average number of lines collected for a group, not the total number.

Either the code or the documentation is wrong?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, that's right. It should be total.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@daipom @ashie

This PR enables us to form different groups based on configuration rules provided by the user. Within each group, one can set the rate limits for log collection. The set rate limit gets uniformly applied to each member (log file) of the group.

The PR description states that the limit set in the configuration distributes the rate uniformly to all the members in a group. That's why the condition checks for average rate limit for a particular group member. I get why this is confusing and we may either change the documentation with an example or change the code itself.

The reason we are using average limit (for a group member) is because it puts a hard upper limit on the rate of logs emitted by a group. For example, a two groups with same limit but one having 1000 containers and the other having 10 containers, both will emit the same number of log lines.
However, if you apply the same limit to all the group members (without having a hard upper limit control) then there's no limit on the amount of data that is sent via fluentd. Considering the previous example, the rate of data sent from one group will be 100x larger.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Pranjal-Gupta2 Thanks!

Considering the previous example, the rate of data sent from one group will be 100x larger.

If we apply the limit to each file independently, it will be 100 times larger, right?

I think the preferred specification is to calculate the total number in the group in #4184.
In this way, it will be the same upper amount independent of the number of files.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! We can write a better articulation of this point in the documentation.
@ashie Can I take this up?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will think some more about what to do with #4184.
If it looks good to fix the document, I will fix it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup! We can write a better articulation of this point in the documentation. @ashie Can I take this up?

Oh, sorry. I missed this comment.
At this point, fixing the document would be a good idea!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We welcome the document modifications of course!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @daipom @ashie!
Will start working on this.

kavirajk pushed a commit to grafana/loki that referenced this pull request Oct 10, 2023
…10839)

[![Mend
Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [fluentd](https://www.fluentd.org/)
([source](https://togithub.com/fluent/fluentd)) | `'1.14.2'` ->
`'1.15.3'` |
[![age](https://developer.mend.io/api/mc/badges/age/rubygems/fluentd/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/rubygems/fluentd/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/rubygems/fluentd/'1.14.2'/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/rubygems/fluentd/'1.14.2'/1.15.3?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

### GitHub Vulnerability Alerts

####
[CVE-2022-39379](https://togithub.com/fluent/fluentd/security/advisories/GHSA-fppq-mj76-fpj2)

### Impact
A remote code execution (RCE) vulnerability in non-default
configurations of Fluentd allows unauthenticated attackers to execute
arbitrary code via specially crafted JSON payloads.

Fluentd setups are only affected if the environment variable
`FLUENT_OJ_OPTION_MODE` is explicitly set to `object`.

Please note: The option FLUENT_OJ_OPTION_MODE was introduced in Fluentd
version 1.13.2. Earlier versions of Fluentd are not affected by this
vulnerability.

### Patches
v1.15.3

### Workarounds
Do not use `FLUENT_OJ_OPTION_MODE=object`.

### References

* GHSL-2022-067

---

### Release Notes

<details>
<summary>fluent/fluentd (fluentd)</summary>

###
[`v1.15.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1153---20221102)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.2...v1.15.3)

##### Bug Fix

-   Support glob for `!include` directive in YAML config format

[fluent/fluentd#3917
-   Remove meaningless oj options

[fluent/fluentd#3929
-   Fix log initializer to correctly create per-process files on Windows

[fluent/fluentd#3939
-   out_file: Fix the multi-worker check with `<worker 0-N>` directive

[fluent/fluentd#3942

##### Misc

-   Fix broken tests on Ruby 3.2

[fluent/fluentd#3883

###
[`v1.15.2`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1152---20220822)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.1...v1.15.2)

##### Enhancement

-   Add a new system configuration `enable_jit`

[fluent/fluentd#3857

##### Bug Fix

-   out_file: Fix append mode with `--daemon` flag

[fluent/fluentd#3864
-   child_process: Plug file descriptor leak

[fluent/fluentd#3844

##### Misc

-   Drop win32-api gem to support Ruby 3.2

[fluent/fluentd#3849

###
[`v1.15.1`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1151---20220727)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.15.0...v1.15.1)

##### Bug Fix

-   Add support for concurrent append in out_file

[fluent/fluentd#3808

##### Misc

-   in_tail: Show more information on skipping update_watcher

[fluent/fluentd#3829

###
[`v1.15.0`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1150---20220629)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.6...v1.15.0)

##### Enhancement

-   in_tail: Add log throttling in files based on group rules

[fluent/fluentd#3535
-   Add `dump` command to fluent-ctl

[fluent/fluentd#3680
-   Handle YAML configuration format on configuration file

[fluent/fluentd#3712
- Add `restart_worker_interval` parameter in `<system>` directive to set
interval to restart workers that has stopped for some
reas[fluent/fluentd#3768

##### Bug fixes

-   out_forward: Fix to update timeout of cached sockets

[fluent/fluentd#3711
- in_tail: Fix a possible crash on file rotation when `follow_inodes
true`

[fluent/fluentd#3754
-   output: Fix a possible crash of flush thread

[fluent/fluentd#3755
-   in_tail: Fix crash bugs on Ruby 3.1 on Windows

[fluent/fluentd#3766
- in_tail: Fix a bug that in_tail cannot open non-ascii path on Windows

[fluent/fluentd#3774
- Fix a bug that fluentd doesn't release its own log file even after
rotated by
external
to[fluent/fluentd#3782

##### Misc

-   in_tail: Simplify TargetInfo related code

[fluent/fluentd#3489
-   Fix a wrong issue number in CHANGELOG

[fluent/fluentd#3700
-   server helper: Add comments to linger_timeout behavior about Windows

[fluent/fluentd#3701
-   service_discovery: Fix typo

[fluent/fluentd#3724
-   test: Fix unstable tests and warnings

[fluent/fluentd#3745

###
[`v1.14.6`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1146---20220331)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.5...v1.14.6)

##### Enhancement

-   Enable server plugins to specify socket-option `SO_LINGER`

[fluent/fluentd#3644
-   Add `--umask` command line parameter

[fluent/fluentd#3671

##### Bug fixes

-   Fix metric name typo

[fluent/fluentd#3630
- Apply modifications in pipeline to the records being passed to
`@ERROR` label

[fluent/fluentd#3631
-   Fix wrong calculation of retry interval

[fluent/fluentd#3640
-   Support IPv6 address for `rpc_endpoint` in `system` config

[fluent/fluentd#3641

##### Misc

-   CI: Support Ruby 3.1 except Windows

[fluent/fluentd#3619
-   Switch to GitHub Discussions

[fluent/fluentd#3654
-   Fix CHANGELOG.md heading styles

[fluent/fluentd#3648
-   Declare `null_value_pattern` as `regexp`

[fluent/fluentd#3650

###
[`v1.14.5`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1145---20220209)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.4...v1.14.5)

##### Enhancement

-   Add support for "application/x-ndjson" to `in_http`

[fluent/fluentd#3616
-   Add support for ucrt binary for Windows

[fluent/fluentd#3613

##### Bug fixes

-   Don't retry when `retry_max_times == 0`

[fluent/fluentd#3608
-   Fix hang-up issue during TLS handshake in `out_forward`

[fluent/fluentd#3601
-   Bump up required ServerEngine to v2.2.5

[fluent/fluentd#3599
-   Fix "invalid byte sequence is replaced" warning on Kubernetes

[fluent/fluentd#3596
- Fix "ArgumentError: unknown keyword: :logger" on Windows with Ruby 3.1

[fluent/fluentd#3592

###
[`v1.14.4`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1144---20220106)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.3...v1.14.4)

##### Enhancement

-   `in_tail`: Add option to skip long lines (`max_line_size`)

[fluent/fluentd#3565

##### Bug fix

- Incorrect BufferChunkOverflowError when each event size is <
`chunk_limit_size`

[fluent/fluentd#3560
- On macOS with Ruby 2.7/3.0, `out_file` fails to write events if
`append` is true.

[fluent/fluentd#3579
-   test: Fix unstable test cases

[fluent/fluentd#3574

###
[`v1.14.3`](https://togithub.com/fluent/fluentd/blob/HEAD/CHANGELOG.md#Release-v1143---20211126)

[Compare
Source](https://togithub.com/fluent/fluentd/compare/v1.14.2...v1.14.3)

##### Enhancement

-   Changed to accept `http_parser.rb` 0.8.0.
    `http_parser.rb` 0.8.0 is ready for Ractor.

[fluent/fluentd#3544

##### Bug fix

-   in_tail: Fixed a bug that no new logs are read when
    `enable_stat_watcher true` and `enable_watch_timer false` is set.

[fluent/fluentd#3541
-   in_tail: Fixed a bug that the beginning and initial lines are lost
after startup when `read_from_head false` and path includes wildcard
'\*'.[fluent/fluentd#3542
-   Fixed a bug that processing messages were lost when
    BufferChunkOverflowError was thrown even though only a specific
message size exceeds
chunk_limi[fluent/fluentd#3553

##### Misc

-   Bump up required version of `win32-service` gem.
newer version is required to implement additional `fluent-ctl` commands.

[fluent/fluentd#3556

</details>

---

### Configuration

📅 **Schedule**: Branch creation - "" (UTC), Automerge - At any time (no
schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/grafana/loki).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy44LjEiLCJ1cGRhdGVkSW5WZXIiOiIzNy44LjEiLCJ0YXJnZXRCcmFuY2giOiJtYWluIn0=-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

None yet

8 participants