From 1e6f78f314270b6612d147c9bfcb5911a222c70d Mon Sep 17 00:00:00 2001 From: Pranjal-Gupta2 Date: Mon, 18 Oct 2021 17:38:21 +0530 Subject: [PATCH 1/3] in_tail_with_throttle: Add documentation for `` directives Signed-off-by: Pranjal-Gupta2 --- input/tail_with_throttle.md | 281 ++++++++++++++++++++++++++++++++++++ 1 file changed, 281 insertions(+) create mode 100644 input/tail_with_throttle.md diff --git a/input/tail_with_throttle.md b/input/tail_with_throttle.md new file mode 100644 index 00000000..c8ebe075 --- /dev/null +++ b/input/tail_with_throttle.md @@ -0,0 +1,281 @@ +# tail_with_throttle + +![](../.gitbook/assets/tail.png) + +The `in_tail_with_throttle` Input plugin allows Fluentd to read events from the tail of text files. Its behavior is similar to that of `in_tail` Input Plugin with the only difference being that this plugin rate limits the number of events read on the basis of groups. + +## Example Configuration + +```text + + @type tail_with_throttle + path /var/log/*.log + pos_file /var/log/temp.log.pos + tag applogs + + @type apache2 + + + pattern /(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$ + rate_period 30s + + namespace logging + appname frontend + limit 500 + + + namespace logging + appname backend + limit 1000 + + + limit 20 + + + +``` + +Refer to the [Configuration File](../configuration/config-file.md) article for the basic structure and syntax of the configuration file. + +For ``, see [Parse Section](../configuration/parse-section.md). + +### How It Works + +When Fluentd is configured with `in_tail_with_throttle`, it will form groups based on the rules provided in the configuration file. Based on group formation, log files will be assigned to a group and reading will start from the **tail** of that log, not the beginning. Group limits are equally distributed among the log files in a group. If the group line limit of the log is reached within rate_period time interval, Fluentd will stop reading logs. Once, **rate_period** (time) is over, Fluentd will reset its group counters and restart the reading process. + + +## Plugin Helpers + +* [`timer`](../plugin-helper-overview/api-plugin-helper-timer.md) +* [`event_loop`](../plugin-helper-overview/api-plugin-helper-event_loop.md) +* [`parser`](../plugin-helper-overview/api-plugin-helper-parser.md) +* [`compat_parameters`](../plugin-helper-overview/api-plugin-helper-compat_parameters.md) + +See also: [Linux capability](../deployment/linux-capability.md) + +## Parameters + +See [Common Parameters](../configuration/plugin-common-parameters.md). + +`in_tail_with_throttle` Input Plugin utilizes `in_tail` plugin's configuration. Refer to `in_tail` plugin's parameters [in_tail Parameters](./tail.md) + +The additional parameter which `in_tail_with_throttle` uses is the `` Directive. + +### `` Directive \(required\) + +Grouping rules for the log. + +`in_tail_with_throttle` assigns each log file to a group based on the rules provided in **``** directive. The `limit` parameter controls the total number of lines collected for a group by `tail_with_throttle` Input Plugin. + +Example: + +```text +# group rules + + rate_period 5s + + + namespace shopping + appname frontend + limit 1000 + + + + namespace payment + appname backend + limit 2000 + + +``` + +#### `pattern` + +| type | default | version | +| :--- | :--- | :--- | +| regexp | `/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`| 0.14.1 | + +Specifies the regular expression for extracting metadata (namespace, appname) from log file path. Default value of the pattern regexp follows CRIO path format which gives information about `namespace`, `pod` (appname), `docker_id`, `container` of the log. + +Please add `namespace` and `appname` regex-groups in your custom regex pattern to extract metadata from filenames. If the pattern specified in the configuration fails to identify `namespace` and `appname` information, it will assign the log to the default group. + + +#### `rate_period` + +| type | default | version | +| :--- | :--- | :--- | +| time | 60 \(seconds\) | 0.14.1 | + +Time period in which the group line limit is applied. `in_tail_with_throttle` resets the line collected counter after every `rate_period` interval. + + +### `` Directive \(required\) + +Specifies the rules and limits for a group. + + +#### `namespace` + +| type | default | version | +| :--- | :--- | :--- | +| array | allow all files [\/.\/] | 0.14.1 | + +Namespace criteria to form rules. This parameter internally converts each string element in the array to a regexp so it can be used to generalize a list of namespaces. + +Examples: +- This rule will apply to all files which have **monitoring** as a substring in their namespace. + +```text + + namespace monitoring + +``` + +- You can also specify multiple namespaces in your configuration file. + +```text + + namespace monitoring, logging + +``` +#### `appname` + +| type | default | version | +| :--- | :--- | :--- | +| array | allow all files [\/.\/] | 0.14.1 | + +Appname criteria to form rules. This parameter, like `namespace`, internally converts each string element in the array to a regexp so it can be used to generalize a list of appnames. + +Examples: +- This rule will apply to all files which have **heavy-stress** as a substring in their appname. +```text + + appname heavy-stress + +``` + +- You can also specify multiple appnames in your configuration files. + +```text + + appname heavy-stress, backend-payment + +``` + + +#### `limit` + +| type | default | version | +| :--- | :--- | :--- | +| integer | -1 | 0.14.1 | + +Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. + + +## Group Rules Example + +### Generic Rules +This rule will assign files (that do not satisfy any rule) to a group limit of 100 lines. + +Example - + +```text + + limit 100 + +``` + +### Context-Specifc Rules + +#### Namespace based grouping + +This configuration will limit all files that have substring as **monitoring** present in their namespace, and will not check for appname in the filename. + +```text + + namespace monitoring + limit 100 + +``` + +#### Appname based grouping + +Similarly, this configuration will limit all files containing 'frontend` substring present in their appname, and will not check for the namespace in the filename. + +```text + + appname frontend + limit 200 + +``` + +#### Namespace and Appname based grouping + +Examples: + +- This rule will limit all files which have `monitoring` substring and `frontend` substring in namespace and appname fields, respectively. + +```text + + namespace monitoring + appname frontend + limit 50 + +``` +- If we omit the `limit` parameter in a rule, then there will be no rate limiting applied to the files falling in that group. + +```text + + namespace monitoring + appname frontend + +``` + +- When using multiple namespaces and appnames in a single rule, the group line limits are distributed uniformly. + + +```text + + namespace monitoring, logging + appname frontend, backend + limit 4000 + +``` + +The list of groups formed from this rule are: + +| Namespace | Appname | Group Limit | +|:------------: |:----------: |:-----------: | +| /monitoring/ | /frontend/ | 1000 | +| /monitoring/ | /backend/ | 1000 | +| /logging/ | /frontend/ | 1000 | +| /logging/ | /backend/ | 1000 | + +## FAQ + +### What happens when a file can be assigned to both, context-specific and generic rules? + +By default, context-specific rules always precede generic rule applied to default group. Hence the file will be assigned to context-specific rule. + +### What happens when we have the same rule repeated in the configuration file? + +Example, + +```text + + namespace monitoring + appname logger + limit 100 + + + + namespace monitoring + appname logger + limit 2000 + +``` + +In this case, limits are overridden by `in_tail_with_throttle` Input Plugin. So, the final limit of the group for namespace `monitoring` and appname `logger` will be 2000. + +### Can we run `tail` and `tail_with_throttle` simultaneously ? + +Yes, we can run both input plugins simultaneously. \ No newline at end of file From 3ee0c71bf674635ea4395db16a67bfda56a676f3 Mon Sep 17 00:00:00 2001 From: Pranjal-Gupta2 Date: Tue, 26 Apr 2022 01:32:52 +0530 Subject: [PATCH 2/3] Update `in_tail` with group directive Signed-off-by: Pranjal-Gupta2 --- input/tail.md | 106 ++++++++++++++ input/tail_with_throttle.md | 281 ------------------------------------ 2 files changed, 106 insertions(+), 281 deletions(-) delete mode 100644 input/tail_with_throttle.md diff --git a/input/tail.md b/input/tail.md index a6c71b61..11a265ec 100644 --- a/input/tail.md +++ b/input/tail.md @@ -17,6 +17,16 @@ It is included in Fluentd's core. @type apache2 + + rate_period 30s + + match { + namespace: /space1|space2|space2/, + podname: /app.*/, + } + limit 200 + + ``` @@ -402,6 +412,80 @@ The `@log_level` option allows the user to set different levels of logging for e Refer to the [Logging](../deployment/logging.md) for more details. + +### `` Section \(highly recommended\) + +The `in_tail` plugin can assign each log file to a group, based on user defined rules. The `limit` parameter controls the total number of lines collected for a group within a `rate_period` time interval. + +Example: + +```text +# group rules -- 1 + + rate_period 5s + + + match { + namespace: /shopping/, + podname: /frontend/, + } + limit 1000 + + + +# group rules -- 2 + + + match { + directoy: /payment/ + } + limit 2000 + + +``` + +#### `pattern` + +| type | default | version | +| :--- | :--- | :--- | +| regexp | `/^\/var\/log\/containers\/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$/`| 1.15 | + +Specifies the regular expression for extracting metadata (namespace, podname) from log file path. Default value of the pattern regexp extracts information about `namespace`, `podname`, `docker_id`, `container` of the log (K8s specific). + +You can also add custom named captures in `pattern` for custom grouping of log files. For example, +```text + pattern /^\/home\/logs\/(?.+)\.log$/ +``` +In this example, filename will be extracted and used to form groups. + +#### `rate_period` + +| type | default | version | +| :--- | :--- | :--- | +| time | 60 \(seconds\) | 1.15 | + +Time period in which the group line limit is applied. `in_tail` resets the counter after every `rate_period` interval. + +#### `` Section \(required\) + +Grouping rules for log files. + +##### match + +| type | default | version | +| :--- | :--- | :--- | +| hash | {"namespace": /./, "podname": /./} | 1.15 | + +`match` parameter is used to check if a file belongs to a particular group based on hash keys (named captures from `pattern`) and hash values (regexp) + +##### limit + +| type | default | version | +| :--- | :--- | :--- | +| integer | -1 | 1.15 | + +Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. + ## Learn More * [Input Plugin Overview](./) @@ -469,3 +553,25 @@ path C:\\path\\to\\*\\foo.log If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License. +### What happens when a file can be assigned to more than one group? + +Example, + +```text + ## Rule1 + match { + namespace: /monitoring/ + } + limit 100 + + + ## Rule2 + match { + namespace: /monitoring/, + podname: /logger/, + } + limit 2000 + +``` + +In this case, rules with more constraints, i.e., greater number of `match` hash keys will be given a higher priority. So a file will be assigned to `Rule2` if it can be assigned to both `Rule1` and `Rule2`. \ No newline at end of file diff --git a/input/tail_with_throttle.md b/input/tail_with_throttle.md deleted file mode 100644 index c8ebe075..00000000 --- a/input/tail_with_throttle.md +++ /dev/null @@ -1,281 +0,0 @@ -# tail_with_throttle - -![](../.gitbook/assets/tail.png) - -The `in_tail_with_throttle` Input plugin allows Fluentd to read events from the tail of text files. Its behavior is similar to that of `in_tail` Input Plugin with the only difference being that this plugin rate limits the number of events read on the basis of groups. - -## Example Configuration - -```text - - @type tail_with_throttle - path /var/log/*.log - pos_file /var/log/temp.log.pos - tag applogs - - @type apache2 - - - pattern /(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$ - rate_period 30s - - namespace logging - appname frontend - limit 500 - - - namespace logging - appname backend - limit 1000 - - - limit 20 - - - -``` - -Refer to the [Configuration File](../configuration/config-file.md) article for the basic structure and syntax of the configuration file. - -For ``, see [Parse Section](../configuration/parse-section.md). - -### How It Works - -When Fluentd is configured with `in_tail_with_throttle`, it will form groups based on the rules provided in the configuration file. Based on group formation, log files will be assigned to a group and reading will start from the **tail** of that log, not the beginning. Group limits are equally distributed among the log files in a group. If the group line limit of the log is reached within rate_period time interval, Fluentd will stop reading logs. Once, **rate_period** (time) is over, Fluentd will reset its group counters and restart the reading process. - - -## Plugin Helpers - -* [`timer`](../plugin-helper-overview/api-plugin-helper-timer.md) -* [`event_loop`](../plugin-helper-overview/api-plugin-helper-event_loop.md) -* [`parser`](../plugin-helper-overview/api-plugin-helper-parser.md) -* [`compat_parameters`](../plugin-helper-overview/api-plugin-helper-compat_parameters.md) - -See also: [Linux capability](../deployment/linux-capability.md) - -## Parameters - -See [Common Parameters](../configuration/plugin-common-parameters.md). - -`in_tail_with_throttle` Input Plugin utilizes `in_tail` plugin's configuration. Refer to `in_tail` plugin's parameters [in_tail Parameters](./tail.md) - -The additional parameter which `in_tail_with_throttle` uses is the `` Directive. - -### `` Directive \(required\) - -Grouping rules for the log. - -`in_tail_with_throttle` assigns each log file to a group based on the rules provided in **``** directive. The `limit` parameter controls the total number of lines collected for a group by `tail_with_throttle` Input Plugin. - -Example: - -```text -# group rules - - rate_period 5s - - - namespace shopping - appname frontend - limit 1000 - - - - namespace payment - appname backend - limit 2000 - - -``` - -#### `pattern` - -| type | default | version | -| :--- | :--- | :--- | -| regexp | `/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`| 0.14.1 | - -Specifies the regular expression for extracting metadata (namespace, appname) from log file path. Default value of the pattern regexp follows CRIO path format which gives information about `namespace`, `pod` (appname), `docker_id`, `container` of the log. - -Please add `namespace` and `appname` regex-groups in your custom regex pattern to extract metadata from filenames. If the pattern specified in the configuration fails to identify `namespace` and `appname` information, it will assign the log to the default group. - - -#### `rate_period` - -| type | default | version | -| :--- | :--- | :--- | -| time | 60 \(seconds\) | 0.14.1 | - -Time period in which the group line limit is applied. `in_tail_with_throttle` resets the line collected counter after every `rate_period` interval. - - -### `` Directive \(required\) - -Specifies the rules and limits for a group. - - -#### `namespace` - -| type | default | version | -| :--- | :--- | :--- | -| array | allow all files [\/.\/] | 0.14.1 | - -Namespace criteria to form rules. This parameter internally converts each string element in the array to a regexp so it can be used to generalize a list of namespaces. - -Examples: -- This rule will apply to all files which have **monitoring** as a substring in their namespace. - -```text - - namespace monitoring - -``` - -- You can also specify multiple namespaces in your configuration file. - -```text - - namespace monitoring, logging - -``` -#### `appname` - -| type | default | version | -| :--- | :--- | :--- | -| array | allow all files [\/.\/] | 0.14.1 | - -Appname criteria to form rules. This parameter, like `namespace`, internally converts each string element in the array to a regexp so it can be used to generalize a list of appnames. - -Examples: -- This rule will apply to all files which have **heavy-stress** as a substring in their appname. -```text - - appname heavy-stress - -``` - -- You can also specify multiple appnames in your configuration files. - -```text - - appname heavy-stress, backend-payment - -``` - - -#### `limit` - -| type | default | version | -| :--- | :--- | :--- | -| integer | -1 | 0.14.1 | - -Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. - - -## Group Rules Example - -### Generic Rules -This rule will assign files (that do not satisfy any rule) to a group limit of 100 lines. - -Example - - -```text - - limit 100 - -``` - -### Context-Specifc Rules - -#### Namespace based grouping - -This configuration will limit all files that have substring as **monitoring** present in their namespace, and will not check for appname in the filename. - -```text - - namespace monitoring - limit 100 - -``` - -#### Appname based grouping - -Similarly, this configuration will limit all files containing 'frontend` substring present in their appname, and will not check for the namespace in the filename. - -```text - - appname frontend - limit 200 - -``` - -#### Namespace and Appname based grouping - -Examples: - -- This rule will limit all files which have `monitoring` substring and `frontend` substring in namespace and appname fields, respectively. - -```text - - namespace monitoring - appname frontend - limit 50 - -``` -- If we omit the `limit` parameter in a rule, then there will be no rate limiting applied to the files falling in that group. - -```text - - namespace monitoring - appname frontend - -``` - -- When using multiple namespaces and appnames in a single rule, the group line limits are distributed uniformly. - - -```text - - namespace monitoring, logging - appname frontend, backend - limit 4000 - -``` - -The list of groups formed from this rule are: - -| Namespace | Appname | Group Limit | -|:------------: |:----------: |:-----------: | -| /monitoring/ | /frontend/ | 1000 | -| /monitoring/ | /backend/ | 1000 | -| /logging/ | /frontend/ | 1000 | -| /logging/ | /backend/ | 1000 | - -## FAQ - -### What happens when a file can be assigned to both, context-specific and generic rules? - -By default, context-specific rules always precede generic rule applied to default group. Hence the file will be assigned to context-specific rule. - -### What happens when we have the same rule repeated in the configuration file? - -Example, - -```text - - namespace monitoring - appname logger - limit 100 - - - - namespace monitoring - appname logger - limit 2000 - -``` - -In this case, limits are overridden by `in_tail_with_throttle` Input Plugin. So, the final limit of the group for namespace `monitoring` and appname `logger` will be 2000. - -### Can we run `tail` and `tail_with_throttle` simultaneously ? - -Yes, we can run both input plugins simultaneously. \ No newline at end of file From c0a9c1e35276d9e33ec8280b6a33f6aa9c46c253 Mon Sep 17 00:00:00 2001 From: Takuro Ashie Date: Wed, 29 Jun 2022 16:37:07 +0900 Subject: [PATCH 3/3] in_tail: Remove `` section in example config In addition, remove `higly recommended` label for it. The priority of this option depends on usage and purpose of users. Signed-off-by: Takuro Ashie --- input/tail.md | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/input/tail.md b/input/tail.md index 11a265ec..a6e3b321 100644 --- a/input/tail.md +++ b/input/tail.md @@ -17,16 +17,6 @@ It is included in Fluentd's core. @type apache2 - - rate_period 30s - - match { - namespace: /space1|space2|space2/, - podname: /app.*/, - } - limit 200 - - ``` @@ -413,7 +403,7 @@ The `@log_level` option allows the user to set different levels of logging for e Refer to the [Logging](../deployment/logging.md) for more details. -### `` Section \(highly recommended\) +### `` Section The `in_tail` plugin can assign each log file to a group, based on user defined rules. The `limit` parameter controls the total number of lines collected for a group within a `rate_period` time interval. @@ -574,4 +564,4 @@ Example, ``` -In this case, rules with more constraints, i.e., greater number of `match` hash keys will be given a higher priority. So a file will be assigned to `Rule2` if it can be assigned to both `Rule1` and `Rule2`. \ No newline at end of file +In this case, rules with more constraints, i.e., greater number of `match` hash keys will be given a higher priority. So a file will be assigned to `Rule2` if it can be assigned to both `Rule1` and `Rule2`.