From f3f001c94a7171899324afaa0eb392bdef3f8bc8 Mon Sep 17 00:00:00 2001 From: Pranjal-Gupta2 Date: Mon, 18 Oct 2021 17:38:21 +0530 Subject: [PATCH] in_tail_with_throttle: Add documentation for `` directives Signed-off-by: Pranjal-Gupta2 --- input/tail_with_throttle.md | 281 ++++++++++++++++++++++++++++++++++++ 1 file changed, 281 insertions(+) create mode 100644 input/tail_with_throttle.md diff --git a/input/tail_with_throttle.md b/input/tail_with_throttle.md new file mode 100644 index 00000000..c8ebe075 --- /dev/null +++ b/input/tail_with_throttle.md @@ -0,0 +1,281 @@ +# tail_with_throttle + +![](../.gitbook/assets/tail.png) + +The `in_tail_with_throttle` Input plugin allows Fluentd to read events from the tail of text files. Its behavior is similar to that of `in_tail` Input Plugin with the only difference being that this plugin rate limits the number of events read on the basis of groups. + +## Example Configuration + +```text + + @type tail_with_throttle + path /var/log/*.log + pos_file /var/log/temp.log.pos + tag applogs + + @type apache2 + + + pattern /(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$ + rate_period 30s + + namespace logging + appname frontend + limit 500 + + + namespace logging + appname backend + limit 1000 + + + limit 20 + + + +``` + +Refer to the [Configuration File](../configuration/config-file.md) article for the basic structure and syntax of the configuration file. + +For ``, see [Parse Section](../configuration/parse-section.md). + +### How It Works + +When Fluentd is configured with `in_tail_with_throttle`, it will form groups based on the rules provided in the configuration file. Based on group formation, log files will be assigned to a group and reading will start from the **tail** of that log, not the beginning. Group limits are equally distributed among the log files in a group. If the group line limit of the log is reached within rate_period time interval, Fluentd will stop reading logs. Once, **rate_period** (time) is over, Fluentd will reset its group counters and restart the reading process. + + +## Plugin Helpers + +* [`timer`](../plugin-helper-overview/api-plugin-helper-timer.md) +* [`event_loop`](../plugin-helper-overview/api-plugin-helper-event_loop.md) +* [`parser`](../plugin-helper-overview/api-plugin-helper-parser.md) +* [`compat_parameters`](../plugin-helper-overview/api-plugin-helper-compat_parameters.md) + +See also: [Linux capability](../deployment/linux-capability.md) + +## Parameters + +See [Common Parameters](../configuration/plugin-common-parameters.md). + +`in_tail_with_throttle` Input Plugin utilizes `in_tail` plugin's configuration. Refer to `in_tail` plugin's parameters [in_tail Parameters](./tail.md) + +The additional parameter which `in_tail_with_throttle` uses is the `` Directive. + +### `` Directive \(required\) + +Grouping rules for the log. + +`in_tail_with_throttle` assigns each log file to a group based on the rules provided in **``** directive. The `limit` parameter controls the total number of lines collected for a group by `tail_with_throttle` Input Plugin. + +Example: + +```text +# group rules + + rate_period 5s + + + namespace shopping + appname frontend + limit 1000 + + + + namespace payment + appname backend + limit 2000 + + +``` + +#### `pattern` + +| type | default | version | +| :--- | :--- | :--- | +| regexp | `/(?[a-z0-9]([-a-z0-9]*[a-z0-9])?(\/[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?[^_]+)_(?.+)-(?[a-z0-9]{64})\.log$`| 0.14.1 | + +Specifies the regular expression for extracting metadata (namespace, appname) from log file path. Default value of the pattern regexp follows CRIO path format which gives information about `namespace`, `pod` (appname), `docker_id`, `container` of the log. + +Please add `namespace` and `appname` regex-groups in your custom regex pattern to extract metadata from filenames. If the pattern specified in the configuration fails to identify `namespace` and `appname` information, it will assign the log to the default group. + + +#### `rate_period` + +| type | default | version | +| :--- | :--- | :--- | +| time | 60 \(seconds\) | 0.14.1 | + +Time period in which the group line limit is applied. `in_tail_with_throttle` resets the line collected counter after every `rate_period` interval. + + +### `` Directive \(required\) + +Specifies the rules and limits for a group. + + +#### `namespace` + +| type | default | version | +| :--- | :--- | :--- | +| array | allow all files [\/.\/] | 0.14.1 | + +Namespace criteria to form rules. This parameter internally converts each string element in the array to a regexp so it can be used to generalize a list of namespaces. + +Examples: +- This rule will apply to all files which have **monitoring** as a substring in their namespace. + +```text + + namespace monitoring + +``` + +- You can also specify multiple namespaces in your configuration file. + +```text + + namespace monitoring, logging + +``` +#### `appname` + +| type | default | version | +| :--- | :--- | :--- | +| array | allow all files [\/.\/] | 0.14.1 | + +Appname criteria to form rules. This parameter, like `namespace`, internally converts each string element in the array to a regexp so it can be used to generalize a list of appnames. + +Examples: +- This rule will apply to all files which have **heavy-stress** as a substring in their appname. +```text + + appname heavy-stress + +``` + +- You can also specify multiple appnames in your configuration files. + +```text + + appname heavy-stress, backend-payment + +``` + + +#### `limit` + +| type | default | version | +| :--- | :--- | :--- | +| integer | -1 | 0.14.1 | + +Maximum number of lines allowed from a group in `rate_period` time interval. The default value of `-1` doesn't throttle log files of that group. + + +## Group Rules Example + +### Generic Rules +This rule will assign files (that do not satisfy any rule) to a group limit of 100 lines. + +Example - + +```text + + limit 100 + +``` + +### Context-Specifc Rules + +#### Namespace based grouping + +This configuration will limit all files that have substring as **monitoring** present in their namespace, and will not check for appname in the filename. + +```text + + namespace monitoring + limit 100 + +``` + +#### Appname based grouping + +Similarly, this configuration will limit all files containing 'frontend` substring present in their appname, and will not check for the namespace in the filename. + +```text + + appname frontend + limit 200 + +``` + +#### Namespace and Appname based grouping + +Examples: + +- This rule will limit all files which have `monitoring` substring and `frontend` substring in namespace and appname fields, respectively. + +```text + + namespace monitoring + appname frontend + limit 50 + +``` +- If we omit the `limit` parameter in a rule, then there will be no rate limiting applied to the files falling in that group. + +```text + + namespace monitoring + appname frontend + +``` + +- When using multiple namespaces and appnames in a single rule, the group line limits are distributed uniformly. + + +```text + + namespace monitoring, logging + appname frontend, backend + limit 4000 + +``` + +The list of groups formed from this rule are: + +| Namespace | Appname | Group Limit | +|:------------: |:----------: |:-----------: | +| /monitoring/ | /frontend/ | 1000 | +| /monitoring/ | /backend/ | 1000 | +| /logging/ | /frontend/ | 1000 | +| /logging/ | /backend/ | 1000 | + +## FAQ + +### What happens when a file can be assigned to both, context-specific and generic rules? + +By default, context-specific rules always precede generic rule applied to default group. Hence the file will be assigned to context-specific rule. + +### What happens when we have the same rule repeated in the configuration file? + +Example, + +```text + + namespace monitoring + appname logger + limit 100 + + + + namespace monitoring + appname logger + limit 2000 + +``` + +In this case, limits are overridden by `in_tail_with_throttle` Input Plugin. So, the final limit of the group for namespace `monitoring` and appname `logger` will be 2000. + +### Can we run `tail` and `tail_with_throttle` simultaneously ? + +Yes, we can run both input plugins simultaneously. \ No newline at end of file