Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Performance and deprecation refactorings (#347)
* fix 320. Remove de_dot functionality * fix #343. Remove deprecated journal functionality * fix 344. optimize cache lookup to use fluent event time * fix 345. Remove serialization and deserialization of meta info * fix #346. Fix dump_stats and allow configuration to disable * add benchmark results
- Loading branch information
Showing
27 changed files
with
735 additions
and
636 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -7,7 +7,6 @@ vendor/ | |
InstalledFiles | ||
_yardoc | ||
coverage | ||
doc/ | ||
lib/bundler/man | ||
pkg | ||
rdoc | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
|
||
# collector Functionl Benchmark Results | ||
## Options | ||
* Image: quay.io/openshift-logging/fluentd:1.14.6 | ||
* Total Log Stressors: 1 | ||
* Lines Per Second: 2500 | ||
* Run Duration: 5m | ||
* Payload Source: synthetic | ||
|
||
## Latency of logs collected based on the time the log was generated and ingested | ||
|
||
Total Msg| Size | Elapsed (s) | Mean (s)| Min(s) | Max (s)| Median (s) | ||
---------|------|-------------|---------|--------|--------|--- | ||
705733|256|5m0s|16.929|1.826|36.399|14.956 | ||
|
||
![](cpu.png) | ||
|
||
![](mem.png) | ||
|
||
![](latency.png) | ||
|
||
![](loss.png) | ||
|
||
## Percent logs lost between first and last collected sequence ids | ||
Stream | Min Seq | Max Seq | Purged | Collected | Percent Collected | | ||
-------| ---------| --------| -------|-----------|--------------| | ||
| functional.0.0000000000000000DBCED7977B8F643F|0|716850|0|705733|98.4% | ||
|
||
|
||
## Config | ||
|
||
``` | ||
<system> | ||
log_level debug | ||
</system> | ||
<source> | ||
@type tail | ||
@id container-input | ||
path /var/log/pods/testhack-clwn63sc_*/loader-*/* | ||
exclude_path ["/var/log/pods/**/*/*.gz","/var/log/pods/**/*/*.tmp"] | ||
pos_file "/var/lib/fluentd/pos/containers-app" | ||
refresh_interval 5 | ||
rotate_wait 5 | ||
tag kubernetes.* | ||
read_from_head "true" | ||
<parse> | ||
@type regexp | ||
expression /^(?<@timestamp>[^\s]+) (?<stream>stdout|stderr) (?<logtag>[F|P]) (?<message>.*)$/ | ||
time_format '%Y-%m-%dT%H:%M:%S.%N%:z' | ||
keep_time_key true | ||
</parse> | ||
</source> | ||
<filter kubernetes.**> | ||
@type concat | ||
key message | ||
partial_key logtag | ||
partial_value P | ||
separator '' | ||
</filter> | ||
<match **> | ||
@type forward | ||
heartbeat_type none | ||
keepalive true | ||
<buffer> | ||
flush_mode interval | ||
flush_interval 5s | ||
flush_at_shutdown true | ||
flush_thread_count 2 | ||
retry_type exponential_backoff | ||
retry_wait 1s | ||
retry_max_interval 60s | ||
retry_forever true | ||
overflow_action block | ||
</buffer> | ||
<server> | ||
host 0.0.0.0 | ||
port 24224 | ||
</server> | ||
</match> | ||
``` | ||
|
127 changes: 127 additions & 0 deletions
127
doc/benchmark/5m-1-2500lps-256b-baseline-01/results.html
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,127 @@ | ||
|
||
<html> | ||
<div> | ||
<div><b>Options</b><div> | ||
<div>Image: quay.io/openshift-logging/fluentd:1.14.6</div> | ||
<div>Total Log Stressors: 1</div> | ||
<div>Lines Per Second: 2500</div> | ||
<div>Run Duration: 5m</div> | ||
<div>Payload Source: synthetic</div> | ||
</div> | ||
<div> | ||
Latency of logs collected based on the time the log was generated and ingested | ||
</div> | ||
<table border="1"> | ||
<tr> | ||
<th>Total</th> | ||
<th>Size</th> | ||
<th>Elapsed</th> | ||
<th>Mean</th> | ||
<th>Min</th> | ||
<th>Max</th> | ||
<th>Median</th> | ||
</tr> | ||
<tr> | ||
<th>Msg</th> | ||
<th></th> | ||
<th>(s)</th> | ||
<th>(s)</th> | ||
<th>(s)</th> | ||
<th>(s)</th> | ||
<th>(s)</th> | ||
</tr> | ||
<tr> | ||
<td>705733</td> | ||
<td>256</td> | ||
<td>5m0s</td> | ||
<td>16.929</td> | ||
<td>1.826</td> | ||
<td>36.399</td> | ||
<td>14.956</td> | ||
</tr> | ||
</table> | ||
<div> | ||
<img src="cpu.png"> | ||
</div> | ||
<div> | ||
<img src="mem.png"> | ||
</div> | ||
<div> | ||
<img src="latency.png"> | ||
</div> | ||
<div> | ||
<img src="loss.png"> | ||
</div> | ||
<div> | ||
<table border="1"> | ||
<tr> | ||
<th>Stream</th> | ||
<th>Min Seq</th> | ||
<th>Max Seq</th> | ||
<th>Purged</th> | ||
<th>Collected</th> | ||
<th>Percent Collected</th> | ||
</tr> | ||
<tr> | ||
<tr><td>functional.0.0000000000000000DBCED7977B8F643F</td><td>0</td><td>716850</td><td>0</td><td>705733</td><td>98.4%</td><tr> | ||
</table> | ||
</div> | ||
<div> | ||
<code style="display:block;white-space:pre-wrap"> | ||
|
||
<system> | ||
log_level debug | ||
</system> | ||
|
||
<source> | ||
@type tail | ||
@id container-input | ||
path /var/log/pods/testhack-clwn63sc_*/loader-*/* | ||
exclude_path ["/var/log/pods/**/*/*.gz","/var/log/pods/**/*/*.tmp"] | ||
pos_file "/var/lib/fluentd/pos/containers-app" | ||
refresh_interval 5 | ||
rotate_wait 5 | ||
tag kubernetes.* | ||
read_from_head "true" | ||
<parse> | ||
@type regexp | ||
expression /^(?<@timestamp>[^\s]+) (?<stream>stdout|stderr) (?<logtag>[F|P]) (?<message>.*)$/ | ||
time_format '%Y-%m-%dT%H:%M:%S.%N%:z' | ||
keep_time_key true | ||
</parse> | ||
</source> | ||
|
||
<filter kubernetes.**> | ||
@type concat | ||
key message | ||
partial_key logtag | ||
partial_value P | ||
separator '' | ||
</filter> | ||
|
||
<match **> | ||
@type forward | ||
heartbeat_type none | ||
keepalive true | ||
|
||
<buffer> | ||
flush_mode interval | ||
flush_interval 5s | ||
flush_at_shutdown true | ||
flush_thread_count 2 | ||
retry_type exponential_backoff | ||
retry_wait 1s | ||
retry_max_interval 60s | ||
retry_forever true | ||
overflow_action block | ||
</buffer> | ||
|
||
<server> | ||
host 0.0.0.0 | ||
port 24224 | ||
</server> | ||
</match> | ||
|
||
</code> | ||
</div> | ||
</html> |
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Oops, something went wrong.