Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEATURE: Make log message max length configurable and bump the default to 2000 #109

Merged
merged 1 commit into from Feb 26, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Expand Up @@ -48,7 +48,9 @@ Logster can be configured using `Logster.config`:

- `Logster.config.allow_grouping` : Enable grouping of similar messages into one messages with an array of `env` of the grouped messages. Similar messages are messages that have identical backtraces, severity and log message.

- `Logster.config.maximum_message_size_bytes` : set a maximum size for messages. Default value is 10,000. If a message size exceeds this limit, Logster will first remove all occurrences of `gems_dir` (more on this config below) from the backtrace and computes the size again; if the message is still above the limit, Logster will remove as many as character as needed from the backtrace to bring the size below the limit. It's discouraged to set this config to a really low value (e.g. less than 2000) because a message needs a minimum amount of data in order to make sense (the minimum amount varies per message), so the closer the limit is to the mimimum amount of space needed, the more of the backtrace will be removed. Keep this in mind when tweaking this config.
- `Logster.config.maximum_message_length` : set a maximum length for log messages that are shown inside the `info` tab and in the message rows in the UI. Messages that exceed the specified length will be truncated and an ellipsis will be appended to indicate that the message has been truncated. Default value is 2000.

- `Logster.config.maximum_message_size_bytes` : set a maximum size for message objects. Default value is 10,000. If a message size exceeds this limit, Logster will first remove all occurrences of `gems_dir` (more on this config below) from the backtrace and computes the size again; if the message is still above the limit, Logster will remove as many as character as needed from the backtrace to bring the size below the limit. It's discouraged to set this config to a really low value (e.g. less than 2000) because a message needs a minimum amount of data in order to make sense (the minimum amount varies per message), so the closer the limit is to the mimimum amount of space needed, the more of the backtrace will be removed. Keep this in mind when tweaking this config.

- `Logster.config.max_env_bytes` : set a maximum size for `env`. Default value is 1000. In case `env` is an array of hashes, this limit applies to the individual hashes in the array rather than the whole array. If an `env` hash exceeds this limit, Logster will take the biggest subset of key-value pairs whose size is below the limit. If the hash has a key with the name `time`, it will always be included.

Expand Down
7 changes: 7 additions & 0 deletions lib/logster/base_store.rb
Expand Up @@ -152,6 +152,8 @@ def report(severity, progname, msg, opts = {})
return if (!msg || (String === msg && msg.empty?)) && skip_empty
return if level && severity < level

msg = msg.inspect unless String === msg
msg = truncate_message(msg)
message = Logster::Message.new(severity, progname, msg, opts[:timestamp], count: opts[:count])

env = opts[:env] || {}
Expand Down Expand Up @@ -245,6 +247,11 @@ def clear_patterns_cache(key)

private

def truncate_message(msg)
cap = Logster.config.maximum_message_length
msg.size <= cap ? msg : msg[0...cap] + "..."
end

def not_implemented
raise "Not Implemented"
end
Expand Down
4 changes: 3 additions & 1 deletion lib/logster/configuration.rb
Expand Up @@ -17,7 +17,8 @@ class Configuration
:enable_backtrace_links,
:gems_dir,
:max_env_bytes,
:max_env_count_per_message
:max_env_count_per_message,
:maximum_message_length
)

attr_writer :subdirectory
Expand All @@ -37,6 +38,7 @@ def initialize
@project_directories = []
@enable_backtrace_links = true
@gems_dir = Gem.dir + "/gems/"
@maximum_message_length = 2000

@allow_grouping = false

Expand Down
14 changes: 1 addition & 13 deletions lib/logster/message.rb
Expand Up @@ -4,8 +4,6 @@
require 'securerandom'

module Logster
MAX_MESSAGE_LENGTH = 600

class Message
LOGSTER_ENV = "_logster_env".freeze
ALLOWED_ENV = %w{
Expand All @@ -30,7 +28,7 @@ def initialize(severity, progname, message, timestamp = nil, key = nil, count: 1
@timestamp = timestamp || get_timestamp
@severity = severity
@progname = progname
@message = truncate_message(message)
@message = message
@key = key || SecureRandom.hex
@backtrace = nil
@count = count || 1
Expand All @@ -57,10 +55,6 @@ def to_h(exclude_env: false)
h
end

def message=(m)
@message = truncate_message(m)
end

def to_json(opts = nil)
exclude_env = Hash === opts && opts.delete(:exclude_env)
JSON.fast_generate(to_h(exclude_env: exclude_env), opts)
Expand Down Expand Up @@ -311,12 +305,6 @@ def truncate_env(env, limit)
end
end

def truncate_message(msg)
return msg unless msg
msg = msg.inspect unless String === msg
msg.size <= MAX_MESSAGE_LENGTH ? msg : msg[0...MAX_MESSAGE_LENGTH] + "..."
end

def get_timestamp
(Time.new.to_f * 1000).to_i
end
Expand Down
14 changes: 14 additions & 0 deletions test/logster/test_base_store.rb
Expand Up @@ -162,4 +162,18 @@ def test_chained_loggers_dont_have_superfluous_frames_in_backtrace
assert_includes(message.backtrace.lines.first, __method__.to_s)
end
end

def test_log_message_is_truncated_when_above_maximum_message_length
orig = Logster.config.maximum_message_length
Logster.config.maximum_message_length = 300
msg = @store.report(Logger::WARN, '', 'a' * 400)
# 3 is the ... at the end to indicate truncated message
assert_equal(300 + 3, msg.message.size)

Logster.config.maximum_message_length = 100
msg = @store.report(Logger::WARN, '', 'a' * 200)
assert_equal(100 + 3, msg.message.size)
ensure
Logster.config.maximum_message_length = orig
end
end
6 changes: 0 additions & 6 deletions test/logster/test_message.rb
Expand Up @@ -146,12 +146,6 @@ def test_message_to_json_respects_params
refute_includes(msg.to_json(exclude_env: true), test_hash.to_json)
end

def test_title_is_truncated_when_too_large
msg = Logster::Message.new(0, "", "a" * 1000)
# 3 is the ... at the end to indicate truncated message
assert_equal(600 + 3, msg.message.size)
end

def test_drop_redundant_envs
message = Logster::Message.new(Logger::WARN, '', 'message')
message.env = [{ a: 4 }] * 10
Expand Down