logwarn.1.in

.\"  -*- nroff -*-
.\"
.\" Logwarn - Utility for finding interesting messages in log files
.\"
.\" Copyright (C) 2010-2011 Archie L. Cobbs. All rights reserved.
.\"
.\" Licensed under the Apache License, Version 2.0 (the "License");
.\" you may not use this file except in compliance with the License.
.\" You may obtain a copy of the License at
.\"
.\"     http://www.apache.org/licenses/LICENSE-2.0
.\"
.\" Unless required by applicable law or agreed to in writing, software
.\" distributed under the License is distributed on an "AS IS" BASIS,
.\" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
.\" See the License for the specific language governing permissions and
.\" limitations under the License.
.\"
.Dd January 31, 2011
.Dt LOGWARN 1
.Os
.Sh NAME
.Nm logwarn
.Nd utility for finding interesting messages in log files
.Sh SYNOPSIS
.Nm logwarn
.Bk -words
.Op Fl achlnpqRvz
.Op Fl d Ar dir | Fl f Ar file
.Op Fl m Ar firstpat
.Op Fl r Ar sufpat
.Op Fl L Ar maxlines
.Op Fl M Ar maxprint
.Op Fl N Ar maxerrors
.Ar logfile
.Op Fl T Ar num/secs
.Ar [!]pattern ...
.Ek
.Pp
.Nm logwarn
.Bk -words
.Fl i
.Op Fl d Ar dir | Fl f Ar file
.Ar logfile
.Ek
.Sh DESCRIPTION
.Nm
searches for interesting messages in log files, where ``interesting'' is defined by an
user-supplied list of positive and negative extended regular expressions
provided on the command line.
A line that matches neither a positive nor a negative pattern is considered a match
(use the
.Fl p
flag to change this default).
.Pp
Each log message is compared against each
.Ar pattern
in the order given.
Negative patterns are specified with a ``!'' prefix.
If the log message matches a positive
.Ar pattern
before matching a negative
.Ar !pattern ,
or if none of the patterns match, then it's printed to standard output.
.Pp
.Nm
keeps track of its current position in the log file between invocations, so each matching line is only ever output once.
This information is kept in a separate `state file' (see the
.Fl d
and
.Fl f
flags below).
By default,
.Nm
interprets a missing state file as if the log file were previously empty,
resulting in a scan of its entire contents; this behavior can be changed with the
.Fl a
and
.Fl i
flags.
.Pp
.Nm
will find messages in log files that have been rotated (and possibly compressed)
since the previous invocation, as long as the rotated file has a name equal to
.Ar logfile
followed by a suffix matching the rotated log file suffix pattern (see the
.Fl r
and
.Fl R
flags below).
.Pp
By default, each line in the log file is considered to be a separate log message.
Log messages spanning multiple lines are supported with the use of the
.Fl m
flag.
.Pp
If
.Ar logfile
is `-' then standard input is scanned.
In this mode, multiple invocations of
.Nm
can be pipelined together, optionally with intervening processing, to perform
transformations of the raw log input prior to matching.
Typically the
.Fl z
flag is used in this scenario.
.Sh OPTIONS
.Bl -tag -width Ds
.It Fl a
Auto-initialize when the state file does not exist.
This flag turns on the
.Fl i
intialization behavior in cases where
.Ar logfile
exists but the state file does not.
Normally this is not what you want, but this can be helpful in cases where it's important to
avoid a flood of repeated log messages caused by state files somehow disappearing between invocations.
.It Fl c
Match each
.Ar pattern
and
.Ar firstpat
(if any) case-insensitively.
.It Fl d
Specify
.Ar dir
as the directory in which
.Nm
will store state information between invocations.
When this flag is used, the name of the state file is automatically generated from
.Ar logfile .
.Pp
The default state directory is
.Pa @DEFAULT_CACHE_DIR@ .
.Pp
It is an error to use this flag and
.Fl f
at the same time.
.It Fl f
Specify the state file used to store state information between invocations.
Each
.Ar logfile
should have its own state file.
.Pp
To run in stateless mode, use
.Fl f Ar /dev/null .
.Pp
It is an error to use this flag and
.Fl d
at the same time.
.It Fl h
Output help message and exit.
.It Fl i
Initialize the saved state for
.Ar logfile
as `up to date'.
This causes the next invocation to start its scan at the current
(as of this invocation) end of
.Ar logfile .
.It Fl L
Produce at most
.Ar maxlines
lines of output for any single log message; continuation lines beyond
this limit are suppressed.
This flag is not needed unless multi-line messages are being detected using
.Fl m .
.Pp
Setting
.Ar maxlines
to zero has the same effect as
.Fl q .
.It Fl l
Prefix each output line with the line number from the log file.
.It Fl m
Enable multi-line support using
.Ar firstpat
to match the first line of new log messages.
.Pp
Without this flag, each line in
.Ar logfile
is considered a separate log message.
With this flag, each line in
.Ar logfile
is matched against the extended regular expression
.Ar firstpat ;
non-matching lines are considered continuations of the previous line.
.Pp
Multi-line mode will work correctly even if a message crosses a log file rotation boundary.
.Pp
Be careful with this flag: if you get the pattern wrong, it's possible nothing will ever match.
.It Fl M
Output at most
.Ar maxprint
log messages; log messages beyond this limit are processed normally but their output is suppressed.
.Pp
Messages that span multiple lines (see
.Fl m )
only count once.
.Pp
Setting
.Ar maxprint
zero has the same effect as
.Fl q .
.It Fl N
Process at most
.Ar maxerrors
log messages.
.Pp
In contrast to
.Fl M ,
which limits the number of log messages output without affecting how
many are processed, the
.Fl N
flag causes
.Nm
to stop processing after encountering the specified number of log messages,
possibly before reaching the end of the log file.
.Pp
As a result, repeated invocations of
.Nm
may be required to output all outstanding log messages.
When using the
.Fl N
flag, be sure to invoke
.Nm
frequently enough so that it doesn't fall behind.
.Pp
Messages that span multiple lines (see
.Fl m )
only count once.
.Pp
The value of
.Ar maxerrors
must be at least one.
.It Fl n
Normally, if the
.Ar logfile
does not exist,
.Nm
will exit with an error.
This flag causes
.Nm
to treat a non-existent
.Ar logfile
as if it were empty.
.It Fl p
Change default match behavior to non-matching.
By default, if a log message doesn't match any of the positive or negative patterns, it is considered a match.
This flag reverses this behavior so that these messages are considered non-matches.
.It Fl q
Disable the printing of matching log messages to standard output.
The process exit value can still be used to detect whether there were any matches (see below).
.It Fl r
Make
the extended regular expression
.Ar sufpat
the rotated log file suffix pattern.
.Pp
When
.Nm
detects that a log file has been rotated, it searches for the rotated log file by finding
files in the same directory that have the same name as
.Ar logfile
plus a suffix matching the rotated log file suffix pattern.
.Pp
The default rotated log file suffix pattern is
.Pa ^(-[[:digit:]]{8}|\e\.[01])(\e\.(gz|xz|bz2))?$
.It Fl R
When multiple files match the rotated log file suffix pattern,
norally the first one in sorting order is chosen.
When this flag is given, the last one is chosen instead.
.Pp
This option is appropriate when the suffix is formatted as a timestamp.
.It Fl T
Suppress output until
.Ar num
matching lines appear within a
.Ar secs
second interval.
.Pp
This flag comes after the
.Ar logfile .
Each time it occurs, it applies to all subsequent patterns, or up until the next occurrence.
Use
.Fl T Ar 1/0
to revert to normal behavior, in which every matching occurrence is reported.
.Pp
The affected patterns are treated as a single group for counting purposes, i.e.,
a single counter tracks lines matching any of the patterns.
When the last of
.Ar num
matching lines is seen in a
.Ar secs
second window, it is printed, and the associated tracking is reset, so at least another
.Ar num
lines must be seen before another match is printed.
.Pp
Note that the calculated timing of occurrences is whenever
.Nm
reads the file.
Therefore
.Nm
should be invoked more frequently than your smallest
.Ar secs
time interval, to provide the required time resolution.
.It Fl v
Output version information and exit.
.It Fl z
Always start reading from the beginning of the file, even if the state file says otherwise.
This option is useful when reading from standard input.
.El
.Sh DETAILS
.Nm
treats missing state files as if the log file were previously empty, and it
never creates new state files for nonexistent log files.
The result is that when the
.Fl n
flag is used,
.Nm
correctly functions even if the log file doesn't come into existence until after the first few runs.
.Pp
Log file rotation is detected by comparing filesystem inode numbers, so
.Nm
may exhibit incorrect behavior if (for example) an existing log file is replaced by a copy of itself.
In this situation, use the
.Fl i
flag to reinitialize the saved state.
.Pp
Currently, the supported compression types for rotated files are
.Xr gzip 1 ,
.Xr bzip2 1 ,
and
.Xr xz 1 .
The corresponding executables
.Xr gunzip 1 ,
.Xr bunzip2 1 ,
and
.Xr unxz 1
must be present on the user's
.Pa $PATH .
.Pp
When the
.Fl f
flag is not specified, the state files in the state directory have names created by taking the
.Ar logfile
from the command line and replacing ``/'' characters with ``_''.
Therefore, referring to the same log file using different pathnames can result in inconsistent behavior.
.Pp
The matching patterns are only applied to the first line of a multi-line log message.
However, if the first line is a match, then entire log message including all continuation lines will be output.
.Pp
In order to avoid the race condition where
.Nm
reads a partially written line, if the last line of
.Ar logfile
does not end in a newline character, it is not processed.
.Pp
The maximum supported length for a single line is 100,000 characters;
longer lines will be split and treated as multiple lines.
.Pp
If
.Nm
detects that a file's inode number has not changed but its size has decreased since the previous invocation,
it assumes that the file has been truncated in place and scans from the beginning of the file.
.Sh EXAMPLES
.Pp
.Bl -tag -width xxx
.It logwarn /var/log/warn
.Pp
Show all syslog warnings since the previous invocation.
.Pp
.It logwarn /var/log/warn | uniq -f 5 | logwarn -
.Pp
Same as above, compressing repeated instances of the same message into a single line of output.
.Pp
.It logwarn -p /var/log/apache2/access_log '^.* "[^"]+" 5[0-9]{2}'
.Pp
Show any Apache 5xx server errors since the previous invocation.
.It logwarn -p -m '^myprog: ' '!retrying' 'ERROR'
.Pp
Show lines not containing `retrying' but containing `ERROR', as well as any subsequent lines in a multi-line log message,
assuming the `myprog: ' prefix marks the start of each new log message.
.It logwarn /var/log/warn -T 3/60 pat1 -T 10/300 pat2 pat3
.Pp
Match three or more occurrences of
.Ar pat1
within a one minute interval,
or ten or more occurrences of either
.Ar pat2
or
.Ar pat3
within a five minute interval.
.El
.Sh RETURN VALUES
.Nm
exits with one of the following values:
.Bl -tag -width Ds
.It 0
No matching log messages were found.
.It 1
One or more matching log messages were found.
.It 2
An error occurred.
.El
.Sh SEE ALSO
.Rs
.%T "Logwarn: Utility for finding interesting messages in log files"
.%O https://github.com/archiecobbs/logwarn
.Re
.Sh AUTHOR
.An Archie L. Cobbs Aq archie.cobbs@gmail.com