Skip to content

Latest commit

 

History

History
271 lines (213 loc) · 29.9 KB

check_vmware_alarms.md

File metadata and controls

271 lines (213 loc) · 29.9 KB
Raw

check-vmware | check_vmware_alarms plugin

Table of Contents

Overview

Nagios plugin used to monitor for Triggered Alarms in one or more datacenters.

  • Explicit exclusions take priority over either implicit or explicit inclusions.
  • All filtering is currently applied in batches/bulk.

It helps to think of the process working this way for each filter in the "pipeline":

  1. Explicit inclusions are applied, marking matching triggered alarms as explicitly included and non-matches as implicitly excluded
  2. Explicit exclusions are applied, marking matching triggered alarms as explicitly excluded, permanently "dropping" the triggered alarm from further evaluation
  3. After all filters have finished processing, any triggered alarms marked as excluded (implicit or explicit) are removed from final evaluation (i.e., ignored and not reported as a problem).

Filtering is available for explicitly including or excluding based on:

  • Acknowledged status
  • Managed Entity type (e.g., Datastore, VirtualMachine) associated with the Triggered Alarm
  • Inventory object name (e.g., node1.example.com, vc1.example.com) associated with the Triggered Alarm
  • Alarm Name field substring match
  • Alarm Description field substring match
  • Triggered Alarm Status (e.g., red, yellow, gray)
  • Resource Pool for the Managed Entity type (e.g., ResourcePool, VirtualMachine) associated with the Triggered Alarm

Output

The output for these plugins is designed to provide the one-line summary needed by Nagios for quick identification of a problem while providing longer, more detailed information for display within the web UI, use in email and Teams notifications (atc0005/send2teams).

See the main project README for details.

Performance Data

Background

Initial support has been added for emitting Performance Data / Metrics, but refinement suggestions are welcome.

Consult the list below for the metrics implemented thus far, the original discussion thread and the Add Performance Data / Metrics support project board for an index of the initial implementation work.

Please add to an existing Discussion thread or open a new one with any feedback that you may have. Thanks in advance!

Supported metrics

NOTE: These metrics are based on the visibility of the service account used to login to the target VMware environment. If the service account cannot see a resource, it cannot evaluate the resource.

Metric Unit of Measurement Description
time milliseconds plugin runtime
datacenters all (visible) datacenters in the inventory
triggered_alarms all (visible) triggered alarms for specified datacenters
triggered_alarms_included triggered alarms remaining after they have been implicitly or explicitly excluded
triggered_alarms_excluded triggered alarms that have been implicitly or explicitly excluded
triggered_alarms_critical triggered alarms in the collection are considered to be in a CRITICAL state
triggered_alarms_warning triggered alarms in the collection are considered to be in a WARNING state
triggered_alarms_unknown triggered alarms in the collection are considered to be in an UNKNOWN state
triggered_alarms_ok triggered alarms in the collection are considered to be in an OK state

Optional evaluation

Some plugins provide optional support to limit evaluation of VMs to specific Resource Pools (explicitly including or excluding) and power states (on or off). Other plugins support similar filtering options (e.g., Acknowledged state of Triggered Alarms). See the configuration options, examples and contrib sections for more information.

Installation

See the main project README for details.

Configuration options

Threshold calculations

Nagios State Description
OK Ideal state, no non-excluded Triggered Alarms detected.
WARNING One or more non-excluded alarms with a yellow status.
CRITICAL One or more non-excluded alarms with a red status.

Command-line arguments

  • Use the -h or --help flag to display current usage information.
  • Flags marked as required must be set via CLI flag.
  • Flags not marked as required are for settings where a useful default is already defined, but may be overridden if desired.
Flag Required Default Repeat Possible Description
branding No false No branding Toggles emission of branding details with plugin status details. This output is disabled by default.
h, help No false No h, help Show Help text along with the list of supported flags.
v, version No false No v, version Whether to display application version and then immediately exit application.
ll, log-level No info No disabled, panic, fatal, error, warn, info, debug, trace Log message priority filter. Log messages with a lower level are ignored. Log messages are sent to stderr by default. See Output for more information.
p, port No 443 No positive whole number between 1-65535, inclusive TCP port of the remote ESXi host or vCenter instance. This is usually 443 (HTTPS).
t, timeout No 10 No positive whole number of seconds Timeout value in seconds allowed before a plugin execution attempt is abandoned and an error returned.
s, server Yes No fully-qualified domain name or IP Address The fully-qualified domain name or IP Address of the remote ESXi host or vCenter instance.
u, username Yes No valid username Username with permission to access specified ESXi host or vCenter instance.
pw, password Yes No valid password Password used to login to ESXi host or vCenter instance.
domain No No valid user domain (Optional) domain for user account used to login to ESXi host or vCenter instance. This is needed for user accounts residing in a non-default domain (e.g., SSO specific domain).
trust-cert No false No true, false Whether the certificate should be trusted as-is without validation. WARNING: TLS is susceptible to man-in-the-middle attacks if enabling this option.
dc-name No No comma-separated list of valid vSphere datacenter names Specifies the name of one or more vSphere Datacenters. If not specified, applicable plugins will attempt to evaluate all visible datacenters found in the vSphere environment. Not applicable to standalone ESXi hosts.
include-entity-type No No comma-separated list of valid managed object type keywords If specified, triggered alarms will only be evaluated if the associated entity type (e.g., Datastore) matches one of the specified values; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-entity-type No No comma-separated list of valid managed object type keywords If specified, triggered alarms will only be evaluated if the associated entity type (e.g., Datastore) does NOT match one of the specified values; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
include-entity-name No No comma-separated list of vSphere inventory object names If specified, triggered alarms will only be evaluated if the associated entity name (e.g., node1.example.com) matches one of the specified values; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-entity-name No No comma-separated list of vSphere inventory object names If specified, triggered alarms will only be evaluated if the associated entity name (e.g., node1.example.com) does NOT match one of the specified values; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
include-entity-rp No No comma-separated list of resource pool names If specified, triggered alarms will only be evaluated if the associated entity is part of one of the specified Resource Pools (case-insensitive match on the name) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-entity-rp No No comma-separated list of resource pool names If specified, triggered alarms will only be evaluated if the associated entity is NOT part of one of the specified Resource Pools (case-insensitive match on the name) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
eval-acknowledged No false No true, false Toggles evaluation of acknowledged triggered alarms in addition to unacknowledged triggered alarms. Evaluation of acknowledged alarms is disabled by default.
include-name No No valid custom or default alarm names If specified, triggered alarms will only be evaluated if the alarm name (e.g., Datastore usage on disk) case-insensitively matches one of the specified substring values (e.g., datastore or datastore usage) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-name No No valid custom or default alarm names If specified, triggered alarms will only be evaluated if the alarm name (e.g., Datastore usage on disk) DOES NOT case-insensitively match one of the specified substring values (e.g., datastore or datastore usage) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
include-desc No No valid custom or default alarm descriptions If specified, triggered alarms will only be evaluated if the alarm description (e.g., Default alarm to monitor datastore disk usage) case-insensitively matches one of the specified substring values (e.g., datastore disk or monitor datastore) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-desc No No valid custom or default alarm descriptions If specified, triggered alarms will only be evaluated if the alarm description (e.g., Default alarm to monitor datastore disk usage) DOES NOT case-insensitively match one of the specified substring values (e.g., datastore disk or monitor datastore) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
include-status No No valid managed entity status (excluding green) or Nagios state (excluding OK) (WARNING, CRITICAL , UNKNOwN) If specified, triggered alarms will only be evaluated if the alarm status (e.g., yellow) case-insensitively matches one of the specified keywords (e.g., yellow or warning) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.
exclude-status No No valid managed entity status If specified, triggered alarms will only be evaluated if the alarm status (e.g., yellow) DOES NOT case-insensitively match one of the specified keywords (e.g., yellow or warning) and is not explicitly excluded by another filter in the pipeline; while multiple explicit inclusions are allowed, explicit exclusions have precedence over explicit inclusions and will exclude the triggered alarm from further evaluation.

Configuration file

Not currently supported. This feature may be added later if there is sufficient interest.

Contrib

See the main project README for details.

Examples

While entries in this section attempt to provide a brief overview of usage, it is recommended that you review the provided command definitions and other Nagios configuration files within the contrib directory for more complete examples.

See the configuration options section for all command-line settings supported by this plugin along with descriptions of each.

CLI invocation

/usr/lib/nagios/plugins/check_vmware_alarms --username SERVICE_ACCOUNT_NAME --password "SERVICE_ACCOUNT_PASSWORD" --server vc1.example.com  --trust-cert --log-level info

See the configuration options section for all command-line settings supported by this plugin along with descriptions of each. See the contrib section for information regarding example command definitions and Nagios configuration files.

Of note:

  • Triggered alarms are evaluated for all detected datacenters
    • due to lack of specified datacenter name (or names)
  • Triggered alarms are not filtered based on associated managed object (aka, managed entity) type
    • due to lack of explicit exclusions or inclusions
  • Triggered alarms are not filtered based on associated managed object (aka, managed entity) name
    • due to lack of explicit exclusions or inclusions
  • Triggered alarms are not filtered based on associated managed object (aka, managed entity) resource pool
    • due to lack of explicit exclusions or inclusions
  • Triggered alarms that were previously acknowledged are ignored
  • Triggered alarms are not filtered based on defined Alarm name
    • due to lack of explicit exclusions or inclusions
  • Triggered alarms are not filtered based on defined Alarm description
    • due to lack of explicit exclusions or inclusions
  • Triggered alarms are not filtered based on Triggered Alarm status
    • due to lack of explicit exclusions or inclusions
  • Certificate warnings are ignored.
    • not best practice, but many vCenter instances use self-signed certs per various freely available guides
  • Service Check results output is sent to stdout
  • Logging output is enabled at the info level.
    • logging output is sent to stderr by default
    • logging output is intended to be seen when invoking the plugin directly via CLI (often for troubleshooting)
      • see the Output section of the main README for potential conflicts with some monitoring systems

Command definition

# /etc/nagios-plugins/config/vmware-alarms.cfg

# Look at triggered alarms across all detected datacenters, do not evaluate
# any triggered alarms which have been previously acknowledged.
define command{
    command_name    check_vmware_alarms
    command_line    $USER1$/check_vmware_alarms --server '$HOSTNAME$' --domain '$ARG1$' --username '$ARG2$' --password '$ARG3$' --trust-cert --log-level info
    }

# Look at triggered alarms within specified datacenters. Do not evaluate any
# triggered alarms which have been previously acknowledged.
define command{
    command_name    check_vmware_alarms_specific_dc
    command_line    $USER1$/check_vmware_alarms --server '$HOSTNAME$' --domain '$ARG1$' --username '$ARG2$' --password '$ARG3$' --dc-name '$ARG4$' --trust-cert --log-level info
    }

License

See the main project README for details.

References