MyGovBC-Splunk-Forwarder

A NodeJS based Splunk forwarder for MyGovBC-MSP user interfaces.

Features:

1. Receive events on a given port from a variety of sources.
2. Log events locally in a rotating log structure in the Gluster PV attached.
3. Forwards it to HIBC Splunk server.
4. Allow remote access to local logs via a simple REST lookup protected by username and password.

Developer Prerequisites

• Node.js 4.2.5 or later.
• Splunk Enterprise 6.3.0 or later, or Splunk Cloud.
• An HTTP Event Collector token from your Splunk Enterprise server.

First, update npm to the latest version by running:

sudo npm install npm -g

Then run:

npm start server

To test:

• curl -XPOST -H "Authorization: Splunk XXX" -H "Content-Type: application/json" -d '{"body": "xyz"}' localhost:5504/log

where XXX is the SERVICE_AUTH_TOKEN environment variable passed to the service.

Splunk Server Configuration

Splunk Server version 3.2.1 or greater is required.

1. Turn on HTTP Event Collector (HEC) by enabling its endpoint as it is not enabled by default.
2. Generate an HEC token.
3. Configure the HEC token in the configuration environment variable of the Splunk Forwarder.

For more information, pleast consult the Event Collector component of Splunk.

Splunk Forwarder Configuration

All configuration is done via a user's shell environment variable and read in NodeJS via process.env.

These are:

Environment Variable	Description
SERVICE_IP (string)	IP address of where the service runs, defaults to 'localhost'
SERVICE_PORT (number)	port for the service, Default: 8080
SERVICE_USE_AUTH (boolean)	Splunk Forwarder uses a token to authorize connection from a service
SERVICE_AUTH_TOKEN (string)	security token used by services to connect to Splunk Forwarder
SPLUNK_AUTH_TOKEN (string)	security token used by the remote Splunk Server (generated by the Splunk HEC)
FILE_LOG_LEVEL (string)	log level for local log file, defaults to 'info'
LOG_DIR_NAME (string)	Directory path to store local log files - typically local PV mount point, ie. /var/logs
USE_SPLUNK (boolean)	use a remote Splunk Server or just log locally (useful for testing)
SPLUNK_URL (string)	url of remote Splunk Server
RETRY_COUNT (number)	the number of times to retry the Splunk Server
ONLY_LOG_WHEN_SPLUNK_FAILS (boolean)	only write to local file system when there's an error sending to Splunk. Ignored if USE_SPLUNK is false
MONITOR_USERNAME (string)	username for the /monitor REST endpoint to view and download local logs
MONITOR_PASSWORD (string)	password for the /monitor REST endpoint to view and download local logs
CA_CERT (string)	The CA cert of the remote Splunk Server (or proxy to the remote Splunk Server)
MAX_FILES (number)	total number of log files to rotate over. Default: 10
MAX_BYTE_SIZE_PER_FILE (number)	total number of each log file. Default: (1024 * 1024 * 75) = 75mb
APPEND_POD_NAME_TO_FILE (boolean)	Whether the pod name should be appended to the local log file name

The max storage size used for the entire splunk-forwarder will be MAX_FILES * MAX_BYTES_PER_FILE. It's default storage size is 750mb.

A note on the two different AUTH tokens. The idea is to not expose the remote Splunk Server's authorization token to the service clients of the Splunk Forwarder. These use SERVICE_AUTH_TOKEN, while the Splunk Forwarder itself uses SPLUNK_AUTH_TOKEN generated by HEC.

To view and download local log files created by the Splunk Forwarder, go to URL of the route of the Splunk Forwarder followed by "/monitor". For example: https://gcpe-mygovbc-splunk-forwader-dev.pathfinder.gov.bc.ca/monitor This is often useful when the remote Splunk Service is unavailable.

The username and password should match the environment variables configured.

Splunk Forwarder Client

An application client wishing to use the Splunk-Forwarder, will make an HTTP POST request to the host/port that the Splunk Forwarder listens on. Typically, the following environment values are used in the client:

Environment Variable	Description
LOGGER_HOST (string)	name of the Splunk Forwarder service. In OpenShift this is splunk-forwarder. Can be an IP address.
LOGGER_PORT (number)	port for the service, Default: 8080
SPLUNK_AUTH_TOKEN (string)	security token used by clients to connect to Splunk Forwarder

The client creates an http POST request. An example of a function that posts a message string in javascript:

function logSplunkError (message) {

var body = JSON.stringify({
   message: message
})

var options = {
  hostname: process.env.LOGGER_HOST,
  port: process.env.LOGGER_PORT,
  path: '/log',
  method: 'POST',
  headers: {
     'Content-Type': 'application/json',
     'Authorization': 'Splunk ' + process.env.SPLUNK_AUTH_TOKEN,
     'Content-Length': Buffer.byteLength(body),
     'logsource': process.env.HOSTNAME,
     'timestamp': moment().format('DD-MMM-YYYY'),
     'program': 'name of the client application',
     'serverity': 'error'
   }
};

var req = http.request(options, function (res) {
   res.setEncoding('utf8');
   res.on('data', function (chunk) {
      console.log("Body chunk: " + JSON.stringify(chunk));
   });
   res.on('end', function () {
      console.log('End of chunks');
   });
});

req.on('error', function (e) {
   console.error("error sending to splunk-forwarder: " + e.message);
});

// write data to request body
req.write(body);
req.end();
}

Notice that all POSTs are made to the path "/log" of the splunk-forwarder.

Also notice the format of the Authorization header as required by Splunk HEC.

Production Setup

See Deploy to OpenShift docs.