Skip to content

Latest commit

 

History

History
698 lines (497 loc) · 20.4 KB

transports.md

File metadata and controls

698 lines (497 loc) · 20.4 KB
Raw

Transports

Pino transports can be used for both transmitting and transforming log output.

The way Pino generates logs:

  1. Reduces the impact of logging on an application to the absolute minimum.
  2. Gives greater flexibility in how logs are processed and stored.

It is recommended that any log transformation or transmission is performed either in a separate thread or a separate process.

Prior to Pino v7 transports would ideally operate in a separate process - these are now referred to as Legacy Transports.

From Pino v7 and upwards transports can also operate inside a Worker Thread, and can be used or configured via the options object passed to pino on initialization.

v7+ Transports

A transport is a module that exports a default function which returns a writable stream:

import { Writable } from 'stream'
export default (options) => {
  const myTransportStream = new Writable({
    autoDestroy: true, // Needed for Node v12+
    write (chunk, enc, cb) {
    // apply a transform and send to stdout
    console.log(chunk.toString().toUpperCase())
    cb()
    }
  })
  return myTransportStream
}

Let's imagine the above defines our "transport" as the file my-transport.mjs (ESM files are supported even if the project is written in CJS).

We would set up our transport by creating a transport stream with pino.transport and passing it to the pino function:

const pino = require('pino')
const transport = pino.transport({
  target: '/absolute/path/to/my-transport.mjs'
})
pino(transport)

The transport code will be executed in a separate worker thread. The main thread will write logs to the worker thread, which will write them to the stream returned from the function exported from the transport file/module.

The exported function can also be async. Imagine the following transport:

import fs from 'fs'
import { once } from 'events'
export default async (options) => {
  const stream = fs.createWriteStream(options.destination)
  await once(stream, 'open')
  return stream
}

While initializing the stream we're able to use await to perform asynchronous operations. In this case waiting for the write streams open event.

Let's imagine the above was published to npm with the module name some-file-transport.

The options.destination value can be set when the creating the transport stream with pino.transport like so:

const pino = require('pino')
const transport = pino.transport({
  target: 'some-file-transport',
  options: { destination: '/dev/null' }
})
pino(transport)

Note here we've specified a module by package rather than by relative path. The options object we provide is serialized and injected into the transport worker thread, then passed to the module's exported function. This means that the options object can only contain types that are supported by the Structured Clone Algorithm which is used to (de)serializing objects between threads.

What if we wanted to use both transports, but send only error logs to some-file-transport while sending all logs to my-transport.mjs? We can use the pino.transport function's destinations option:

const pino = require('pino')
const transport = pino.transport({
  targets: [
    { target: '/absolute/path/to/my-transport.mjs', level: 'error' },
    { target: 'some-file-transport', options: { destination: '/dev/null' }
  ]
})
pino(transport)

For more details on pino.transport see the API docs for pino.transport.

Writing a Transport

The module pino-abstract-transport provides a simple utility to parse each line. Its usage is highly recommended.

You can see an example using a async iterator with ESM:

import build from 'pino-abstract-transport'

export default async function (opts) {
  return build(async function (source) {
    for await (let obj of source) {
      console.log(obj)
    }
  })
}

or using Node.js streams and CommonJS:

'use strict'

const build = require('pino-abstract-transport')

module.exports = function (opts) {
  return build(function (source) {
    source.on('data', function (obj) {
      console.log(obj)
    })
  })
}

(It is possible to use the async iterators with CommonJS and streams with ESM.)

To consume async iterators in batches, consider using the hwp library.

Creating a transport pipeline

As an example, the following transport returns a Transform stream:

import build from 'pino-abstract-transport'
import { pipeline, Transform } from 'stream'
export default async function (options) {
  return build(function (source) {
    const myTransportStream = new Transform({
      // Make sue autoDestroy is set,
      // this is needed in Node v12 or when using the
      // readable-stream module.
      autoDestroy: true,

      objectMode: true,
      transform (chunk, enc, cb) {

        // modifies the payload somehow
        chunk.service = 'pino'

        // stringify the payload again
        this.push(JSON.stringify(chunk))
        cb()
      }
    })
    pipeline(source, myTransportStream, () => {})
    return myTransportStream
  }, {
    // This is needed to be able to pipeline transports.
    enablePipelining: true
  })
}

Then you can pipeline them with:

import pino from 'pino'

const logger = pino({
  transport: {
    pipeline: [{
      target: './my-transform.js'
    }, {
      // Use target: 'pino/file' to write to stdout
      // without any change.
      target: 'pino-pretty'
    }]
  }
})

logger.info('hello world')

NOTE: there is no "default" destination for a pipeline but a terminating target, i.e. a Writable stream.

Notable transports

pino/file

The pino/file transport routes logs to a file (or file descriptor).

The options.destination property may be set to specify the desired file destination.

const pino = require('pino')
const transport = pino.transport({
  target: 'pino/file',
  options: { destination: '/path/to/file' }
})
pino(transport)

The options.destination property may also be a number to represent a file descriptor. Typically this would be 1 to write to STDOUT or 2 to write to STDERR. If options.destination is not set, it defaults to 1 which means logs will be written to STDOUT.

The difference between using the pino/file transport builtin and using pino.destination is that pino.destination runs in the main thread, whereas pino/file sets up pino.destination in a worker thread.

pino-pretty

The pino-pretty transport prettifies logs.

By default the pino-pretty builtin logs to STDOUT.

The options.destination property may be set to log pretty logs to a file descriptor or file. The following would send the prettified logs to STDERR:

const pino = require('pino')
const transport = pino.transport({
  target: 'pino-pretty',
  options: { destination: 1 } // use 2 for stderr
})
pino(transport)

Asynchronous startup

The new transports boot asynchronously and calling process.exit() before the transport started will cause logs to not be delivered.

const pino = require('pino')
const transport = pino.transport({
  targets: [
    { target: '/absolute/path/to/my-transport.mjs', level: 'error' },
    { target: 'some-file-transport', options: { destination: '/dev/null' }
  ]
})
const logger = pino(transport)

logger.info('hello')

// If logs are printed before the transport is ready when process.exit(0) is called,
// they will be lost.
transport.on('ready', function () {
  process.exit(0)
})

Legacy Transports

A legacy Pino "transport" is a supplementary tool which consumes Pino logs.

Consider the following example for creating a transport:

const { pipeline, Writable } = require('stream')
const split = require('split2')

const myTransportStream = new Writable({
  write (chunk, enc, cb) {
  // apply a transform and send to stdout
  console.log(chunk.toString().toUpperCase())
  cb()
  }
})

pipeline(process.stdin, split(JSON.parse), myTransportStream)

The above defines our "transport" as the file my-transport-process.js.

Logs can now be consumed using shell piping:

node my-app-which-logs-stuff-to-stdout.js | node my-transport-process.js

Ideally, a transport should consume logs in a separate process to the application, Using transports in the same process causes unnecessary load and slows down Node's single threaded event loop.

Known Transports

PR's to this document are welcome for any new transports!

Pino v7+ Compatible

Legacy

pino-applicationinsights

The pino-applicationinsights module is a transport that will forward logs to Azure Application Insights.

Given an application foo that logs via pino, you would use pino-applicationinsights like so:

$ node foo | pino-applicationinsights --key blablabla

For full documentation of command line switches read readme

pino-azuretable

The pino-azuretable module is a transport that will forward logs to the Azure Table Storage.

Given an application foo that logs via pino, you would use pino-azuretable like so:

$ node foo | pino-azuretable --account storageaccount --key blablabla

For full documentation of command line switches read readme

pino-cloudwatch

pino-cloudwatch is a transport that buffers and forwards logs to Amazon CloudWatch.

$ node app.js | pino-cloudwatch --group my-log-group

pino-couch

pino-couch uploads each log line as a CouchDB document.

$ node app.js | pino-couch -U https://couch-server -d mylogs

pino-datadog

The pino-datadog module is a transport that will forward logs to DataDog through it's API.

Given an application foo that logs via pino, you would use pino-datadog like so:

$ node foo | pino-datadog --key blablabla

For full documentation of command line switches read readme

pino-elasticsearch

pino-elasticsearch uploads the log lines in bulk to Elasticsearch, to be displayed in Kibana.

It is extremely simple to use and setup

$ node app.js | pino-elasticsearch

Assuming Elasticsearch is running on localhost.

To connect to an external elasticsearch instance (recommended for production):

$ node app.js | pino-elasticsearch --node http://192.168.1.42:9200

Assuming Elasticsearch is running on 192.168.1.42.

To connect to AWS Elasticsearch:

$ node app.js | pino-elasticsearch --node https://es-url.us-east-1.es.amazonaws.com --es-version 6

Then create an index pattern on 'pino' (the default index key for pino-elasticsearch) on the Kibana instance.

pino-gelf

Pino GELF (pino-gelf) is a transport for the Pino logger. Pino GELF receives Pino logs from stdin and transforms them into GELF format before sending them to a remote Graylog server via UDP.

$ node your-app.js | pino-gelf log

pino-http-send

pino-http-send is a configurable and low overhead transport that will batch logs and send to a specified URL.

$ node app.js | pino-http-send -u http://localhost:8080/logs

pino-kafka

pino-kafka transport to send logs to Apache Kafka.

$ node index.js | pino-kafka -b 10.10.10.5:9200 -d mytopic

pino-logdna

pino-logdna transport to send logs to LogDNA.

$ node index.js | pino-logdna --key YOUR_INGESTION_KEY

Tags and other metadata can be included using the available command line options. See the pino-logdna readme for a full list.

pino-logflare

pino-logflare transport to send logs to a Logflare source.

$ node index.js | pino-logflare --key YOUR_KEY --source YOUR_SOURCE

pino-mq

The pino-mq transport will take all messages received on process.stdin and send them over a message bus using JSON serialization.

This useful for:

  • moving backpressure from application to broker
  • transforming messages pressure to another component
node app.js | pino-mq -u "amqp://guest:guest@localhost/" -q "pino-logs"

Alternatively a configuration file can be used:

node app.js | pino-mq -c pino-mq.json

A base configuration file can be initialized with:

pino-mq -g

For full documentation of command line switches and configuration see the pino-mq readme

pino-papertrail

pino-papertrail is a transport that will forward logs to the papertrail log service through an UDPv4 socket.

Given an application foo that logs via pino, and a papertrail destination that collects logs on port UDP 12345 on address bar.papertrailapp.com, you would use pino-papertrail like so:

node yourapp.js | pino-papertrail --host bar.papertrailapp.com --port 12345 --appname foo

for full documentation of command line switches read readme

pino-pg

pino-pg stores logs into PostgreSQL. Full documentation in the readme.

pino-mysql

pino-mysql loads pino logs into MySQL and MariaDB.

$ node app.js | pino-mysql -c db-configuration.json

pino-mysql can extract and save log fields into corresponding database field and/or save the entire log stream as a JSON Data Type.

For full documentation and command line switches read the readme.

pino-redis

pino-redis loads pino logs into Redis.

$ node app.js | pino-redis -U redis://username:password@localhost:6379

pino-sentry

pino-sentry loads pino logs into Sentry.

$ node app.js | pino-sentry --dsn=https://******@sentry.io/12345

For full documentation of command line switches see the pino-sentry readme

pino-seq

pino-seq supports both out-of-process and in-process log forwarding to Seq.

$ node app.js | pino-seq --serverUrl http://localhost:5341 --apiKey 1234567890 --property applicationName=MyNodeApp

pino-socket

pino-socket is a transport that will forward logs to a IPv4 UDP or TCP socket.

As an example, use socat to fake a listener:

$ socat -v udp4-recvfrom:6000,fork exec:'/bin/cat'

Then run an application that uses pino for logging:

$ node app.js | pino-socket -p 6000

Logs from the application should be observed on both consoles.

Logstash

The pino-socket module can also be used to upload logs to Logstash via:

$ node app.js | pino-socket -a 127.0.0.1 -p 5000 -m tcp

Assuming logstash is running on the same host and configured as follows:

input {
  tcp {
    port => 5000
  }
}

filter {
  json {
    source => "message"
  }
}

output {
  elasticsearch {
    hosts => "127.0.0.1:9200"
  }
}

See https://www.elastic.co/guide/en/kibana/current/setup.html to learn how to setup Kibana.

For Docker users, see https://github.com/deviantony/docker-elk to setup an ELK stack.

pino-stackdriver

The pino-stackdriver module is a transport that will forward logs to the Google Stackdriver log service through it's API.

Given an application foo that logs via pino, a stackdriver log project bar and credentials in the file /credentials.json, you would use pino-stackdriver like so:

$ node foo | pino-stackdriver --project bar --credentials /credentials.json

For full documentation of command line switches read readme

pino-syslog

pino-syslog is a transforming transport that converts pino NDJSON logs to RFC3164 compatible log messages. The pino-syslog module does not forward the logs anywhere, it merely re-writes the messages to stdout. But when used in combination with pino-socket the log messages can be relayed to a syslog server:

$ node app.js | pino-syslog | pino-socket -a syslog.example.com

Example output for the "hello world" log:

<134>Apr  1 16:44:58 MacBook-Pro-3 none[94473]: {"pid":94473,"hostname":"MacBook-Pro-3","level":30,"msg":"hello world","time":1459529098958}

pino-websocket

pino-websocket is a transport that will forward each log line to a websocket server.

$ node app.js | pino-websocket -a my-websocket-server.example.com -p 3004

For full documentation of command line switches read the README.