Skip to content

msgpack schema for msgpack payloads of cmetrics and ctraces

Jump to bottom Edit New page
Hiroshi Hatake edited this page Nov 7, 2023 · 2 revisions

This is the first specification of observability contexts that are leveraged with cmetrics and ctraces. They are used in observability contexts on Fluent Bit.

This is the first version of observability contexts of specification.

Changes

  • November 7, 2023: the first release of this spec.

Abstract

This specification describes the recently implemented observability contexts in Fluent Bit. Fluent Bit v2 implemented capabilities for metrics and traces.

To implement them in Fluent Bit, cmetrics and ctraces provide the functionalities to implement observability. These are used for storing and updating values of observable quantities which are collated by the running Fluent Bit agents.

Terminology

The keywords "MUST", "MUST NOT", "SHOULD", "SHOULD NOT" and "MAY" in this document are to be interpreted as described in RFC 2119. The following terms are also used:

  • msgpack: a light weight binary representation of serialized objects.
  • Prometheus: An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database, and modern alerting approach.
  • OpenTelemetry: High-quality, ubiquitous, and portable telemetry to enable effective observability.
  • cmetrics: a standalone library to create and manipulate metrics in C.
  • ctraces: Library to create and manipulate traces in C.
  • Hash: HashMap data structure.

Msgpack Encoding Specification for Both Contexts

cmetrics

Cmetrics is built with C and it provides functionalities to handle quantities that are typed as counter, gauge, untyped, summary, and histogram types. They are based on Prometheus' type system of quantities. Originally, Prometheus' quantity type system provides counter, gauge, summary, and histogram types. In more detail, please refer to the documentation for the type system: https://prometheus.io/docs/tutorials/understanding_metric_types/. The untyped type of metrics is the cmetrics specific type system that provides a least common set of operations for counter and gauge types.

This quantity handling system brings to Prometheus compatible operation of observabilities. Then, this specification will focus the msgpack encoding format on the current cmetrics.

From the cmetrics' source, cmetrics uses to encode the following schema.

In short, the cmetrics' msgpack encoding is compounded by two parts, header and metrics.

The header is always added and metrics MAY able to zero or more than one.

{
  HEADER,
  METRICS*
}

The header is encoded as:

'meta' => {
  'cmetrics' => {
     'producer': STRING
   },
   'external' => { ... },
   'processing' => {
     'static_labels' =>  [
       [STRING, STRING], ...
     ]
   }
},

This indicates that the header section in meta is composed with three elements:

name type content
cmetrics Hash internal metadata
external Hash external metadata
processing Hash Hash contains static labels

Metrics is built by two parts which are a header part that includes the meta key and a metrics part that includes the values key:

'metrics' => [
    {
        'meta' => {
            'ver'  => INTEGER
            'type' => INTEGER
            '0' = counter
            '1' = gauge
            '2' = histogram
            '3' = summary
            '4' = untyped
            'opts' => {
                'ns'          => ns
                'subsystem'   => subsystem
                'name'        => name
                'description' => description
            },
            'label_keys' => [STRING, ...],
            'buckets' => [n, ...],
            'quantiles' => [n, ...]
        },
        'values' => [
            {
                'ts'   : nanosec timestamp,
                'value': float64 value,
                'label_values': [STRING, ...],
                'histogram':{
                    'sum': float64,
                    'count': uint64,
                    'buckets': [n, ...]
                },
                'summary':  {
                    'sum': float64,
                    'count': uint64,
                    'quantiles': [n, ...],
                    'quantiles_set': uint64
                },
                'hash': uint64 value
            }
        ]
    }, ...
]

This indicates that the metrics contains actual metrics:

name type content
metrics Array metrics data

metrics can contains zero or more actual metrics:

name type content
meta Hash metadata of the actual metrics
values Hash The actual data of metrics

meta has the following fields:

name type content
ver int metadata version of the actual metrics
type int show the type of metrics. (counter, gauge, histogram, summary, untyped)
opts Hash The description for Prometheus compatible metrics
label_keys Hash The actual data of key for labels
buckets [n, ...] An array of count and associated upper bounds. meta.type = 2.
quantiles [n, ...] An array of count and associated upper bounds. meta.type = 3.

A msgpack payloads of cmetrics MAY contain other keys in meta part in metrics key.

values has the following fields. Ond of the value, histogram, and summary is required.

name type content
ts uint64 timestamp of a metrics
value double The double number of the actual data of metrics. This field is used for counter, gauge, and untyped type of metrics.
label_values Array The values of dynamic labels for the actual data of metrics
hash uint64 The Hash value for the actual data of metrics
histogram Hash A histogram values. Only used for summary type of metrics. meta.type = 2.
summary Hash A summary values Only used for histogram type of metrics. meta.type = 3.

histogram contains the three fields to represent its values:

name type content
sum double A sum of summary
count uint64 a count of summary values
buckets [n, ...] An array of count and associated upper bounds.

summary contains the three fields to represent its values:

name type content
sum double A sum of summary
count uint64 a count of summary values
quantiles [n, ...] An array of count and associated upper bounds.
quantiles_set uint64 Quantive values

Then, the final combined format is:

{
    'meta' => {
        'cmetrics' => {
            'producer': STRING
        },
        'external' => { ... },
        'processing' => {
            'static_labels' =>  [
                [STRING, STRING], ...
            ]
        }
    },
    'metrics' =>    [
        {
            'meta' => {
                'ver'  => INTEGER
                'type' => INTEGER
                '0' = counter
                '1' = gauge
                '2' = histogram
                '3' = summary
                '4' = untyped
                'opts' => {
                    'ns'          => ns
                    'subsystem'   => subsystem
                    'name'        => name
                    'description' => description
                },
                'label_keys' => [STRING, ...],
                'buckets' => [n, ...],
                'quantiles' => [n, ...]
            },
            'values' => [
                {
                    'ts'   : nanosec timestamp,
                    'value': float64 value,
                    'label_values': [STRING, ...],
                    'histogram':{
                        'sum': float64,
                        'count': uint64,
                        'buckets': [n, ...]
                    },
                    'summary':  {
                        'sum': float64,
                        'count': uint64,
                        'quantiles': [n, ...],
                        'quantiles_set': uint64
                    },
                    'hash': uint64 value
                }
            ]
        }, ...
    ]
}
/*
 *
 *
 * The following fields are metric type specific and are only
 * included for histograms :
 *      meta->buckets
 *      values[n]->buckets
 *      values[n]->count
 *      values[n]->sum
 *
 * The following fields are metric type specific and are only
 * included for summary :
 *      meta->quantiles
 *      values[n]->quantiles
 *      values[n]->quantiles_set
 *      values[n]->count
 *      values[n]->sum
 */

This specification is derived from the implementation of msgpack encoder in cmetrics v0.6.4.

ctraces

Ctraces is built with C and it provides functionalities to handle traces of applications that are used in OpenTelemetry, In more detail, please refer to the documentation for the traces on OpenTelemetry: https://opentelemetry.io/docs/concepts/signals/traces/

These traces handling system brings to OpenTelemerty compatible operation of observabilities. Then, this specification will focus the msgpack encoding format on the current ctraces.

From the ctraces' source, ctraces uses to encode the following schema. ctraces' msgpack schema refers to the definition of protocol buffers on OpenTelemetry: https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto

Fluent Bit uses the msgpack format as an internal representation for re-encoding to JSON.

This schema is automatically generated from the protobuf definition. Fluent Bit uses the generated C sources from that definition.

Currently, the generated source was used this revision of protobuf of OpenTelemetry: https://github.com/open-telemetry/opentelemetry-proto/commit/d1468b7700309cec0a3fdfffbfba4e84acf94072

Add a custom footer
Add a custom sidebar
Clone this wiki locally