From 04e9bcc51e1097a31c7ab34361a3ebed7487250a Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 11:42:31 -0700 Subject: [PATCH 01/12] add heatmap type --- data/contract_docs/contract.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md index f4226d649..34de47d92 100644 --- a/data/contract_docs/contract.md +++ b/data/contract_docs/contract.md @@ -14,6 +14,20 @@ There are logical **_kinds_** (like Time Series Data, Numeric, Histogram, etc), A **_data type_** definition or declaration in this framework includes both a kind and format. For example, "TimeSeriesWide" is: kind: "Time Series", format: "Wide". +* [Time series](./timeseries.md) + * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) + * [Long](./timeseries.md#time-series-wide-format-timeserieswide) + * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) +* [Numeric](./numeric.md) + * [Wide](./numeric.md#numeric-wide-format-numericwide) + * [Many](./numeric.md#numeric-many-format-numericmany) +* [Heatmap](./heatmap.md) + * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) + * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) + * [Sparse](./heatmap.md#heatmap-sparse-heatmapsparse) + + + ## Dimensional Set Based Kinds Within a data type (kind+format), there can be multiple **_items_** of data that are uniquely identified. This forms a **_set_** of data items. For example, in the numeric kind there can be a set of numbers, or, in the time series kind, a set of time series-es :-). From 5d8a9c72c737e063b44fd862a7c375764c07b296 Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 11:44:09 -0700 Subject: [PATCH 02/12] add heatmap type --- data/contract_docs/heatmap.md | 196 ++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 data/contract_docs/heatmap.md diff --git a/data/contract_docs/heatmap.md b/data/contract_docs/heatmap.md new file mode 100644 index 000000000..f46e0999e --- /dev/null +++ b/data/contract_docs/heatmap.md @@ -0,0 +1,196 @@ +# Heatmap + +Status: EARLY Draft/Proposal + +Heatmaps are used to show the magnitude of a phenomenon as color in two dimensions. The variation in color +may give visual cues about how the phenomenon is clustered or varies over space. + + +## Heatmap buckets (HeatmapBuckets) + +The first field represents the X axis, the rest of the fields indicate rows in the heatmap. +The true numeric range of each bucket can be indicated using an "le" label. When absent, +The field display is used for the bucket label. + +Example: + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Type: Time +

+ Name: Time +

+ 		+ Type: Number +

+ Name: +

+

+ Labels: {"le": "10"} +

+ 		+ Type: Number +

+ Name: +

+

+ Labels: {"le": "20"} +

+ 		+ Type: Number +

+ Name: +

+

+ Labels: {"le": "+Inf"} +

+
1653416391000678
1653416391000678
1653416391000678
+ + +## Heatmap scanlines (HeatmapScanlines) + +In this format, each row in the frame indicates the value of a single cell in a heatmap. +There exists a row for every cell in the heatmap. + +**Example:** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Type: Time +

+ Name: xMax +

+ 		+ Type: Number +

+ Name: yMax +

+ 		+ Type: Number +

+ Name: Value +

+
16534163910001001
16534163910002002
16534163910003003
16534163920001004
16534163920002005
16534163920003006
+ +This format requires uniform cell sizing + + +## Heatmap sparse (HeatmapSparse) + +This format is simplar to Heatmap scanlines, except that each cell is independent from its adjacent values. +Unlike scanlines, this allows resolutions to change over time. When + +Example: + + + + + + + + + + + + + + + + + + + + + + + + +
+ Type: Time +

+ Name: xMin +

+ 		+ Type: Time +

+ Name: xMax +

+ 		+ Type: Number +

+ Name: yMin +

+ 		+ Type: Number +

+ Name: yMax +

+ 		+ Type: Number +

+ Name: Value +

+
165341639100016534163920001002001
165341639200016534163930002004002
+ +* For high resolution with many gaps, this will require less space +* This format is much less optomized for fast render and lookup than the uniform "scanlines" approach From 58560ac7b9deed6fa951f318ac8c51e80dabc920 Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 11:50:32 -0700 Subject: [PATCH 03/12] rename contract.md --- data/contract_docs/contract.md | 93 ---------------------------------- data/contract_docs/heatmap.md | 28 +++++++++- 2 files changed, 27 insertions(+), 94 deletions(-) delete mode 100644 data/contract_docs/contract.md diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md deleted file mode 100644 index 34de47d92..000000000 --- a/data/contract_docs/contract.md +++ /dev/null @@ -1,93 +0,0 @@ -# Data Plane Contract - Technical Specification - -Status: Draft/Proposal - -## Doc Objective - -Define in detail common query response schemas for data returned from data sources. This improves the experience for developers of both features and datasources. This will also improve the experience for users through more reliability and quality - which leads to more development time spent more towards improving experience. - -Current Backend [proof of concept code](https://github.com/grafana/grafana-plugin-sdk-go/pull/440). - -## Kinds and Formats - -There are logical **_kinds_** (like Time Series Data, Numeric, Histogram, etc), and there are **_formats_** that a kind can be in. - -A **_data type_** definition or declaration in this framework includes both a kind and format. For example, "TimeSeriesWide" is: kind: "Time Series", format: "Wide". - -* [Time series](./timeseries.md) - * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) - * [Long](./timeseries.md#time-series-wide-format-timeserieswide) - * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) -* [Numeric](./numeric.md) - * [Wide](./numeric.md#numeric-wide-format-numericwide) - * [Many](./numeric.md#numeric-many-format-numericmany) -* [Heatmap](./heatmap.md) - * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) - * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) - * [Sparse](./heatmap.md#heatmap-sparse-heatmapsparse) - - - -## Dimensional Set Based Kinds - -Within a data type (kind+format), there can be multiple **_items_** of data that are uniquely identified. This forms a **_set_** of data items. For example, in the numeric kind there can be a set of numbers, or, in the time series kind, a set of time series-es :-). - -Each item of data in a set is uniquely identified by its **_name_** and **_dimensions_**. - -Dimensions are facets of data (such as "location" or "host") with a corresponding value. For example, {"host"="a", "location"="new york"}. - -Within a dataframe, dimensions are in either a field's Labels property or in string field(s). - -### Properties Shared by all Dimensional Set Based Kinds - -* When there are multiple items that have the same name, they should have different dimensions (e.g. labels) that uniquely identifies each item[^1]. -* The item name should appear in the Name property of each value (numeric or bool typed) Field, as should any Labels[^2] -* A response can have different item names in the response (Note: SSE doesn't currently handle this) - -## Remainder Data - -Data is encoded into dataframe(s), therefore all types are implemented as an array of data.Frames. - -There can be data in dataframe(s) that is not part of the data type's data. This extra data is **_remainder data_**. What readers choose to do with this data is open. However, libraries based on this contract must clearly delineate remainder data from data that is part of the type. - -What data becomes remainder data is dependent on and specified in the data type. Generally, it can be additional frames and/or additional fields of a certain field type. - -## Invalid Data - -Although there is remainder data, there are still cases where the reader should error. The situation for this is when the data type specifier exists on the frame(s), but rules about that type are not followed. - -## "No Data" and Empty - -There are two named cases for when a response is lacking data and also doesn't have an error. - - **_"No Data"_** is for when we retrieve a response from a datasource but the response has no data items. The encoding for the form of a type is a single frame, with the data type declaration, and a zero length of fields (null or []). This is for the case when the entire set has no items. - -We retrieve one or more data items from a datasource but an item has no values, that item is said to be an "**_Empty value_**". In this case, the required dataframe fields should still be present (but the fields themselves each have no values). - -## Multi Data Type Responses - -The case where a response has multiple data types in a single result (Within a RefID) exists but is currently out of scope for this version of the spec. - -However, it needs to be possible to add support for this case. For now, the following logic is suggested: - -* Per data type, within a response, only one format should be used. For example: There may be TimeSeriesWide and NumericLong, but there should _not_ be TimeSeriesWide and TimeSeriesLong. -* The borders between the types are derived from adjacent frames (within the array of frames) that share the same data type. -* If a reader does not opt-in into multi-type responses, it should be able to get the first data type that matches what the reader is looking for. - -## Considerations to Add / Todo - -* Meta-data (Frame and Field) -* If the type/schema is declared, do we need to support the case where, for whatever reason, the type can be considered multiple Kinds at once? -* So far ordering is ignored (For example, the order of Value Fields in TimeSeriesWide or the order of Frames in TimeSeriesMulti). Need to decide if ordering as any symantec meaning, if so what it is, and consider it properties of converting between formats - * Note: Issue on ordering [https://github.com/grafana/grafana-plugin-sdk-go/issues/366](https://github.com/grafana/grafana-plugin-sdk-go/issues/366) , not sure if it is display issue or not at this time - - -## Notes - -[^1]: - - In theory they can still be passed for things like visualization because Fields do have a numeric ordering within the frame, but this won't work with things like SSE/alerting. - -[^2]: - - Using Field Name keeps naming consistent with the TimeSeriesMulti format (vs using the Frame Name) diff --git a/data/contract_docs/heatmap.md b/data/contract_docs/heatmap.md index f46e0999e..f7e5c2654 100644 --- a/data/contract_docs/heatmap.md +++ b/data/contract_docs/heatmap.md @@ -95,7 +95,19 @@ There exists a row for every cell in the heatmap. Type: Number

- Name: Value + Name: Count +

+ + + Type: Number +

+ Name: Total +

+ + + Type: Number +

+ Name: Speed

@@ -103,37 +115,51 @@ There exists a row for every cell in the heatmap. 1653416391000 100 1 + 1 + 1 1653416391000 200 2 + 2 + 2 1653416391000 300 3 + 3 + 3 1653416392000 100 4 + 4 + 4 1653416392000 200 5 + 5 + 5 1653416392000 300 6 + 6 + 6 This format requires uniform cell sizing +Note that multiple "value" fields can included to represent multiple dimensions within the same cell. +The first value field is used in the display, unless explicilty configured ## Heatmap sparse (HeatmapSparse) From 59741af0f1fb1adcdccc28f8bc94f2e406bffbfb Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 11:51:09 -0700 Subject: [PATCH 04/12] rename contract.md --- data/contract_docs/README.md | 93 ++++++++++++++++++++++++++++++++++++ 1 file changed, 93 insertions(+) create mode 100644 data/contract_docs/README.md diff --git a/data/contract_docs/README.md b/data/contract_docs/README.md new file mode 100644 index 000000000..34de47d92 --- /dev/null +++ b/data/contract_docs/README.md @@ -0,0 +1,93 @@ +# Data Plane Contract - Technical Specification + +Status: Draft/Proposal + +## Doc Objective + +Define in detail common query response schemas for data returned from data sources. This improves the experience for developers of both features and datasources. This will also improve the experience for users through more reliability and quality - which leads to more development time spent more towards improving experience. + +Current Backend [proof of concept code](https://github.com/grafana/grafana-plugin-sdk-go/pull/440). + +## Kinds and Formats + +There are logical **_kinds_** (like Time Series Data, Numeric, Histogram, etc), and there are **_formats_** that a kind can be in. + +A **_data type_** definition or declaration in this framework includes both a kind and format. For example, "TimeSeriesWide" is: kind: "Time Series", format: "Wide". + +* [Time series](./timeseries.md) + * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) + * [Long](./timeseries.md#time-series-wide-format-timeserieswide) + * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) +* [Numeric](./numeric.md) + * [Wide](./numeric.md#numeric-wide-format-numericwide) + * [Many](./numeric.md#numeric-many-format-numericmany) +* [Heatmap](./heatmap.md) + * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) + * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) + * [Sparse](./heatmap.md#heatmap-sparse-heatmapsparse) + + + +## Dimensional Set Based Kinds + +Within a data type (kind+format), there can be multiple **_items_** of data that are uniquely identified. This forms a **_set_** of data items. For example, in the numeric kind there can be a set of numbers, or, in the time series kind, a set of time series-es :-). + +Each item of data in a set is uniquely identified by its **_name_** and **_dimensions_**. + +Dimensions are facets of data (such as "location" or "host") with a corresponding value. For example, {"host"="a", "location"="new york"}. + +Within a dataframe, dimensions are in either a field's Labels property or in string field(s). + +### Properties Shared by all Dimensional Set Based Kinds + +* When there are multiple items that have the same name, they should have different dimensions (e.g. labels) that uniquely identifies each item[^1]. +* The item name should appear in the Name property of each value (numeric or bool typed) Field, as should any Labels[^2] +* A response can have different item names in the response (Note: SSE doesn't currently handle this) + +## Remainder Data + +Data is encoded into dataframe(s), therefore all types are implemented as an array of data.Frames. + +There can be data in dataframe(s) that is not part of the data type's data. This extra data is **_remainder data_**. What readers choose to do with this data is open. However, libraries based on this contract must clearly delineate remainder data from data that is part of the type. + +What data becomes remainder data is dependent on and specified in the data type. Generally, it can be additional frames and/or additional fields of a certain field type. + +## Invalid Data + +Although there is remainder data, there are still cases where the reader should error. The situation for this is when the data type specifier exists on the frame(s), but rules about that type are not followed. + +## "No Data" and Empty + +There are two named cases for when a response is lacking data and also doesn't have an error. + + **_"No Data"_** is for when we retrieve a response from a datasource but the response has no data items. The encoding for the form of a type is a single frame, with the data type declaration, and a zero length of fields (null or []). This is for the case when the entire set has no items. + +We retrieve one or more data items from a datasource but an item has no values, that item is said to be an "**_Empty value_**". In this case, the required dataframe fields should still be present (but the fields themselves each have no values). + +## Multi Data Type Responses + +The case where a response has multiple data types in a single result (Within a RefID) exists but is currently out of scope for this version of the spec. + +However, it needs to be possible to add support for this case. For now, the following logic is suggested: + +* Per data type, within a response, only one format should be used. For example: There may be TimeSeriesWide and NumericLong, but there should _not_ be TimeSeriesWide and TimeSeriesLong. +* The borders between the types are derived from adjacent frames (within the array of frames) that share the same data type. +* If a reader does not opt-in into multi-type responses, it should be able to get the first data type that matches what the reader is looking for. + +## Considerations to Add / Todo + +* Meta-data (Frame and Field) +* If the type/schema is declared, do we need to support the case where, for whatever reason, the type can be considered multiple Kinds at once? +* So far ordering is ignored (For example, the order of Value Fields in TimeSeriesWide or the order of Frames in TimeSeriesMulti). Need to decide if ordering as any symantec meaning, if so what it is, and consider it properties of converting between formats + * Note: Issue on ordering [https://github.com/grafana/grafana-plugin-sdk-go/issues/366](https://github.com/grafana/grafana-plugin-sdk-go/issues/366) , not sure if it is display issue or not at this time + + +## Notes + +[^1]: + + In theory they can still be passed for things like visualization because Fields do have a numeric ordering within the frame, but this won't work with things like SSE/alerting. + +[^2]: + + Using Field Name keeps naming consistent with the TimeSeriesMulti format (vs using the Frame Name) From 6db7efdcdfc5e31d8a7a0fa198c7a9f53e4b8978 Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 11:57:16 -0700 Subject: [PATCH 05/12] yMin/yMax/y --- data/contract_docs/README.md | 2 +- data/contract_docs/heatmap.md | 10 ++++++++-- 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/data/contract_docs/README.md b/data/contract_docs/README.md index 34de47d92..5c418dcc3 100644 --- a/data/contract_docs/README.md +++ b/data/contract_docs/README.md @@ -16,7 +16,7 @@ A **_data type_** definition or declaration in this framework includes both a ki * [Time series](./timeseries.md) * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) - * [Long](./timeseries.md#time-series-wide-format-timeserieswide) + * [Long](./timeseries.md#time-series-long-format-timeserieslong-sql-like) * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) * [Numeric](./numeric.md) * [Wide](./numeric.md#numeric-wide-format-numericwide) diff --git a/data/contract_docs/heatmap.md b/data/contract_docs/heatmap.md index f7e5c2654..6d2b23e50 100644 --- a/data/contract_docs/heatmap.md +++ b/data/contract_docs/heatmap.md @@ -83,13 +83,13 @@ There exists a row for every cell in the heatmap. Type: Time

- Name: xMax + Name: xMax|xMin|x

Type: Number

- Name: yMax + Name: yMax|yMin|y

@@ -161,6 +161,12 @@ This format requires uniform cell sizing Note that multiple "value" fields can included to represent multiple dimensions within the same cell. The first value field is used in the display, unless explicilty configured +The field names for yMax|yMin|y indicate the aggregation period or the supplied values. +* yMax: the values are from the bucket below +* yMin: the values are from to bucket above +* y: the values are in the middle of the bucket + + ## Heatmap sparse (HeatmapSparse) This format is simplar to Heatmap scanlines, except that each cell is independent from its adjacent values. From 7a1534e723c26d0311e57711b7e2ae264c7ddea0 Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 24 May 2022 12:18:42 -0700 Subject: [PATCH 06/12] timeseries wide --- data/contract_docs/heatmap.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/data/contract_docs/heatmap.md b/data/contract_docs/heatmap.md index 6d2b23e50..973fefde2 100644 --- a/data/contract_docs/heatmap.md +++ b/data/contract_docs/heatmap.md @@ -71,6 +71,10 @@ Example: +Note: [Timeseries wide](./timeseries.md#time-series-wide-format-timeserieswide) can be used directly +as heatmap-buckets, in this case each value field becomes a row in the heatmap. + + ## Heatmap scanlines (HeatmapScanlines) In this format, each row in the frame indicates the value of a single cell in a heatmap. From 0ba186a02a643fa428487a5194a7ed4e6ca8523b Mon Sep 17 00:00:00 2001 From: Ryan McKinley Date: Tue, 8 Nov 2022 15:37:21 -0800 Subject: [PATCH 07/12] add numeric long --- data/contract_docs/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/data/contract_docs/README.md b/data/contract_docs/README.md index 101fbf34f..4c5dfd21c 100644 --- a/data/contract_docs/README.md +++ b/data/contract_docs/README.md @@ -21,6 +21,7 @@ A **_data type_** definition or declaration in this framework includes both a ki * [Numeric](./numeric.md) * [Wide](./numeric.md#numeric-wide-format-numericwide) * [Many](./numeric.md#numeric-many-format-numericmany) + * [Long](./numeric.md#numeric-many-format-numericlong) * [Heatmap](./heatmap.md) * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) From 84ec97d88acf5fbb9f2946b4042199f5332e0a6c Mon Sep 17 00:00:00 2001 From: Brendan O'Handley Date: Wed, 9 Nov 2022 12:33:15 -0500 Subject: [PATCH 08/12] DataPlane: Update heatmap docs (#554) * rename README to contract * update docs for scanline uniform cells and difference between sparse and scanlines --- data/contract_docs/contract.md | 102 +++++++++++++++++++++++++++++++++ data/contract_docs/heatmap.md | 4 +- 2 files changed, 104 insertions(+), 2 deletions(-) create mode 100644 data/contract_docs/contract.md diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md new file mode 100644 index 000000000..4c5dfd21c --- /dev/null +++ b/data/contract_docs/contract.md @@ -0,0 +1,102 @@ +# Data Plane Contract - Technical Specification + +Status: Draft/Proposal + +## Doc Objective + +Define in detail common query response schemas for data returned from data sources. This improves the experience for developers of both features and datasources. This will also improve the experience for users through more reliability and quality - which leads to more development time spent more towards improving experience. + +Current Backend [proof of concept code](https://github.com/grafana/grafana-plugin-sdk-go/pull/440). + +## Kinds and Formats + +There are logical **_kinds_** (like Time Series Data, Numeric, Histogram, etc), and there are **_formats_** that a kind can be in. + +A **_data type_** definition or declaration in this framework includes both a kind and format. For example, "TimeSeriesWide" is: kind: "Time Series", format: "Wide". + +* [Time series](./timeseries.md) + * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) + * [Long](./timeseries.md#time-series-long-format-timeserieslong-sql-like) + * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) +* [Numeric](./numeric.md) + * [Wide](./numeric.md#numeric-wide-format-numericwide) + * [Many](./numeric.md#numeric-many-format-numericmany) + * [Long](./numeric.md#numeric-many-format-numericlong) +* [Heatmap](./heatmap.md) + * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) + * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) + * [Sparse](./heatmap.md#heatmap-sparse-heatmapsparse) + + + +## Dimensional Set Based Kinds + +Within a data type (kind+format), there can be multiple **_items_** of data that are uniquely identified. This forms a **_set_** of data items. For example, in the numeric kind there can be a set of numbers, or, in the time series kind, a set of time series-es :-). + +Each item of data in a set is uniquely identified by its **_name_** and **_dimensions_**. + +Dimensions are facets of data (such as "location" or "host") with a corresponding value. For example, {"host"="a", "location"="new york"}. + +Within a dataframe, dimensions are in either a field's Labels property or in string field(s). + +### Properties Shared by all Dimensional Set Based Kinds + +* When there are multiple items that have the same name, they should have different dimensions (e.g. labels) that uniquely identifies each item[^1]. +* The item name should appear in the Name property of each value (numeric or bool typed) Field, as should any Labels[^2] +* A response can have different item names in the response (Note: SSE doesn't currently handle this) + +## Remainder Data + +Data is encoded into dataframe(s), therefore all types are implemented as an array of data.Frames. + +There can be data in dataframe(s) that is not part of the data type's data. This extra data is **_remainder data_**. What readers choose to do with this data is open. However, libraries based on this contract must clearly delineate remainder data from data that is part of the type. + +What data becomes remainder data is dependent on and specified in the data type. Generally, it can be additional frames and/or additional fields of a certain field type. + +## Invalid Data + +Although there is remainder data, there are still cases where the reader should error. The situation for this is when the data type specifier exists on the frame(s), but rules about that type are not followed. + +## "No Data" and Empty + +There are two named cases for when a response is lacking data and also doesn't have an error. + + **_"No Data"_** is for when we retrieve a response from a datasource but the response has no data items. The encoding for the form of a type is a single frame, with the data type declaration, and a zero length of fields (null or []). This is for the case when the entire set has no items. + +We retrieve one or more data items from a datasource but an item has no values, that item is said to be an "**_Empty value_**". In this case, the required dataframe fields should still be present (but the fields themselves each have no values). + +## Error Responses + +An error is returned from outside the dataframes using the `Error` and `Status` properties on a [DataResponse](https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go/backend#DataResponse). + +When an error is returned with the DataResponse, a single frame with no fields may be included as well. If the error is present, this will not be considered "No Data". This frame is included so that metadata, in particular a Frame's `ExecutedQueryString`, can be returned to the caller with an error. + +Note: In a backend plugin an error can be returned from a [`DataQueryHandler`](https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go/backend#QueryDataHandler) call. This should only be used when the entire request (all queries) will fail. + +## Multi Data Type Responses + +The case where a response has multiple data types in a single result (Within a RefID) exists but is currently out of scope for this version of the spec. + +However, it needs to be possible to add support for this case. For now, the following logic is suggested: + +* Per data type, within a response, only one format should be used. For example: There may be TimeSeriesWide and NumericLong, but there should _not_ be TimeSeriesWide and TimeSeriesLong. +* The borders between the types are derived from adjacent frames (within the array of frames) that share the same data type. +* If a reader does not opt-in into multi-type responses, it should be able to get the first data type that matches what the reader is looking for. + +## Considerations to Add / Todo + +* Meta-data (Frame and Field) +* If the type/schema is declared, do we need to support the case where, for whatever reason, the type can be considered multiple Kinds at once? +* So far ordering is ignored (For example, the order of Value Fields in TimeSeriesWide or the order of Frames in TimeSeriesMulti). Need to decide if ordering as any symantec meaning, if so what it is, and consider it properties of converting between formats + * Note: Issue on ordering [https://github.com/grafana/grafana-plugin-sdk-go/issues/366](https://github.com/grafana/grafana-plugin-sdk-go/issues/366) , not sure if it is display issue or not at this time + + +## Notes + +[^1]: + + In theory they can still be passed for things like visualization because Fields do have a numeric ordering within the frame, but this won't work with things like SSE/alerting. + +[^2]: + + Using Field Name keeps naming consistent with the TimeSeriesMulti format (vs using the Frame Name) diff --git a/data/contract_docs/heatmap.md b/data/contract_docs/heatmap.md index 973fefde2..8f55365db 100644 --- a/data/contract_docs/heatmap.md +++ b/data/contract_docs/heatmap.md @@ -160,7 +160,7 @@ There exists a row for every cell in the heatmap. -This format requires uniform cell sizing +This format requires uniform cell sizing. The size of the cell is defined by the columns in each row that are chosen as the xMax|xMin|x and the yMax|yMin|y. We can see that the Number column(yMax|yMin|y) increases by 100(each cell is roughly 100 higher than the previous cell on the y axis) for each row containing a similar Time value(these stacked cells all have roughly the same location along the x axis). This produces a uniform cell size. Note that multiple "value" fields can included to represent multiple dimensions within the same cell. The first value field is used in the display, unless explicilty configured @@ -174,7 +174,7 @@ The field names for yMax|yMin|y indicate the aggregation period or the supplied ## Heatmap sparse (HeatmapSparse) This format is simplar to Heatmap scanlines, except that each cell is independent from its adjacent values. -Unlike scanlines, this allows resolutions to change over time. When +Unlike scanlines, this allows resolutions to change over time. Where scanline has uniformity of cells over time, heatmap sparse allows for variability of cells along the x axis(Time). Example: From 30f89fac5f33e926fffa68c72fb2aee71a61e958 Mon Sep 17 00:00:00 2001 From: bohandley Date: Wed, 9 Nov 2022 12:47:35 -0500 Subject: [PATCH 09/12] remove readme will change this later --- data/contract_docs/README.md | 102 ----------------------------------- 1 file changed, 102 deletions(-) delete mode 100644 data/contract_docs/README.md diff --git a/data/contract_docs/README.md b/data/contract_docs/README.md deleted file mode 100644 index 4c5dfd21c..000000000 --- a/data/contract_docs/README.md +++ /dev/null @@ -1,102 +0,0 @@ -# Data Plane Contract - Technical Specification - -Status: Draft/Proposal - -## Doc Objective - -Define in detail common query response schemas for data returned from data sources. This improves the experience for developers of both features and datasources. This will also improve the experience for users through more reliability and quality - which leads to more development time spent more towards improving experience. - -Current Backend [proof of concept code](https://github.com/grafana/grafana-plugin-sdk-go/pull/440). - -## Kinds and Formats - -There are logical **_kinds_** (like Time Series Data, Numeric, Histogram, etc), and there are **_formats_** that a kind can be in. - -A **_data type_** definition or declaration in this framework includes both a kind and format. For example, "TimeSeriesWide" is: kind: "Time Series", format: "Wide". - -* [Time series](./timeseries.md) - * [Wide](./timeseries.md#time-series-wide-format-timeserieswide) - * [Long](./timeseries.md#time-series-long-format-timeserieslong-sql-like) - * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) -* [Numeric](./numeric.md) - * [Wide](./numeric.md#numeric-wide-format-numericwide) - * [Many](./numeric.md#numeric-many-format-numericmany) - * [Long](./numeric.md#numeric-many-format-numericlong) -* [Heatmap](./heatmap.md) - * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) - * [Scanlines](./heatmap.md#heatmap-scanlines-heatmapscanlines) - * [Sparse](./heatmap.md#heatmap-sparse-heatmapsparse) - - - -## Dimensional Set Based Kinds - -Within a data type (kind+format), there can be multiple **_items_** of data that are uniquely identified. This forms a **_set_** of data items. For example, in the numeric kind there can be a set of numbers, or, in the time series kind, a set of time series-es :-). - -Each item of data in a set is uniquely identified by its **_name_** and **_dimensions_**. - -Dimensions are facets of data (such as "location" or "host") with a corresponding value. For example, {"host"="a", "location"="new york"}. - -Within a dataframe, dimensions are in either a field's Labels property or in string field(s). - -### Properties Shared by all Dimensional Set Based Kinds - -* When there are multiple items that have the same name, they should have different dimensions (e.g. labels) that uniquely identifies each item[^1]. -* The item name should appear in the Name property of each value (numeric or bool typed) Field, as should any Labels[^2] -* A response can have different item names in the response (Note: SSE doesn't currently handle this) - -## Remainder Data - -Data is encoded into dataframe(s), therefore all types are implemented as an array of data.Frames. - -There can be data in dataframe(s) that is not part of the data type's data. This extra data is **_remainder data_**. What readers choose to do with this data is open. However, libraries based on this contract must clearly delineate remainder data from data that is part of the type. - -What data becomes remainder data is dependent on and specified in the data type. Generally, it can be additional frames and/or additional fields of a certain field type. - -## Invalid Data - -Although there is remainder data, there are still cases where the reader should error. The situation for this is when the data type specifier exists on the frame(s), but rules about that type are not followed. - -## "No Data" and Empty - -There are two named cases for when a response is lacking data and also doesn't have an error. - - **_"No Data"_** is for when we retrieve a response from a datasource but the response has no data items. The encoding for the form of a type is a single frame, with the data type declaration, and a zero length of fields (null or []). This is for the case when the entire set has no items. - -We retrieve one or more data items from a datasource but an item has no values, that item is said to be an "**_Empty value_**". In this case, the required dataframe fields should still be present (but the fields themselves each have no values). - -## Error Responses - -An error is returned from outside the dataframes using the `Error` and `Status` properties on a [DataResponse](https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go/backend#DataResponse). - -When an error is returned with the DataResponse, a single frame with no fields may be included as well. If the error is present, this will not be considered "No Data". This frame is included so that metadata, in particular a Frame's `ExecutedQueryString`, can be returned to the caller with an error. - -Note: In a backend plugin an error can be returned from a [`DataQueryHandler`](https://pkg.go.dev/github.com/grafana/grafana-plugin-sdk-go/backend#QueryDataHandler) call. This should only be used when the entire request (all queries) will fail. - -## Multi Data Type Responses - -The case where a response has multiple data types in a single result (Within a RefID) exists but is currently out of scope for this version of the spec. - -However, it needs to be possible to add support for this case. For now, the following logic is suggested: - -* Per data type, within a response, only one format should be used. For example: There may be TimeSeriesWide and NumericLong, but there should _not_ be TimeSeriesWide and TimeSeriesLong. -* The borders between the types are derived from adjacent frames (within the array of frames) that share the same data type. -* If a reader does not opt-in into multi-type responses, it should be able to get the first data type that matches what the reader is looking for. - -## Considerations to Add / Todo - -* Meta-data (Frame and Field) -* If the type/schema is declared, do we need to support the case where, for whatever reason, the type can be considered multiple Kinds at once? -* So far ordering is ignored (For example, the order of Value Fields in TimeSeriesWide or the order of Frames in TimeSeriesMulti). Need to decide if ordering as any symantec meaning, if so what it is, and consider it properties of converting between formats - * Note: Issue on ordering [https://github.com/grafana/grafana-plugin-sdk-go/issues/366](https://github.com/grafana/grafana-plugin-sdk-go/issues/366) , not sure if it is display issue or not at this time - - -## Notes - -[^1]: - - In theory they can still be passed for things like visualization because Fields do have a numeric ordering within the frame, but this won't work with things like SSE/alerting. - -[^2]: - - Using Field Name keeps naming consistent with the TimeSeriesMulti format (vs using the Frame Name) From 3823ce898d377d552890ff56d17f25fc42a8f984 Mon Sep 17 00:00:00 2001 From: bohandley Date: Wed, 9 Nov 2022 12:59:35 -0500 Subject: [PATCH 10/12] change numeric many to numeric multi --- data/contract_docs/contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md index 4c5dfd21c..ad0e278c1 100644 --- a/data/contract_docs/contract.md +++ b/data/contract_docs/contract.md @@ -20,7 +20,7 @@ A **_data type_** definition or declaration in this framework includes both a ki * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) * [Numeric](./numeric.md) * [Wide](./numeric.md#numeric-wide-format-numericwide) - * [Many](./numeric.md#numeric-many-format-numericmany) + * [Multi](./numeric.md#numeric-many-format-numericmany) * [Long](./numeric.md#numeric-many-format-numericlong) * [Heatmap](./heatmap.md) * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) From bc701529b2732a34f656cef634fce50b29a24427 Mon Sep 17 00:00:00 2001 From: bohandley Date: Wed, 9 Nov 2022 13:01:05 -0500 Subject: [PATCH 11/12] change numeric many to numeric multi --- data/contract_docs/contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md index ad0e278c1..589a29111 100644 --- a/data/contract_docs/contract.md +++ b/data/contract_docs/contract.md @@ -20,7 +20,7 @@ A **_data type_** definition or declaration in this framework includes both a ki * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) * [Numeric](./numeric.md) * [Wide](./numeric.md#numeric-wide-format-numericwide) - * [Multi](./numeric.md#numeric-many-format-numericmany) + * [Many](./numeric.md#numeric-multi-format-numericmulti) * [Long](./numeric.md#numeric-many-format-numericlong) * [Heatmap](./heatmap.md) * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets) From d3c058e469c4a8dc2832270ee88baa6886aae1d1 Mon Sep 17 00:00:00 2001 From: bohandley Date: Wed, 9 Nov 2022 13:07:41 -0500 Subject: [PATCH 12/12] change numeric many to numeric multi --- data/contract_docs/contract.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/data/contract_docs/contract.md b/data/contract_docs/contract.md index 589a29111..db7059913 100644 --- a/data/contract_docs/contract.md +++ b/data/contract_docs/contract.md @@ -20,7 +20,7 @@ A **_data type_** definition or declaration in this framework includes both a ki * [Multi](./timeseries.md#time-series-multi-format-timeseriesmulti) * [Numeric](./numeric.md) * [Wide](./numeric.md#numeric-wide-format-numericwide) - * [Many](./numeric.md#numeric-multi-format-numericmulti) + * [Multi](./numeric.md#numeric-multi-format-numericmulti) * [Long](./numeric.md#numeric-many-format-numericlong) * [Heatmap](./heatmap.md) * [Buckets](./heatmap.md#heatmap-buckets-heatmapbuckets)