Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inconsistent results get from GreptimeDB and Prometheus in irate of PromQL #3790

Open
Taylor-lagrange opened this issue Apr 24, 2024 · 1 comment
Open

Inconsistent results get from GreptimeDB and Prometheus in irate of PromQL #3790

Taylor-lagrange opened this issue Apr 24, 2024 · 1 comment
Assignees
@waynexia

Comments

@Taylor-lagrange
Copy link
Collaborator

Taylor-lagrange commented Apr 24, 2024 •
edited

What type of bug is this?

Incorrect result

What subsystems are affected?

Query Engine

Minimal reproduce step

I send a range query to both GreptimeDB and Prometheus, with

  • start: current_timestamp -2m - 20m
  • end: current_timestamp -2m
  • step: 2m
  • $__rate_interval：1m

But get different output:

irate(node_network_receive_bytes_total{instance="100.96.135.81:9100",job="node_exporter"}[1m])*8

What did you expect to see?

The same result

What did you see instead?

Inconsist result

What operating system did you use?

MacOS 14.4

What version of GreptimeDB did you use?

0.7.2

Relevant log output and stack trace

Response from GreptimeDB:

{
    "status": "success",
    "data": {
        "resultType": "matrix",
        "result": [
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter",
                    "device": "br-3381aa7373cf"
                },
                "values": [
                    [
                        1713941445.0,
                        36.57142857142857
                    ],
                    [
                        1713941565.0,
                        "76.57142857142857"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "device": "br-e377d2753209",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "14886.285714285714"
                    ],
                    [
                        1713941565.0,
                        "28031.428571428572"
                    ],
                    [
                        1713941685.0,
                        "14336"
                    ]
                ]
            },
            {
                "metric": {
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "docker0",
                    "instance": "100.96.135.81:9100"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "device": "enp4s0",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "26734.85714285714"
                    ],
                    [
                        1713941565.0,
                        "40889.142857142855"
                    ],
                    [
                        1713941685.0,
                        "27256"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter",
                    "device": "lo"
                },
                "values": [
                    [
                        1713941445.0,
                        "3381.714285714286"
                    ],
                    [
                        1713941565.0,
                        "608"
                    ],
                    [
                        1713941685.0,
                        "704.5333333333333"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "device": "tailscale0",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "1026.857142857143"
                    ],
                    [
                        1713941565.0,
                        "848.5714285714286"
                    ],
                    [
                        1713941685.0,
                        "819.7333333333333"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth1d05ba5",
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "instance": "100.96.135.81:9100"
                },
                "values": [
                    [
                        1713941445.0,
                        "137.14285714285714"
                    ],
                    [
                        1713941565.0,
                        "137.14285714285714"
                    ],
                    [
                        1713941685.0,
                        "105.6"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "veth1d05e95",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "44.57142857142857"
                    ],
                    [
                        1713941565.0,
                        "92.57142857142857"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "veth317a2c6"
                },
                "values": [
                    [
                        1713941445.0,
                        "40"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter",
                    "device": "veth386d553"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth41aa8c8",
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter",
                    "instance": "100.96.135.81:9100",
                    "device": "veth4a79cc2"
                },
                "values": [
                    [
                        1713941445.0,
                        "12770.285714285714"
                    ],
                    [
                        1713941565.0,
                        "26155.428571428572"
                    ],
                    [
                        1713941685.0,
                        "12488"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth622841d",
                    "__name__": "node_network_receive_bytes_total",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "veth719414a"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth73c9db7",
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "137.14285714285714"
                    ],
                    [
                        1713941565.0,
                        "137.14285714285714"
                    ],
                    [
                        1713941685.0,
                        "105.6"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "device": "veth76e1e19",
                    "job": "node_exporter",
                    "instance": "100.96.135.81:9100"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter",
                    "device": "veth7a06cf3"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "job": "node_exporter",
                    "instance": "100.96.135.81:9100",
                    "device": "veth8195fc8",
                    "__name__": "node_network_receive_bytes_total"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "veth893ac6d"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "__name__": "node_network_receive_bytes_total",
                    "device": "vethb57df15",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "vethdfd7faa",
                    "instance": "100.96.135.81:9100",
                    "__name__": "node_network_receive_bytes_total",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445.0,
                        "0"
                    ],
                    [
                        1713941565.0,
                        "0"
                    ],
                    [
                        1713941685.0,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "job": "node_exporter",
                    "__name__": "node_network_receive_bytes_total",
                    "device": "vethf024f1e",
                    "instance": "100.96.135.81:9100"
                },
                "values": [
                    [
                        1713941445.0,
                        "2740"
                    ],
                    [
                        1713941565.0,
                        "2740"
                    ],
                    [
                        1713941685.0,
                        "2557.3333333333335"
                    ]
                ]
            }
        ]
    }
}

Response from Prometheus:

{
    "status": "success",
    "data": {
        "resultType": "matrix",
        "result": [
            {
                "metric": {
                    "device": "br-3381aa7373cf",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        34.14699213018541
                    ],
                    [
                        1713941565,
                        "71.47143142876192"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "br-e377d2753209",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "13899.426437241564"
                    ],
                    [
                        1713941565,
                        "26164.410960730715"
                    ],
                    [
                        1713941685,
                        "14331.222925691436"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "docker0",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "enp4s0",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "24962.5183406696"
                    ],
                    [
                        1713941565,
                        "38165.74438295886"
                    ],
                    [
                        1713941685,
                        "27246.917694101965"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "lo",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "3157.529678538082"
                    ],
                    [
                        1713941565,
                        "567.50450030002"
                    ],
                    [
                        1713941685,
                        "704.2985671442852"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "tailscale0",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "958.7835134053622"
                    ],
                    [
                        1713941565,
                        "792.0528035202346"
                    ],
                    [
                        1713941685,
                        "819.46017994002"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth1d05ba5",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "128.05122048819527"
                    ],
                    [
                        1713941565,
                        "128.00853390226015"
                    ],
                    [
                        1713941685,
                        "105.56481172942352"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth1d05e95",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "41.61664665866346"
                    ],
                    [
                        1713941565,
                        "86.4057603840256"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth317a2c6",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "37.34827264239029"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth386d553",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth41aa8c8",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth4a79cc2",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "11923.702814459117"
                    ],
                    [
                        1713941565,
                        "24413.360890726046"
                    ],
                    [
                        1713941685,
                        "12483.838720426524"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth622841d",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth719414a",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth73c9db7",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "128.05122048819527"
                    ],
                    [
                        1713941565,
                        "128.00853390226015"
                    ],
                    [
                        1713941685,
                        "105.56481172942352"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth76e1e19",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth7a06cf3",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth8195fc8",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "veth893ac6d",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "vethb57df15",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "vethdfd7faa",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "0"
                    ],
                    [
                        1713941565,
                        "0"
                    ],
                    [
                        1713941685,
                        "0"
                    ]
                ]
            },
            {
                "metric": {
                    "device": "vethf024f1e",
                    "instance": "100.96.135.81:9100",
                    "job": "node_exporter"
                },
                "values": [
                    [
                        1713941445,
                        "2558.3566760037347"
                    ],
                    [
                        1713941565,
                        "2557.5038335889058"
                    ],
                    [
                        1713941685,
                        "2556.4811729423523"
                    ]
                ]
            }
        ]
    }
}
@waynexia waynexia self-assigned this Apr 24, 2024
@killme2008
Copy link
Contributor

Is it fixed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@waynexia waynexia

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@killme2008 @waynexia @Taylor-lagrange