Failed to scrape Prometheus endpoint "scrape_timestamp": 1692630513175

[Herman19890]

-Hi.
when i run otelcol-contrib --config=file:/skywalking/otel-collector-config.yaml ;
it become some warn information as blow:

2023-08-21T23:08:33.176+0800    warn    internal/transaction.go:111     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1692630513175, "target_labels": "{__name__=\"up\", host_name=\"centos7u9\", instance=\"oap:1534\", job=\"skywalking-so11y\", service=\"oap-server\"}"}
2023-08-21T23:08:33.294+0800    info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 1, "metrics": 5, "data points": 5}
2023-08-21T23:08:33.294+0800    info    ResourceMetrics #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(skywalking-so11y)
     -> net.host.name: Str(oap)
     -> service.instance.id: Str(oap:1534)
     -> net.host.port: Str(1534)
     -> http.scheme: Str(http)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope otelcol/prometheusreceiver 0.83.0
Metric #0
Descriptor:
     -> Name: scrape_samples_post_metric_relabeling
     -> Description: The number of samples remaining after metric relabeling was applied
     -> Unit:
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host_name: Str(centos7u9)
     -> service: Str(oap-server)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-21 15:08:33.175 +0000 UTC
Value: 0.000000
Metric #1
Descriptor:
     -> Name: scrape_series_added
     -> Description: The approximate number of new series in this scrape
     -> Unit:
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host_name: Str(centos7u9)
     -> service: Str(oap-server)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-21 15:08:33.175 +0000 UTC
Value: 0.000000
Metric #2
Descriptor:
     -> Name: up
     -> Description: The scraping was successful
     -> Unit:
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host_name: Str(centos7u9)
     -> service: Str(oap-server)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-21 15:08:33.175 +0000 UTC
Value: 0.000000
Metric #3
Descriptor:
     -> Name: scrape_duration_seconds
     -> Description: Duration of the scrape
     -> Unit: seconds
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host_name: Str(centos7u9)
     -> service: Str(oap-server)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-21 15:08:33.175 +0000 UTC
Value: 0.000617
Metric #4
Descriptor:
     -> Name: scrape_samples_scraped
     -> Description: The number of samples the target exposed
     -> Unit:
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> host_name: Str(centos7u9)
     -> service: Str(oap-server)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-08-21 15:08:33.175 +0000 UTC
Value: 0.000000
        {"kind": "exporter", "data_type": "metrics", "name": "logging"}

the yaml contents is :

[root@centos7u9 ~]# cat /skywalking/otel-collector-config.yaml
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'skywalking-so11y'
scrape_interval: 5s
static_configs:
- targets: ['oap:1534']
labels:
#host_name: http://192.168.200.118:1534
host_name: centos7u9
service: oap-server
processors:
batch:

exporters:
otlp:
endpoint: 192.168.200.118:11800
tls:
insecure: true
logging:
verbosity: detailed
sampling_initial: 5
sampling_thereafter: 200
service:
pipelines:
metrics:
receivers:
- prometheus
processors:
- batch
exporters:
- otlp
- logging
[root@centos7u9 ~]#

the information for my hosts is

[root@centos7u9 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ens192: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:be:a7:6e brd ff:ff:ff:ff:ff:ff
inet 192.168.200.118/24 brd 192.168.200.255 scope global noprefixroute ens192
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:f0:cf:cf brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:f0:cf:cf brd ff:ff:ff:ff:ff:ff
[root@centos7u9 ~]# netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 1777/dnsmasq
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1095/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1090/cupsd
tcp 0 0 0.0.0.0:11800 0.0.0.0:* LISTEN 74932/java
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1348/master
tcp 0 0 192.168.200.118:1534 0.0.0.0:* LISTEN 74932/java
tcp 0 0 0.0.0.0:12800 0.0.0.0:* LISTEN 74932/java
tcp 0 0 0.0.0.0:12801 0.0.0.0:* LISTEN 74932/java
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 74932/java
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 799/rpcbind
tcp 0 0 0.0.0.0:9010 0.0.0.0:* LISTEN 75079/java
tcp6 0 0 :::22 :::* LISTEN 1095/sshd
tcp6 0 0 :::33060 :::* LISTEN 74665/mysqld
tcp6 0 0 :::3306 :::* LISTEN 74665/mysqld
tcp6 0 0 :::111 :::* LISTEN 799/rpcbind
udp 0 0 192.168.122.1:53 0.0.0.0:* 1777/dnsmasq
udp 0 0 0.0.0.0:67 0.0.0.0:* 1777/dnsmasq
udp 0 0 0.0.0.0:111 0.0.0.0:* 799/rpcbind
udp 0 0 0.0.0.0:965 0.0.0.0:* 799/rpcbind
udp 0 0 0.0.0.0:5353 0.0.0.0:* 790/avahi-daemon: r
udp 0 0 0.0.0.0:36614 0.0.0.0:* 790/avahi-daemon: r
udp6 0 0 :::111 :::* 799/rpcbind
udp6 0 0 :::965 :::* 799/rpcbind
[root@centos7u9 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.200.118 centos7u9
[root@centos7u9 ~]#

[Charlotte-br560]

It seems like the Prometheus endpoint scraping is failing due to an issue with the timestamp. The timestamp value "1692630513175" seems to be incorrect or invalid, leading to the failure.

You might want to verify the timestamp format or ensure that the timestamp being generated is within an acceptable range. Additionally, check if there are any misconfigurations related to the timestamp handling in your setup.