EC2 - No change in target state when deleting a security rule preventing connection to an instance

[reamasesa]

Hi,

First time writing here, if there are any guidelines I'm not following let me know and I'll edit my post

I'm using a Prometheus docker container to scan my EC2 instances based on tags and I've discovered an issue I'm not sure how to resolve

If there is no security rule for a specified port when Prometheus is launched - the instance is marked as down
If a security rule for the specified port is added after the instance was marked down and Prometheus is already running - the instance is marked as up
However, if Prometheus is running and the instance is marked as up and the security rule for the specified port is removed, the instance is still marked as up and nothing is showing up on the Prometheus logs, even in debug

I tested this with v2.21.0, v2.22.0 and v2.30.2 official docker containers, all produced the same results

Here's my configuration file
`global:
scrape_interval: 10s
evaluation_interval: 15s

alerting:
alertmanagers:

• static_configs:
  • targets:
    • 'alertmanager:9093'

rule_files:

• /etc/prometheus/rules.d/*

scrape_configs:

#############################################################################

EC2 Node Exporter Discovery

#############################################################################

• job_name: 'ec2-auto_9500'
  metrics_path: '/metrics'
  scheme: 'http'
  ec2_sd_configs:
  • region: us-east-1
    profile: arn:aws:iam::XxXxXxXxX:role/DefaultInstance
    port: 9500
    filters:
    • name: tag:Alert
      values:
      • 'Yes'
    • name: tag:NodeExporter9500
      values:
      • 'Yes'
        relabel_configs:
    • source_labels: [__meta_ec2_tag_Alert]
      target_label: Alert
    • source_labels: [__meta_ec2_tag_Name]
      target_label: Hostname
    • source_labels: [__meta_ec2_tag_Environment]
      target_label: Environment
    • source_labels: [__meta_ec2_tag_ServiceType]
      target_label: ServiceType
    • source_labels: [__meta_ec2_instance_id]
      target_label: InstanceID
    • source_labels: [__meta_ec2_public_dns_name]
      target_label: PublicDNSName
    • source_labels: [__meta_ec2_public_ip]
      target_label: ElastiIP
    • source_labels: [address]
      target_label: ram_check
      replacement: 'intensive'
    • source_labels: [address]
      target_label: cpu_check
      replacement: 'intensive'
      #############################################################################`

Here's a screen shot of the issue and cURL output from a shell

cURL output

Thanks

[roidelapluie]

I am not familiar with how EC2 works, but Prometheus uses keepalive to keep the HTTP connection to the targets. I expect that when you change security rules, it does not kill the existing connections.

[reamasesa]

Hi,

I would guess so, as well, but I'm not sure how Prometheus is handling the keepalive requests, if I delete the security rule and the EC2 tag and verify that the tagert has been removed from the targets list, add the EC2 tag only, the target is still shown as up
This, indeed, strengthen your point so, I guees, my next question is, how long should I wait for the keepalive requests to stop hammering the removed (security rule) port? Five minutes? Ten minutes before I can re-use the job and port again?

Thanks

[roidelapluie]

I think it is 5 minutes.

[reamasesa]

Thanks, my boss is too click-savvy to be bothered with waiting, I'll make a note about it in our documentation so he won't be able to complain, at least not about this :)

Thanks again

[roidelapluie]

Are you doing natting or redirecting with the security rule?

[reamasesa]

Nope, just opening a port for NodeExporter