http_conn_manager_prefix stat tag is not matched accordingly

[sschepens]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

All of our http filter stats do not correctly include the http_conn_manager_prefix tag. Here is an example:

/stats

http.InboundPassthroughClusterIpv4.jwt_authn.allowed: 0
http.InboundPassthroughClusterIpv4.jwt_authn.denied: 0
http.InboundPassthroughClusterIpv4.rbac.allowed: 0
http.InboundPassthroughClusterIpv4.rbac.denied: 0
http.InboundPassthroughClusterIpv4.rbac.shadow_allowed: 0
http.InboundPassthroughClusterIpv4.rbac.shadow_denied: 0
http.inbound_10.38.36.249_8080.jwt_authn.allowed: 11486
http.inbound_10.38.36.249_8080.jwt_authn.denied: 0
http.inbound_10.38.36.249_8080.rbac.allowed: 11486
http.inbound_10.38.36.249_8080.rbac.denied: 0
http.inbound_10.38.36.249_8080.rbac.shadow_allowed: 11486
http.inbound_10.38.36.249_8080.rbac.shadow_denied: 0
http.inbound_10.38.36.249_8443.jwt_authn.allowed: 154936
http.inbound_10.38.36.249_8443.jwt_authn.denied: 0
http.inbound_10.38.36.249_8443.rbac.allowed: 154936
http.inbound_10.38.36.249_8443.rbac.denied: 0
http.inbound_10.38.36.249_8443.rbac.shadow_allowed: 154936
http.inbound_10.38.36.249_8443.rbac.shadow_denied: 0

/stats/prometheus

envoy_http_InboundPassthroughClusterIpv4_jwt_authn_allowed{} 0
envoy_http_InboundPassthroughClusterIpv4_jwt_authn_denied{} 0
envoy_http_InboundPassthroughClusterIpv4_rbac{authz_enforce_result="allowed"} 0
envoy_http_InboundPassthroughClusterIpv4_rbac{authz_enforce_result="denied"} 0
envoy_http_InboundPassthroughClusterIpv4_rbac{authz_dry_run_result="allowed"} 0
envoy_http_InboundPassthroughClusterIpv4_rbac{authz_dry_run_result="denied"} 0
envoy_http_inbound_10_38_36_249_8080_jwt_authn_allowed{} 11618
envoy_http_inbound_10_38_36_249_8080_jwt_authn_denied{} 0
envoy_http_inbound_10_38_36_249_8080_rbac{authz_enforce_result="allowed"} 11618
envoy_http_inbound_10_38_36_249_8080_rbac{authz_enforce_result="denied"} 0
envoy_http_inbound_10_38_36_249_8080_rbac{authz_dry_run_result="allowed"} 11618
envoy_http_inbound_10_38_36_249_8080_rbac{authz_dry_run_result="denied"} 0
envoy_http_inbound_10_38_36_249_8443_jwt_authn_allowed{} 156829
envoy_http_inbound_10_38_36_249_8443_jwt_authn_denied{} 0
envoy_http_inbound_10_38_36_249_8443_rbac{authz_enforce_result="allowed"} 156829
envoy_http_inbound_10_38_36_249_8443_rbac{authz_enforce_result="denied"} 0
envoy_http_inbound_10_38_36_249_8443_rbac{authz_dry_run_result="allowed"} 156829
envoy_http_inbound_10_38_36_249_8443_rbac{authz_dry_run_result="denied"} 0

Theoretically InboundPassthroughClusterIpv4 and inbound_10.38.36.249_8080 should be extracted as a tag http_conn_manager_prefix which is included in istio bootstrap templates

We were using BDS to fix this which was deprecated in #48693 and now are left with only tweaking istio bootstrap templates which we were hoping to avoid.

Now this seems like a bug in Istio because the tag extracion regex is defined but does not match values generated by istio itself. Maybe some time ago Istio changed the format used for generating http connection manager names and never adjusted the tag regex accordingly?

FYI, we're using Sidecar CRD to define ingress ports.

Version

1.21.2

Additional Information

No response

[howardjohn]

@keithmattix @kyessenov @zirain is there not some way to have envoy do structured stats without performing a regex against a flat string? I think I have asked this before and was told "no", but it just seems kind of hard to accept?

[sschepens]

@keithmattix @kyessenov @zirain is there not some way to have envoy do structured stats without performing a regex against a flat string? I think I have asked this before and was told "no", but it just seems kind of hard to accept?

There are some ways to generate structured stats but internally in envoy and I think they're note used for most stuff, only used for some filters.

Another way to look at this is probably do something like we do with Istio custom stats. Istio custom stats generate tag keys and values with a specific pattern which is then easy to match in regexes. Maybe we can do the same for stat_prefixes to ensure we can always extract those tags

[keithmattix]

Yeah I think stats is only through regex in Envoy

[kyessenov]

Envoy stats pre-date labels so labels can only be added in post-processing.

Yes, I agree we should default to something sane for regexes. Not sure how it ended up in this state. We should set "stat_prefix" to something that's regex friendly or elide superficial details, like port numbers, to avoid a cardinality explosion.