Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

serviceentry / virtual service not work when target to outside server. #50997

Open
2 tasks done
johnzheng1975 opened this issue May 11, 2024 · 25 comments
Open
2 tasks done

serviceentry / virtual service not work when target to outside server. #50997

johnzheng1975 opened this issue May 11, 2024 · 25 comments
Assignees
@hzxuzhonghu
Labels
area/networking

Comments

@johnzheng1975
Copy link
Member

johnzheng1975 commented May 11, 2024 •
edited

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

I need let ingress gateway as a proxy, for outside domain. It works for http://httpbin.org, but cannot work for https://.
(mtls is set strict in cluster)

Happy case

# configure
kubectl replace --force -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: httpbin-ext
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin-onecloudsol-dev.api.dev-sol-us.company.io/ip
{
  "origin": "192.x.x.x,10.212.186.101, 54.148.123.253"
}

# clean up
k delete se httpbin-ext
k delete vs  httpbin-ext

Fail case 1 for https:

# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
 
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>

# clean up all
k delete se httpbin
k delete vs httpbin-ext

Fail case 2 for https:

# configure
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
spec:
  gateways:
  - istio-system/apigee-gateway
  hosts:
  - httpbin443-onecloudsol-dev.api.dev-sol-us.company.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF

# Test
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

# delete DestinationRule and test again
$ k delete dr originate-tls
destinationrule.networking.istio.io "originate-tls" deleted
 
$ curl https://httpbin443-onecloudsol-dev.api.dev-sol-us.company.io/ip
<html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
</body>
</html>

# clean up all
k delete se httpbin
k delete vs httpbin-ext

Version

$ istio verison 1.20
$ kubernetes version: 1.27

Additional Information

mtls is enabled with strict
@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 11, 2024 •
edited

Error log:
{
"authority": "httpbin443-onecloudsol-dev.api.sandbox-uw2.company.io",
"bytes_received": 0,
"bytes_sent": 253,
"connection_termination_details": null,
"downstream_local_address": "10.202.155.69:443",
"downstream_remote_address": "10.202.168.106:28283",
"duration": 430,
"grpc_status_number": null,
"method": "GET",
"oc_basic_client_id": "nil",
"oc_internal_act_sa": "nil",
"oc_internal_tenants": "[]",
"oc_path_client_id": "nil",
"oc_token_client_id": "nil",
"oc_token_sa": "nil",
"path": "/ip",
"protocol": "HTTP/1.1",
"request_id": "e5a9a6d8-5d17-9d34-9438-e380c12caf68",
"requested_server_name": null,
"response_code": 503,
"response_code_details": "upstream_reset_before_response_started{remote_connection_failure,TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end}",
"response_flags": "UF,URX",
"route_name": null,
"start_time": "2024-05-11T15:17:25.048Z",
"upstream_cluster": "outbound|443||httpbin.org",
"upstream_host": "3.233.6.75:443",
"upstream_local_address": null,
"upstream_service_time": null,
"upstream_transport_failure_reason": "TLS_error:|268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end",
"user_agent": "curl/7.68.0",
"x_b3_traceid": "c07d33673d93a6fe29516eb2fed3948b",
"x_forwarded_for": "192.56.99.15,10.202.168.106"
}

@johnzheng1975 johnzheng1975 mentioned this issue May 11, 2024
Closed
@hzxuzhonghu
Copy link
Member

what is the gateway created and xds configuration generated?

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 13, 2024 •
edited

Hi @hzxuzhonghu
Here is my gateway configure, is this the root reason caused fail?

k get gateway -n istio-system   apigee-gateway  -oyaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: istio-gateways
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: apigee-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway-apigee
  servers:
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingress-tls
      mode: SIMPLE

Attached is the config_dump for fail 1 case.
config_dump.zip

@johnzheng1975
Copy link
Member Author

Here is log when api is invoked:
$ curl https://httpbin443-onecloudsol-dev.web.dev-sol-us.company.io/ip?id=11111111111111

...
{"level":"debug","time":"2024-05-13T03:47:37.761400Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"824\",\"StreamId\":\"17754883692509463399\"] cluster 'outbound|80||httpbin.org' match for URL '/ip?id=11111111111111'","caller":"external/envoy/source/common/router/router.cc:514","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761486Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"824\",\"StreamId\":\"17754883692509463399\"] router decoding headers:\n':authority', 'httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io'\n':path', '/ip?id=11111111111111'\n':method', 'GET'\n':scheme', 'https'\n'x-forwarded-for', '15.65.204.33,10.212.152.57'\n'x-forwarded-proto', 'https'\n'x-forwarded-port', '443'\n'x-amzn-trace-id', 'Root=1-66418d59-78b1fc6150486f954300eb51'\n'user-agent', 'curl/7.35.0'\n'accept', '*/*'\n'x-envoy-external-address', '10.212.152.57'\n'x-request-id', '562280e1-0a6d-9740-a774-ef6f847b5577'\n'x-envoy-decorator-operation', 'httpbin.org:80/*'\n'x-envoy-peer-metadata-id', 'router~10.212.127.217~istio-ingressgateway-web-56d5b6d676-klp75.istio-system~istio-system.svc.cluster.local'\n'x-envoy-peer-metadata', 'ChQKDkFQUF9DT05UQUlORVJTEgIaAAoaCgpDTFVTVEVSX0lEEgwaCkt1YmVybmV0ZXMKIAoMSU5TVEFOQ0VfSVBTEhAaDjEwLjIxMi4xMjcuMjE3ChkKDUlTVElPX1ZFUlNJT04SCBoGMS4yMS4wCuMBCgZMQUJFTFMS2AEq1QEKIQoDYXBwEhoaGGlzdGlvLWluZ3Jlc3NnYXRld2F5LXdlYgodCgVpc3RpbxIUGhJpbmdyZXNzZ2F0ZXdheS13ZWIKPQofc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtbmFtZRIaGhhpc3Rpby1pbmdyZXNzZ2F0ZXdheS13ZWIKLwojc2VydmljZS5pc3Rpby5pby9jYW5vbmljYWwtcmV2aXNpb24SCBoGbGF0ZXN0CiEKF3NpZGVjYXIuaXN0aW8uaW8vaW5qZWN0EgYaBHRydWUKGgoHTUVTSF9JRBIPGg1jbHVzdGVyLmxvY2FsCjMKBE5BTUUSKxopaXN0aW8taW5ncmVzc2dhdGV3YXktd2ViLTU2ZDViNmQ2NzYta2xwNzUKGwoJTkFNRVNQQUNFEg4aDGlzdGlvLXN5c3RlbQphCgVPV05FUhJYGlZrdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvaXN0aW8tc3lzdGVtL2RlcGxveW1lbnRzL2lzdGlvLWluZ3Jlc3NnYXRld2F5LXdlYgqDAQoRUExBVEZPUk1fTUVUQURBVEESbipsCiUKFWF3c19hdmFpbGFiaWxpdHlfem9uZRIMGgp1cy13ZXN0LTJhCigKD2F3c19pbnN0YW5jZV9pZBIVGhNpLTA1OTE2ZGI0NWVhODk5MDUwChkKCmF3c19yZWdpb24SCxoJdXMtd2VzdC0yCisKDVdPUktMT0FEX05BTUUSGhoYaXN0aW8taW5ncmVzc2dhdGV3YXktd2Vi'\n'ratelimit-service', 'httpbin443-onecloudsol-dev'\n'ratelimit-source-ip', '15.65.204.33'\n'ratelimit-enabled-secure', 'true'\n'test-cloud-tenant', 'myheadervalue'\n'internal_act_sa', 'nil'\n'oc-token-sa', 'nil'\n'oc-token-client-id', 'nil'\n'internal_tenants', '[]'\n'oc-path-client-id', 'nil'\n'oc-basic-client-id', 'nil'\n'x-envoy-expected-rq-timeout-ms', '3000'\n'x-envoy-attempt-count', '1'\n","caller":"external/envoy/source/common/router/router.cc:731","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761531Z","scope":"envoy pool","msg":"queueing stream due to no available connections (ready=0 busy=0 connecting=0)","caller":"external/envoy/source/common/http/conn_pool_base.cc:78","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761537Z","scope":"envoy pool","msg":"trying to create new connection","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:291","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761542Z","scope":"envoy pool","msg":"creating a new connection (connecting=0)","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:145","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761597Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] current connecting state: true","caller":"external/envoy/source/common/network/connection_impl.h:98","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761608Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"825\"] connecting","caller":"external/envoy/source/common/http/codec_client.cc:57","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761614Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connecting to 54.160.164.209:443","caller":"external/envoy/source/common/network/connection_impl.cc:1009","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761714Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connection in progress","caller":"external/envoy/source/common/network/connection_impl.cc:1028","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.824986Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connected","caller":"external/envoy/source/common/network/connection_impl.cc:746","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891499Z","scope":"envoy connection","msg":"verify cert failed: verify SAN list","caller":"external/envoy/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc:237","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891568Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] remote address:54.160.164.209:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:241","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891581Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] closing socket: 0","caller":"external/envoy/source/common/network/connection_impl.cc:278","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891611Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] remote address:54.160.164.209:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:241","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891664Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"825\"] disconnect. resetting 0 pending requests","caller":"external/envoy/source/common/http/codec_client.cc:107","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891679Z","scope":"envoy pool","msg":"[Tags: \"ConnectionId\":\"825\"] client disconnected, failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:495","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891700Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"824\",\"StreamId\":\"17754883692509463399\"] upstream reset: reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end","caller":"external/envoy/source/common/router/router.cc:1332","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891729Z","scope":"envoy pool","msg":"invoking 1 idle callback(s) - is_draining_for_deletion_=false","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:463","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.894883Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"824\",\"StreamId\":\"17754883692509463399\"] performing retry","caller":"external/envoy/source/common/router/router.cc:1955","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.894942Z","scope":"envoy pool","msg":"queueing stream due to no available connections (ready=0 busy=0 connecting=0)","caller":"external/envoy/source/common/http/conn_pool_base.cc:78","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.894950Z","scope":"envoy pool","msg":"trying to create new connection","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:291","thread":24}

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 13, 2024 •
edited 

# From here, seems it try to create connection to "outbound|80||httpbin.org"
{"level":"debug","time":"2024-05-13T03:47:37.761400Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"824\",\"StreamId\":\"17754883692509463399\"] cluster 'outbound|80||httpbin.org' match for URL '/ip?id=11111111111111'","caller":"external/envoy/source/common/router/router.cc:514","thread":24}
 
# From here, seems it connecting to 54.160.164.209 (IP of httpbin.org)
{"level":"debug","time":"2024-05-13T03:47:37.761537Z","scope":"envoy pool","msg":"trying to create new connection","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:291","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761542Z","scope":"envoy pool","msg":"creating a new connection (connecting=0)","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:145","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761597Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] current connecting state: true","caller":"external/envoy/source/common/network/connection_impl.h:98","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761608Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"825\"] connecting","caller":"external/envoy/source/common/http/codec_client.cc:57","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.761614Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connecting to 54.160.164.209:443","caller":"external/envoy/source/common/network/connection_impl.cc:1009","thread":24}

# From here, seems it `verify cert failed:`
{"level":"debug","time":"2024-05-13T03:47:37.761714Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connection in progress","caller":"external/envoy/source/common/network/connection_impl.cc:1028","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.824986Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] connected","caller":"external/envoy/source/common/network/connection_impl.cc:746","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891499Z","scope":"envoy connection","msg":"verify cert failed: verify SAN list","caller":"external/envoy/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc:237","thread":24}
{"level":"debug","time":"2024-05-13T03:47:37.891568Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"825\"] remote address:54.160.164.209:443,TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:241","thread":24}

Does that means ingressgateway try to connect "outbound|80||httpbin.org" whose `server ip` is `54.160.164.209` and `cert subject alternative name` is `*.httpbin.org`, then think server cert is incorrect? Thanks.

@hzxuzhonghu
Copy link
Member

i can not even searched outbound|80||httpbin.org in yout your configdum, and not even a dynamic listeners there?

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 13, 2024 •
edited

@hzxuzhonghu , I cannot find also previously. Maybe I got from wrong pod, or others... Sorry for the confusion.

This time I put everything in istio-system, and test with another ingress gateway. as below.
You can find the outbound|80||httpbin.org

# Configure as below:
kubectl create -f - <<EOF
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
  namespace: istio-system
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
  namespace: istio-system
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
  namespace: istio-system
spec:
  gateways:
  - istio-system/web-gateway
  hosts:
  - httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io
  http:
  - timeout: 3s
    route:
    - destination:
        host: httpbin.org
      weight: 100
EOF


# Test:
$ curl https://httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

# restart all pods in istio-system
$ k rollout restart deploy -n istio-system
deployment.apps/ingressgateway restarted
deployment.apps/istio-ingressgateway-apigee restarted
deployment.apps/istio-ingressgateway-private restarted
deployment.apps/istio-ingressgateway-public restarted
deployment.apps/istio-ingressgateway-public-grpc restarted
deployment.apps/istio-ingressgateway-web restarted
deployment.apps/istiod restarted


~$ k get dr -A
NAMESPACE      NAME                            HOST                                                      AGE
istio-system   originate-tls                   httpbin.org                                               2m48s
... ...

$ k get se -A
NAMESPACE      NAME                  HOSTS                                                         LOCATION        RESOLUTION   AGE
istio-system   httpbin               ["httpbin.org"]                                               MESH_EXTERNAL   DNS          3m18s
... ...

(dev-sol-uw2-blue):  oc-dev-blue@PS0IMMjohn:~$ k get vs -n is
NAME               GATEWAYS                          HOSTS                                                                                                                  AGE
httpbin-ext        ["istio-system/web-gateway"]      ["httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io"]                                                            3m39s
... ...

# Test again
$ curl https://httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io/ip
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end(dev-sol-uw2-blue): 

# Get config_dump
$ k exec -ti  istio-ingressgateway-web-675599fb67-m7mnw  -n istio-system -c istio-proxy  -- curl localhost:15000/config_dump  > new_web_confg_dump.yaml

new_web_confg_dump.zip

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 13, 2024 •
edited

Here is test and full log for ingress gateway container, include container start and error during api call. FYI

$ curl https://httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io/ip?id=33333333333333333333333
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure, transport failure reason: TLS_error:|268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED:TLS_error_end:TLS_error_end

test.zip

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 14, 2024 •
edited

Seems ingressgateway try to connect "outbound|80||httpbin.org" whose cert subject alternative name is *.httpbin.org, then think server cert is incorrect?
Thanks.

Note: SAN of httpbin.org is *.httpbin.org
image

@johnzheng1975
Copy link
Member Author

Note that I have not installed Egress gateway, is this reason caused the issue? Thanks.

@hzxuzhonghu
Copy link
Member

Egress gateway is not a must， actually ingress gateway Can act as egress gateway too if you want.

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 14, 2024 •
edited

Thanks very much. @hzxuzhonghu .
Do you know why this issue happens?
Does that means Istio do not support such usage? Is there any workaround in case Istio do not support this case?

Here is background for my try:
I want to use istio as an api gateway, not only proxy services inside cluster, but also proxy external server. Then add jwt verify(ra/ authorationpolicy), ratelimit, whitelist, CORS and so on... This cannot be achieved until this issue is resolved.

Appreciate your support.

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 16, 2024 •
edited

@howardjohn Can you also help to take a look, thanks a lot.

@johnzheng1975 johnzheng1975 mentioned this issue May 16, 2024
Closed
@hzxuzhonghu
Copy link
Member

I doubt if this is caused by VERIFY_CERTIFICATE_AT_CLIENT default to true now. Or try set InsecureSkipVerify = false

@johnzheng1975
Copy link
Member Author

Thanks so much! @howardjohn
Let me try :-)

@johnzheng1975
Copy link
Member Author

@hzxuzhonghu , you are completely right.
with insecureSkipVerify set to true, it works!

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
  namespace: istio-system
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
      insecureSkipVerify: true

$ curl https://httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io/ip
{
"origin": "192.56.99.15,10.212.106.162, 34.208.140.168"
}

@howardjohn
Copy link
Member

are you really using httpbin in your tests or something else? I don't see why httpbin would fail to verify..

@johnzheng1975
Copy link
Member Author

Yes, I am using httpbin, actually the code is exact as #50997 (comment)
I even do not change my company domain :-)

@johnzheng1975
Copy link
Member Author

johnzheng1975 commented May 17, 2024 •
edited

Thanks so much @hzxuzhonghu @howardjohn
The final issue is:
For production level solution, do we have a better way to resolve this instead of "insecureSkipVerify: true". (I am not sure company's cybersecurity will challenge me)
Like, change "outbound|80||httpbin.org" to "httpbin.org" before sent out to server.
Can I change this in envoyfilter? we have sidecar_outbound type which may useful for sidecar. but no gateway_outbound. Can I change it in gateway?
image

Or, we can improve in istio code directly? like, add a flag similar with insecureSkipVerify, called "removeOutbound" in destinationrule. Then change the "outbound|80||httpbin.org" to "httpbin.org"

Appreciate your comments.

@hzxuzhonghu
Copy link
Member

Previoulsy we do not verify, this is now changed by VERIFY_CERTIFICATE_AT_CLIENT, you can tune it to false, i believe it can also work foryou, so you donot need a envoyfilter to do that

@johnzheng1975
Copy link
Member Author

Thanks. Close this issue.

@howardjohn
Copy link
Member

Full reproducer:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin
  namespace: istio-system
spec:
  hosts:
  - httpbin.org
  ports:
  - number: 80
    name: http
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: originate-tls
  namespace: istio-system
spec:
  host: httpbin.org
  trafficPolicy:
    tls:
      mode: SIMPLE
      sni: httpbin.org # DOES FIX IT
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-ext
  namespace: istio-system
spec:
  gateways:
  - istio-system/web-gateway
  hosts:
  - httpbin443-onecloudsol-dev.web.dev-sol-us.hponecloud.io
  http:
  - timeout: 3s
    rewrite:
      authority: httpbin.org # DOES NOT FIX IT
    route:
    - destination:
        host: httpbin.org
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: web-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---

A few questions here...

  • Do we really need auto_sni when we have a FQDN DR? Why not just hardcode sni to httpbin.org here?
  • Does rewrite.authority in route not impact auto_sni?

cc @kfaseela @jacob-delgado

@howardjohn howardjohn reopened this May 23, 2024
@kfaseela
Copy link
Member

@leosarra could you please take a look at this?

@leosarra
Copy link
Contributor

Sure I will

@MengjiaLiang
Copy link

MengjiaLiang commented May 24, 2024 •
edited

@johnzheng1975 Could you please try with what John Howard mentioned above to have the hardcoded SNI in Destination�Rule? If that works, you don't need turn on insecureSkipVerify

My team is doing exactly what you are trying to achieve and we have hardcoded SNI in all external endpoints' destination rule. It works well from Istio 1.15 to Istio 1.19. We didn't maintain any clusters with Istio >= 1.20 yet though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hzxuzhonghu hzxuzhonghu

Labels
area/networking
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@howardjohn @kfaseela @leosarra @hzxuzhonghu @johnzheng1975 @istio-policy-bot @MengjiaLiang