(aws-eks): Construct Library custom resources doesn't use proxy properly

[oleksii-boiko-ua]

I'm trying to test new feature related to provision within the VPC lambda functions related to EKS configuration like ClusterHandler. Lambdas are placed to vpc and it's great. But I got error connecting to EKS api via proxy

Reproduction Steps

self.cluster = aws_eks.Cluster(
            scope=self,
            id='cluster',
            cluster_name="cluster-" + environment,
            endpoint_access=aws_eks.EndpointAccess.PUBLIC_AND_PRIVATE,
            default_capacity=0,
            vpc=vpc,
            vpc_subnets=[aws_ec2.SubnetSelection(subnets=[subnet_a_eks, subnet_b_eks, subnet_c_eks])],
            # issue with 3 subnet
            place_cluster_handler_in_vpc=True,
            version=cluster_version,
            cluster_handler_environment={
                "http_proxy": "http://login:pass@proxy.cloud.local:8080/"
            },
            kubectl_environment={
                "http_proxy": "http://login:pass@proxy.cloud.local:8080/"
            },
            security_group=eks_control_plane_sg,
            role=eks_control_plane_role,
        )

What did you expect to happen?

successful cluster creation

What actually happened?

Cloudwatch log of ProviderframeworkonEvent. function:
2021-01-09T14:37:27.905Z e785fa78-c5f8-471c-a495-59d75389a6c6 INFO [provider-framework] submit response to cloudformation { "Status": "FAILED", "Reason": "Error: connect ETIMEDOUT 63.32.73.253:443\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)", "StackId": "arn:aws:cloudformation:eu-west-1:accountid:stack/eks-stack-develop-cdk/276db000-5285-11eb-ab35-0615947f7f49", "RequestId": "ce8a03d7-fdf3-4def-acb4-6219fb352732", "PhysicalResourceId": "AWSCDK::CustomResourceProviderFramework::CREATE_FAILED", "LogicalResourceId": "clusterC5B25D0D" }

Proxy is fine i tested it against same endpoint ( strange that it calls ec2 and nor eks api

[ec2-user@ip-10-60-233-255 ~]$ curl -vk https://ec2-63-32-73-253.eu-west-1.compute.amazonaws.com.
* Rebuilt URL to: https://ec2-63-32-73-253.eu-west-1.compute.amazonaws.com./
* Uses proxy env variable https_proxy == 'http://user:password@proxy.cloud.local:8080/'
*   Trying 10.60.249.170...
* TCP_NODELAY set
* Connected to proxy.cloud.local (10.60.249.170) port 8080 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443
* Proxy auth using Basic with user 'user'
> CONNECT ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443 HTTP/1.1
> Host: ec2-63-32-73-253.eu-west-1.compute.amazonaws.com:443
> Proxy-Authorization: Basic token
> User-Agent: curl/7.61.1
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=lambda.eu-west-1.amazonaws.com
*  start date: Dec 23 00:00:00 2020 GMT
*  expire date: Jan 21 23:59:59 2022 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x17f8190)
> GET / HTTP/2
> Host: ec2-63-32-73-253.eu-west-1.compute.amazonaws.com
> User-Agent: curl/7.61.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 403 
< date: Tue, 12 Jan 2021 13:31:59 GMT
< content-length: 127
< x-amzn-requestid: c6c6badc-56de-4e6a-8266-0d7971505c84
< 
<MissingAuthenticationTokenException>
  <Message>Missing Authentication Token</Message>
</MissingAuthenticationTokenException>
* Connection #0 to host proxy.cloud.local left intact

Environment

• **CDK CLI Version :1.83
• Framework Version:
• **Node.js Version: v14.13.0
• **OS :macOS
• **Language (Version): Python (3.9)

Other

I noticed that 5 lambda functions are created but only 1 of them "OnEventHandler" receives proxy configuration, but looks like it only one which interacts with api

---

This is 🐛 Bug Report

[oleksii-boiko-ua]

@iliapolo could you take a look when have time please, could it be related to tls check, can we somehow disable it?

[iliapolo]

@alexey-boyko

I noticed that 5 lambda functions are created but only 1 of them "OnEventHandler" receives proxy configuration, but looks like it only one which interacts with api

Yes, all functions should be connected to the VPC but only the one interacting with the EKS API receives the cluster_handler_environment. This is the intended behavior.

could it be related to tls check, can we somehow disable it?

Not sure what you mean by that, disable what?

Are you able to create an EKS cluster via the proxy using the SDK? we use the createCluster API call, I think this would be the ultimate test to isolate the problem, i'm not sure yet this is a CDK issue.

[oleksii-boiko-ua]

thanks i will try it right now to test via api call, but i noticed that call is made from ProviderframeworkonEvent lambda function, which doesn't have proxy set during deploy, only OnEventHandler lambda function

[iliapolo]

@alexey-boyko The ProviderframeworkonEvent only invokes the OnEventHandler and responds to CFN with the status. You should look at the CloudWatch of the OnEventHandler function.

[oleksii-boiko-ua]

@iliapolo unfortunately i can't make it work( i tried to create lambda function in vpc with code below, but anyway i got same timeout as using OnEventHandler Task timed out after 3.00 seconds. I saw many issue about aws-sdk-javascript and proxy( people experiencing the same) looks like it bypass proxy. But when i use python3 and boto3 i can successfully create or list cluster

const AWS = require('aws-sdk')
// Set the region 
AWS.config.update({region: 'eu-west-1'});
const eks = new AWS.EKS()

var proxy = require('proxy-agent');
AWS.config.update({
  httpOptions: { agent: proxy('http://login:password@proxy.cloud.local:8080/') }
});

exports.handler = async function(event) {
    var params = { 
      maxResults: '5',
    }

    console.log("Checkpoint 1");

    let cluster

    try {
      cluster = await eks.listClusters(params).promise();
      console.log(cluster)
    } catch (e) {
      console.log(e)
    }

    console.log("Checkpoint 2");
    
    return {
        statusCode: 200,
        body: JSON.stringify(cluster || {message: 'Nothing'})
    }
}

[oleksii-boiko-ua]

anything i can try, can i increase log level?, also i wonder why timeout to ec2-63-32-73-253.eu-west-1.compute.amazonaws.com. not eks.eu-west-1.compute.amazonaws.com, and in cloudwatch only logs from ProviderframeworkonEven

[iliapolo]

@alexey-boyko The ProviderframeworkonEvent function invokes the OnEventHandler, if you only see logs from ProviderframeworkonEvent function, it means that the framework couldn't invoke the function.

Do the subnets you are using have outgoing internet access? If not, you will need to configure VPC endpoints.

See #12171 for more details. This will allow the function calling EKS api's to be invoked, from that point on your proxy configuration should be applied.

As for the proxy configuration itself, I'm afraid I don't have any insight on this...I suggest we take it step by step by first making sure that the OnEventHandler function actually gets invoked.

Thanks

[oleksii-boiko-ua]

hey @iliapolo thanks a lot for answer, "Do the subnets you are using have outgoing internet access? If not, you will need to configure VPC endpoints." we don't have internet, we use proxy for accessing aws api's which don't have vpc endpoints, for rest we use vpc endpoints. I have several lambda function which use proxy to call eks api, but they use python3 and boto3,
I don't know what I can test more to debug this issue((( i can't even make it work with lambda function above( Did someone test it via proxy? also why step function logs are disabled by default? anyway maybe i need to investigate more. thanks

[iliapolo]

for rest we use vpc endpoints

Makes sense, so just make sure you configure all the necessary endpoints as mentioned here.

[oleksii-boiko-ua]

@iliapolo i was missing lambda and step function endpoints, now OnEventHandler lambda gets invoked but anyway i receive not verbose error(( so it hard to understand what is happening
OnEventHandler lambda log:

2021-01-18T16:32:48.849Z	cbee5717-0921-4d2c-acbc-08b5e857e3d0	INFO	onCreate: creating cluster with options: 
{
    "resourcesVpcConfig": {
        "endpointPrivateAccess": true,
        "securityGroupIds": [
            "sg-0fccfa9d6ds9"
        ],
        "endpointPublicAccess": true,
        "subnetIds": [
            "subnet-086ds54e9073",
            "subnet-0d3c5as3c0869f91"
        ]
    },
    "roleArn": "arn:aws:iam::accountId:role/test-develop-cdk-control-plane",
    "name": "cluster-develop-cdk",
    "version": "1.18"
}

2021-01-18T16:33:48.795Z cbee5717-0921-4d2c-acbc-08b5e857e3d0 Task timed out after 60.06 seconds

Copy
2021-01-18T16:33:48.795Z cbee5717-0921-4d2c-acbc-08b5e857e3d0 Task timed out after 60.06 seconds

ProviderframeworkonEvent log:

2021-01-18T16:33:48.817Z 71f60ed5-face-4ef4-842a-8f02e491d969 INFO [provider-framework] user function response: { "StatusCode": 200, "FunctionError": "Unhandled", "ExecutedVersion": "$LATEST", "Payload": "{\"errorMessage\":\"2021-01-18T16:33:48.795Z cbee5717-0921-4d2c-acbc-08b5e857e3d0 Task timed out after 60.06 seconds\"}" } object

Copy
2021-01-18T16:33:48.817Z	71f60ed5-face-4ef4-842a-8f02e491d969	INFO	[provider-framework] user function response: 
{
    "StatusCode": 200,
    "FunctionError": "Unhandled",
    "ExecutedVersion": "$LATEST",
    "Payload": "{\"errorMessage\":\"2021-01-18T16:33:48.795Z cbee5717-0921-4d2c-acbc-08b5e857e3d0 Task timed out after 60.06 seconds\"}"
}

It happens anyway even if i add proxy variable or not

[iliapolo]

@alexey-boyko

It happens anyway even if i add proxy variable or not

I guess that makes in case the proxy isn't being applied correctly. I did a little digging and it seems that unlike python, nodejs doesn't use any global env variables to configures proxy passes. Looks like every http client implements this independently.

We might need to do add explicit support for this in the library. I'll try adding some more information soon.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.