http function with egress through vpc-connector does not egress traffic to internet when called from external IP

[davidrdm]

What I’m trying to do:

• Configure a VPC connector, serverless vpc access, firewall rules, and any other necessary infrastructure, so that I can call an http function from whitelisted ip addresses including those from outside of GCP, but ensure that egress to port 443 (https) for all ip's instances is blocked. (The function requires https.)
• I am agnostic as to whether other non-whitelisted IPs inside my VPC are blocked.

What’s happening:

• After configuring my vpc connector and creating firewall rules, the http function times out without any response received by the instance calling the function, regardless of whether that instance is inside or outside of the Google network, including the instances whose IP I have attempted to whitelist.
• When I check the Stackdriver logs for the subset associated with the vpc-connector, the DENY actions all list a dest_ip that is controlled by Google, which I verified by running whois on every dest_ip logged. The external ip I'm using to call call the function appears nowhere in the subnet logs.

What I have done:

• Create a new vpc-connector on the default network in us-central1 with subnet 10.8.0.0/28
  Followed all instructions, granted all of the roles and permissions, and created all of the firewall rules, described at https://cloud.google.com/vpc/docs/configure-serverless-vpc-access
• Added the following additional firewall rules:
  Deny egress to TCP and UDP port 443, all destinations, priority 65535 (lowest possible), target tag: vpc-connector
  Allow egress to TCP and UDP port 443, destination IP ranges: 107.178.230.64/26, 35.199.224.0/19, and an external IP that I control, priority 900, target tag: vpc-connector
• Configured my http function to route all egress traffic through the vpc-connector (the function is also in us-central1), and to allow all ingress
• Confirmed that a route exist from the default network to 0.0.0.0/0 with next hop set to default internet gateway
• Confirmed that the function succeeds from my whitelisted external IP and from the Google Cloud Shell when the rule used to block all egress to TCP and UDP ports 443 from the vpc-connector is disabled
• Confirmed that IAP proxy is not enabled on my GCP project

What I expected to happen:

• I would have thought that calls to the function made from the external IP would still succeed after enabling the firewall rule to block all agreed to TCP and UDP port 443 from the vpc-connector. It’s a lower priority than the rule to allow egress from the whitelisted IP. Instead it times out, as do calls made from the Google Cloud Shell. (I am not surprised that the Cloud Shell failed, since its IP block wasn’t whitelisted).
• I also tried adding a 2nd route from the default network to 0.0.0.0/0 with next hop set to default internet gateway, this time adding instance tag: vpc-connector, but this did not make a difference

When my external ip calls the function, why is egress traffic from the vpc-connector try to go to a different ip on the Google network (and getting blocked) rather than directly back to my external ip?

More generally, can someone explain to me what I’m doing wrong, and what the right way is to block egress to the internet of my http function except for whitelisted ip addresses? (I intend this as a backup security mechanism, as I am also checking the hash of a secret that is passed with the HTTP GET call and returning an error if it doesn’t match.)