When creating a service networking connection resource, check if Create or Update is needed

[zli82016]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

After Terraform google provider 5.0, google_service_networking_connection uses the Create endpoint instead of the Patch endpoint during the creation step. If a service networking connection has been already created for the given network, an error is returned from API when creating another service networking connection because only one google_service_networking_connection is allowed for one network.

Error waiting for Create Service Networking Connection: Error code 9, message: Cannot modify allocated ranges in CreateConnection. Please use UpdateConnection.

The improvement can be made that when creating a service networking connection resource, check if Create or Update is needed.

Edit: For the service networking connection removed with removePeering with Terraform provider earlier than 5.0, when recreating the connection with the same network and different reserved ranges with Terraform provider 5.x, the error is returned from API side. But it is empty to get/list the connections.
Error waiting for Create Service Networking Connection: Error code 9, message: Cannot modify allocated ranges in CreateConnection. Please use UpdateConnection.

To handle this case, we consider changing back to use PATCH when creating the resource in Terraform provider 6.0.

New or Affected Resource(s)

• google_service_networking_connection

Potential Terraform Configuration

# Propose what you think the configuration to take advantage of this feature should look like.
# We may not use it verbatim, but it's helpful in understanding your intent.

References

• #0000

b/331798129

[rileykarson]

Note: we should work w/ the service owners to determine how to untangle these resources. The move from upsert/abandon to create/delete in 5.0.0 had more knock-on effects than anticipated.

[Esquire-gh]

@rileykarson Is there any update on how to fix this.

[rh4ll]

Would reverting it to using patch as prior to #15761 be acceptable?

[Esquire-gh]

For now, that is what we're doing. But doing that prevents us from using newer features like datastore scheduled backups.
Knowing if there is a plan to fix it would help us also plan our next steps.

[zli82016]

Would reverting it to using patch as prior to #15761 be acceptable?

Unfortunately, it is a breaking change to revert it and cannot do in the minor release version.

[zli82016]

For now, that is what we're doing. But doing that prevents us from using newer features like datastore scheduled backups. Knowing if there is a plan to fix it would help us also plan our next steps.

@Esquire-gh, can you please provide more details what are you doing now and how does it prevent you from using the new features?

[rh4ll]

Is there any update on this, this is preventing anyone who recreates service networking connections updating the package past the 5.0 migration

[zli82016]

Forward the ticket to the service team.

[muffl0n]

Is there any known workaround? We're just starting to migrate from MySQL to Postgres and this is a hard showstopper for us.

[zli82016]

@muffl0n , only one networking connection is needed between one network and the service producer.

Private connections are a one-to-one relationship between your VPC network and a service producer. If a single service producer offers multiple services, you only need one private connection for all of the producer's services.

https://cloud.google.com/vpc/docs/configure-private-services-access#creating-connection

Once the networking connection exists, you don't need to create another networking connection for the same network.

[muffl0n]

Thank you very much for the clarification! Must have misunderstood the scope of this issue.

[ametad]

Hi, this problem is bugging me also. I hope the check to create or update is coming soon.

This is my Terraform plan (it is redacted to not include personal data):

Terraform will perform the following actions:
  # module.network.google_compute_global_address.internal_ip_block["service-1"] will be created
  + resource "google_compute_global_address" "internal_ip_block" {
      + address            = (known after apply)
      + address_type       = "INTERNAL"
      + creation_timestamp = (known after apply)
      + id                 = (known after apply)
      + name               = "service-1-private-ip-block"
      + network            = "projects/REDACTED/global/networks/staging-version-2020-network"
      + prefix_length      = 16
      + project            = "REDACTED"
      + purpose            = "VPC_PEERING"
      + self_link          = (known after apply)
    }
  # module.network.google_service_networking_connection.private_vpc_connection[0] will be created
  + resource "google_service_networking_connection" "private_vpc_connection" {
      + id                      = (known after apply)
      + network                 = "projects/REDACTED/global/networks/staging-version-2020-network"
      + peering                 = (known after apply)
      + reserved_peering_ranges = [
          + "service-1-private-ip-block",
        ]
      + service                 = "servicenetworking.googleapis.com"
    }
Plan: 2 to add, 0 to change, 0 to destroy.

But I got this error all the time now:

╷
│ Error: Error waiting for Create Service Networking Connection: Error code 9, message: Cannot modify allocated ranges in CreateConnection. Please use UpdateConnection.
│ Help Token: AWUw39Xxnx9W0-wEB9pxcgch0yLzPZiSo6AeSxwAyEFhd_5PFO5yY80GS9R0f8zF1Lsvc-u6yeIQLhaK7-G-ZWfq9APaxjh4s0abPclWI5Op-Yfe
│ 
│   with module.network.google_service_networking_connection.private_vpc_connection[0],
│   on .terraform/modules/network/main.tf line 87, in resource "google_service_networking_connection" "private_vpc_connection":
│   87: resource "google_service_networking_connection" "private_vpc_connection" {

I do not understand exactly what is going wrong, because I thought the private Service Networking Connection is not yet existing in the project. So how can it complain about not been able to create?

There are no Service network connection's if I look in the console:

And I am able to create the one I want Terraform to make, see:

Am I missing something, or do I misunderstand something? I would really like to know, thank you.

[ametad]

From my previous post, I have tried to create a connection manually trough the console. But there I also got a similar error (to my surprise):

[ametad]

For now I've manually fixed the problem by force modify the connection, like so:

gcloud services vpc-peerings update --service=servicenetworking.googleapis.com --ranges=service-1-private-ip-block --network=staging-version-2020-network --project=REDACTED --force

This resulted in a working configuration:

[ametad]

Strange thing is that before the forces manual modification, I tried to list the connections but I got an empty list:

$ gcloud services vpc-peerings list \
>     --network=staging-version-2020-network \
> --project=REDACTED

After the modification it seems alright:

$ gcloud services vpc-peerings list \
>     --network=staging-version-2020-network \
> --project=REDACTED
---
network: projects/REDACTED/global/networks/staging-version-2020-network
peering: servicenetworking-googleapis-com
reservedPeeringRanges:
- service-1-private-ip-block
service: services/servicenetworking.googleapis.com

I am a bit lost here, what was wrong in the first place then? Why did I get an error saying I could not create the resource but I had to use update the existing resource? It seems to me there was no existing resource!

[ametad]

After manually fixing the configuration within the Google cloud console, I had to import this into Terraform. After that all was correct again. See:

$ terraform import module.network.google_service_networking_connection.private_vpc_connection[0] projects/REDACTED/global/networks/staging-version-2020-network:servicenetworking.googleapis.com
data.google_client_config.default: Reading...
module.network.google_service_networking_connection.private_vpc_connection[0]: Importing from ID "projects/REDACTED/global/networks/staging-version-2020-network:servicenetworking.googleapis.com"...
module.network.google_service_networking_connection.private_vpc_connection[0]: Import prepared!
  Prepared google_service_networking_connection for import
module.network.google_service_networking_connection.private_vpc_connection[0]: Refreshing state... [id=projects/REDACTED/global/networks/staging-version-2020-network:servicenetworking.googleapis.com]
data.google_client_config.default: Read complete after 0s [id=projects/"REDACTED"/regions/"europe-REDACTED"/zones/<null>]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

[ametad]

Not totally as expected... After the import, I tried a terraform plan to see if everything is really up-to-date. But Terraform says it is not correct yet, needs to delete and (re)create the connection:

module.network.google_service_networking_connection.private_vpc_connection[0]: Refreshing state... [id=projects/REDACTED/global/networks/staging-version-2020-network:servicenetworking.googleapis.com]
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement
Terraform will perform the following actions:
  # module.network.google_service_networking_connection.private_vpc_connection[0] must be replaced
-/+ resource "google_service_networking_connection" "private_vpc_connection" {
      ~ id                      = "projects/REDACTED/global/networks/staging-version-2020-network:servicenetworking.googleapis.com" -> (known after apply)
      ~ network                 = "projects/REDACTED/global/networks/staging-version-2020-network" -> "projects/REDACTED/global/networks/staging-version-2020-network" # forces replacement
      ~ peering                 = "servicenetworking-googleapis-com" -> (known after apply)
        # (2 unchanged attributes hidden)
    }
Plan: 1 to add, 0 to change, 1 to destroy.

I am going to try to apply this...

YES it works! The connection was first destroyed successfully and after that successfully created!

[zli82016]

Strange thing is that before the forces manual modification, I tried to list the connections but I got an empty list:

$ gcloud services vpc-peerings list \
>     --network=staging-version-2020-network \
> --project=REDACTED

After the modification it seems alright:

$ gcloud services vpc-peerings list \
>     --network=staging-version-2020-network \
> --project=REDACTED
---
network: projects/REDACTED/global/networks/staging-version-2020-network
peering: servicenetworking-googleapis-com
reservedPeeringRanges:
- service-1-private-ip-block
service: services/servicenetworking.googleapis.com

I am a bit lost here, what was wrong in the first place then? Why did I get an error saying I could not create the resource but I had to use update the existing resource? It seems to me there was no existing resource!

Has the private service networking connection with the same network been created and destroyed with terraform? I wonder if that is the reason. There was a bug that the private service networking connection could be abandoned instead of deleted in Terraform provider.

[zli82016]

Is there any update on this, this is preventing anyone who recreates service networking connections updating the package past the 5.0 migration

@rh4ll , only one networking connection is needed between one network and the service producer.
Once the networking connection exists, you don't need to create another networking connection for the same network.

[ametad]

Has the private service networking connection with the same network been created and destroyed with terraform? I wonder if that is the reason. There was a bug that the private service networking connection could be abandoned instead of deleted in Terraform provider.

Hi @zli82016 Thank you for the clarification!

Yes indeed, the resource had been created, deleted, and re-created again. And I think the first creation and deletions was done by the provider version 4.x.

[Esquire-gh]

Is there any update on this, this is preventing anyone who recreates service networking connections updating the package past the 5.0 migration

@rh4ll , only one networking connection is needed between one network and the service producer. Once the networking connection exists, you don't need to create another networking connection for the same network.

@zli82016 We use service networking connection as part of our config to reserve internal ip addresses. An example config from the docs is show below

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name]
}

Everytime we try to update the reserved perring ranges, we run into the issue of Cannot modify allocated ranges in Create...

What do you suggest in this case?

It would be nice if the resource google_service_networking_connection didn't require the field reserved_peering_ranges and also if there was a specific resource for updating the reserved ip ranges for an existing service networking connection.

If these exist and I'm not aware, I would appreciate a link to their docs. Thank you.

[zli82016]

@zli82016 We use service networking connection as part of our config to reserve internal ip addresses. An example config from the docs is show below

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name]
}

Everytime we try to update the reserved perring ranges, we run into the issue of Cannot modify allocated ranges in Create...

What do you suggest in this case?

It would be nice if the resource google_service_networking_connection didn't require the field reserved_peering_ranges and also if there was a specific resource for updating the reserved ip ranges for an existing service networking connection.

If these exist and I'm not aware, I would appreciate a link to their docs. Thank you.

@Esquire-gh, can you run this command to see if the service networking connection already exists?

gcloud services vpc-peerings list --network=xxxx  --project=xxxx

Or has the service networking connection been created and deleted before with provider 4.x?

[Esquire-gh]

@zli82016 Yes, the service networking connection already exists

[zli82016]

list the connections but I got an empty list

@Esquire-gh , just want to confirm that if the following is your use case?
The service networking connection exists with one configuration and you try to update the connection with another configuration.

[Esquire-gh]

@zli82016 Yes, the service networking connection exists in one project (shared network).
We create other projects with resources that need an ip address reserved in the shared network. (memorystore)
To do this, we need to update the reserved ip ranges for the service networking connection. That is where our problem is.

[zli82016]

@zli82016 Yes, the service networking connection exists in one project (shared network). We create other projects with resources that need an ip address reserved in the shared network. (memorystore) To do this, we need to update the reserved ip ranges for the service networking connection. That is where our problem is.

@Esquire-gh , does updating the reserved ip ranges for the service networking connection in the first project (shared network) work?

[Esquire-gh]

@zli82016 Yes, the service networking connection exists in one project (shared network). We create other projects with resources that need an ip address reserved in the shared network. (memorystore) To do this, we need to update the reserved ip ranges for the service networking connection. That is where our problem is.

@Esquire-gh , does updating the reserved ip ranges for the service networking connection in the first project (shared network) work?

@zli82016 No it does not. that is what we are trying to do.

[zli82016]

@zli82016 Yes, the service networking connection exists in one project (shared network). We create other projects with resources that need an ip address reserved in the shared network. (memorystore) To do this, we need to update the reserved ip ranges for the service networking connection. That is where our problem is.

@Esquire-gh , I don't understand the case completely. If other projects need an ip address reserved in the shared network, can you get the ip address with the data source google_compute_global_address? I don't get the reason that you need to update the ip ranges for the service networking connection?

Can you please provide the configurations in the first project (shared network) and the second project (memorystore) and where you are trying to update the connection? Thanks.

[rh4ll]

In my case, the provider has been previously deleted in 4.x version, so what you are saying is it will be impossible to create a service networking connection in a project where it has been deleted in provider <5.0.0?
The amount of breaking changes from the provider in the last year ahs been a real ball ache

[Esquire-gh]

Can you please provide the configurations in the first project (shared network) and the second project (memorystore) and where you are trying to update the connection? Thanks.

@zli82016 Unfortunately the first project was created manually in the console. But in the second project we have a configuration for redis the is similar to this. https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/redis_instance#example-usage---redis-instance-private-service

We use this because we want to use internal ip addresses instead of external ips. We also have to set the authorized_network to the shared network in the first project becuase this redis instances will be accessed by resources in other projects.

But becuase there isn't a data type configuration for google_service_networking_connection we have to use resource which tries to create instead of update.

Manually updating this will be difficult becuase we create several environments in our system al they all have a redis instance on the shared network.

[zli82016]

@rh4ll and @Esquire-gh , I am so sorry for the issue. We plan to make a fix in the provider 6.0.

There is a workaround in provider 5.x. Hope it work for your cases.

1. Modify the configuration of google_service_networking_connection to have empty or old reserved_peering_ranges and then run terraform apply

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [] // or old reserved ranges
}   
 

2. Modify the configuration of google_service_networking_connection to have updated reserved_peering_ranges and then run terraform apply

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name]
}   
 

[rh4ll]

@rh4ll and @Esquire-gh , I am so sorry for the issue. We plan to make a fix in the provider 6.0.

There is a workaround in provider 5.x. Hope it work for your cases.

1. Modify the configuration of google_service_networking_connection to have empty reserved_peering_ranges and then run terraform apply

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = []
}   
 

2. Modify the configuration of google_service_networking_connection to have updated reserved_peering_ranges and then run terraform apply

resource "google_service_networking_connection" "default" {
  network                 = google_compute_network.peering_network.id
  service                 = "servicenetworking.googleapis.com"
  reserved_peering_ranges = [google_compute_global_address.private_ip_alloc.name]
}   
 

Awesome! Thank you for the work around. I'll give that a go tomorrow.

[Esquire-gh]

so sorry for the issue. We plan to make a fix in the provider 6.0.

@zli82016 Thank you!

[zli82016]

For the service networking connection removed with removePeering with Terraform provider earlier than 5.0, when recreating the connection with the same network and different reserved ranges with Terraform provider 5.x, the error is returned from API side. But it is empty to get/list the connections.
Error waiting for Create Service Networking Connection: Error code 9, message: Cannot modify allocated ranges in CreateConnection. Please use UpdateConnection.

To handle this case, we consider changing back to use PATCH when creating the resource in Terraform provider 6.0.

[Esquire-gh]

@zli82016 Thank you. Looking forward to the upgrade.