Unable to increase number_of_brokers_nodes without rebuilding cluster.

[ryan-dyer-sp]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.20

Affected Resource(s)

aws_msk_cluster

Terraform Configuration Files

number_of_broker_nodes = 3 -> 6

Debug Output

  # aws_msk_cluster.cluster must be replaced
-/+ resource "aws_msk_cluster" "cluster" {
      ~ arn                      = "REDACTED" -> (known after apply)
      ~ bootstrap_brokers        = "REDACTED" -> (known after apply)
      ~ bootstrap_brokers_tls    = "REDACTED" -> (known after apply)
        cluster_name             = "develop-us-east-1-admin"
      ~ current_version          = "K13V1IB3VIYZZH" -> (known after apply)
        enhanced_monitoring      = "PER_TOPIC_PER_BROKER"
      ~ id                       = "REDACTED" -> (known after apply)
        kafka_version            = "2.2.1"
      ~ number_of_broker_nodes   = 3 -> 6 # forces replacement
      ~ zookeeper_connect_string = "REDACTED" -> (known after apply)

        broker_node_group_info {
            az_distribution = "DEFAULT"
            client_subnets  = [
                "subnet-REDACTED",
                "subnet-REDACTED",
                "subnet-REDACTED",
            ]
            ebs_volume_size = 1000
            instance_type   = "kafka.m5.large"
            security_groups = [
                "REDACTED",
            ]
        }

        configuration_info {
            arn      = "REDACTED"
            revision = 1
        }

        encryption_info {
            encryption_at_rest_kms_key_arn = "REDACTED"

            encryption_in_transit {
                client_broker = "TLS_PLAINTEXT"
                in_cluster    = true
            }
        }
    }

Expected Behavior

Should not cause rebuild of MSK cluster from increasing the number of brokers. This is possible through UI/API/CLI

Steps to Reproduce

1. terraform plan

[brettcave]

the number_of_brokers property would need to determine if its an increase to unset ForceNew. (not sure if that sort of state would be possible in the provider. decreases would require a replace).

[ewbankkit]

Announcement.

[ryan-dyer-sp]

ebs_volume_size does not have the ForceNew, but does behave the same way in practice. You can increase this value, but not decrease it. I'm assuming an apply will fail with the API returning some error message if you attempt to shrink the volume. I would prefer this same behavior with the num_brokers. If I am intentional in my desire to shrink either of those #s, I can destroy targetting the cluster resource and then reapply, vs now I can't use tf at all beyond initial cluster creation.

I believe this is a pretty significant flaw in this provider and its ability to support MSK beyond initial cluster creation is severely hampered with this. I'm all for checking values as @brettcave proposed, but can we get the ForceNew removed in the interim post haste.

[ewbankkit]

I think this is the same issue as #10553 and is addressed by #11451.

[bflad]

As @ewbankkit notes above, we already have an open issue and pull request covering this functionality -- please follow those for updates (I'll be reviewing the PR today hopefully 🤞 ).

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!