Increased memory usage with GetMetadata call

[skedia-confluent]

Description

Our application uses the Producer client to make the GetMetadata API call. We have observed that the application's memory usage increases whenever it makes this call. Interestingly, it does not go back to the original value thereafter. The increase appears to be directly proportional to the number of topics.

To confirm that the issue is with the package, we wrote a simple application to just create a new producer client and make the GetMetadata call in a loop. The memory usage graph looks like this

It can be seen that memory usage increased from around 1.5MB before the loop to around 60MB during the loop and then settled at 23MB after the loop.

We tried the same thing with Admin client. The memory usage graph looks like this

It shows a similar pattern with lower values. Memory usage increased from 1MB before the loop to 23MB during the loop and then settled at 19MB after the loop.

How to reproduce

It can be reproduced by creating the client and making the GetMetadata call in a loop.
Code:

func getProducerConfig(namespace string) *kafka.ConfigMap {
	conf := &kafka.ConfigMap{}
	setupSSL(conf, namespace)
	return conf
}

func setupSSL(conf *kafka.ConfigMap, namespace string) {
	conf.SetKey("bootstrap.servers", fmt.Sprintf("kafka.%s.svc.cluster.local:9075", namespace))
	conf.SetKey("security.protocol", "SSL")
	conf.SetKey("ssl.endpoint.identification.algorithm", "none")
}

func GetProducer(namespace string) (*kafka.AdminClient, error) {
	config := getProducerConfig(namespace)
	return kafka.NewAdminClient(config)
}

func main() {
	time.Sleep(3 * time.Minute)
	for i := 1; i <= 15; i++ {
		producer, _ := GetProducer(<namespace>)
		producer.GetMetadata(nil, true, 10000)
		producer.Close()
		time.Sleep(1 * time.Minute)
	}
	select {}
}

Checklist

Please provide the following information:

• confluent-kafka-go and librdkafka version (v2.3.0):
• Apache Kafka broker version:
• Client configuration: ConfigMap{...}

"bootstrap.servers": <bootstrap_server>,
"security.protocol": "SSL",
"ssl.endpoint.identification.algorithm": "none",

• Operating system: Linux
• Provide client logs (with "debug": ".." as necessary)
• Provide broker log excerpts
• Critical issue