Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

out_kafka: increase max kafka dynamic topic length to 249 characters #8789

Merged
merged 1 commit into from May 14, 2024

Conversation

mickymiek
Copy link
Contributor

This commit increases the max length of kafka dynamic topics.

As the limit previously set (64) seemed arbitrary, and kafka allows for a maximum of 249 characters per topic, I think fluentbit should reflect that limit.

Fixes #4337

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

This commit increases the max length of kafka dynamic topics.

As the limit previously set (64) seemed arbitrary, and kafka allows for
a maximum of 249 characters per topic, I think fluentbit should reflect
that limit.

Signed-off-by: Michael Meunier <mmeunier@skilld.cloud>
@cosmo0920
Copy link
Contributor

From the Kafka documentation, this modification should be vaild:

Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room in the folder name for a dash and a potentially 5 digit long partition id.
ref: https://kafka.apache.org/documentation/

@edsiper edsiper merged commit abb3b06 into fluent:master May 14, 2024
45 checks passed
@edsiper
Copy link
Member

edsiper commented May 14, 2024

thank you

@mickymiek mickymiek deleted the increase-kafka-auto-topic-max-char branch May 15, 2024 08:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Unable to automatically create a Kafka topic longer than 64 characters
3 participants