Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to automatically create a Kafka topic longer than 64 characters #4337

Closed
aseev-xx opened this issue Nov 17, 2021 · 6 comments · Fixed by #8789
Closed

Unable to automatically create a Kafka topic longer than 64 characters #4337

aseev-xx opened this issue Nov 17, 2021 · 6 comments · Fixed by #8789
Labels

Comments

@aseev-xx
Copy link

Bug Report

Describe the bug
An attempt to start writing to a topic with a name greater than 64 characters is impossible due to a direct limitation in the code.
https://github.com/fluent/fluent-bit/blob/master/plugins/out_kafka/kafka.c#L221-L225

[2021/11/17 15:37:06] [ warn]  dynamic kafka topic length > 64 not allowed
[2021/11/17 15:37:15] [ warn]  dynamic kafka topic length > 64 not allowed
[2021/11/17 15:37:21] [ warn]  dynamic kafka topic length > 64 not allowed
[2021/11/17 15:37:23] [ warn]  dynamic kafka topic length > 64 not allowed

To Reproduce
You must enable the automatic creation of topics (auto.create.topics.enable=true) and try to write to a topic with a name exceeding 64 characters

Expected behavior
According to the documentation, the length of the topic name should not exceed 249 characters. Therefore, it seems correct to support such a possibilityю

https://kafka.apache.org/documentation/:
"Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room in the folder name for a dash and a potentially 5 digit long partition id."

Your Environment
Reproduced in all versions

Additional context
If such a restriction is not accidental and is not planned to be changed, I would like to hear a detailed answer for what this was done, since this would at least help us to write a specific patch for ourselves that removes this restriction

@github-actions
Copy link
Contributor

github-actions bot commented Mar 6, 2022

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

@github-actions github-actions bot added the Stale label Mar 6, 2022
@github-actions
Copy link
Contributor

This issue was closed because it has been stalled for 5 days with no activity.

@mickymiek
Copy link
Contributor

mickymiek commented May 2, 2024

Hi,
Does someone has an answer? This issue is still relevant even with fluentbit latest version. The limit should be equal to kafka one, or is there something else to consider?

I'm willing to contribute this change if needed.

@aseev-xx
Copy link
Author

aseev-xx commented May 2, 2024

@mickymiek We made an internal patch, so I didn’t push the topic for changes. I haven't looked at the new version for a long time, but if the problem persists, it seems to me that this should be fixed.

@mickymiek
Copy link
Contributor

I see. If this get no answer we'll consider forking/patching this as well, but such a quick (and necessary) fix should normally be in main branch rather quickly.

@patrick-stephens
Copy link
Contributor

I would say the best thing here is to send a PR - we rely on contribution for OSS so if you have a patch then it helps both us and you to get it merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants