New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to automatically create a Kafka topic longer than 64 characters #4337
Comments
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the |
This issue was closed because it has been stalled for 5 days with no activity. |
Hi, I'm willing to contribute this change if needed. |
@mickymiek We made an internal patch, so I didn’t push the topic for changes. I haven't looked at the new version for a long time, but if the problem persists, it seems to me that this should be fixed. |
I see. If this get no answer we'll consider forking/patching this as well, but such a quick (and necessary) fix should normally be in main branch rather quickly. |
I would say the best thing here is to send a PR - we rely on contribution for OSS so if you have a patch then it helps both us and you to get it merged. |
Bug Report
Describe the bug
An attempt to start writing to a topic with a name greater than 64 characters is impossible due to a direct limitation in the code.
https://github.com/fluent/fluent-bit/blob/master/plugins/out_kafka/kafka.c#L221-L225
To Reproduce
You must enable the automatic creation of topics (auto.create.topics.enable=true) and try to write to a topic with a name exceeding 64 characters
Expected behavior
According to the documentation, the length of the topic name should not exceed 249 characters. Therefore, it seems correct to support such a possibilityю
https://kafka.apache.org/documentation/:
"Each sharded partition log is placed into its own folder under the Kafka log directory. The name of such folders consists of the topic name, appended by a dash (-) and the partition id. Since a typical folder name can not be over 255 characters long, there will be a limitation on the length of topic names. We assume the number of partitions will not ever be above 100,000. Therefore, topic names cannot be longer than 249 characters. This leaves just enough room in the folder name for a dash and a potentially 5 digit long partition id."
Your Environment
Reproduced in all versions
Additional context
If such a restriction is not accidental and is not planned to be changed, I would like to hear a detailed answer for what this was done, since this would at least help us to write a specific patch for ourselves that removes this restriction
The text was updated successfully, but these errors were encountered: