Skip to content

n8sty/kafka-stuff

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 

Repository files navigation

Kafka: Interesting, Useful, or Weird

Here's a brief, but growing, collection of media that I've found to be one of the above (or something else) as relates to Kafka and, more broadly, streaming data.

Kafka Pixy REST and gRPC Kafka proxy, which automagically controls consumer group assignment, subscription, and termination. That's cool! From their documentation: "It is designed to hide the complexity of the Kafka client protocol and provide a stupid simple API that is trivial to implement in any language." I like that!

Application Reset Tool From Apache! A blessed tool to force an application that reads Kafka messages to reprocess said messages from the beginning by resetting topic offsets.

Kafka Consumers: Reading Data from Kafka

What determines Kafka consumer offsets? Great StackOverflow answer explaining how consumer offsets get configured. For reference, a consumer offset tells something reading messages from one or more topics where to resume reading from on the occasion that it's execution paused.

How to reset Kafka consumer group offset

Which Knobs To Turn? Short! That's good! Bare minimum configuration changes that need to happen to run Kafka in production.

Configuring Apache Kafka for Performance and Resource Management

Introducing Kafka-Kit: Tools for Scaling Kafka

Effective Strategies for Kafka Topic Partitioning

Handling GDPR with Apache Kafka: How does a log forget? Configure topics that store personally identifying information (PII) to perform log compaction, which (eventually) deletes messages with the same key. Requires that messages be keyed and deletion is not instantaenous, though it's quick from a human's perspective. I've used log compaction in a simple way to keep the currrent state of some entity for the Maestro (Teachable Masters) project.

Kafka, GDPR and Event Sourcing

Faust Like Kafka Streams but for Python!

Streaming data from PostgreSQL to Kafka using Debezium

Publishing with Apache Kafka at The New York Times This is the article that got me interested in using Kafka in novel ways. The New York Times, the Grey Lady, uses Kafka as a durable, forever data store. This is wildand I don't think will ever actually work anywhere else in practice but it's very intersting / shows the power of Kafka and its configurability.

Corrupted Index File

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published