The B-Tree of Horror: Kafka migration tales to chill your partitions
In this session, Patrick from Lenses.io and Kafka expert Ben Gamble step into the darkest corners of Apache Kafka production environments. Whether you are a seasoned Site Reliability Engineer or a Data Engineer, these real-world 'horror stories' provide critical lessons on avoiding common pitfalls in distributed systems.
From mysterious message duplication during migrations to brokers being brought to their knees by 'ancient' clients, we uncover the monsters hiding in your clusters. We dive deep into the technical nuances of MirrorMaker 2, protocol translation, the Zookeeper to KRaft migration path, and the terrifying consequences of clock skew on data retention.
Key Topics Covered:
- Why MirrorMaker 2 offset translation can lead to message duplication.
- The hidden CPU cost of protocol translation with legacy Kafka clients.
- Avoiding the 'limbo' state during Zookeeper to KRaft migrations.
- How NTP and clock drift can cause unexpected data loss or disk exhaustion.
- The complexities of CDC and schema management with Debezium and Clickhouse.
Learn how to build more resilient real-time systems and ensure your production environment remains a dream, not a nightmare.
Check out Lenses.io for better Kafka visibility and try K2K as a MirrorMaker2 alternative!
Chapters:
00:00 - Introduction: Kafka Nightmares Begin
01:26 - Meet Lenses.io: Empowering Real-Time Data Teams
02:02 - Nightmare 1: Vanishing Offsets & MirrorMaker 2 Glitches
04:52 - The Cost of Failure: Flink & Financial Double Payments
07:46 - How to Mitigate Migration & Idempotency Issues
09:04 - Nightmare 2: The Attack of the Ancient Clients (CPU Spikes)
14:12 - Nightmare 3: Stuck in Zookeeper to KRaft Migration Limbo
19:00 - Upgrade Strategies: In-Place vs. Destination Clusters
21:14 - Nightmare 4: The Time-Traveling Producer (Clock Skew & Data Loss)
26:01 - The Dangers of On-Prem vs. Cloud Clock Drift
28:45 - Bonus Story: Schema Horrors & ETL Complexities with Debezium
32:41 - Conclusion: Avoiding the Dark Side of Kafka
#ApacheKafka #Kafka #DistributedSystems #DataEngineering #SoftwareEngineering #Lensesio #KRaft #Zookeeper #Debezium #SRE #TechHorrorStories