Systems | Development | Analytics | API | Testing

If you’ve been running Kafka for a while, with any luck you have quite a few engineering teams onboarded, potentially with hundreds or even thousands of applications. Hopefully the Lenses.io Developer Experience platform helped in this adoption. But finding the right balance between governance and openness can be tricky.

Here at Lenses we’re as always laser-focused on making engineering streaming apps and managing Kafka, not just less stressful, but delightful. 6.1 is another big step forward. It starts with the Lenses MCP Server. It connects your AI assistant such as Cursor and Claude, bringing the knowledge of the internet with the context of your Kafka environment. It has the power to transform the work of engineers building and managing streaming apps.

If 2024 was the year enterprises adopted Generative AI, 2025 is the year Agentic AI became a reality. In the last six months, the conversations I’m having with engineering leaders have quickly shifted from simple chatbots to AI-enabled IDEs, copilots and autonomous systems that take action. This has to a large part been helped thanks to MCP. To date there have been 10,000+ MCP integrations built by the community, so to say it is a success is an understatement.

Imagine an airline system monitoring traffic around an airport. If it detects a major delay, countless systems may need to react instantly: Ground operations to adjust flows. Some of these systems will still connect via API, traditional MQ or iPaaS technologies, but the data’s volume and urgency and the ease of decoupling apps make architecting with Kafka the better fit. The natural question is: should all these applications & systems connect to the same Kafka cluster?

Kafka cluster-to-cluster data replication is critical to many use cases: disaster recovery (DR), cloud or data center migration, testing applications with production-like data, and multi-region data distribution. Easy replication of data between clusters: The business case is clear, but the cost model is not. Some solutions appear free but impose heavy operational burden.

From weeks-long projects to self-service data distribution.

AWS MSK has become popular because it deploys Kafka easily and bills alongside other AWS services. Over the past few years, AWS announced Express Brokers, a new cluster type that offers unlimited storage and separates brokers from storage resources. This simplifies scaling and reduces the time needed to rebalance topics when adding or removing brokers.

You're three hours into debugging a stalled Kafka consumer. The lag is climbing. Customers are complaining. Your logging doesn't show anything useful, and changing the log level requires a deployment approval that won't come until tomorrow morning. Sound familiar? If you're operating Apache Kafka at scale, that sinking feeling when a consumer group stops progressing, and you're left playing detective with insufficient clues.

Devoxx Poland 2025 felt different. Not because of revolutionary new frameworks or another "this changes everything" moment, but because of what didn't happen. The conference had an unusual dose of pragmatism, skepticism, and – dare we say it – common sense. Maybe it's because developers are asking the right questions: "Does this solve a problem?" and "What happens when this inevitably breaks?" Here's what emerged from the sessions we watched, and the people we spoke to.

Let's talk about a problem that might sound simple but gets complex quickly: copying data from one Kafka cluster to another. As our Kafka usage grows, many of us find ourselves managing multiple clusters and needing to share data between them. Or worst still, sharing data to an external cluster. During a London meetup, we explored why this happens, what existing solutions offer, and why we decided to build our own Kafka replicator. Here's what we learned.