Systems | Development | Analytics | API | Testing

Democratizing and sharing data inside and outside your organization, as a real-time data stream, has never been more in demand. Treating data as-a-product and adopting Data Mesh practices is leading the way. Here, we explain the concept through a real-life example of an airline building applications that process data across different domains.

Organizations today face complex data challenges as they scale, with more distributed data architectures and a growing number of teams building streaming applications. They will need to implement Data Mesh principles for sharing data across business domains, ensure data sovereignty across different jurisdictions and clouds, and maintain real-time operations.

Every enterprise is modernizing their business systems and applications to respond to real-time data. Within the next few years, we predict that most of an enterprise's data products will be built using a streaming fabric – a rich tapestry of real-time data, abstracted from the infrastructure it runs on. This streaming fabric spans not just one Apache Kafka cluster, but dozens, hundreds, maybe even thousands of them.

As Kafka evolves in your business, adopting best practices becomes a must. The GitOps methodology makes sure deployments match intended outcomes, anchored by a single source of truth. When integrating Apache Kafka with GitOps, many will think of Strimzi. Strimzi uses the Operator pattern for synchronization. This approach, whilst effective, primarily caters to Kubernetes-based Kafka resources (e.g. Topics). But this isn’t ideal.

It was lovely to see so many of the community and hear about the latest data streaming initiatives at Kafka Summit this year. We always try to distill the sea of content from the industry’s premier event into a digestible blog post. This time we’ll do it slightly differently and summarize some broader learnings, not only from the sessions we saw, but the conversations we had across the two days.

In this age of AI, the demand for real-time data integration is greater than ever. For many, these data pipelines should no longer be configured and deployed by centralized teams, but distributed, so that each owner creates their flows independently. But how to simplify this, whilst practicing good software and data governance? We are introducing Lenses 5.5.

If 2023 was the year we woke up to how generative AI would change our world, 2024 is the year we realize the change. The real-time AI-driven enterprise may not be pixel-perfect yet, but we’re well on the way. Gen AI has a knock-on effect on all the trends and challenges we will see in 2024. Here’s our take.

Amazon S3 is a standout storage service known for its ease of use, power, and affordability. When combined with Apache Kafka, a popular streaming platform, it can significantly reduce costs and enhance service levels. In this post, we’ll explore various ways S3 is put to work in streaming data platforms.

We like to reduce the most mundane, complex and time-consuming work associated with managing a Kafka platform. One such task is backing up topic data. With a growing reliance on Kafka for various workloads, having a solid backup strategy is not just a nice-to-have, but a necessity. If you haven’t backed up your Kafka and you live in fear of disaster striking, worry no more.

An effective data platform thrives on solid data integration, and for Kafka, S3 data flows are paramount. Data engineers often grapple with diverse data requests related to S3. Enter Lenses. By partnering with major enterprises, we've levelled up our S3 connector, making it the market's leading choice. We've also incorporated it into our Lenses 5.3 release, boosting Kafka topic backup/restore.