Systems | Development | Analytics | API | Testing

AWS re:Invent is one of my favorite trade shows. It is one of the biggest technology conferences of the year and is an opportunity to have hundreds of conversations with customers and prospects, listen to their priorities and challenges, hopes, and give them a Cloudera tote bag or a pair of orange sunglasses. What follows is a collection of just a few things I learned and observed during my week in Las Vegas.

As a data engineer who has designed and managed ETL (Extract, Transform, Load) processes, I've witnessed firsthand the transformative impact of cloud-based solutions on data integration. Amazon Web Services (AWS) offers a suite of tools that streamline ETL workflows, enabling mid-market companies to move the big data to data stores such as Snowflake, data lake from different sources depending on use cases.

This December comes with some magic to it with a pivotal milestone: Scale to Zero is now in public preview and available for everyone! We’ve said it, our goal is to provide a serverless experience: Scale to Zero combined to Autoscaling makes serverless real. Starting today, your workloads running on GPU and CPU adapt fully automatically to traffic - they sleep and wake up automatically depending on requests, and scale out horizontally according to your criteria.

Earlier this year, we unveiled our vision for Tableflow to feed Apache Kafka streaming data into data lakes, warehouses, or analytical engines with a single click. Since then, many customers have been exploring, experimenting with, and providing valuable feedback on Tableflow Early Access. Our teams have worked tirelessly to incorporate this feedback and are excited to bring Tableflow Open Preview to you in the near future.

AWS Lambda has become a cornerstone of serverless architecture, enabling developers to run code without provisioning or managing servers. However, as applications grow, managing shared code, dependencies, and runtime configuration across multiple Lambda functions becomes challenging. This is where Lambda Layers step in to streamline development and deployment. In this blog we will explore what Lambda Layers are, why they are needed, how they work, and much more.

Today enterprises can leverage the combination of Cloudera and Snowflake—two best-of-breed tools for ingestion, processing and consumption of data—for a single source of truth across all data, analytics, and AI workloads. But now AWS customers will gain more flexibility, data utility, and complexity, supporting the modern data architecture.

For organizations looking to accelerate their Azure adoption while maintaining control, security and reliability, Fivetran provides a best-in-class solution that consistently outperforms ADF.

Snowflake combines unmatched scalability, performance, and ease of use. It simplifies the complexities of traditional data warehousing, enabling businesses to store and analyze data at scale without the overhead of infrastructure management. But to truly unlock the power of Snowflake, businesses need an efficient and secure way to move data into it.