Systems | Development | Analytics | API | Testing

Greetings, my dear readers. It’s been some time since I’ve posted my last article. This is the third chapter of the introduction to the internal data world of Keboola. In the previous chapters, I’ve posted about an introduction to our internal reporting and communication with our users. Since the last time, a couple things have happened.

Kindred Group is a publicly-traded gambling operator with offices across four continents, offering entertainment options such as online poker, sports betting, and online casinos. Since its founding, Kindred has experienced fast growth acquiring nine different gambling brands over the last 20 years. With over 30 million customers globally and numerous brands to manage, the Kindred team had a pressing need for a good data management system.

In the modern age, everything runs on the cloud. The majority of modern applications are written with cloud technologies - they use public cloud providers for DNS, distributed caching, and distributed data stores. Cloud solutions are so popular among engineers because of their many advantages: But distributed systems are not impervious to breaking. Foursquare’s example is testimony that even the great and mighty experience failure within distributed systems.

Distributed systems have unlocked high performance at a large scale and low latency. You can run your applications worldwide from the comfort of your Amazon Web Services (AWS) platform in California, but the user adding an item to their shopping cart in Japan will not notice any delay or system faults. However, distributed systems - and specifically distributed database systems - also malfunction.

Modern engineering has revolutionized almost every complex human endeavor. From lean manufacturing to globe-wide telecommunications; from software and IT bringing the world to our fingertips to medical devices discovering previously invisible diseases, there is no human endeavor that engineering has not changed for the better. But engineers don’t only build complex systems and tools that help the world run around. They’re also the first line of defense when things turn south.

The acronym ACID stands for atomicity, consistency, isolation, and durability. These describe the set of properties of database transactions that guarantee data integrity despite errors, system failures, power failures, or other issues.

The trouble with marketing initiatives is that it is almost impossible to tell how they impacted the business’s bottom line. As the marketing pioneer John Wanamaker said: A person scrolling through Twitter on their mobile app might have seen your ad, loved your brand, and then logged into their desktop to purchase your product. The gap between needs generated by marketing spans across marketing channels and time.

Data transformations are the backbone of smooth-running data operations. Transformations are used in data replication between databases, data migration from cloud to on-premise, and data cleaning (aggregations, outlier removal, deduplication …) aka all the good stuff that goes into extracting insights from data. But as any data professional can attest, transformation can also be a painful bottleneck. Think scripts that run for an entire day and finish just before the next scheduled job.

As your data operations evolve, they become messier. Diverse data sources and data models at their sources, multiple movements of data throughout your platform, and cobbled-up infrastructure, which has grown in complexity through every deployment have made it hard to identify, trace, classify, and understand your data assets. This can be as simple as an analyst spending hours trying to figure out where a data attribute in a table came from and whether it is trustworthy.

Working with raw or unprocessed data often leads to poor decision-making. This explains why data scientists, engineers, and other analytic professionals spend over 80% of their time finding, cleaning, and organizing data. Accordingly, the ETL process - the foundation of all data pipelines - devotes an entire section to T, transformations: the act of cleaning, molding, and reshaping data into a valuable format.