A new Python package from Fivetran and Astronomer enables connector management in Airflow.
Continuous development is an umbrella term that includes many DevOps processes. For that reason, it is important to understand what is continuous development to help you ensure that each of your DevOps processes are efficient and effective. Read along or jump ahead to the section that interests you the most.
Large portions of software development budgets are dedicated for testing code. A new component may take weeks to thoroughly test, and even then mistakes happen. If you consider software defects as security issues then the concern goes well beyond an application temporarily crashing. Although even minor bugs can cost companies a lot of time to locate the bug, resolve it, retest it in lower environments, then deploy it back to production.
Many technologies of the last century are out of date now, but flat file databases are still very much in use today and likely will be for a long while yet. They’ve stood the test of time for over four decades and are still going strong for a variety of reasons.
Encryption of Data at Rest is a highly desirable or sometimes mandatory requirement for data platforms in a range of industry verticals including HealthCare, Financial & Government organizations. The capability increases security and protects sensitive data from various kinds of attack that could be internal or external to the platform.
Let’s start with a real-world example from one of my past machine learning (ML) projects: We were building a customer churn model. “We urgently need an additional feature related to sentiment analysis of the customer support calls.” Creating the data pipeline to extract this dataset took about 4 months! Preparing, building, and scaling the Spark MLlib code took about 1.5-2 months!
