If you’re a data engineer, then you’ve likely at least heard of Airflow. Apache Airflow is one of the most popular open-source workflow orchestration solutions that gets used for data pipelines. This is what spurred me to write the article “Should You Use Airflow” because there are plenty of people who don’t enjoy Airflow or…
Read more
Photo by Sigmund on Unsplash 2022 is coming to an end. What is the state of data infra? Are Snowflake and Databricks still fighting over total cost of ownership? Is everyone switching to DuckDB? Are data engineers all learning Rust? Let’s try to answer these questions. Building field-level lineage for modern data systems Lineage…
Read more
“Why do we still struggle to answer basic questions like how many customers are still active or what exactly is the company’s churn?” This was a question posed to a panel I was sitting on at the Snowflake Summit. Several panelists provided different answers and all come from very different perspectives. Truth be told I…
Read more
The modern data stack has been discussed heavily in 2020 and 2021. But what wasn’t discussed enough was how to set up a successful data strategy. Truthfully, to drive value from your data you’re going to need to focus less on modernizing and more on your actual business problems. Driving business value with your data…
Read more
Photo by Victoriano Izquierdo on Unsplash Companies that are serious about using raw data for business intelligence need some way to centralize and control that data. Many of them are using specialized vendor products to do so. Fivetran is one choice for getting all of that data wherever it resides and centralizing it in an…
Read more