Someone once called me the “Airflow Guy“. I found it a little strange. I have really only written a few articles on Airflow. But you never know what sticks with people. Airflow was a very popular data orchestration solution in 2020. It was only about five years old at the time. Fast forward a few…
Read more
Let me start out by saying that not every data warehouse uses a columnar storage format. They often do benefit from it though. Data engineering teams often discuss columnar storage. Especially when discussing data warehouses or file formats like Parquet. But why does it matter and why do analytics benefit from columnar storage formats? Whether…
Read more
Not every data pipeline or process needs to be real-time. In fact, in my experience, often times when clients ask for real-time they mean daily or hourly loads. But, every so often there are use cases where real-time is required. Beyond how frequently a data pipeline needs to be processed, some also need to capture…
Read more
There are certain problems that seem to never really go away if you’re leading a data team. Don’t get me wrong. The tech world is currently saturated by AI hype. Not only because every tool you look at now just feels like a thin layer of ChatGPT, but every conversation you have with the business…
Read more
Here is something that remains true in the era of AI. Data is still siloed and hard to access. Are there plenty of tools that can help make data ingestion and ETL easier? Yes, but they aren’t all created equal. One established solution is CData. According to CData, its products provide live connectivity and data…
Read more
Every morning, your team wakes up to over one hundred data quality alerts. I know, I recall this issue at Facebook. Some of them would be fixable issues others should have just been warnings. It’s so easy to build data pipelines and add data quality checks these days that I am sure for some people…
Read more
When you demo a new data tool, everything works perfectly. Account executives are great at making it sound like their product is the perfect product. They make sure their data pipelines run smoothly. The demo dashboards that load in seconds. And of course, the graphs and charts they put together look pretty impressive. So, you…
Read more
Data teams continue to debate which is better: ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform). Meanwhile, they are overspending, pipelines are breaking, or no one is utilizing the data. So, does this debate even matter anymore? Ok perhaps that’s harsh but I do always find it a little funny when people still talk…
Read more
We are several years into the AI Revolution, so to speak, and with that has come an increased demand for data. The increased demand for data comes an increased demand for data infrastructure. Some companies already have reliable data stacks; others are looking to migrate to Snowflake, Databricks, or some other solution(I am sure some…
Read more
The demand for data has grown substantially in this AI-driven world. Meaning, there are more and more data pipelines being created. The funny thing is, when I first started in the data world, no one around me used the term data pipeline. I am sure plenty of data teams used the term data pipeline. But…
Read more