Not every data pipeline or process needs to be real-time. In fact, in my experience, often times when clients ask for real-time they mean daily or hourly loads. But, every so often there are use cases where real-time is required. Beyond how frequently a data pipeline needs to be processed, some also need to capture…
Read more
There are certain problems that seem to never really go away if you’re leading a data team. Don’t get me wrong. The tech world is currently saturated by AI hype. Not only because every tool you look at now just feels like a thin layer of ChatGPT, but every conversation you have with the business…
Read more
I’ve been working in the data world for over a decade and consulting for nearly as long, and some things just don’t change. Back in 2015, every company was trying to figure out how they could use data science to improve their business outcomes. Now there is a similar discussion around AI. Every business always…
Read more
Here is something that remains true in the era of AI. Data is still siloed and hard to access. Are there plenty of tools that can help make data ingestion and ETL easier? Yes, but they aren’t all created equal. One established solution is CData. According to CData, its products provide live connectivity and data…
Read more
Every morning, your team wakes up to over one hundred data quality alerts. I know, I recall this issue at Facebook. Some of them would be fixable issues others should have just been warnings. It’s so easy to build data pipelines and add data quality checks these days that I am sure for some people…
Read more
When you demo a new data tool, everything works perfectly. Account executives are great at making it sound like their product is the perfect product. They make sure their data pipelines run smoothly. The demo dashboards that load in seconds. And of course, the graphs and charts they put together look pretty impressive. So, you…
Read more
Data teams continue to debate which is better: ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform). Meanwhile, they are overspending, pipelines are breaking, or no one is utilizing the data. So, does this debate even matter anymore? Ok perhaps that’s harsh but I do always find it a little funny when people still talk…
Read more
We are several years into the AI Revolution, so to speak, and with that has come an increased demand for data. The increased demand for data comes an increased demand for data infrastructure. Some companies already have reliable data stacks; others are looking to migrate to Snowflake, Databricks, or some other solution(I am sure some…
Read more
The demand for data has grown substantially in this AI-driven world. Meaning, there are more and more data pipelines being created. The funny thing is, when I first started in the data world, no one around me used the term data pipeline. I am sure plenty of data teams used the term data pipeline. But…
Read more
Whether you’re working at a large enterprise or a small business, there’s almost always a need to extract data from source systems, process it, and use it for operational or analytical purposes. That process, moving data from point A to point B, transforming it along the way, and making it usable, is what we typically…
Read more