Every morning, your team wakes up to over one hundred data quality alerts. I know, I recall this issue at Facebook. Some of them would be fixable issues others should have just been warnings. It’s so easy to build data pipelines and add data quality checks these days that I am sure for some people…
Read more
When you demo a new data tool, everything works perfectly. Account executives are great at making it sound like their product is the perfect product. They make sure their data pipelines run smoothly. The demo dashboards that load in seconds. And of course, the graphs and charts they put together look pretty impressive. So, you…
Read more
We are several years into the AI Revolution, so to speak, and with that has come an increased demand for data. The increased demand for data comes an increased demand for data infrastructure. Some companies already have reliable data stacks; others are looking to migrate to Snowflake, Databricks, or some other solution(I am sure some…
Read more
The demand for data has grown substantially in this AI-driven world. Meaning, there are more and more data pipelines being created. The funny thing is, when I first started in the data world, no one around me used the term data pipeline. I am sure plenty of data teams used the term data pipeline. But…
Read more
One of the questions most data engineers need to answer is whether this data pipeline should be real-time or batch. Sometimes posed as streaming vs batch. The tools you might use to do that have changed over the past few years, but that was always the question. The business, of course, would always ask for…
Read more
After working in data for over a decade, one thing that remains the same is the need to create data pipelines. Whether you call them ETLs/ELTs or something else, companies need to move and process data for analytics. The question becomes how companies are actually building their data pipelines. What ETL tools are they actually…
Read more
Leading data teams can be challenging. You’ve got management and non-technical teams constantly reaching out with ad-hoc data requests; you’re likely trying to figure out what tools will work best and not blow the bank. Not to mention, you’ve got to bridge the gap between business and technology. All while trying to grow your data…
Read more
One of the holy grails that many data teams seem to chase is real-time data analytics. After all, if you can have real-time analytics, you can make better decisions faster. However, there often is a conflation between real-time data analytics and stream processing. These are two different concepts that are crucial to understanding how to…
Read more
Running a successful data team is hard. Data teams are expected to juggle a combination of ad-hoc requests, big bet projects, migrations, etc. All while keeping up with the latest changes in technology. In the past few years I have gotten to work with dozens of teams and see how various directors and managers deal…
Read more
Getting data out of source systems and into a data warehouse or data lake is one of the first steps in making it usable by analysts and data scientists. The question is how will your team do that? Will they write custom data connectors, pay for a data connector out of the box or perhaps…
Read more