Whether you’re working at a large enterprise or a small business, there’s almost always a need to extract data from source systems, process it, and use it for operational or analytical purposes. That process, moving data from point A to point B, transforming it along the way, and making it usable, is what we typically…
Read more
One of the questions most data engineers need to answer is whether this data pipeline should be real-time or batch. Sometimes posed as streaming vs batch. The tools you might use to do that have changed over the past few years, but that was always the question. The business, of course, would always ask for…
Read more
Is your data team constantly feeling the pressure to deliver? Do members of your team say they feel like they’re doing work meant for two people? If the answer to either or both of these questions is a resounding yes, you may feel tempted to think, “We just need more hands on deck.” However, hiring…
Read more
After working in data for over a decade, one thing that remains the same is the need to create data pipelines. Whether you call them ETLs/ELTs or something else, companies need to move and process data for analytics. The question becomes how companies are actually building their data pipelines. What ETL tools are they actually…
Read more
Scraping data from PDFs is a right of passage if you work in data. Someone somewhere always needs help getting invoices parsed, contracts read through, or dozens of other use cases. Most of us will turn to Python and our trusty list of Python libraries and start plugging away. Of course, there are many challenges…
Read more
A key responsibility for any data team is to understand the core metrics driving their business. Starting from the top, these metrics often include figures like gross revenue and expenses. However, these high-level metrics can feel too far removed and abstract from the actual business. Many companies, therefore, break down these top-line metrics into more…
Read more
Leading data teams can be challenging. You’ve got management and non-technical teams constantly reaching out with ad-hoc data requests; you’re likely trying to figure out what tools will work best and not blow the bank. Not to mention, you’ve got to bridge the gap between business and technology. All while trying to grow your data…
Read more
What would you do if you became the head or director of data for a 1,000-person company? Yesterday, you were plugging along as an analyst, and now, suddenly, you have all these new responsibilities. Figuring out where to start is part of the job. You’d probably feel a strong temptation to freak out. Who wouldn’t?…
Read more
Recently, I wrote an article diving into what Druid is and which companies are using it. Now I wanted to do a deeper dive into Apache Druid’s architecture. Apache Druid has several unique features that allow it to be used as a real-time OLAP. Everything from its various nodes and processes that each have unique…
Read more
SQL Server Integration Services (SSIS) comes with a lot of functionality useful for extracting, transforming, and loading data. It can also play important roles in application development and other projects. But SSIS is far from the only platform that can provide these services. You might seek alternatives to SSIS because you want a more agile…
Read more