Let me start out by saying that not every data warehouse uses a columnar storage format. They often do benefit from it though. Data engineering teams often discuss columnar storage. Especially when discussing data warehouses or file formats like Parquet. But why does it matter and why do analytics benefit from columnar storage formats? Whether…
Read more
Maybe you’re luckier than me. Maybe you’ve never opened a .sql file or an Airflow DAG only to be greeted by a 5,000+ line query…a true monster of a script that leaves you wondering where to begin. I’ve seen plenty of these, and every time, I ask myself: Why in the world do these exist? And, more…
Read more
Data integration is critical for organizations of all sizes and industries—and one of the leading providers of data integration tools is Talend, which offers the flagship product Talend Studio. In 2023, Talend was acquired by Qlik, combining the two companies’ data integration and analytics tools under one roof. In January 2024, Talend discontinued Talend Open…
Read more
Photo by Lukas As you increase your analytical processes and abilities, you’ll unavoidably increase costs. But there are definite ways to avoid having your costs grow at an unsustainable rate. This is the topic of a panel at the Modern Data Stack Conference featuring Maura Church, ex-director of data science and data engineering from Patreon.…
Read more
Apache Airflow is an extremely popular tool that data engineers rely on. But why? Why do data engineers like Airflow? Also, what does the Apache Airflow event do? In this article, we will answer questions like: What is Airflow? What is a DAG? Why do people use Apache Airflow? Why do we like Airflow?…
Read more
Photo by Pawel Czerwinski on Unsplash Big data is touted as the solution to many problems. However, the truth is, big data has caused a lot of big problems. Yes, big data can provide more context and provide insights we have never had before. It also makes queries slow, data expensive to manage, requires a lot of expensive…
Read more
Big data is the new commodity that companies need to drive the business forward. Having access to all your companies data sets used to be limited to large companies that could afford a whole team of data engineers and software developers. But this is no longer the case. Traditionally you would need to have a…
Read more
Data is an integral aspect of any business. It allows for solution development, metric tracking, and creates a structure for streamlined and integrated processes. Data empowers business decisions. I say that both from the fact that consulting firms like McKinsey have found that in their research companies that are using AI and analytics can attribute 20%…
Read more
Source Apache Airflow is a popular open-source tool that helps teams create, schedule, and monitor sequences of tasks, known as “workflows.” Managed Workflows for Apache Airflow, abbreviated as MWAA, is a managed service designed for orchestration in Apache Airflow. With MWAA, users can easily operate data pipelines at scale, setting up and managing them from…
Read more
Photo by Martin Sanchez on Unsplash There are plenty of clichés about data and its likeness to oil. But it’s far from easy to get value from data. Companies are creating more data than ever before. At our current pace, 2.5 quintillion bytes of data are created every day. Companies went from pulling data solely…
Read more