Photo by Lukas As you increase your analytical processes and abilities, you’ll unavoidably increase costs. But there are definite ways to avoid having your costs grow at an unsustainable rate. This is the topic of a panel at the Modern Data Stack Conference featuring Maura Church, ex-director of data science and data engineering from Patreon.…
Read more
Apache Airflow is an extremely popular tool that data engineers rely on. But why? Why do data engineers like Airflow? Also, what does the Apache Airflow event do? In this article, we will answer questions like: What is Airflow? What is a DAG? Why do people use Apache Airflow? Why do we like Airflow?…
Read more
Photo by Pawel Czerwinski on Unsplash Big data is touted as the solution to many problems. However, the truth is, big data has caused a lot of big problems. Yes, big data can provide more context and provide insights we have never had before. It also makes queries slow, data expensive to manage, requires a lot of expensive…
Read more
gBig data is the new commodity that companies need to drive the business forward. Having access to all your companies data sets used to be limited to large companies that could afford a whole team of data engineers and software developers. But this is no longer the case. Traditionally you would need to have a…
Read more
Data is an integral aspect of any business. It allows for solution development, metric tracking, and creates a structure for streamlined and integrated processes. Data empowers business decisions. I say that both from the fact that consulting firms like McKinsey have found that in their research companies that are using AI and analytics can attribute 20%…
Read more
Source Apache Airflow is a popular open-source tool that helps teams create, schedule, and monitor sequences of tasks, known as “workflows.” Managed Workflows for Apache Airflow, abbreviated as MWAA, is a managed service designed for orchestration in Apache Airflow. With MWAA, users can easily operate data pipelines at scale, setting up and managing them from…
Read more
Photo by Martin Sanchez on Unsplash There are plenty of clichés about data and its likeness to oil. But it’s far from easy to get value from data. Companies are creating more data than ever before. At our current pace, 2.5 quintillion bytes of data are created every day. Companies went from pulling data solely…
Read more
Photo by Fabio Sasso on Unsplash In the business world, cloud technology has become more and more dominant in recent years. Right now, research shows that about 50% of all business data is stored in the cloud, which just demonstrates the importance of external data sources and their place in the modern business environment. In…
Read more