As data increased in volume, velocity, and variety, so, in turn, did the need for tools that could help process and manage those larger data sets coming at us at ever faster speeds. As a result, frameworks such as Apache Spark and Apache Flink became popular due to their abilities to handle big data processing…
Read more
No matter your industry, you’ll often need to make split-second business decisions in the digital age. Real-time data can help you do just that. It’s information that’s made available as soon as it’s created, meaning you don’t need to wait around for the insights you need. Real-time data processing can satisfy the ever-increasing demand for…
Read more
Starting new data engineering projects can be challenging. Data engineers can get stuck on finding the right data for their data engineering project or picking the right tools. And many of my Youtube followers agree as they confirmed in a recent poll that starting a new data engineering project was difficult. Here were the key…
Read more
If you’re relying on your OLTP system to provide analytics, you might be in for a surprise. While it can work initially, these systems aren’t designed to handle complex queries. Adding databases like MongoDB and CassandraDB only makes matters worse, since they’re not SQL-friendly – the language most analysts and data practitioners are used to.…
Read more
Photo by JESHOOTS.COM on Unsplash Modern data stack consultant plays an important role in companies looking to become data-driven. They help companies design and deploy centralized data sets that are easy to use and reliable. They do so by using cloud based solutions that help automate data pipelines and processes with less code than in…
Read more
Photo by Sigmund on Unsplash 2022 is coming to an end. What is the state of data infra? Are Snowflake and Databricks still fighting over total cost of ownership? Is everyone switching to DuckDB? Are data engineers all learning Rust? Let’s try to answer these questions. Building field-level lineage for modern data systems Lineage…
Read more
AWS has jumped on the bandwagon of removing the need for ETLs. Snowflake announced this both with their hybrid tables and their partnership with Salesforce. Now, I do take a little issue with the naming “Zero ETLs”. Because at the very surface the functionality described is often closer to a zero integration future, which probably…
Read more
Photo by Natalie Pedigo on Unsplash Onboarding at most companies would be a great bit for a comedian. For many, we have just accepted the fact that it’s not a great experience where you will likely spend the first two weeks just trying to get access to email. But it doesn’t have to be this…
Read more
Photo by Christopher Gower on Unsplash By Reseun McClendon Data Governance is the set of principles and procedures an organization uses to manage data during its life cycle. More specifically, data governance encompasses all that is done to ensure data is private, secure, accurate, available, and usable. The data life cycle may include the data’s acquisition, use, and disposal…
Read more
Databricks is a company that aims to make managing big data easy. By utilizing Apache Spark as its core processing engine, Databricks aims to make managing and processing big data easy. As one of the leading providers of big data solutions, Databricks has made a name for itself in the industry. What is Databricks? So,…
Read more