What Is Shadow IT?
At my very first job in data, I was an analyst on a finance team. I was working with a senior engineer who had built out several new automated processes and was managing a data warehouse and several other applications.
We were supporting 4-5 financial analysts who were relying on our data to report to the CIO.
Two weeks after I started, the senior engineer left and I became responsible for everything they built. After continuing to manage the entire set of processes and automated reporting, I got a call from the “official data team”. They were calling because they realized our team had been creating a duplicate of the data warehouse every night and using it for reporting that they weren’t aware of, and these reports were suddenly causing issues at executive meetings.
I, the whole time, not knowing it, had become a shadow IT team.
The Problem
If you’ve worked on a data team, you’re constantly fielding requests. There are ad-hoc requests that can come from analysts and PMs. You have larger project asks that come from the executive team, and hey, a vendor you were paying for might be sunsetting a product and now you have to migrate to their new solution.
Everyone wants your time and like every team at a company, you’re understaffed. But other departments need answers and they can’t wait around all day for you. Tools like Excel allowed a broader range of users the ability to answer many of these questions. Instead of waiting for data teams to get back to them with views or ad-hoc queries they could directly connect to the data sources they needed.
Excel – the “self-serve analytics” tool before Tableau – started to run with that term back in 2012 (at least that’s the first time I saw the term).
Those reports and tools then become part of the team’s everyday process and with new SaaS tools, it’s become even easier to pay for a $10 a month Retool subscription that then allows your team to build applications with no oversight.
Another problem that many business teams run into when working with IT is that generally IT doesn’t always understand the problem’s context. At least, not as viscerally as the operations teams that are dealing with the problems on a daily basis. Meaning there is an unavoidable back and forth as both teams try to communicate and collaborate.
So why not just do it, just start a Shadow IT team.
What Is Shadow IT
Ok, we have talked around what shadow IT is for the last few paragraphs. So what is Shadow IT? Well, it is the usage of hardware or software(these days usually SaaS) by a department or individual without the knowledge of the IT or other governance/compliance teams within an organization.
This is becoming an ever-growing issue and estimated that nearly half of all IT spend “lurks in the shadows, as recently reported by a 2019 study from Everest Group.
Ok, but work is getting done right? So why should companies care?
Risks
IT and data teams can feel like they get in the way with all their needs for security compliance and data governance. Their goal is to:
App sprawl
At one of the first meet-ups I went to, I heard a CIO give a presentation that discussed how they had been hired at a company that gave them the initiative to reduce the number of applications in use. They chuckled a bit in the talk as they said they quit 2 years later because in that 2 years they had gone from 155 applications to 200. Teams are constantly requesting new technology, and purchasing duplicative functioning solutions all across an organization. Especially ones that lack any form of IT governance.
Security
Even if the specific app isn’t notoriously insecure, every piece of IT added without planning and consideration represents a possible attack surface. Each new process that moves data unencrypted via a thumb drive or a new single sign-on added into a workflow adds a new attack surface. IT teams already spend plenty of resources on cybersecurity for known applications and attack surfaces. Adding in new applications outside of the known space just amplifies the risk.
Rework
A common pattern that occurs when working with shadow data teams is a non-IT team member builds a process in VBA, Excel, Tableau, or Jupyter Notebooks, and then it becomes difficult to manage. So they then go to IT and ask them to rebuild and automate the process. I have been involved in this process multiple times. In one case the process that had been developed involved 10 different data sets that were pulled and managed in 10 different ways. Trying to simplify and standardize the process was an endeavor.
Cost
Shadow IT and data teams are often a function of companies trying to keep costs low. You avoid hiring more engineers and IT professionals and offloading work to other employees. This, in the short run, may keep costs low. But there are a lot of costs that may be incurred in the future. Such as needing to rework systems that weren’t developed for longevity, pose security risks, or had key person risks where the individual who built the shadow IT process leaves and no one else knows how it works.
All of this leads to unforeseen costs that could be far greater than the original cost of going through IT.
Benefits of Shadow IT
Yes, Shadow IT and data teams pose risks to businesses. So why do they show up everywhere? Whether you’re an SMB or Enterprise, there is a good chance that teams are setting up infrastructure in the form of SaaS applications, custom code, cloud components, and more.
Well here are just a few benefits of shadow IT.
- Faster delivery of data, automated processes, and other IT deliverables
- Reducing costs in the short term.
- Empowerment of business users and often less back and forth between IT and the business
Final Thoughts
To some degree shadow, IT and data teams are unavoidable. With the growth of SaaS and tools that are increasingly affordable to pay for monthly subscriptions, employees are going to continue to use said solutions.
And yes, it’s nice to think that we should build systems, data models, and processes that follow best practices. I just don’t know if we can close the Pandora’s box that is Shadow IT at this point.
As more individuals learn SQL and use tools like Retool and Airtable to build solutions without even reaching out to IT or data teams, then maybe the new problem to be solved is how to ensure they don’t put the company at risk.
Even if shadow IT keep costs low, these projects still require some planning and thinking through to ensure they don’t run up costs, send data to the wrong place or simply break when an edge case is hit.
If you enjoyed this article, then check out some of these articles and videos.
Databases Vs Data Warehouses Vs Data Lakes
How to Choose the Right ELT Tool
How To Build a Data-Based Business Strategy in 2022
Onboarding For Data Teams – How to set-up a streamlined onboarding experience