While installing and configuring some new software on my Windows server, I noticed that the IT department forgot to remove some previous software components from my server.
I remember seeing the notice that the software was being uninstalled and replaced by another package.
I could have removed the left over components myself (I am admin on the server), but I wanted to see if they would ever be removed. Did the Windows server team forget about this, or did the team not concern itself with such things? Maybe the procedures don’t include a process to ensure all components are removed.
I waited about 2 months, but the components were not removed.
Previously Noted
This event reminded me of something I wrote about 2 years ago: A Sneaky Way to Analyze IT Controls. In that post, I noted that one of the best ways to understand how a company’s IT controls work (or don’t) is to place yourself in the computing path.
In other words, own and use some of the infrastructure components around which the controls operate: servers, firewall rules, system IDs, etc. See the post for more info.
What was the Problem?
So what did I find left over on my server?
Some scheduled tasks were still running in Task Scheduler that should have been removed when a previous application (installed and managed by the Windows Server team) was uninstalled.
The risk of this situation depends on the original application that was installed, and the possible attack surface that these scheduled tasks, which were still running, presented. This particular application was a security app, which concerned me on multiple levels.
I always like to ask, “What’s the risk?” (my favorite question) and think through the possibilities.
First, being a security app means that if an attacker was targeting my server, this app would probably get the most attention, as it would probably needed to be dealt with as part of the attack.
Second, since some pieces were left behind, could they be exploited? In my particular case, it didn’t look like it.
Third, most uninstall routines should take care of things like scheduled tasks, IF the application installed those tasks in the first place. The fact they were left behind made me wonder if the IT team added a script to the install routine to create these scheduled tasks. (This was a popular security package that I would expect to clean itself up properly during install.)
In my mind, if that was true (that someone internally wrote this script), then a higher chance of that script having a flaw or working incorrectly exists; that means the scheduled tasks could be malicious or ineffective. Not likely, but certainly possible. Maybe the uninstall job just didn’t run to completion (not a better alternative).
Fourth, if the IT team uninstalls security apps without removing everything, that makes me wonder how careful they are installing/uninstalling other apps. My neck hairs are tingling way too much at this point.
Fifth, this app was deployed to all if not almost all Windows servers in the environment (more on this later). Some servers run all kinds of apps that may interact with this app or its leftovers, meaning you never know what problem A plus problem B on a server will produce.
Sixth, at the very least, these scheduled tasks were chewing up CPU cycles across thousands of servers, and nobody evidently knew or cared. The load might be noticeable for a server that is 5 years old that had 10 apps uninstalled and various pieces were left behind.
Now, I’ll be the first to admit that I might be pushing the risk boundaries a bit, and several dominos would have to line up before they starting falling with a loud clatter. But isn’t that how vulnerabilities work?
If you have enough, you’re going to find that eventually a problem will occur.
Taking Action
Again, I could have fixed the problem on my server myself, but instead did 2 things which were a lot more useful and fun:
- Sent an email to the Windows server team explaining what I found, and asked the team to uninstall the scheduled tasks on my server, and consider whether other servers had the same issue.
- Sent an email to the IT audit manager who sets the audit schedule for the next reporting period so that she would consider whether testing this would be appropriate.
Response
About 6 weeks after I sent the emails, I got a notice (sent to all Windows server owners, like myself) that these scheduled tasks were being uninstalled by IT. So that’s how I know the problem existed across the enterprise.
And sure enough, the tasks were removed. At least from my server.
Never heard from the IT audit manager. Time to follow up, but I doubt it goes on the audit radar.
Last Thoughts
I find all kinds of risks just by being alert, thinking critically about the situation, and asking, “Could this have an impact across the enterprise?” (my second favorite question).
And unlike this person, I then poke the bear.
ITauditSecurity 