My friend Brenda is an auditor who came to work one day and couldn’t connect to her department’s audit server.
Brenda called the help desk and told them she couldn’t connect. She wasn’t sure what the server’s name was, so the help desk had a bit of trouble finding it.
The help desk called back later to note that the server (which resided in the data center) was installed and managed entirely by the Audit department, and that IT was “not responsible for anything related to it other than the hardware.”
Brenda managed to locate the phone number of the person who installed and managed the server (and left the company some months ago), and he provided the correct server name, and reminded Brenda that he wrote extensive documentation regarding how the server was configured and how to install the applications on it.
“Just give the documentation to IT,” he said, chuckling. “They should be able to restore it rather quickly. Remember that last meeting I had before I left, when I walked everyone through the documentation and where it was stored on the shared drive?”
After some hunting, Brenda located the document. A day later, after IT “got the server running again,” Brenda heard that a junior IT staffer has wiped the server because “it never appeared to be used.”
What issues do you see with this event? What normal IT activities should have prevented this unfortunate occurrence?
See the comments below, and the follow-up post, Conclusion: Audit Server Disappeared.
ITauditSecurity 
Appropriate access rights per chance? Sorry, can’t get excited about servers and bits of copper cable, especially on my weekend off.
LikeLike
AuditMonkey,
No, not an access rights issue. Think bigger.
I can think of at least 5 items that were overlooked, both on the IT and Audit side.
Thanks for your comments.
LikeLike
A couple of observations. The ability to ‘delete’ the server wasn’t restricted and any wiping should be authorised. On face value, no-one knows how many servers are maintained and having separately maintained servers makes it difficult to apply an encompassing IT security policy to ensure the integrity of data and prevent unauthorised access.
LikeLike
Audit Monkey,
Thanks for the input. Authorization is huge. Can you expand on your comment, “having separately maintained servers makes it difficult to apply an encompassing IT security policy.” I’m not sure I understood it completely.
LikeLike
1- Brenda not being able to connect to the server might mean that it was taken off network, this shows lack of port security; a security issue.
2- Lack of IT resources in the Audit team, if the last guy was able to do it then there should have been someone who took over the responsibility after he left. This would also help to maintain the access control to the Audit server since IT wouldn’t need to have access to it.
3- Lack of policy implementation; data wiping policy. There must be some conditions to be met before anyone can wipe any data off.
4- Lack of asset identification. The guy probably had no idea who the asset owner was and probably didn’t bother finding out since there might not be such a procedure enforced.
Just a few things I could see wrong with the whole incident. I am interested to see your input and Audit’s action on the incident.
LikeLike
coffeeking,
Great input. My comments: 1) port security wasn’t something this company dealt with, but a good point (do you find that many companies go to this level? I haven’t). 2) Excellent. I didn’t think anyone would catch the lack of Audit appointing a new auditor for this task. 3) Right on. No wipe policy enacted. 4) Absolutely.
You covered many of the issues, so thanks again.
For the other issues, think about how IT should have been alerted without Audit ever calling…
LikeLike
ITauditSecurity,
Thanks.
You are right, not many companies get to the point of deploying port security, unless security has a good back from management, which is the case at my current workplace and this is one of the ways we would identify an internal incident.
Regarding the issue how IT should have been informed of the incident, one way I can think of is there should have been some sort of monitoring in place that would provide status updates on the servers online, something like a SCOM solution, this would report server related issues in real-time.
Was this in place at the company in address?
LikeLike
Nope. One of the other interesting things about this company is that my friend Brenda (the auditor) is the only one who has a password on her company-issued smart phone. Not even the infosec manager uses one. That tells you all you need to know…
LikeLike
So it appears that your friend is more sec savvy then the infosec manager himself. In my opinion it should been a security issue before it became audit issue, security guys are supposed to know all the infrastructure additions and removals before anyone else.
I am taking a wild guess at what the audit did in response to the incident, they issued a report against the lack of security enforcement in the organization, the report was targeted at the security department.
Is my guess close?
LikeLike
coffeeking,
I’m going to hold my comments for a little while longer, as this discussion is still heating up. Stay tuned. Thanks.
LikeLike
If you have separate servers maintained by different departments, would it be more difficult to ensure server maintenance and apply a consistent IT security policy? Thinking out of sight, out of mind.
LikeLike
Audit Monkey,
Ok, now I understand. The audit server was in the data center, it was just managed day to day by Internal Audit (originally, at least). Audit added users, updated the software, did initial trouble-shooting evidently. However, the hardware and everything but the application was still the responsibility of IT.
Regardless of who manages the day-to-day operations, if it’s in the data center, the server is still IT’s responsibility overall. It similar to a PeopleSoft implementation where the DBA manages the database, the application admin manages PeopleSoft application, Finance manages the data, and IT manages everything else. IT security policy should be the same regardless–if it’s on the network, it needs to follow company policy (if it has one). Also, most companies have multiple data centers (some call them closets), so most devices, servers, and applications (including those in the cloud) are spread around all over the place.
Unfortunately, some things are right under IT’s nose (uh, and Audit’s nose) and they still don’t manage them. Too many companies forget that all the departments need to run as a team; they fall and rise together, silo or no silo.
LikeLike
Many good comments here. I’ll put in my 2-cents-worth, from a security perspective.
Logging & monitoring is the most glaring security issue. It may have been partially in place or configured to look at the wrong things — “never appeared to be used”, based on what?
Also from a security perspective :
1. P&P implementation and enforcement is an issue if a junior IT person wipes an entire server. Perhaps training and communication too (even a junior should know, or have been told, to double-check before wiping production data, esp. wholesale).
2. Why did it take a whole day to restore the server? Is the backup strategy up-to-date and tested? What about BCP/DRP? If huge volume of data, older data should be kept near-/off-line instead.
LikeLike
2Hats,
Very good points, totally agree with you on these.
LikeLike
Excellent points throughout, although I’ll play devil’s advocate on it taking “a day” to come back online – sounds like they may have had to rebuild a system, and it probably wasn’t a super high priority request (audit? come on… it’s not front line sales). :P
LikeLike
Krupo,
No it wasn’t even a crucial audit server. But it was one that was used almost every day. You know the old line, always keep payroll and audit happy…
LikeLike