In most organizations, there’s a lot of friction between the IT guys and the security guys. This is due to a shared interest in the same company assets, but a difference in responsibilities for those assets.
The Role of IT Operations
The IT guys are generally tasked with keeping systems up and running. It’s their job to make sure that applications are accessible to the users at all times. If a server is down, that means an application is unusable. This leads to unhappy users, support tickets and ultimately, more work for the IT guys. Then they have to answer to management as to why they failed at their job.
The Role of Security Operations
The security guys exist to help mitigate risks. It’s their job to make sure that people aren’t accessing computers or data that they’re not supposed to. That’s the case whether they’re guarding data against hackers from outside the company or just random people inside the company who happen to have access to data that they shouldn’t.
And therein lies the problem. The IT people see it as their job to make sure all services are available at all times while security comes in the door and says “Yes, but you need to add another layer on top of your operations and filter out all of this other stuff so it’s ONLY accessible to this subset of people.”
Basically, the end result is that the security people are creating more work for the IT people. And when the IT people ignores the security team is saying, they are creating insecurities that the security team is responsible for. It creates an unfortunate situation where both teams have a job to do and one of them can't usually directly do it.
It’s not a big surprise that most IT people and security people butt heads so often. It’s hard enough sometimes to do your job. When you have to answer to an external team when there appears to be very little upside to doing what they say, it’s going to be an uphill battle.
Bridging the Divide
The solution to this little dilemma is actually an education problem, not a technical problem. Neither group seems to truly understand the role of the other in the organization. The reality is that both groups are trying to do the same thing: Make IT services available to the people who need them.
What IT needs to know about security
Believe it or not, the job of the security group isn’t to make your life difficult. In fact, you share many of the same goals. To make sure IT services are accessible to the people who need them. The key part of that phrase is “the people who need them”. People not working for your company don’t need access to your corporate data. Nor do internal employees who want to see data that they shouldn't be seeing.
Consider for a moment what’s going to happen if there’s a security breach of any kind. Do you know what the first thing you’re supposed to do is? That’s right. Cut the machine off from the network. For virtual servers, that generally means killing the network cards. Taking a snapshot and then analyzing it offline is sometimes an option, but if you have a hacker inside the machine, the last thing you want to do is leave it publicly available so they can siphon more data out of it.
So you have to take it offline. And that means taking your IT service offline. Which means unhappy users. Then the CIO needs to get involved to figure out what to do about the security breach and how best to get a new server online. Then there's the endless follow up, the insurance, the legal team, the forensics people, law enforcement, etc. The list goes on.
What’s that? Just restore the machine from a backup? Well, depending on how long the machine was compromised, that might or might not be an option. If your backup strategy includes off-site backups on a monthly basis, how far back do you think you can go with what you have on-site? I suspect a couple of weeks at the most. And it’s going to take time to restore the machine. And can you ever be 100% sure that you know exactly when a security breach happened. Are you sure you restored from before they got in the door? Are you sure you closed off the holes so they can't do it again as soon as the machine comes back online? Many times, you’re much better off starting from scratch so that everything is a known quantity.
That’s going to take you several days, which means several days of downtime. Depending on the service, that’s not always an easy pill to swallow.
In a way, the security team is trying to help you make sure the IT services are available for your users. Sure, they’re creating work for you to do. But the cost of that work more like operational overhead, while the cost of a security breach is an “all hands on deck, drop EVERYTHING that you’re doing and help deal with this RIGHT NOW” kind of event.
Do you know how liability insurance works for security breaches? Generally, you get one chance. If you suffer a major breach, then insurance is going to cover it, but then your insurance premiums are going to skyrocket because you’ve proven that you’re a risk to the insurer. Most companies will survive their first security breach, but not all of them.
Suffer a second security breach and you’re almost certainly un-insurable. It’s going to be really hard to survive that because the insurance is only going to cover so much and your company is on the hook for the rest. You'll need to prove that you did everything right and there was nothing you could have done. Most companies don’t survive a second security breach.
Indirectly, this means that the security group is tasked with helping to make sure you still get a paycheck from one week to the next. It might be a good idea to work with them and pay attention to their recommendations, if for no other reason than job security. But there's the added benefit that keeping the systems up and running at all times is good for everyone.
What the security team needs to know about IT operations
The IT team already has plenty of things to do. Don’t drop a bombshell report on their desk and tell them ALL of the things that are wrong with the computers on the network. If you do, that report is going to be stuffed in a desk until the end of time… or until the company goes out of business after the IT team ignores everything you told them.
Instead, you need to start small.
Target just one simple thing at first. Something basic, like reviewing user accounts that are on all of the computers. Make sure those accounts are supposed to be there. If not, make sure they’re disabled or deleted.
Now that you have that list, create a process around remediating those things. The best thing to do is to attach this process to something like the patch management process. Ideally, there’s already a patch management process in place that involves scheduled maintenance and scheduled downtime of some kind.
This allows you to get security changes onto the schedule in such a way that it doesn’t impact the users of the system. If you don’t have scheduled maintenance windows, this would be a really good time to look into implementing them.
These maintenance windows don’t need to be extensive or invasive. Maybe an hour or two once each week or perhaps twice per month. Again, start small. Ideally, you’d be able to get to a point where you will have a daily maintenance window available, whether it’s used or not. However keep in mind that some systems you just can’t bring down on a regular basis like that.
Now that you have a set of maintenance windows
Build a process around notifying the users of your regular maintenance windows. Schedule that downtime and make the changes that you feel are necessary. Make sure that someone is responsible for approving changes that are going into your environment.
You don’t need to go out and purchase a change management system just yet. Tools aren’t the answer. Process is. You can use something as simple as an Excel spreadsheet to track changes being proposed in the environment. To do that, just track the following things:
- Date Proposed
- Proposed By
- Target Computers
- Target Change Date
- Testing Process
- Rollback Process
- Approved By
- Approved Date
- Implementation Date
We're all on the same team
Conflicts between groups within an organization are fairly common. Fortunately, the thing to keep in mind is this. We're all on the same team.