Twitter Feed Popout byInfofru

IT Operations vs. Security - FIGHT!

In most organizations, there’s a lot of friction between the IT guys and the security guys. This is due to a shared interest in the same company assets, but a difference in responsibilities for those assets.

The Role of IT Operations

The IT guys are generally tasked with keeping systems up and running. It’s their job to make sure that applications are accessible to the users at all times. If a server is down, that means an application is unusable. This leads to unhappy users, support tickets and ultimately, more work for the IT guys. Then they have to answer to management as to why they failed at their job.

The Role of Security Operations

The security guys exist to help mitigate risks. It’s their job to make sure that people aren’t accessing computers or data that they’re not supposed to. That’s the case whether they’re guarding data against hackers from outside the company or just random people inside the company who happen to have access to data that they shouldn’t.

The Conflict

And therein lies the problem. The IT people see it as their job to make sure all services are available at all times while security comes in the door and says “Yes, but you need to add another layer on top of your operations and filter out all of this other stuff so it’s ONLY accessible to this subset of people.”

Basically, the end result is that the security people are creating more work for the IT people. And when the IT people ignores the security team is saying, they are creating insecurities that the security team is responsible for. It creates an unfortunate situation where both teams have a job to do and one of them can't usually directly do it.

It’s not a big surprise that most IT people and security people butt heads so often. It’s hard enough sometimes to do your job. When you have to answer to an external team when there appears to be very little upside to doing what they say, it’s going to be an uphill battle.

Bridging the Divide

The solution to this little dilemma is actually an education problem, not a technical problem. Neither group seems to truly understand the role of the other in the organization. The reality is that both groups are trying to do the same thing: Make IT services available to the people who need them.

What IT needs to know about security

Believe it or not, the job of the security group isn’t to make your life difficult. In fact, you share many of the same goals. To make sure IT services are accessible to the people who need them. The key part of that phrase is “the people who need them”. People not working for your company don’t need access to your corporate data. Nor do internal employees who want to see data that they shouldn't be seeing.

Consider for a moment what’s going to happen if there’s a security breach of any kind. Do you know what the first thing you’re supposed to do is? That’s right. Cut the machine off from the network. For virtual servers, that generally means killing the network cards. Taking a snapshot and then analyzing it offline is sometimes an option, but if you have a hacker inside the machine, the last thing you want to do is leave it publicly available so they can siphon more data out of it.

So you have to take it offline. And that means taking your IT service offline. Which means unhappy users. Then the CIO needs to get involved to figure out what to do about the security breach and how best to get a new server online. Then there's the endless follow up, the insurance, the legal team, the forensics people, law enforcement, etc. The list goes on.

What’s that? Just restore the machine from a backup? Well, depending on how long the machine was compromised, that might or might not be an option. If your backup strategy includes off-site backups on a monthly basis, how far back do you think you can go with what you have on-site? I suspect a couple of weeks at the most. And it’s going to take time to restore the machine. And can you ever be 100% sure that you know exactly when a security breach happened. Are you sure you restored from before they got in the door? Are you sure you closed off the holes so they can't do it again as soon as the machine comes back online? Many times, you’re much better off starting from scratch so that everything is a known quantity.

That’s going to take you several days, which means several days of downtime. Depending on the service, that’s not always an easy pill to swallow.

In a way, the security team is trying to help you make sure the IT services are available for your users. Sure, they’re creating work for you to do. But the cost of that work more like operational overhead, while the cost of a security breach is an “all hands on deck, drop EVERYTHING that you’re doing and help deal with this RIGHT NOW” kind of event.

Do you know how liability insurance works for security breaches? Generally, you get one chance. If you suffer a major breach, then insurance is going to cover it, but then your insurance premiums are going to skyrocket because you’ve proven that you’re a risk to the insurer. Most companies will survive their first security breach, but not all of them.

Suffer a second security breach and you’re almost certainly un-insurable. It’s going to be really hard to survive that because the insurance is only going to cover so much and your company is on the hook for the rest. You'll need to prove that you did everything right and there was nothing you could have done. Most companies don’t survive a second security breach.

Indirectly, this means that the security group is tasked with helping to make sure you still get a paycheck from one week to the next. It might be a good idea to work with them and pay attention to their recommendations, if for no other reason than job security. But there's the added benefit that keeping the systems up and running at all times is good for everyone.

What the security team needs to know about IT operations

The IT team already has plenty of things to do. Don’t drop a bombshell report on their desk and tell them ALL of the things that are wrong with the computers on the network. If you do, that report is going to be stuffed in a desk until the end of time… or until the company goes out of business after the IT team ignores everything you told them.

Instead, you need to start small.

Target just one simple thing at first. Something basic, like reviewing user accounts that are on all of the computers. Make sure those accounts are supposed to be there. If not, make sure they’re disabled or deleted.

Now that you have that list, create a process around remediating those things. The best thing to do is to attach this process to something like the patch management process. Ideally, there’s already a patch management process in place that involves scheduled maintenance and scheduled downtime of some kind.

This allows you to get security changes onto the schedule in such a way that it doesn’t impact the users of the system. If you don’t have scheduled maintenance windows, this would be a really good time to look into implementing them.

These maintenance windows don’t need to be extensive or invasive. Maybe an hour or two once each week or perhaps twice per month. Again, start small. Ideally, you’d be able to get to a point where you will have a daily maintenance window available, whether it’s used or not. However keep in mind that some systems you just can’t bring down on a regular basis like that.

Now that you have a set of maintenance windows

Build a process around notifying the users of your regular maintenance windows. Schedule that downtime and make the changes that you feel are necessary. Make sure that someone is responsible for approving changes that are going into your environment.

You don’t need to go out and purchase a change management system just yet. Tools aren’t the answer. Process is. You can use something as simple as an Excel spreadsheet to track changes being proposed in the environment. To do that, just track the following things:

  • Date Proposed
  • Proposed By
  • Description
  • Target Computers
  • Target Change Date
  • Testing Process
  • Rollback Process
  • Approved By
  • Approved Date
  • Result
  • Implementation Date

We're all on the same team

Conflicts between groups within an organization are fairly common. Fortunately, the thing to keep in mind is this. We're all on the same team.

Five Common Security Myths

Today, we start with just a little bit of education. Let's look at some common myths and misconceptions people have about server security.

Myth #1: My company is too small to be a target for hackers.
Reality: Most people believe this to be true, but the problem is that hackers are opportunistic people. It's true that hackers are probably not going to actively seek you out and try to hack into your servers. But the kinds of people who actively hack servers as a profession often don't target specific businesses. They run software that scans IP addresses and subnets, looking for valid machines. When they find one, the software tries to identify it, and then exploit that device. So whether you're a big company or not, it's almost a given that your servers are being scanned by the bad guys and they're trying to get in. They might not know who you are, but they don't care either.

Myth #2: I bought "such and such" software from Symantec/McAfee/Kaspersky/etc, so I should be good.
Reality: Security is a process, not a piece of software. Repeat that because it's extremely important to understand that. Anyone who tells you their software will protect you from everything is lying AND is probably selling something. There's not a piece of software in the world that will save your data if you set things up incorrectly. There's no substitute for using a structured process of making sure your systems are reasonably secure to begin with, and systematically verifying that it's still secure on a regular basis.

Myth #3: I'm already running a firewall and anti-virus software. I should be good.
Reality: There's no doubt that using anti-virus software and a firewall are critical components to helping to secure your servers, but it's not enough. Servers plugged into a network have holes in the firewall by design. If they didn't, they wouldn't be able to communicate with other servers or your users. Often, these firewalls are open for web server traffic, database traffic, remote access, or other types of incoming requests. This is often the entry point that hackers need to exploit misconfigurations in either the operating system or the applications running on those ports.

Is the operating system fully patched? How do you know? Did you check it today?
Is all of the software you're running patched? How do you know? Did you check it today?
Was a vulnerability recently discovered that you should know about because when combined with traffic over port 443 it can run arbitrary code? How do you know? Did you check it today?

What happens if an application is compromised and it creates a local user account or installs a system service that you didn't know about? How long would it be before you found out about it? A week? A month? Six months?

The entire time that account exists or service is installed, your data is at risk. Your customers' data is at risk. It could be credit card information, personal health information, email addresses, mailing addresses, phone numbers, etc. I know an email service provider that had a server breach and hackers stole lists of email addresses. It took quite a bit of time before customers of their customers started complaining about being spammed after signing up for Company XYZ's mailing lists. 

The longer you don't know about a breach, the worse it becomes. These are called long-dwell hackers. Firewalls and anti-virus software are a good start, but it's a bare minimum.

Myth #4: These servers aren't important enough to protect.
Reality: When a server is hacked, details of the server are often inventoried and sent off to the hacker. They then use that information to determine what other parts of your environment are connected to it and will use that server as a launching point. Since it's a part of your infrastructure, there is often some implicit trust from the other servers from this one. It could be as simple as using the same local administrator password on each server, or it could be firewall ports that are open when traffic is received from this server. When you remotely connect to a server, sometimes information is stored about who was connecting to it and from where. 

Hackers can often trace paths back through an environment once they get a foothold. Don't discount the importance of a server, simply because you don't have sensitive data directly on it. This is the same reason that a perimeter firewall isn't going to prevent all attacks against your servers.

Myth #5: I have a small team, so I know what's going on with my servers.
Reality: It's nice to be able to point fingers at who might have caused a problem, but in no way does that protect you from misconfiguring something on the server. All it takes is a single mistake and a hacker could be on your system. Even if it wasn't an accident made by an employee, did you know that when certain patches are installed, they run in kernel mode so they can make permissions changes to the file system to ensure that the patches are installed correctly? Did you also know that the patches often don't put the permissions back the way they were originally configured?

This can do one of two things. It could break an existing application that you have running. But far worse, is that it could introduce a vulnerability in the operating system which could then be exploited in the future. And unless you're regularly checking things like local users and groups, running services and installed software, it could be weeks, months, or even *gasp* years before you find out about it.

The software that hackers use these days has become so complicated that it can be really hard to detect. So how do you know that the removal utility that you used really did the job? The most widely accepted solution to fixing a server that's been compromised is to rebuild it from the ground up. I don't know about you, but I'd prefer not to spend my evenings rebuilding production servers.

Bonus Myth: You're selling something!
Reality: Not a myth. That one is true. I have a security product, but this email course is free and you're free to cross-check the information. Each year, Verizon publishes the Verizon Data Breach Report, which analyzes security breaches from around the world and compiles information from law enforcement agencies from around the world, including Australia, the Netherlands, Ireland, the UK and the USA. Here are some highlights of the 2012 Data Breach Report.

* 79% of victims were targets of opportunity
* 96% of attacks were not highly difficult
* 94% of data compromised involved servers
* 85% of breaches took weeks or more to discover
* 92% of incidents were discovered by a third party
* 97% of breaches were avoidable through simple or intermediate controls

"That data is more than a year old!" you might say. It's ancient in "internet years". Perhaps, but perhaps not. Check out the Verizon Data Breach Report for 2013, because it only gets worse.

* 66% of attacks took months or more to discover (note that they say "months", not "weeks")
* Increase of internal attacks from 4% to 14%
* Malware & hacking are still the most common, but social attacks have quadrupled
* Decrease from 855 breaches to 621, but more than 47,000 security incidents in 2013 did not meet the definition of a "breach"
* 48% of the security incidents were caused by user error
* Data included from Microsoft indicates that 50% of security breaches they're aware of are caused by configuration errors

If you're not scared yet, you should be. These threats are real and ignoring them doesn't make them go away. But the real question is: How do we fix this?

The first step is to do more to protect our data by establishing some security standards that you apply to your environment. If you're not using a set of security standards, how do you know what to fix in order to make it more difficult for hackers to get into your servers and cause damage to your business?
© 2011-2019 Moon River Software Inc. All rights reserved.