Signup/Sign In
Ask Question
Not satisfied by the Answer? Still looking for a better solution?

How do I deal with a compromised server?

Canonical Version
I suspect that one or more of my servers is compromised by a hacker, virus, or other mechanisms:

What are my first steps? When I arrive on site should I disconnect the server, preserve "evidence", are there other initial considerations?
How do I go about getting services back online?
How do I prevent the same thing from happening immediately again?
Are there best practices or methodologies for learning from this incident?
If I wanted to put an Incident Response Plan together, where would I start? Should this be part of my Disaster Recovery or Business Continuity Planning?
by

1 Answer

pankajshivnani123
It sounds like are in slightly over your head; that's ok. Call your boss and start negotiating for an emergency security response budget. $10,000 might be a good place to start. Then you need to get somebody (a PFY, a coworker, a manager) to start calling companies that specialize in security incident response. Many can respond within 24 hours, and sometimes even faster if they have an office in your city.

You also need somebody to triage customers; Doubtless, somebody already is. Somebody needs to be on the phone with them to explain what is going on, what is being done to handle the situation, and to answer their questions.

Then, you need to...

1. Stay calm. If you are in charge of incident response, what you do now needs to demonstrate the utmost professionalism and leadership. Document everything you do, and keep your manager and executive team apprised of major actions you take; this includes working with a response team, disabling servers, backing up data, and bringing things online again. They don't need gory details, but they should hear from you every 30 minutes or so.

2. Be realistic. You aren't a security professional, and there are things you don't know. That's ok. When logging in to servers and looking at data, you need to understand your limits. Tread gently. In the course of your investigation, make sure you don't stomp on vital information or change something that might be needed later. If you feel uncomfortable or that you are guessing, that's a good place to stop and get an experienced professional to take over.

3. Get a clean USB stick and spare hard drives. You will collect evidence here. Make backups of everything you feel may be relevant; communication with your ISP, network dumps, etc. Even if law enforcement doesn't get involved, in case of lawsuit you will want this evidence to prove that your company handled the security incident in a professional and appropriate manner.

4. Most important is to stop loss. Identify and cut off access to compromised services, data, and machines. Preferably, you should pull their network cable; if you cannot, then pull the power.

5. Next, you need to remove the attacker and close the hole(s). Presumably, the attacker no longer has interactive access because you pulled the network. You now need to identify, document (with backups, screenshots, and your own personal observational notes; or preferably even by removing the drives from the affected servers and making a full disk image copy), and then remove any code and processes he left behind. This next part will suck if you don't have backups; You can try to untangle the attacker from the system by hand, but you will never be sure that you got everything he left behind. Rootkits are vicious, and not all are detectable. The best response will be to identify the vulnerability he used to get in, make image copies of the affected disks, and then wipe the affected systems and reload from a known good backup. Don't blindly trust your backup; verify it! Repair or close the vulnerability before the new host goes on the network again, and then bring it online.

6. Organize all of your data into a report. At this point the vulnerability is closed and you have some time to breath. Don't be tempted to skip this step; it is even more important than the rest of the process. In the report, you need to identify what went wrong, how your team responded, and the steps you are taking to prevent this incident from occurring again. Be as detailed as you can; this isn't just for you, but for your management and as a defense in a potential lawsuit.

That's a sky-high review of what to do; most of the work is simply documentation and backup handling. Don't panic, you can do that stuff. I strongly recommend you get professional security help. Even if you can handle what's going on, their help will be invaluable and they usually come with equipment to make the process easier and faster. If your boss balks at the cost, remind him that it's very small when compared to handling a lawsuit.

You have my consolations for your situation. Good luck.

Login / Signup to Answer the Question.