Throwing Water on the Fire

Looks like we’ve got another worm on our hands.  Many Administrators are getting sleep tonight because they know their systems have been patched for a bit now.  They have set up the proper processes and/or technologies throughout the enterprise to be proactive and patch before something like this happens.  But what about those Administrators that haven’t patched their systems yet (this isn’t necessarily a failure on an Administrators part but could have been a political, business, or other issue that caused this) and are stuck fighting the worm?

There are good practices and procedures that should be followed for remediation across the Enterprise as well.  Similar to how you handle Management across your systems, if you aren’t organized with a good plan and using the proper tools you will be spinning your wheels during remediation as well.

The first step that needs to be done is organization and ownership of resources.  Many times large organizations have clients and servers spread across different buildings and groups that are assumed to be managed however are not.  The best step in this process is to have a contact person or ‘Incident Owner’ that can designate which groups are responsible for what machines and when they need to address them by.  Usually this person in a large company is a member of the Security Team and also owns other processes such as vulnerability scanning.

After organization and delegation of duties the method of worm/virus cleaning and patching should be agreed upon as well.  Usually most large Anti Virus vendors provide tools to clean systems but a good move more recently has been made by Microsoft to provide tools as well for this process.  Almost all come with silent command line switches so you can easily incorporate them into an SMS package and distribute remotely however another option is to use a scripting language such as VBS or Perl to loop through systems, copy the tool locally and execute it (such as seen here).  Its important to note that a mistake made by many is to immediately disconnect or ‘blackhole’ vulnerable systems from the network when an outbreak occurs in order to contain it.  Unless the number of unpatched systems is small, this is usually the most counterproductive move that can be made.  By doing this you will not only impact business greatly but also cause your desktop support personnel and possibly your Administrative staff to ‘Sneakernet’ there way to possibly hundreds of systems to clean and patch them.  A more effective means of containing a worm like this is to attempt cleaning and patching an infected or vulnerable system remotely (as stated previously with SMS or a script – the tools are there for you to do this!) before removing it from the network.  Not only does this save workload from your staff but can also be much more efficient in resolving the problem faster.  Another strategy depending on the size of the outbreak is to only disable infected systems while still continuing remote patching efforts.

Eventually there will almost always need to be a physical visit to some systems when you have an outbreak.  Usually your Desktop Support Staff will be dispatched to clean the virus and patch the system.  I cannot stress enough here how important it is here to communicate instructions on how to do this to all vested parties.  Many times Administrators will understand a worm and create tools to remediate those systems however never communicate this information to the support staff that may need it the most. 

Finally I’d like to talk about scanning tools.  If you are in a pinch here you may not have the proper tools to find vulnerable systems still left on your network.  Foundstone has a great one out but its not the best for reporting.  There are also the new MBSA scripts which have awesome reporting but the performance isn’t that great.  Just remember to keep scanning your networks for a bit after remediation efforts are complete so that you know you’ve put this beast to bed.

After all this schedule a post mortem-type meeting with all parties to discuss improvements for next time – especially why your Patch Management tool/process didn’t cover all your systems!