The state of patching

On February 19, 2007, in Security, by

“I had a meeting with a partner last week…. they said that they can’t deal with the patch takes them three weeks to recover from patch breakage….so they shut off automatic updates”

“The recommended fix, if the problem reappears, is to turn off automatic updates, and just manually go to Windows Update periodically and update your machine.”  (no the recommended fix is KB927891)

Microsoft Update trying to ruin our machines?”

This week I was asked if the monthly patching change that was announced by Steve Ballmer back in October of 2003 was working… and in general I think it is…..there are advances in patching in Vista now that we certainly didn’t have back then…but I don’t think I’d be speaking out of turn if I didn’t say that patching isn’t enough.  Oh sure for those that do patch it’s easier that it’s a monthly patch schedule, but the problem with the typical VAR/VAP was evidenced the other day when someone was having issues with HP director software that suddenly “stopped working”.

Was it due to patches?  He couldn’t say for sure as he didn’t have complete and total control and information about the timing of the “when it stopped working”.   Unless you have enough monitors in place, it’s hard to ‘debug’ the underlying cause or issue.

Look how many times these days I’m telling people to say “go check the ASP.NET tab in IIS and make sure the default web sites of SBS stay “default” on .NET 1.1 and if they’ve accidentally flipped to .NET 2.0 due to the installation of a patch or R2, flip it back.

Just like the case of the VIA storage driver that looks like it came down on Friday, we can say in a haughty voice “oh sure, veteran patchers don’t install hardware drivers”… but … doesn’t that imply that you need expertise to run a computer to know what to do?

Do I have solutions?  Nope.

Other than to say and give you a few of those “experienced patcher” guidelines.  I’m sure you have your own standards and routines:

  • When patching a server, reboot to ensure any installer routines left behind are cleared out.

  • Have a good backup.

  • Never install drivers from Windows Update.

  • If you don’t need the application, remove it.

  • In general, most patches do not cause issues, so when issues occur, read the bulletins.  They should give a clue of the type of issues you might see with them.  IE patches will impact Web applications.  But I’ve seen too many times normal server/network issues being blamed on patching when there’s no way that the patch could be the cause.  When an issue occurs, go back and read the bulletins.

  • If you are a DIYer I’ll forgive you for saying “I don’t know what got installed on my systems”.  If you are a var/vap and one offering up a managed service, KNOW.  Control them.  If you don’t want to patch, then use mitigations, group policy, non administrator and lock those systems down.

  • Know which systems are expendable, and which ones aren’t.  Most experienced patchers won’t let servers install automatically, but will let desktops install automatically.

  • Understand that every day, every hour you have “change” occuring in your network with antivirus changes, thus know that your only change in your network isn’t “just” patching.

  • With the recent zero days, understand that many (all) are very targeted and as Scottie would say “Cap’tan the shields are holding”…that is to say, while there are some specifically targeted, the vast majority of folks are not and thus we should review the risks accordingly.

“Patching is critical, but patching is insufficient, for all the reasons I highlighted in terms of the speed with which new vulnerabilities are coming out. What we really want to do is make our customers resilient to attack, even when patches are not installed. Does that make sense? You should be able to have a kind of perimeter around you that protects you so that you can install patches on your own schedule — I’m not saying patching becomes irrelevant — but you should be able to apply patches on your own schedule, not on the schedules of the hackers.

Our goal essentially is to make seven out of every 10 of the patches we’ve ever done or ever will do installable when you want to install them, as opposed to us putting out a bulletin that says, “Now, now, now!” We can say if you have this perimeter defense in, you’re okay. If you have this safety measure in place, you’re okay — you can install this at your comfort within the next month or so. “

Steve Ballmer at WWPC – October of 2003

Well Steve… we’re not there yet…


One Response to The state of patching

  1. Vlad Mazek says:

    Whatever happened to that “bloggers ask” trip you were on last week? ๐Ÿ™‚ And you pasted in our personal conversation WITHOUT permission and without attribution!

    I demand credit and a percentage of revenues this post brings in! ๐Ÿ™‚