Does Microsoft think SMS Software Packaging is dead?


A few weeks ago I was doing Q&A at the SMS booth with some Microsoft employees at TechEd and a woman approached us about the best way to roll out a particular piece of software. If I remember correctly, she currently uses InstallSheild to call the vendor’s silent installation routine but her question was in regards to the package reporting fields within the SMS Administrator’s console. After she walked away the comment was made to me “People still package software before rolling it out, huh?”.


WOW. I was blown away that there are people inside the SMS Product group don’t really think that packaging isn’t a major piece of a solid SMS Infrastructure. I’d actually go out on a limb and say it’s one of the most critical pieces of a good SMS implementation. This made me do a little research into some of the documentation that’s out there for SMS2K3 to find some information on software packaging. Low and behold there really isn’t much out there. In fact this is pretty much all there is:


 


SMS Installer


You can use SMS Installer to create an executable file that you can add to a package and advertise to clients. SMS Installer creates a self-extracting file or Windows Installer file that includes the data and files for the software application and the installation script that you created using SMS Installer. By using the SMS Installer Script Editor, you can modify the installation script that SMS Installer creates.


SMS Installer does not create the package, distribution points, or advertisements within SMS, so you must use another method to perform these tasks. SMS Installer creates a package definition file that can be imported into SMS with either the Distribute Software Wizard or the Create Package from Definition Wizard. For more information about SMS Installer, see Chapter 7, “Creating Software Installation Packages with SMS Installer.”


Really when I think back at all the Microsoft demo’s and conference sessions you almost always see them call an ISV’s silent installation directly in the program command line or an MSI. Now I know that there are many people within the SMS Product group that don’t think app’s should be deployed like this. In fact I was even talking with one of them (names not divulged to protect the innocent!) when I was in Redmond and he went on for about 20 minutes at lunch telling me about one customer that started taking people with high levels of software development in their backgrounds and made them packagers for SMS. When this started happening the quality of packages and success rates of deployments went up by a dramatic level. Why is this? IMHO for a couple of reasons:


1. Developers are usually more detail orientated. They make sure that a package is fully tested across all platforms and all bugs are worked out before deployment in the same manner that they would test a new product that they developed before selling it to a customer.


2. Developers account for unknown variables in the environment. Just sending out a package isn’t enough as you need to think about what might cause your installation to fail as well as fallback on experience of things you know have impacted your installations in the past.


There are also other factors which tell me not everyone believes software packaging is dead. Take a look at the SMS2K3 SDK. If the SMS Product group didn’t think that packaging was still important why would they create functions like InstallStatusMIF?


And what about the line of business applications that don’t support silent installation parameters? Smaller organizations with budget constraints are stuck with using SMS Installer for repackaging which, lets face it, is getting to be an antiquated tool that hasn’t been updated in quite some time. They don’t have the resources to put towards an enterprise packaging toolset like Wise or InstallSheild in order to do quality deployments.


Time for an update!


So what needs to change? I think its Microsoft’s perception of the SMS Administrator to be perfectly honest, as it is the SMS Admin’s toolsets have needed to grow for SMS 2003. The days of the 2.0 Admin are gone with just needing SMS and some domain knowledge. SMS 2003 ties in more closely with AD for one so knowledge of LDAP communication with the directory, etc is needed (and don’t even get me started on the amount of time people spend troubleshooting MP communication issues because of SPN registration issues or Computer access/authentication problems) as well as IIS configuration and so forth. Its really funny when I see a vendor at a trade show talking about how their management technology can solve all of your problems because they have all these built in reports, etc, so you won’t need to waste time on that stuff. No way any ISV can do that. The purpose of reporting is so it can be customized for your organization so the knowledge to do that stuff has to be in-house, especially with really large organizations. The same thing goes for Software Packaging. Really good packaging teams have reporting built into their installers to make troubleshooting easy and reporting integrated into your SMS site. Some common things done to improve software packages are:


– Standard package logging location with easy to follow flow and error messages


– Registry reporting for inventorying


– Custom SMS Status Messages


– Package Versioning for tracking of bugs


– Anti-Virus Software stopped before installation and restarted after (a common recommendation from ISV’s)


– Checking for backups (in the case of major updates like Service Packs)


Most importantly however is the ability to include checks in your scripts for problems. An example would be a problem I ran into with SP4 for Win2K. We had this application called Win Driver which placed a library, Windrvr.sys I believe, in the %sys32%\Drivers directory. Turns out that on every system if it wasn’t at the latest version the system would blue screen. Now, if you have 200+ machines that this application is on that could be an issue. So what does a good software packager do here? He (or she) can check to make sure that the file version of Windrvr.sys is recent enough and if its not either abort the installation (and use the good packaging practices above to log this efficiently and report back to SMS) or automatically update the file if its not in use and continue with the SP4 installation.


That’s just one of many examples that can be done through good packaging which is really one of the most lucrative pieces of SMS where effective enterprise management can be seen. Its just a shame that Microsoft won’t be providing the tools to do this in the foreseeable future so we are stuck with limited functionality or the purchase of high-end (read: expensive) tools to get the job done. Additionally these 3rd party tools have no impact into the future versions of SMS and integration into the management structure. Yeah, many are gold level partners but lets not kid ourselves here – Microsoft has the biggest impact on Microsoft so if you want to see some cool integration with a software packaging tool they most likely need to be doing it. Why not SMS Installer 3.0 with some new actions like:


– Add new Advanced Client Policy


– Compile data class to WMI (yes! No need to copy your custom sms_def.mof to the client and compile)


– Initiate HINV/SINV


– Initiate Machine Policy Retrieval Cycle


– Update Get System Information to get the SMS Site, Default MP, AD site, etc


When I think about what a major complaint from many Microsoft customers is about SMS, I consistently hear that Software Distributions can be difficult. Now to be fair they aren’t claiming there are other products that are hands down easier, just that its problematic with SMS. If the scripting interface were more robust and easier to configure anyone would be able to create awesome packages that could be deployed with great success.


 


 


 

New SMS SUSFP Guide

Some may know already, but I put together a guide consisting of a compilation of the SUSFP knowledge from the community.  You can get it here.  What an awesome experience I’ve had making this guide!  Essentially this is the culmination of about 2-3 years using SMS for patch management with and without the SUSFP.  I’ve had the good fortune of getting known as the “SMS Patch Management Guy” in the SMS Community which has not only pushed my SMS skills to their limit at times but also given me the opportunity to work with A LOT of different people across a multitude of environments.  Thinking back on this now, I guess I kind of fell into the role by being the first one to backwards engineer a lot of what the Microsoft folks did for the sheer fact that the geek inside me needs to always tear apart stuff and know what’s under the hood.  I also have been very lucky in meeting the right people behind the scenes who give me answers to questions to made me look like a smart guy and who I will never be able to thank enough, as well as the other SMS Jedi that hang around the community and have helped me along the way.  The best part about this has been all the good friends I’ve made by helping them come to grips with the SUSFP and develop comprehensive PM strategies for their organizations.  I can’t even count the number of people that I’ve talked to on different lists, forums, newsgroups, conferences, offline emails, and even phone conversations in the past year about the SUSFP.  This guide is really that – a summary of everything I’ve learned and experienced with this toolset over the past couple of years for any people that I haven’t reached yet and to get them up to speed quick.  Mostly I’d like to thank the myITforum.com community and those dedicated to supporting the SMS community as a whole because without you this wouldn’t exist!


 


Take this guide to be exactly that – a guide.  As with any documentation it may not contain all the answers you are looking for in your environment or be 100% accurate but hopefully will point you in the right direction.  I hope to do one in the future for SMS 2003 as well, but many of the tools and process in this guide can be used for 2003 for the time being.

I’m Back with TechEd Notes!

Back from vacation now – sorry its been quiet here for a couple weeks. I actually split my time between relatives visiting in town and helping out the SMS Product group by manning the SMS booth at TechEd last week (oh yeah and a 1 night trip to Vegas). What an interesting experience at TechEd! My name tag actually listed that I worked for Microsoft along with having to wear one of the traditional geeky blue shirts so everyone thought I worked for Microsoft 🙂 Its really interesting seeing how fast people do a flip-flop when talking to you and they find out you aren’t a Microsoft employee – for whatever reason its like you get more credibility because they don’t think you are another member of the Borg. I also had people coming up to me left and right just to vent about a failed SMS 2.0 implementation or go on about some struggles they were having with their SMS2K3 deployment. Many complaints were just about political issues within their organizations and not technical at all. I think it just says that people like to talk to others about their problems and get things off their chests. What I got a kick out of the most were people that came up to me saying “you’re probably just a salesperson that doesn’t know a deep technical answer to my question, but let me ask it anyway” and then I’d actually give them a deep technical answer back and watch their eyes glaze over. It was a lot of fun though hearing some real-life issues that people had and working to give them ideas for resolution. The face-to-face stuff like that is a lot of fun because you get past the awkwardness that sometimes happens in email or public forums.

Overall I’d have to say that SMS2K3 has a really good name right now – much better than 2.0 ever did. I even had one guy give me a success story about his 20K client infrastructure upgrade from 2.0 to 2K3 in one weekend without any issue. A lot of organizations are also making the move from a competing management product such as Tivoli, Altiris, etc. Even more interesting was hearing that some people were rolling out SMS2K3 just for the servers they manage and letting the desktop team use another product for workstations. Now that’s a change in mentality!

MOM was generating an insane amount of buzz at TechEd as well. There were times when 10-20 people were waiting in line at the product booth to get a question answered from the MOM team (actually questions and interest in the product, not just to get Zoo tickets). This is another example of how seriously Management technologies are being taken today in companies of all sizes. Administrators seem to be realizing how these products tie into the general direction of IT today: Lifecycle Management, Security and Patch Management, etc. So what does this mean? A lot if you have expertise in the area. I foresee a big need for knowledgeable professionals in the field of Management going forward. Not only will you see a need to debunk a lot of bad information that will be out there by overnight experts but more theory, process, and techniques will need to be developed in order to adapt to all those companies out there that will be embracing SMS, MOM, and the like going forward. For me its looking like an exciting time going forward.

Fasten your seatbelts folks – hopefully the ride isn’t too bumpy!

Uggghhhh!!!!! My Sync Tool isn’t working either!!!!

Man oh man doesn’t it just suck when you work so hard on automating your Patch Management infrastructure and its someone else’s software configuration screwing up that is causing you headaches. Again this month reports of peoples Security Update Bulletin Catalog (MSSECURE.CAB) aren’t downloading the latest copy. This is due to ISP’s caching the old one. Unfortunately the only way to fix is to call up your ISP, tell them to clear the cache, and tell them (try and be nice) to stop caching going forward.

Now….I know there are a lot of smart fellers up there in Redmond. A lot of them are buddies of mine. Perhaps there is something we can do in the future in the Sync Tool code to work around this issue (wink, wink, nudge, nudge). 😉

Anyone in the community got an idea? I’d love to hear them? How about renaming the cab to reflect the month? Send them to me or post them!

Reaction Times

Thinking about feature sets in products, its really evident that there will always be give and take on both ends. How much is to much though? Back in the day SMS Administrators used to have a pretty rough time creating big packages for patch management. There was detection of the OS, seeing which updates were needed, installing the proper ones, verifying that they were installed, etc, etc, etc. What a pain that was. I find myself asking though – are we better off having simplicity given by Microsoft into the product if the accuracy is lower than using our in-house developed tools?

Don’t get me wrong I LOVE the new Software Updates features of SMS 2.0 and especially the UI interaction in SMS2K3. But I can also throw down with the best of them when it comes to packaging and scripting. And my Hotfix packages never used to have the detection problems like we see today with the MBSA. So basically what this means is you have a bunch of very technical scripters and packagers that used to code and are now given a solution set from Microsoft because of product feedback but now when there is an issue with detection, etc, they have to sit in front of management and explain that although their jobs are easier to do now they have more hassles to deal with.

Not only that, but add to the fact that its not always that there is an MBSA detection issue but many times it flat out can’t detect it and you are no better off than you were before. How long has the SUSFP for 2.0 been out now? Over 1 1/2 years? Personally I’m a person that does something all or nothing and expect the same from members of my team. Its very frustrating when I’m paying money for a product and it only does 85% of what it claims to do.

And I’m not just slamming MS or the MBSA here. Shavlik and other vendors have the same trouble (case in point, no one detects OE or IM updates which is a joke since OE is part of the Operating System really). Yes I realize that file checksums and registry checks must work together to make sure everything was installed fine, but what would rock would be a system that you could bypass one or the other and use the intrinsic SMS functions to detect needed updates. Like instead of SMS_DEF.MOF editing if the Software Updates Scan Tool had an INI file that could have lines added for registry checks that would roll up into SMS.

I just hate having something that only works 85% of the time when I could do it myself (yes with a bit more work) and be more accurate.  Microsoft and other ISVs know about these issues plauging the field.  Its about time to react and fix them instead of contstantly hearing about roadmaps.

Bogus TCO

A lot of discussion goes into TCO for organizations.  Choosing this Management product over this other one because it does X,Y, and Z while the other only does X and Y will produce a lower TCO.  Running this Operating System as opposed to this other one will give you a lower TCO in this scenario.  Who came up with this new age version of Voodoo Economics?  IT departments need to constantly get feedback from their user community as to what they need to do their jobs better.  They should actually be going out of their ways to learn the daily processes and job functions of the customer base they are supporting so that they can be better informed to make decisions that will benefit the company to the greatest extent.  Does having a snazzy web reporting module showing disk queue length for all your servers really benefit your customers?  Does this justify paying more for a product when the other one does the same thing through an Administrators Console?  How about Patch Management features in your product?  Does it allow you to customize patching, reboots, etc around your customer’s schedules or does it abruptly interrupt work?  Doesn’t this work stoppage work against TCO and in fact raise it?  Its all about the ‘X’ factor.  Management of your enterprise is about having product sets that are flexible enough to adjust to your organizations needs.  Leave the bells and whistles for Hallmark.

Throwing Water on the Fire

Looks like we’ve got another worm on our hands.  Many Administrators are getting sleep tonight because they know their systems have been patched for a bit now.  They have set up the proper processes and/or technologies throughout the enterprise to be proactive and patch before something like this happens.  But what about those Administrators that haven’t patched their systems yet (this isn’t necessarily a failure on an Administrators part but could have been a political, business, or other issue that caused this) and are stuck fighting the worm?

There are good practices and procedures that should be followed for remediation across the Enterprise as well.  Similar to how you handle Management across your systems, if you aren’t organized with a good plan and using the proper tools you will be spinning your wheels during remediation as well.

The first step that needs to be done is organization and ownership of resources.  Many times large organizations have clients and servers spread across different buildings and groups that are assumed to be managed however are not.  The best step in this process is to have a contact person or ‘Incident Owner’ that can designate which groups are responsible for what machines and when they need to address them by.  Usually this person in a large company is a member of the Security Team and also owns other processes such as vulnerability scanning.

After organization and delegation of duties the method of worm/virus cleaning and patching should be agreed upon as well.  Usually most large Anti Virus vendors provide tools to clean systems but a good move more recently has been made by Microsoft to provide tools as well for this process.  Almost all come with silent command line switches so you can easily incorporate them into an SMS package and distribute remotely however another option is to use a scripting language such as VBS or Perl to loop through systems, copy the tool locally and execute it (such as seen here).  Its important to note that a mistake made by many is to immediately disconnect or ‘blackhole’ vulnerable systems from the network when an outbreak occurs in order to contain it.  Unless the number of unpatched systems is small, this is usually the most counterproductive move that can be made.  By doing this you will not only impact business greatly but also cause your desktop support personnel and possibly your Administrative staff to ‘Sneakernet’ there way to possibly hundreds of systems to clean and patch them.  A more effective means of containing a worm like this is to attempt cleaning and patching an infected or vulnerable system remotely (as stated previously with SMS or a script – the tools are there for you to do this!) before removing it from the network.  Not only does this save workload from your staff but can also be much more efficient in resolving the problem faster.  Another strategy depending on the size of the outbreak is to only disable infected systems while still continuing remote patching efforts.

Eventually there will almost always need to be a physical visit to some systems when you have an outbreak.  Usually your Desktop Support Staff will be dispatched to clean the virus and patch the system.  I cannot stress enough here how important it is here to communicate instructions on how to do this to all vested parties.  Many times Administrators will understand a worm and create tools to remediate those systems however never communicate this information to the support staff that may need it the most. 

Finally I’d like to talk about scanning tools.  If you are in a pinch here you may not have the proper tools to find vulnerable systems still left on your network.  Foundstone has a great one out but its not the best for reporting.  There are also the new MBSA scripts which have awesome reporting but the performance isn’t that great.  Just remember to keep scanning your networks for a bit after remediation efforts are complete so that you know you’ve put this beast to bed.

After all this schedule a post mortem-type meeting with all parties to discuss improvements for next time – especially why your Patch Management tool/process didn’t cover all your systems!

Single Point of Failure?

We posted an interesting survey yesterday on myITforum to see if certain Administrators were the Single Point of Failure for Patch Management in their company.  A surprising amount of people responded that they were, and an even more surprising amount sent me emails offline about it.  Some responded to me saying that it has been 3-6 months since a patch had been deployed at their organization because they were taken from their SMS responsibilities and on to something else!  At first I was astounded by this.  In today’s day of the SUSFP and the native Software Updates features in SMS 2003 this should be easy right?

Well, maybe not.  I got to thinking about this and while this may be the case for organizations with (and I’ll ballpark here) 5000 or more clients that are able to have multiple staff members for their SMS infrastructures its not always the case for smaller ones utilizing SMS with just a single Administrator.  If I were the single SMS Admin at a smaller site I wouldn’t be that happy if my boss was calling me to create a patch package while I was on vacation because I was the only one, or even worse if I had to schedule my vacations around the Tuesday patch release schedule.

So what’s the answer to this?  A couple of things.  One is possibly product improvements for SMS Software Updates features.  Yes the DSUW integration into the UI is a great feature but maybe a simple web page tied into the SMS Provider that can do the same thing for managers, etc, to use in an emergency with minimal training would suffice as well?  This wouldn’t be the tool you use to manage your entire Software Updates management strategy but it would allow others to be able to secure systems while you are away.  Along with this would be better status messages for non-SMS Admins to read when they used this tool set.  Seeing “A non-zero Exit Code was returned” means nothing to a manager, or even a technical person not familiar with SMS.  In a lot of ways SMS does a good job at making simple things difficult such as status messages.

Some things that are outside of the SMS product groups control could be improved upon as well to help this situation.  One is standardization of hotfix installers.  This process is already happening inside Microsoft but to the chagrin of us all it is happening much to slow.  Once this happens it may lead to the other improvement which is standard silent hotfix command line parameters across the board so that the non-SMS Amin can easily distribute updates when they are OoO.  The telling fact about this is that many new SMS Administrators have failed distributions because if incorrect command line switches.  Yes they aren’t doing the correct thing by not testing switch functionality first but news flash:  They are correct in thinking most, if not all, Microsoft generated hotfix installers should act the same.

Hopefully as the hotfix installer technology catches up with the rest of Microsoft products, SMS and other Software Update Management technologies like WUS can have even more robust tool sets.

New Management Blog

Welcome to my new Blog covering Management technologies, processes, and thoughts. Hopefully this can serve as another avenue to spread ideas in this area on what others in the industry are doing and also deliver new information as it happens.

So what are Manageability Products? Traditionally these are products such as SMS, Tivoli, MOM, etc that enable Administrators to better ‘Manage’ their network resources and lower TCO. This area, at least as Microsoft goes, seems to be expanding more into other areas. Take authentication now. AD has many Intellimirror features that give some organizations a small and limited subset of Manageability like Software Distribution or control through GPO, etc. Its funny how products like SMS traverse many fields like Security and Auditing and really shows the power and control that Administrators have over their networks.

Management Products Rule. There are a lot of issues with them though such as deployment, configuration, and knowledge that have to be overcome for success. With proper research, testing, and learning by your staff most products should be successful in any environment. I often find myself wondering in this area if Administrators are driving the direction of the products or if vendors themselves are. Many times I see new features added or changed and I have no idea where the thought process for this comes from. The real answer for the most part is that customer requests (and demands) force product change over time and vendors can only hope that this leads to a more robust feature set. It seems though that Administrators of larger networks are 180 degrees different than developers in that they resist change even if they asked for it. This is a big reason why deployment of products like SMS fail is because of the closed-mindedness that many Administrators have. If you read, learn, and ask questions first rather than argue and belittle technology your grasp of it will come much faster.