On January 23, Jeff Jones, Director of Security at Microsoft, published his “One Year Vulnerability Report” for Windows Vista. In the report, he analyzed whether Windows Vista had fewer vulnerabilities in its first year than it’s predecessor, Windows XP had in its first year. Jeff also compared Vista to Red Hat, Ubuntu, and Mac OS and how they did in their first year.
Predictably, the report has generated the expected amount of controversy. Thomas Claburn, of Information Week, promptly wrote an article about it, which, in my summary, essentially says “Microsoft makes up statistics to show that Vista is secure. Nobody else believes them.” Austin Wilson, another Director of Security at Microsoft, meanwhile, published a blog post about how good Vista is. Austin’s argument is largely centered around disproving the myth that Windows Vista made no real security advances over XP. In fact, I have heard several people claim that Vista has just as many security flaws as XP.
That lead me to wonder about Jeff’s One Year Vulnerability Report. It compares how Vista did in its first year, relative to how other products did in their first year. I would be very disappointed if Microsoft had made no progress in security issues from a product released in August 2001 to a product released in November 2006. Rather, I think it would be more interesting to know whether Windows Vista users had more security problems than Windows XP users, in the same time frame. That to me gets right at the arguments against Vista: that it does not provide better security. The fact that Vista had less problems in its first year than XP had in its first year is really not that relevant to me if I am an IT manager deciding whether to replace XP with Vista today. What I want to know is what I can expect in terms of vulnerabilities today, not what I had to put up with six years ago. Therefore, I set out to do some quick analysis on the current version of Vista versus the current version of XP. I picked, somewhat arbitrarily, the January to December 2007 timeframe, rather than Jeff’s December through November timeframe that corresponds with the Vista release plus one year.
Specifically, I wanted to see if some of the comments in the Information Week article were actually correct. The press has been extremely critical of Windows Vista. Infoworld, three days ago, even declared Vista the second biggest technology flop of all time, beaten only by security. Because of that I was very interested to analyze this for myself. I decided to start with the various claims in the InfoWeek article. InfoWeek is a respected publication. Clearly, they would not print claims that were unsubstantiated. However, there is precious little data in the article itself. Nevertheless, it contains quotes such as:
“When you start counting vulnerabilities, it’s a matter of defining vulnerabilities. For example, if a bulletin is released for Internet Explorer, that’s one patch for IE. Microsoft may have broken it out to say there are five distinct issues fixed in this patch. Is that five vulnerabilities or is that one vulnerability because it’s one patch?” – Eric Schultze, CTO, Shavlik Technologies
Well, Jeff actually does address that in the report. I have attached a spreadsheet to this post that shows all the bulletins Microsoft ascribes to either Windows Vista or Windows XP since Windows Vista shipped. As you can see, in the December 2006 to November 2007 timeframe covered by Jeff Jones, there were 17 bulletins issued that announced updates for Windows Vista. The updates announced in those bulletins covered 36 vulnerabilities as reflected by the CVE codes. As for the other operating systems, it is not clear if “vulnerability” equates to a CVE number in Jeff’s report, but since there are numbers quoted for both advisories and vulnerabilities, it seems logical that they do. I must conclude, therefore, that Eric Schultze’s claim is just a general statement, not a specific concern regarding the methodology in Jeff’s report. Eric, of course, was also one of the founders of the Trustworthy Computing Initiative at Microsoft before he left the company in 2002 to join Shavlik. One must assume he has some inherent pride in making sure that Windows XP, a product he helped secure both as a member of Trustworthy Computing Team and in his prior role in the Microsoft Security Response Center, is represented fairly.
Rather than get into the debate of how to compare vulnerability counts in Linux and Mac OS to Windows – a debate that seems to never die – I thought it was more interesting to see if Windows Vista users experienced fewer problems than Windows users on an earlier version of Windows. That would be a good measure of whether Windows Vista lived up to its promise – a promise that InfoWorld has already declared as failed. That analysis would also be a useful decision factor when contemplating whether to upgrade from Windows XP to Windows Vista. Clearly, while Jeff Jones’ purpose in his report was dual: to show Windows Vista’s superiority both to Windows XP and to competing operating systems from other vendors; Austin Wilson’s purpose in his blog post was to argue for the premise that Windows Vista users are more secure than Windows XP users.
Thus, let us turn to the comparison of various versions of Windows. In the section where the InfoWeek article discusses Vista in comparison to XP, Claburn again quotes Eric Schultze:
“What [Austin Wilson] states is accurate, but he’s only presenting the numbers that come out in a favorable light. He’s not presenting the numbers that come out in an unfavorable light. For example, he claims that there are a certain number of vulnerabilities for which, on Vista, there was lower severity than Windows XP. But he’s not telling you about the number of patches which were more critical on Vista than on Windows XP.”
OK. That is a fair criticism. Austin’s blog post does not address that. I will, however.
Another very interesting comment in the InfoWeek article came from Dave Marcus, at McAfee Avert Labs:
…”while Vista was superior to Microsoft’s previous operating systems from a security standpoint, many of the security features were only available in 64-bit versions of the operating system and many organizations would be disinclined to purchase new hardware to use those features.”
“We think 2008 will be the year that Vista finally joins the malware party”
Of course. McAfee, just like Eric Schultze’s employer Shavlik, are entirely reliant on security problems, real or perceived, in Windows for their business model. If, by some magic incantation, all security problems in Windows were to disappear over night, most of McAfee’s anti-malware business and Shavlik’s patch management business would go by the by as well. Do not be fooled for a second into thinking that anyone is impartial when it comes to security. In fact, I can virtually guarantee that this very blog post will get a number of comments about how I once worked for Microsoft and therefore am not impartial either. However, unlike both Schultze and Marcus, I do not derive revenue directly from Microsoft’s failures to provide completely secure products. Nor do I sell advertising based on the number of eyeballs you get because you make claims about how badly Windows Vista has failed on its promises, unlike InfoWeek.
Finally, the InfoWeek article closes with a quote by Eric Schultze.
“But [the security research community’s focus on Windows Vista] has yet to offer much clarity. ‘This is a matter of Microsoft bending the statistics for their own purposes,’ said Schulze. ‘We could just as easily create the same number of statistics that puts Windows Vista security in a negative light.'”
Well, let’s try to do that then, shall we?
Based on the claims in the InfoWeek article, we can extrapolate several very intersting, testable hypotheses:
64-bit Vista had fewer security vulnerabilities than 32-bit Vista
Windows Vista had more security vulnerabilities than Windows XP
Windows Vista’s security vulnerabilities were more severe than those from Windows XP
Windows Vista had the same number of security updating events as Windows XP
These are all derived from statements made in the article, either by Claburn himself, or by Schultze and Marcus, in relation to the apparent overall claim that Microsoft created statistics to show what it wanted to show. If you desire, you can postulate your own hypotheses and use the data in the attached spreadsheet to analyze them, or even compile your own data from Microsoft’s Security Bulletin site.
To analyze these hypotheses I created a spreadsheet (attached to this blog post so that anyone can verify the numbers for themselves) that lists all bulletins Microsoft issued for either Windows XP or Windows Vista since Windows Vista was released. I removed two bulletins from the comparison, which Jeff actually did count. The first was MS07-005, which affects Windows XP users that have the Step-by-Step training installed. Because it does not affect anything that ships with the operating system itself I thought it was not relevant. Next I removed MS07-055. It affects only Windows XP users that upgraded from Windows 2000, and the affected component is not one written by Microsoft. Finally, I did not count a bulletin that issued a patch for Services for Unix as an XP issue because Services for Unix does not ship with Windows XP. All of those revisions actually served to make Windows XP look better, which should make our hypotheses stronger.
Next, while I include all bulletins since Windows Vista was released in the spreadsheet, I decided to count them by calendar year. In other words, the hypotheses all relate to the experience of users of each operating system during calendar year 2007. I think that is a bit cleaner, and also makes Windows XP look better since there were no security bulletins issued for Windows Vista during 2006. I also included Windows XP Service Pack 2 with Internet Explorer (IE) 6 and 7 as two separate products. I thought that made more sense because the two are really quite different.
Finally, the analysis was done with Microsoft’s data from the Microsoft security bulletin web site. I have not reanalyzed any of the data to change criticality rankings. Nor have I gone to third-party sites to obtain data on issues that Microsoft may not classify as a vulnerability, or to get different vulnerability-product mappings. I rely entirely on Microsoft’s data. Thus, one possible shortcoming in the analysis is that I have trusted Microsoft’s judgement with respect to the criticality and applicability of an issue. However, as I, and probably you, already trust Microsoft to provide me with an operating system, I think that is quite fair.
Hypothesis 1 is the easiest to address: the data fails to prove it. There were exactly the same number of vulnerabilities for Windows Vista 32-bit edition as for Windows Vista 64-bit edition. Thus, while there are some security features, notably kernel protection and the requirement for signed drivers, that you only get with 64-bit edition, the security experience with the two was identical during 2007. In fact, the only security feature you cannot get with 32-bit Windows Vista that you get with 64-bit is the kernel patch protection.
The data also fails to support hypothesis 2, that Vista has more security vulnerabilities than XP. During 2007 there were 45 vulnerabilities for Windows Vista. There were 61 for Windows XP with IE 7 and 62 for Windows XP with IE 6. In other words, Windows Vista users had 26% fewer security vulnerabilities than Windows XP users with IE 7 and 27% fewer than Windows XP users with IE 6.
Hypothesis 3 states that the Windows Vista vulnerabilities were more severe than the Windows XP vulnerabilities. This also cannot be confirmed by the data. See this table:
|XP with IE7|
|XP with IE6|
The table shows the number of vulnerabilities of each severity rating for each operating system. All four had 2 low vulnerabilities. Windows Vista had the most moderate vulnerabilities. However, on the two higher categories, Important and Critical, Windows XP clearly led the way. Windows Vista users had 20% fewer Important severity issues than users of Windows XP with IE7; although they had two more than Windows XP users on IE 6. Windows Vista users had 41% fewer Critical security issues than Windows XP users with IE 7, and 55% fewer than Windows XP users with IE 6. The data, in fact, seems to support the exact opposite of hypothesis 3: that Windows Vista users had fewer high-severity vulnerabilities than Windows XP users.
Finally, hypothesis 4, that Windows Vista needed to be updated the same number of times as Windows XP, is also unsupported by the data. Windows Vista updates were issued on the following dates during 2007:
February 13, 2007
April 3, 2007
April 10, 2007
May 8, 2007
June 12, 2007
July 10, 2007
August 14, 2007
September 11, 2007 (this bulletin was for SFU, which does not ship with XP but is an optional component for Vista)
October 9, 2007
December 11, 2007
For Windows XP, updates were issued on the following dates:
January 9, 2007
February 13, 2007
April 3, 2007
April 10, 2007
May 8, 2007
June 12, 2007
July 10, 2007
August 14, 2007
October 9, 2007
November 13, 2007
December 11, 2007
In other words, Windows XP users had one more security updating event than Windows Vista users. It is certainly not a very large difference, however. While an IT administrator running Windows XP might have to analyze and test more updates than one running Windows Vista, the number of actual times they have to invoke the response process is quite similar.
From this analysis I think it is safe to conclude that the hypotheses are universally falsified. Windows XP users certainly were no better off than Windows Vista users. In fact, I would argue that the reverse hypotheses are proven by this data: Windows Vista users were subject to fewer vulnerabilities than Windows XP users. They had fewer updates to install, and marginally fewer dates when they needed to think about installing updates. Certainly this will change at some point, as more of the security research moves off of Windows XP, but for now, the data is quite clear: Windows Vista users have fewer vulnerabilities to patch.
But, I run Firefox. Am I more secure?
Many people look at this data and say that it is not relevant to them because they run Firefox as their web browser. I do on some of my computers because I really like some of the developer add-ins for it. Although, I still do patch IE even on those computers. Firefox depends heavily on OS functionality, as its history this year proved, so I would recommend you still patch everything that comes with the OS, including IE.
Nevertheless, let’s say you do run Firefox instead of IE, and you never touch IE, and hope that no vulnerable IE functionality ever touches Firefox. Would you be much better off? There are a couple more testable hypotheses:
5. Users who only use Firefox had fewer security patching events than those who use IE
6. Users who only use Firefox were subject to fewer vulnerabilities that those who use IE
Mozilla publishes very good data on vulnerabilities. They have been very open about this in the past, something they should be lauded for. The only problem I ran into was that I was unable to determine exactly when their fixes were published. Therefore, most dates in this section (those denoted by *) are derived from the date the release was code signed, not the date it was actually issued.
I also decided to look only at Mozilla Firefox 2.x. Firefox 1.5 support ended on May 29, 2007, about 18 months after it was originally released. Firefox 2.0 was released on October 24, 2006. These are quite short support cycles compared to Microsoft.
Another difference is that Firefox updates come as complete browser installations as opposed to only patches, like they do for IE. However, the impact on users is roughly the same. There were a total of 10 releases of Firefox 2.x in 2007, whereof 8 contained security fixes. I analyzed only those 8, under the assumption that if you only care about security you would not have installed the two that did not have security fixes (220.127.116.11 and 18.104.22.168). I did not count version 22.214.171.124 since it was issued in 2006.
I also decided to not count two vulnerabilities that were related to SSL 2.0 (MFSA 2007-06, CVE-2007-0008 and CVE-2007-0009) because SSL 2.0 is not enabled by default in Firefox.
Finally, Mozilla will, just like Microsoft, issue a single advisory with multiple vulnerabilities (as tracked by CVE codes). However, unlike Microsoft, Mozilla does not assign separate severity ratings to the different vulnerabilities. Therefore, in the case where one advisory covers multiple vulnerabilities I have used the severity Mozilla assigned to the advisory as the severity of each of the vulnerabilities in the advisory.
During 2007, Mozilla released the following updates:
126.96.36.199 – Nov. 26, 2007*
188.8.131.52 – Oct. 18, 2007*
184.108.40.206 – Sept. 18, 2007
220.127.116.11 – July 30, 2007
18.104.22.168 – July 17, 2007*
22.214.171.124 – May 29, 2007*
126.96.36.199 – March 19, 2007*
188.8.131.52 – Feb. 23, 2007*
The complete analysis is in the attached spreadsheet. The summary statistics are as follows:
|XP with IE7|
|XP with IE6|
Based on this, I do not believe we can say that either of the hypotheses were confirmed. In fact, both were falsified. Firefox users had three more patching events than users of IE on Vista, and two more than users of IE on XP. IE users on Vista had only 14% as many advisories to analyze as Firefox users, while IE users on XP had 19% as many. IE users on Vista had only 39% as many vulnerabilities to consider as Firefox users. IE users on XP had roughly 45% as many vulnerabilities as Firefox users.
The one bright spot for Firefox users is that, while Firefox users had more vulnerabilities overall than IE 6 users on XP, IE 6 on XP had almost 33% more critical vulnerabilities than Firefox. However, Firefox had 66% more critical vulnerabilities than IE on Windows Vista.
The first thing I would like to point out is that vulnerability counts is generally not a great final measure of overall security. For example, while it is clear from the data that Firefox had more vulnerabilities to fix during 2007 than IE did, most people still claim that IE remains more commonly attacked than Firefox. (I did not bother to go find data to back up that claim). Whether any of this makes you more secure if you use <IE|Firefox> depends in great measure on your surfing habits. Which brings me to the second point: a computer is largely vulnerable through its usage pattern.
Vulnerability counts do not matter if the user invites malicious code onto the computer. In general, patching the end user is considerably more difficult than patching either the OS or the browser. You can have a completely up to date computer, but if the end user installs malicious code, the computer is hacked. Whether that code was downloaded by Firefox or IE matters little. The actual delivery vehicle matters only insofar as the malicious code was automatically installed through some vulnerability. In short, security updates and vulnerabilities are but one component of risk, but I would not even go so far as to say that it is a measure of risk because there are many other components that need to be considered.
What vulnerability counts do, however, measure quite well is workload. For an IT department that was managing Windows Vista, the workload to analyze, test, and deploy security updates during 2007 was lower than for an IT department that managed Windows XP with IE 6. The workload for an IT department that managed either platform went up if they also had to manage Firefox.
Finally, do not forget manageability in this discussion. I would argue that the additional security features in Windows Vista, such as ASLR, BitLocker, UAC to help me run as a non-administrator, Service Hardening, Protected Mode IE, and so on, all contribute to more security. In fact, they probably contribute as much or more than the sheer reduction in vulnerabilities, which is good for all of us, whether we run Firefox or IE, as long as we run Windows.