The study’s methodology is however, greatly flawed at several key points, making its conclusions open to interpretation which should be the case when making such comparative tests.
For starters, NSS Labs undertook a rather minimalistic approach towards the definition of web malware. In this study, the malware URLs they’re using are basically “links that directly lead to a download that delivers a malicious payload“, a decision that directly undermines the statement of “block rate” in times when client-side vulnerabilities are massively abused courtesy of web malware exploitation kits. And since no live exploit URLs were taken into consideration, the DEP/NX Memory Protection feature within IE8 was naturally not benchmarked against known exploits-serving sites, or at least wasn’t mentioned in the report.
Moreover, the competing browsers’ use of SafeBrowsing’s API, a combination of automatic (honey clients) and community-driven efforts to analyze a web site in a much broader “malicious” sense has a higher potential to maintain a more comprehensive database of known badware sites. It also comes as a surprise that Firefox, Safari and Chrome have such a varying block rates given that the browsers take advantage of the SafeBrowsing project’s database. Basically, having a set of ten malicious URLs and running it against the browsers is supposed to return identical results due to the centralized database of known badware sites.
Interestingly, the study used Apple Safari v3 in order to come up with the 24% block rate, which excludes the built-in anti-phishing and anti-malware features introduced in Safari v4. The report is released prior ot IE8’s debut, but even if NSS’s study is in fact relevant in a real-life attack scenario, does it really matter that IE8’s outperforms the rest of the browsers in times when IE8 users are downgrading to IE7? That very same IE7 which according to the study is offering “practically no protection against malware”?
Anyway, consider going through the report, with a salt shaker in hand
See the blog of IE on the study: http://blogs.msdn.com/ie/archive/2009/03/25/ie8-security-part-ix-anti-malware-protection-with-ie8-s-smartscreen-filter.aspx