Spyware Sucks
“There is no magic fairy dust protecting Macs" – Dai Zovi, author of “The Mac Hacker’s Handbook"

Gone Phishing: Evaluating Anti-Phishing tools for Windows

September 28th 2006 in Uncategorized

3sharp, a Redmond based technical services company, has been commissioned by Microsoft to undertake a competitive study of various anti-phishing technologies.  The results of that study were released just minutes ago.

The IE team comment on the study:

Before we proceed, I will say, right at the outset, that the only safe antiphishing technology is one that *BLOCKS* access to known phishing sites.  Why?  Because in its July report (released on 11 September 2006), the Anti-Phishing Working Group reported 182 unique websites hosting password stealing trojans, 1850 sites hosting password stealing malicious code (exploits) and a large increase in traffic redirecting, also known as pharming:
In short, it is not enough to simply warn a user that a Web site is a known phishing site yet still display the page.  Just opening a phishing site in your Web browser can be dangerous, even if you have absolutely no intention of entering any information on a page, if that site attempts to infect your system with a trojan, keylogger or other nasty.  Please keep this in mind when deciding which protective technology you wish to use.  I cannot recommend strongly enough that you choose a product that BLOCKS access to known phishing sites.

Unfortunately IE7 allows sites to continue to load while the phishing filter makes its checks, meaning that it is still theoretically possible for a site to infect a PC even when "blocked" by IE, but any hostile activity that requires user interaction is neutralised.  Your security settings would have to be lowered allowing automatic execution of code or active x, or a exploitable vulernerability would have to be used, and we know that IE7 has been immune to virtually all vulnerabilities.

Ok, now to the results….

The products were tested using 100 known phishing URLs (which had to be tested within 48 hours of collection) and 500 known good URLs.

The "winner", with the best overall performance, and a composite accuracy score of 172 out of 200 was Internet Explorer 7 Beta 3 (V7.0.5450.33).

2nd place went to the NetCraft toolbar (V1.6.2) with IE6 with a score of 168 out of 200.

A distant third was Google's Toolbar for Firefox with "Safe Browsing" (V2.0) with Firefox with a score of 106 out of 200.

The remaining products rated:

eBay's toolbar with AccountGuard (V2.3.1) with IE692 out of 200 (note, eBay restricts itself to eBay and PayPal spoofs and will not detect any other type of phish)

Earthlink's ScamBlocker (V3.1.5) with IE676 out of 200

GeoTrusts TrustWatch (V3b1) with IE667 out of 200

Netscape (V8.1)56 out of 200

McAfee SiteAdvisor (V1.5.0.0 build 3083) with IE63 out of 200

Total catch rate for known phish URLs – pay particular attention to the block versus warn percentages

Mistakes made on known "good" URLs

Important tidbits

  • Although GeoTrust did very well with a 99% catch rate, it also had a very high rate of false positives at 32.2%.  Not only that, it does not block access to known phishing sites.
  • When scoring results, a false block on a good site was scored as twice as bad as a false warning.  Allowing a good site had zero value.
  • The known phishing URLs were not taken from any feeds from known third-party data providers or end users to the Microsoft Phishing Filter Service in IE7.
  • Known good URLS were pulled from a feed of randomly selected traffic-weighted URLs provided by Microsoft and were independent of, and confirmed not to be included in, the Microsoft Phishing Filter system (they are not in the Phishing Filter white list).

The full report, and associated Press Release, can be found the URL below. The report provides comprehensive information about how the products were tested, the rules under which the tests were conducted, how and where the phishing URLs and good URLs were sourced and how scores were calculated, and a full list of the URLs used during testing is also included.


FAQs about "Gone Phishing: Evaluating Anti-Phishing Tools for Windows"


The most important thing to me is that users are safe when browsing the Internet.  That is why I am doing all I can to encourage users to update their copy of Internet Explorer to IE7 Release Candidate 1.  I also strongly recommend that, if you are using IE7, you enable the Phishing Filter.

If, for whatever reason, you are not able to run IE6 then I recommend that you download and install the NetCraft's toolbar.

Quick statistics about IE7's phishing filter

  1. The Phishing Filter is a “real time” service that does not require a user to download or regularly update a list of “bad” sites.
  2. Microsoft has been adding up to 17,000 URLS a month to its Phishing Filter service.
  3. From February to Mid Aug 2006 the Phishing Filter helped block over 800,000 instances of people trying to access reported phishing websites using IE7 or MSN/Windows Live Toolbar.  This figure includes almost 500,000 blocks since IE7 Beta 2 was released.
  4. IE7 users are reporting up to 4,500 potential phishing sites per week.

3 comments to...
“Gone Phishing: Evaluating Anti-Phishing tools for Windows”

Shane Keats

This is Shane from SiteAdvisor here. We’re not surprised to find out that we came in last in Microsoft’s anti-phishing study.

Why? Because we don’t offer anti-phishing.

We test for a lot of important things that no one else does, like whether a site’s e-mail practices result in spam, or whether an offered download bundles spyware, or whether the site attempts to breach browser security, or whether the site agressively links to known bad sites.

But we don’t offer anti-phishing protection, at least not yet. We’re pretty explicit about that too:

“SiteAdvisor’s software does not currently provide automated or real-time phishing detection.”


Ok, so you warn of "fraudulent practices" and have tested "sites representing more than 95% of worldwide Web traffic" and performs "tens of thousands" of tests every day, but phishing sites aren't included:


No exclusion of phishing sites here either:


Perhaps you should be more specific about what these "fraudulent practices" are (fraud, but not phishing, despite phishing being a type of fraud) and add a mention about not covering phishing in the FAQ in addition to the Support Centre (which people won't go to unless they have problems).


Seen on the Microsoft Switzerland Security Blog:http://news.bbc.co.uk/1/hi/technology/5371078.stm"Analysis of the net addresses where the e-mail messages originated showed that more than 100,000 hijacked home computers [my emphasis] spread across 119 nations had been used to despatch the junk mail."Do you have a home computer? A broadband connection? Then the spammers want your machine, and if you give […]

Previous Entry

Note: despite all the headlines to the contrary, this is not an IE vulnerability, although IE is an exploit vector – it is a vulnerability in the Windows Shell.Edit: MS Security Advisory here – patch due by October 10http://www.microsoft.com/technet/security/advisory/926043.mspx Secunia and FRSIRT have released information about a new IE vulnerability:http://secunia.com/advisories/22159/http://www.frsirt.com/english/advisories/2006/2882My tests indicate that not only does the demonstration page crash Internet Explorer 7 […]

Next Entry