Why is PKI so hard?

Apple’s “goto fail” SSL issue–how do you avoid it?

Context – Apple releases security fix; everyone sees what they fixed


Last week, Apple released a security update for iOS, indicating that the vulnerability being fixed is one that allows SSL / TLS connections to continue even though the server should not be authenticated. This is how they described it:

Impact: An attacker with a privileged network position may capture or modify data in sessions protected by SSL/TLS

Description: Secure Transport failed to validate the authenticity of the connection. This issue was addressed by restoring missing validation steps.

Secure Transport is their library for handling SSL / TLS, meaning that the bulk of applications written for these platforms would not adequately validate the authenticity of servers to which they are connected.

Ignore “An attacker with a privileged network position” – this is the very definition of a Man-in-the-Middle (MITM) attacker, and whereas we used to be more blasé about this in the past, when networking was done with wires, now that much of our use is wireless (possibly ALL in the case of iOS), the MITM attacker can easily insert themselves in the privileged position on the network.

The other reason to ignore that terminology is that SSL / TLS takes as its core assumption that it is protecting against exactly such a MITM. By using SSL / TLS in your service, you are noting that there is a significant risk that an attacker has assumed just such a privileged network position.

Also note that “failed to validate the authenticity of the connection” means “allowed the attacker to attack you through an encrypted channel which you believed to be secure”. If the attacker can force your authentication to incorrectly succeed, you believe you are talking to the right server, and you open an encrypted channel to the attacker. That attacker can then open an encrypted channel to the server to which you meant to connect, and echo your information straight on to the server, so you get the same behaviour you expect, but the attacker can see everything that goes on between you and your server, and modify whatever parts of that communication they choose.

So this lack of authentication is essentially a complete failure of your secure connection.

As always happens when a patch is released, within hours (minutes?) of the release, the patch has been reverse engineered, and others are offering their description of the changes made, and how they might have come about.

In this case, the reverse engineering was made easier by the availability of open source copies of the source code in use. Note that this is not an intimation that open source is, in this case, any less secure than closed source, because the patches can be reverse engineered quickly – but it does give us a better insight into exactly the code as it’s seen by Apple’s developers.

Here’s the code:

    if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)
        goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)
        goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)
        goto fail;
    if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)
        goto fail;
        goto fail;
    if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)
        goto fail;

Yes, that’s a second “goto fail”, which means that the last “if” never gets called, and the failure case is always executed. Because of the condition before it, however, the ‘fail’ label gets executed with ‘err’ set to 0.

Initial reaction – lots of haha, and suggestions of finger pointing

So, of course, the Internet being what it is, the first reaction is to laugh at the clowns who made such a simple mistake, that looks so obvious.

T-shirts are printed with “goto fail; goto fail;” on them. Nearly 200 have been sold already (not for me – I don’t generally wear black t-shirts).

But really, these are smart guys – “be smarter” is not the answer

This is SSL code. You don’t get let loose on SSL code unless you’re pretty smart to begin with. You don’t get to work as a developer at Apple on SSL code unless you’re very smart.

Clearly “be smart” is already in evidence.

There is a possibility that this is too much in evidence – that the arrogance of those with experience and a track record may have led these guys to avoid some standard protective measures. The evidence certainly fits that view, but then many developers start with that perspective anyway, so in the spirit of working with the developers you have, rather than the ones you theorise might be possible, let’s see how to address this issue long term:

Here’s my suggested answers – what are yours?

Enforce indentation in your IDE / check-in process

OK, so it’s considered macho to not rely on an IDE. I’ve never understood that. It’s rather like saying how much you prefer pounding nails in with your bare fists, because it demonstrates how much more of a man you are than the guy with a hammer. It doesn’t make sense when you compare how fast the job gets done, or the silly and obvious errors that turn up clearly when the IDE handles your indenting, colouring, and style for you.

Yes, colouring. I know, colour-blind people exist – and those people should adjust the colours in the IDE so that they make sense. Even a colour-blind person can get shade information to help them. I know syntax colouring often helps me spot when an XSS injection is just about ready to work, when I would otherwise have missed it in all the surrounding garbage of HTML code. The same is true when building code, you can spot when keywords are being interpreted as values, when string delimiters are accidentally unescaped, etc.

The same is true for indentation. Indentation, when it’s caused by your IDE based on parsing your code, rather than by yourself pounding the space bar, is a valuable indication of program flow. If your indentation doesn’t match control flow, it’s because you aren’t enforcing indentation with an automated tool.

What the heck, enforce all kinds of style

Your IDE and your check-in process are a great place to enforce style standards to ensure that code is not confusing to the other developers on your team – or to yourself.

A little secret – one of the reasons I’m in this country in the first place is that I sent an eight-page fax to my bosses in the US, criticising their programming style and blaming (rightly) a number of bugs on the use of poor and inconsistent coding standards. This was true two decades ago using Fortran, and it’s true today in any number of different languages.

The style that was missed in this case – put braces around all your conditionally-executed statements.

I have other style recommendations that have worked for me in the past – meaningful variable names, enforced indenting, maximum level of indenting, comment guidelines, constant-on-the-left of comparisons, don’t include comparisons and assignments in the same line, one line does one thing, etc, etc.

Make sure you back the style requirements with statements as to what you are trying to do with the style recommendation. “Make the code look the same across the team” is a good enough reason, but “prevent incorrect flow” is better.

Make sure your compiler warns on unreachable code

gcc has the option “-Wunreachable-code”.

gcc disabled the option in 2010.

gcc silently disabled the option, because they didn’t want anyone’s build to fail.

This is not (IMHO) a smart choice. If someone has a warning enabled, and has enabled the setting to produce a fatal error on warnings, they WANT their build to fail if that warning is triggered, and they WANT to know when that warning can no longer be relied upon.

So, without a warning on unreachable code, you’re basically screwed when it comes to control flow going where you don’t want it to.

Compile with warnings set to fatal errors

And of course there’s the trouble that’s caused when you have dozens and dozens of warnings, so warnings are ignored. Don’t get into this state – every warning is a place where the compiler is confused enough by your code that it doesn’t know whether you intended to do that bad thing.

Let me stress – if you have a warning, you have confused the compiler.

This is a bad thing.

You can individually silence warnings (with much comments in your code, please!) if you are truly in need of a confusing operation, but for the most part, it’s a great saving on your code cleanliness and clarity if you address the warnings in a smart and simple fashion.

Don’t over-optimise or over-clean your code

The compiler has an optimiser.

It’s really good at its job.

It’s better than you are at optimising code, unless you’re going to get more than a 10-20% improvement in speed.

Making code shorter in its source form does not make it run faster. It may make it harder to read. For instance, this is a perfectly workable form of strstr:

const char * strstr(const char *s1, const char *s2)


  return (!s1||!s2||!*s2)?s1:((!*s1)?0:((*s1==*s2&&s1==strstr(s1+1,s2+1)-1)?s1:strstr(s1+1,s2)));


Can you tell me if it has any bugs in it?

What’s its memory usage? Processor usage? How would you change it to make it work on case-insensitive comparisons? Does it overflow buffers?

Better still: does it compile to smaller or more performant code, if you rewrite it so that an entry-level developer can understand how it works?

Now go and read the implementation from your CRT. It’s much clearer, isn’t it?

Release / announce patches when your customers can patch

Releasing the patch on Friday for iOS and on Tuesday for OS X may have actually been the correct move – but it brings home the point that you should release patches when you maximise the payoff between having your customers patch the issue and having your attackers reverse engineer it and build attacks.

Make your security announcements findable

Where is the security announcement at Apple? I go to apple.com and search for “iOS 7.0.6 security update”, and I get nothing. It’d be really nice to find the bulletin right there. If it’s easier to find your documentation from outside your web site than from inside, you have a bad search engine.

Finally, a personal note

People who know me may have the impression that I hate Apple. It’s a little more nuanced than that.

I accept that other people love their Apple devices. In many ways, I can understand why.

I have previously owned Apple devices – and I have tried desperately to love them, and to find why other people are so devoted to them. I have failed. My attempts at devotion are unrequited, and the device stubbornly avoids helping me do anything useful.

Instead of a MacBook Pro, I now use a ThinkPad. Instead of an iPad (remember, I won one for free!), I now use a Surface 2.

I feel like Steve Jobs turned to me and quoted Dr Frank N Furter: “I didn’t make him for you.”

So, no, I don’t like Apple products FOR ME. I’m fine if other people want to use them.

This article is simply about a really quick and easy example of how simple faults cause major errors, and what you can do, even as an experienced developer, to prevent them from happening to you.

DigiNotar – why is Google special?

So, you’ve probably heard about the recent flap concerning a Dutch Certificate Authority, DigiNotar, who was apparently hacked into, allowing for the hackers to issue certificates for sites such as Yahoo, Mozilla and Tor.

I’ve been reading a few comments on this topic, and one thing just seems to stick out like a sore thumb.

DigiNotar’s servers issued over 200 fraudulent certificates. These certificates were revoked – but, as with all certificate revocations, you can’t really get a list of the names related to those revoked certificates to go back and see which sites you visited recently that you might want to reconsider. [You can only check to see if a certificate you’re offered matches one that was revoked.]

What behaviour would you reconsider on recently visited sites? Well, I’d start by changing my passwords at those sites, at the very least, perhaps even checking to make sure nobody had used my account in my stead.

But that’s not the thing that sticks out

What does stick out is that DigiNotar’s own certificate was removed from, well, just about everyone’s list of trusted root Certificate Authorities, once it was discovered that a fraudulent certificate in the name of *.google.com had been issued, and had not yet been revoked.

Yeah, given the title of my blog posting, I’m sure you could guess that this was the thing that I was concerned about.

So, why is Google so special?

I’m not sure I buy that DigiNotar’s removal from the trusted certificate list was simply because they failed to find one fraudulently issued certificate. It seems like, if that fraudulent certificate was for Joe Schmoe Electrical Repair, it would just have been revoked like all of the other certificates.

Removing a CA from the trusted list, after all, is pretty much going to kill that CA – every certificate ever issued by them will suddenly fail. All of their customers will have to install a new certificate, and what’s the chance those customers will go back to the CA that caused them a sudden outage to their secure web site?

So it’s not something that companies like Google, Microsoft, Mozilla, etc would do at the drop of a hat.

It certainly seems like Google is special.

The argument I would buy

There is an argument I would buy, but no one is making it. It goes something like this:

“Back in July, when we first discovered the fraudulent certificates, we had no evidence that anyone was using them in the wild, and the CRL publication schedule allowed us to quietly and easily render unusable the certificates we had discovered. Anyone visiting a fraudulent web site would simply have seen the usual certificate error.

“Then, when we discovered in August that there was still one undiscovered certificate, and it was being used in the wild, it was not appropriate to revoke the certificate, because the CRL publishing schedule wasn’t going to bring it to people’s desks in time to prevent them from being abused. So, we had to look for other ways to prevent people from being abused by this certificate.

“We could have trusted to OCSP, but it’s unlikely that the fraudulent certificate pointed to a valid OCSP server. Besides, the use of a fraudulent certificate pretty much requires that you are a man-in-the-middle and can redirect your target to any site you like.

“We could have added this certificate to the ‘untrusted’ certificate list, but only Microsoft has a way to quickly publish that – the other browser and app vendors have to release new versions of their software, because they have a hard coded untrusted certificates list.

“And maybe there’s another certificate – or pile of certificates – that we missed.

“So we chose, in the interests of securing the Internet, and at the risk of adversely affecting valid customers, we chose to remove this one certificate authority from everyone’s list of trusted roots.”

I’ve indented that as if it’s a quote, but as I said, this is an argument that no one is making. So it’s just a fantasy quote.

Is there another possible argument I might be missing, but willing to accept?

Woot got my Zune, Zune can’t get my woot!

Quite some time ago, my wife was very sneaky. Oh, she’s sneaky again and again, but this is the piece of sneakiness that is appropriate for this post.

I logged on to woot.com one day, as I often do, and saw that there was a 30GB Zune for sale – refurbished, and quite a bit cheaper than most places had it for sale, but still more than I could plonk down without blinking.

I told my wife about it, and she told me that no, I was right, we couldn’t really afford it even at that price.

Then, months later, I found that my birthday present was a 30GB Zune – the very one from woot that she said we couldn’t afford.

Ever since then, I’ve been a strong fan of Zune and woot alike.

The other day, though, it dawned on me that I could use my Zune (now I have a Zune HD 32GB) to keep up with woot’s occasional “woot-off” events, where they proceed throughout the day to offer several deals. Unfortunately, I can’t actually buy anything from woot on the Zune.

I couldn’t figure this out for a while, and assumed that it was simply a lack of Flash support.

Sidebar: Why the Zune and iPhone Don’t Have Flash Support

It’s not immediately obvious that there’s a difference between the Zune having no Flash support, and the iPhone having no Flash support.

But there is – and it’s a little subtle.

The Zune doesn’t have Flash support because Adobe haven’t built it.

The iPod doesn’t have Flash support because Apple won’t let Adobe build it.

Back to the main story – why my Zune can’t woot!

I did a little experimenting, and it’s not that woot requires Flash.

I tried to logon directly to the account page at https://sslwww.woot.com/Member/YourAccount.aspx (peculiar that, the URL says “Your Account”, but it’s my account, not yours, that I see there. That’s why you shouldn’t use personal pronouns in folder names).

That failed with a cryptic error – “Can’t load the page you requested. OK”

No, it’s not actually OK that you can’t load the page, but thanks for telling me what the problem was.

Oh, that’s right, you didn’t, you just told me “failed”. Takes me right back to the days of “Error 4/10”.

The best I can reckon is that, since the Zune can visit other SSL sites, and other browsers have no problem with this SSL site, the Zune simply doesn’t have trust in the certificate chain.

That should be easy to fix, all I have to do on my PC, or on any number of web browsers, is to add the site’s root certificate from its certificate chain to my Trusted Root store.

Sadly, I can find no way to do this for my Zune. So, no woot.

Would this be a feature other people would want?

I think this would – for a start, it would mean that users could add web sites that were previously unavailable to them – including test web sites that they might be working on, which are supported by self-signed test certificates.

But more than that, adding a new root certificate to the trusted root certificate store on the Zune is a vital feature for another functionality that people have been begging for. Without adding a root certificate, it is often impossible to support WPA2 Enterprise wireless mode. So, the “add certificate to my Zune’s Trusted Root store” feature would be a step toward providing WPA2 Enterprise support.

How would that interface look on the Zune?

I’m not sure that the interface would have to be on the Zune itself – but perhaps the Zune could stock up failed certificate matches to pass to the Zune software, and then ask the operator of the Zune software at the next Sync, “do you want to trust these certificates to enable browsing to these sites?”

Similarly, for the WPA Enterprise mode, it could ask the Zune software user “do you want to connect to this WPA Enterprise network in future?”

TLS Renegotiation attack – Microsoft workaround/patch

Hidden by the smoke and noise of thirteen (13! count them!) security bulletins, with updates for 26 vulnerabilities and a further 4 third-party ActiveX Killbits (software that other companies have asked Microsoft to kill because of security flaws), we find the following, a mere security advisory:

Microsoft Security Advisory (977377): Vulnerability in TLS/SSL Could Allow Spoofing

It’s been a long time coming, this workaround – which disables TLS / SSL renegotiation in Windows, not just IIS.

Disabling renegotiation in IIS is pretty easy – you simply disable client certificates or mutual authentication on the web server. This patch gives you the ability to disable renegotiation system-wide, even in the case where the renegotiation you’re disabling is on the client side. I can’t imagine for the moment why you might need that, but when deploying fixes for symmetrical behaviour, it’s best to control it using switches that work in either direction.

The long-term fix is yet to arrive – and that’s the creation and implementation of a new renegotiation method that takes into account the traffic that has gone on before.

To my mind, even this is a bit of a concession to bad design of HTTPS, in that HTTPS causes a “TOC/TOU” (Time-of-check/Time-of-use) vulnerability, by not recognising that correct use of TLS/SSL requires authentication and then resource request, rather than the other way around. But that’s a debate that has enough clever adherents on both sides to render any argument futile.

Suffice it to say that this can be fixed most easily by tightening up renegotiation at the TLS layer, and so that’s where it will be fixed.

Should I apply this patch to my servers?

I’ll fall back to my standard answer to all questions: it depends.

If your servers do not use client auth / mutual auth, you don’t need this patch. Your server simply isn’t going to accept a renegotiation request.

If your servers do use client authentication / mutual authentication, you can either apply this patch, or you can set the earlier available SSLAlwaysNegoClientCert setting to require client authentication to occur on initial connection to the web server.

One or other of these methods – the patch, or the SSLAlwaysNegoClientCert setting – will work for your application, unless your application strictly requires renegotiation in order to perform client auth. In that case, go change your application, and point them to documentation of the attack, so that they can see the extent of the problem.

Be sure to read the accompanying KB article to find out not only how to turn on or off the feature to disable renegotiation, but also to see which apps are, or may be, affected adversely by this change – to date, DirectAccess, Exchange ActiveSync, IIS and IE.

How is Microsoft’s response?


I would have to say that on the speed front, I would have liked to see Microsoft make this change far quicker. Disabling TLS/SSL renegotiation should not be a huge amount of code, and while it has some repercussions, and will impact some applications, as long as the change did not cause instability, there may be some institutions who would want to disable renegotiation lock, stock and barrel in a hurry out of a heightened sense of fear.

I’m usually the first to defend Microsoft’s perceived slowness to patch, on the basis that they do a really good job of testing the fixes, but for this, I have to wonder if Microsoft wasn’t a little over-cautious.


While I have no quibbles with the bulletin, there are a couple of statements in the MSRC blog entry that I would have to disagree with:

IIS 6, IIS 7, IIS 7.5 not affected in default configuration

Customers using Internet Information Services (IIS) 6, 7 or 7.5 are not affected in their default configuration. These versions of IIS do not support client-initiated renegotiation, and will also not perform a server-initiated renegotiation. If there is no renegotiation, the vulnerability does not exist. The only situation in which these versions of the IIS web server are affected is when the server is configured for certificate-based mutual authentication, which is not a common setting.

Well, of course – in the default setting on most Windows systems, IIS is not installed, so it’s not vulnerable.

That’s clearly not what they meant.

Did they mean “the default configuration with IIS installed and turned on, with a certificate installed”?

Clearly, but that’s hardly “the default configuration”. It may not even be the most commonly used configuration for IIS, as many sites escape without needing to use certificates.

Sadly, if I add “and mutual authentication enabled”, we’re only one checkbox away from the “default configuration” to which this article refers, and we’re suddenly into vulnerable territory.

In other words, if you require client / mutual authentication, then the default configuration of IIS that will achieve that is vulnerable, and you have to make a decided change to non-default configuration (the SSLAlwaysNegoClientCert setting), in order to remain non-vulnerable without the 977377 patch.

The other concern I have is over the language in the section “Likelihood of the vulnerability being exploited in general case”, which discusses only the original CSRF-like behaviour exploited under the initial reports of this problem.

There are other ways to exploit this, some of which require a little asinine behaviour on the part of the administrator, and others of which are quite surprisingly efficient. I was particularly struck by the ability to redirect a client, and make it appear that the server is the one doing the redirection.

I think that Eric and Maarten understate the likelihood of exploit – and they do not sufficiently emphasise that the chief reason this won’t be exploited is that it requires a MITM (Man-in-the-middle) attack to have already successfully taken place without being noticed. That’s not trivial or common – although there are numerous viruses and bots that achieve it in a number of ways.


It’s a little unclear on first reading the advisory whether this affects just IIS or all TLS/SSL users on the affected system. I’ve asked if this can be addressed, and I’m hoping to see the advisory change in the coming days.


I’ve rambled on for long enough – the point here is that if you’re worried about SSL / TLS client certificate renegotiation issues that I’ve reported about in posts 1, 2 and 3 of my series, by all means download and try this patch.

Be warned that it may kill behaviour your application relies upon – if that is the case, then sorry, you’ll have to wait until TLS is fixed, and then drag your server and your clients up to date with that fix.

The release of this advisory is by no means the end of the story for this vulnerability – there will eventually be a supported and tested protocol fix, which will probably also be a mere advisory, followed by updates and eventually a gradual move to switch to the new TLS versions that will support this change.

This isn’t a world-busting change, but it should demonstrate adequately that changes to encryption protocols are not something that can happen overnight – or even in a few short months.

Ten key truths

In the spirit of "ten unavoidable security truths", and numerous other top-ten lists, here’s a list of ten key truths that apply to public / private key pairs:

  1. Your private key has to be private to you. It cannot be created by anyone else.
  2. Anyone who has your private key is you, for the purposes to which that key is applied.
  3. If you have a private key that was generated by your employer, then that key identifies you as a part of the employer. It cannot be used to uniquely identify you, because the key was generated under your employer’s control.
  4. Keys associated with expired or revoked certificates are not always useless – you can use them to decrypt a file that they encrypted a long time ago; you can also verify the time-stamped signature of a document, if the certificate was valid at the time of the signature.
  5. A key is a number – it cannot expire, it is the associated certificate that expires. Similarly, the certificate, not the key, is what is revoked after exposure.
  6. A key is not a certificate. A certificate is a statement (usually of ownership) about a key pair, and contains the public key.
  7. Too short of a key is no key at all, and an exposed private key is no key at all.
  8. Protecting a password or key by encrypting it may do nothing more than extend the problem by a level – now you have to protect the key used to encrypt the password / key.
  9. Revocation of a certificate applies only to trusting parties who actually bother to check for revocation.
  10. A key derived from a password has the same strength as the password, no matter how long the key.

Note that this list describes what happens when cryptography is working perfectly. There are other key facts that apply to broken cryptography and broken process:

  1. You cannot increase entropy by repeating bits, or by padding with constant or predictable values.
  2. If someone creates a private key for you, they can become you using that key.
  3. Protecting anything by using a password or key to feed a yes/no gate requires the attacker to only fool the yes/no gate. Encrypted USB drive manufacturers found this out recently to their embarrassment.
  4. Inventing your own crypto – or the processes around it – is always a bad idea. Research others’ ideas and use them, particularly published standards.
  5. Even the longest strongest key in the world can be defeated by a big enough bribe to the key-holder.
    1. If the price of a card number on the black market is $1, then access to a million card numbers is equivalent to a million dollars. Who do you pay well enough that they can ignore that value?
  6. A key or password that is never updated as it passes through many hands is vulnerable to abuse by each of those hands.
  7. Broken cryptographic algorithms cannot be made better by increasing the length of the key or by applying the algorithm more times.
  8. All cryptographic algorithms become broken in time. The trick is to choose one that lasts longer than your need to protect your data.
  9. When you lose the private key through insufficient key management, you have lost the data it protected.
  10. Even a good cryptographic algorithm can be destroyed by a poor key-generation algorithm.

Note that these lists are rather arbitrarily scoped to ten – there may be more important truths I’ve forgotten, or items I’ve included that aren’t really so important.

Corrections to Thierry Zoller’s Whitepaper

Thanks to Thierry Zoller for mentioning me in the FTP section of his whitepaper summary of the TLS renegotiation attacks on various protocols. I’m glad he also spells my name right – you’d be surprised how many people get that wrong, although I’m sure Thierry gets his own share of people unable to spell his name.

The whitepaper itself contains some really nice and simple documentation of the SSL MITM renegotiation attack, and how it works. It’s well worth reading if you’re looking for some insight into how this works.

First, though, a couple of corrections to Thierry’s summary – while he’s working on revising his whitepaper, I’ll post them here:

  • The name of my FTP server is “WFTPD Server”, or “WFTPD Pro Server”, as you will see from http://www.wftpd.com. Of the two, only the WFTPD Pro Server has FTP over TLS capabilities.
  • SFTP is not “FTP over SSH”. SFTP is a completely different protocol – a sub-protocol of SSH. For example, where FTP uses commands of three or four letters, SFTP uses binary single byte instructions. There is no one-to-one mapping between SFTP commands and FTP commands, and there are many other differences that aren’t really worth going into.
  • I don’t see the use of CCC as a reason to “use SFTP over FTPS” – I see it as a reason to not use FTPS clients that require, or FTPS servers that support, the CCC command. There are better ways to surmount the NAT problem in FTPS – the best is IPv6, but even in IPv4, the use of block-mode FTP over the default data channel removes NATs from being a problem for FTPS.
  • The language in the “Client Certificate Authentication” FTPS section appears to be an incomplete sentence. What I think he is trying to say is that when authentication is performed in FTPS, it resets the command state, so that commands entered prior to authentication, if they are executed at all, are executed in the context that existed at that time, rather than the newly-authenticated context.

I think that where FTPS has problems that are thrown into sharp relief with the SSL MITM renegotiation attacks I’ve been discussing for a while now, it has had those problems before. If an attacker can monitor and modify the FTP control channel (because the client requested CCC and the server allowed it), the attacker can easily upload whatever data they like in place of the client’s bona fide upload.

The renegotiation attack simply makes it easier for the attacker to hide the attack. It’s the use of CCC which facilitates the MITM attack, far more than the renegotiation does.

To address one further comment I’ve heard with regard to SSL MITM attacks, I hear “yeah, but getting to be a man-in-the-middle is so difficult anyway, that even a really simple attack is unlikely”. That’s a true comment – for the most part, there is little chance of a man-in-the-middle attack occurring on the general Internet in a bulk situation. The ‘last mile’ of home wireless, coffee bars and other public wireless hangouts, or the possibility of DNS hijacking, HOSTS file editing, broadband router hacking, or just plain viruses and worms, are the place where most man-in-the-middle entry points exist.

However, if you’re going to assert that it’s truly unlikely that an attacker can insert himself into your network stream, you basically have no reason whatever to use SSL / TLS – without a potential for that interception and modification of your traffic, there’s really no need to authenticate it, encrypt it, or monitor its integrity along the path.

The fact that a protocol or application uses SSL /TLS means that it tacitly assumes the existence of a man in the middle. If SSL / TLS allows a man-in-the-middle attack at all, it fails in its basic raison d’etre.

Next post, I promise something other than SSL renegotiation attacks.

My take on the SSL MITM Attacks – part 3 – the FTPS attacks

[Note – for previous parts in this series, see Part 1 and Part 2.]

FTP, and FTP over SSL, are my specialist subject, having written one of the first FTP servers for Windows to support FTP over SSL (and the first standalone FTP server for Windows!)

Rescorla and others have concentrated on the SSL MITM attacks and their effects on HTTPS, declining to discuss other protocols about which they know relatively far less. OK, time to step up and assume the mantle of expert, so that someone with more imagination can shoot me down.

FTPS is not vulnerable to this attack.

No, that’s plainly rubbish. If you start thinking along those lines in the security world, you’ve lost it. You might as well throw in the security towel and go into a job where you can assume everybody loves you and will do nothing to harm you. Be a developer of web-based applications, say. :-)

FTPS has a number of possible vulnerabilities

And they are all dependent on the features, design and implementation of your individual FTPS server and/or client. That’s why I say “possible”.

Attack 1 – renegotiation with client certificates

The obvious attack – renegotiation for client certificates – is likely to fail, because FTPS starts its TLS sessions in a different way from HTTPS.

In HTTPS, you open an unauthenticated SSL session, request a protected resource, and the server prompts for your client certificate.

In FTPS, when you connect to the control channel, you provide your credentials at the first SSL negotiation or not at all. There’s no need to renegotiate, and certainly there’s no language in the FTPS standard that allows the server to query for more credentials part way into the transaction. The best the server can do is refuse a request and say you need different or better credentials.

Attack 2 – unsolicited renegotiation without credentials

A renegotiation attack on the control channel that doesn’t rely on making the server ask for client credentials is similarly unlikely to succeed – when the TLS session is started with an AUTH TLS command, the server puts the connection into the ‘reinitialised’ state, waiting for a USER and PASS command to supply credentials. Request splitting across the renegotiation boundary might get the user name, but the password wouldn’t be put into anywhere the attacker could get to.

Attack 3 – renegotiating the data connection

At first sight, the data connection, too, is difficult or impossible to attack – an attacker would have to guess which transaction was an upload in order to be able to prepend his own content to the upload.

But that’s betting without the effect that NATs had on the FTP protocol.

Because the PORT and PASV commands involve sending an IP address across the control channel, and because NAT devices have to modify these commands and their responses, in many implementations of FTPS, after credentials have been negotiated on the control channel, the client issues a “CCC” command, to drop the control channel back into clear-text mode.

Yes, that’s right, after negotiating SSL with the server, the client may throw away the protection on the control channel, so the MitM attacker can easily see what files are going to be accessed over what ports and IP addresses, and if the server supports SSL renegotiation, the attacker can put his data in at the start of the upload before renegotiating to hand off to the legitimate client. Because the client thinks everything is fine, and the server just assumes a renegotiation is fine, there’s no reason for either one to doubt the quality of the file that’s been uploaded.

How could this be abused? Imagine that you are uploading an EXE file, and the hacker prepends it with his own code. That’s how I wrote code for a ‘dongle’ check in a program I worked on over twenty years ago, and the same trick could still work easily today. Instant Trojan.

There are many formats of file that would allow abuse by prepending data. CSV files, most exploitable buffer overflow graphic formats, etc.

Attack 3.5 – truncation attacks

While I’m on FTP over SSL implementations and the data connection, there’s also the issue that most clients don’t properly terminate the SSL connection in FTPS data transfers.

As a result, the server can’t afford to report as an error when a MitM closes the TCP connection underneath them with an unexpected TCP FIN.

That’s bad – but combine it with FTP’s ability to resume a transfer from part-way into a file, and you realize that an MitM could actually stuff data into the middle of a file by allowing the upload to start, interrupting it after a few segments, and then when the client resumed, interjecting the data using the renegotiation attack.

The attacker wouldn’t even need to be able to insert the FIN at exactly the byte mark he wanted – after all, the client will be sending the REST command in clear-text thanks to the CCC command. That means the attacker can modify it, to pick where his data is going to sit.

Not as earth-shattering as the HTTPS attacks, but worth considering if you rely on FTPS for data security.

How does WFTPD Pro get around these attacks?

1. I never bothered implementing SSL / TLS renegotiation – didn’t see it as necessary; never had the feature requested. Implementing unnecessary complexity is often cause for a security failure.

2. I didn’t like the CCC command, and so I didn’t implement that, either. I prefer to push people towards using Block instead of Stream mode to get around NAT restrictions.

I know, it’s merely fortunate that I made those decisions, rather than that I had any particular foresight, but it’s nice to be able to say that my software is not vulnerable to the obvious attacks.

I’ve yet to run this by other SSL and FTP experts to see whether I’m still vulnerable to something I haven’t thought of, but my thinking so far makes me happy – and makes me wonder what other FTPS developers have done.

I wanted to contact one or two to see if they’ve thought of attacks that I haven’t considered, or that I haven’t covered. So far, however, I’ve either received no response, or I’ve discovered that they are no longer working on their FTPS software.

Let me know if you have any input of your own on this issue.

My take on the SSL MitM Attacks – part 2 – clarifications

Since the last post I made on the topic of SSL renegotiation attacks, I’ve had a few questions in email. Let’s see how well I can answer them:

Q. Some stories talk about SSL, others about TLS, what’s the difference?

A. For trademark reasons, when SSL became an open standard, it had to change its name from SSL to TLS. TLS 1.0 is essentially SSL 3.1 – it even claims to be version “3.1” in its communication. I’ll just call it SSL from here on out to remind you that it’s a problem with SSL and TLS both.

Q. All the press coverage seems to be talking about HTTPS – is this limited to HTTPS?

A. No, this isn’t an HTTPS-only attack, although it is true that most people’s exposure to SSL is through HTTPS. There are many other protocols that use SSL to protect their connections and traffic, and they each may be vulnerable in their own special ways.

Q. I’ve seen some posts saying that SSH and SFTP are not vulnerable – how did they manage that?

A. Simply by being “not SSL”. SFTP is a protocol on top of SSH, and SSH is not related to SSL. That’s why it’s not affected by this issue. Of course, if there’s a vulnerability discovered in SSH, it’ll affect SSH and SFTP, but won’t affect SSL or SSL-based protocols such as HTTPS and FTPS.

Q. Is it OK to disable SSL renegotiation to fix this bug?

A. Obviously, if SSL didn’t need renegotiation at all, it wouldn’t be there. So, in some respects, if you disable SSL renegotiation, you may be killing functionality. There are a few reasons that you might be using SSL renegotiation:

  1. Because that’s how client authentication works – while you can do client authentication without renegotiation, most HTTPS implementations use renegotiation to request the client certificate. Disabling renegotiation will generally prevent most clients from authenticating with client authentication.
  2. After 10 hours, renegotiation is required, so as to refresh the session key. Do you have SSL connections lasting 10 hours? You probably should be looking at some disconnect/reconnect scenario instead.
  3. Because you can’t disable SSL renegotiation in all cases. In OpenSSL, you can only disable renegotiation if you download and install the new version, and in other SSL implementations, there is no way to disable renegotiation outside of modifying the application.

Q. Since this attack requires the attacker to become a man-in-the-middle, doesn’t that make it fundamentally difficult, esoteric, or close to impossible?

A. If becoming a man-in-the-middle (MitM) was impossible or difficult, there would be little-to-no need for SSL in the first place. SSL is designed specifically to protect against MitM attacks by authenticating and encrypting the channel. If a MitM can alter traffic and make it seem as if everything’s secure between client and server over SSL, then there’s a failure in SSL’s basic goal of protecting against men-in-the-middle.

Once you assume that an attacker can intercept, read, and modify (but not decrypt) the SSL traffic, this attack is actually relatively easy. There are demonstration programs available already to show how to exploit it.

I was asked earlier today how someone could become a man-in-the-middle, and off the top of my head I came up with six ways that are either recently or frequently used to do just that.

Q. Am I safe at a coffee shop using the wifi?

A. No, not really – over wifi is the easiest way for an attacker to insert himself into your stream.

When using a public wifi spot, always connect as soon as possible to a secured VPN. Ironically, of course, most VPNs are SSL-based, these days, and so you’re relying on SSL to protect you against possible attacks that might lead to SSL issues. This is not nearly as daft as it sounds.

Q. Is this really the most important vulnerability we face right now?

A. No, it just happens to be one that I understood quickly and can blather on about. I think it’s under-discussed, and I don’t think we’ve seen the last entertaining use of it. I’d like to make sure developers of SSL-dependent applications are at least thinking about what attacks can be performed against them using this step, and how they can prevent these attacks. I know I’m working to do something with WFTPD Pro.

Q. Isn’t the solution to avoid executing commands outside the encrypted tunnel?

A. Very nearly, yes. The answer is to avoid executing commands sent across two encrypted sessions, and to deal harshly with those connections who try to send part of their content in one session and the rest in a differently negotiated session.

In testing WFTPD Pro out against FTPS clients, I found that some would send two encrypted packets for each command – one containing the command itself, the other containing the carriage return and linefeed. This is bad in itself, but if the two packets straddle either side of a renegotiation, disconnect the client. That should prevent the HTTPS Request-Splitting using renegotiation.

One key behaviour HTTPS has is that when you request a protected resource, it will ask for authentication and then hand you the resource. What it should probably be doing is to ask for authentication and then wait for you to re-request the resource. That action alone would have prevented the client-certificate attacks discussed so far.

Q. What is the proposed solution?

A. The proposed solution, as I understand it, is for client and server to state in their renegotiation handshake what the last negotiated session state was. That way, an interloper cannot hand off a previously negotiated session to the victim client without the client noticing.

Note that, because this is implemented as a TLS handshake extension, it cannot be implemented in SSLv3. Those of you who just got done with mandating SSLv2 removal throughout your organisations, prepare for the future requirement that SSLv3 be similarly disabled.

Q. Can we apply the solution today?

A. It’s not been ratified as a standard yet, and there needs to be some discussion to avoid rushing into a solution that might, in retrospect, turn out to be no better – or perhaps worse – than the problem it’s trying to solve.

Even when the solution is made available, consider that PCI auditors are still working hard to persuade their customers to stop using SSLv2, which was deprecated over twelve years ago. I keep thinking that this is rather akin to debating whether we should disable the Latin language portion of our web pages.

However, it does demonstrate that users and server operators alike do not like to change their existing systems. No doubt IDS and IPS vendors will step up and provide modules that can disconnect unwarranted renegotiations.

Update: Read Part 3 for a discussion of the possible threats to FTPS.

My take on the SSL MITM Attacks – part 1 – the HTTPS attack

If you’re in the security world, you’ve probably heard a lot lately about new and deadly flaws in the SSL and TLS protocols – so-called “Man in the Middle” attacks (aka MITM).

These aren’t the same as old-style MITM attacks, which relied on the attacker somehow pretending strongly to be the secure site being connected to – those attacks allowed the attacker to get the entire content of the transmission, but they required the attacker to already have some significant level of access. The access required included that the attacker had to be able to intercept and change the network traffic as it passed through him, and also that the attacker had to provide a completely trusted certificate representing himself as the secure server. [Note – you can always perform a man-in-the-middle attack if you own a trusted certificate authority.]

The current SSL MITM attack follows a different pattern, because of the way HTTPS authentication works in practice. This means it has more limited effect, but requires less in the way of access. You gain some security advantage, you lose some. The attacker still needs to be able to intercept and modify the traffic between client and server, but does not get to see the content of traffic between client and server. All the attacker gets to do is to submit data to the server before the client gets its turn.

Imagine you’re ordering a pizza over the phone. Normally, the procedure is that you call and tell them what the pizza order is (type of pizza, delivery address), and they ask you for your credit card number as verification. Sometimes, though, the phone operator asks for your credit card number first, and then takes your order. So, you’re comfortable working either way.

Now, suppose an attacker can hijack your call to the pizza restaurant and mimic your voice. While playing you a ringing tone to keep you on the line, he talks to the phone operator, specifying the pizza he wants and the address to which it is to be delivered. Immediately after that, he connects you to your pizza restaurant, you’re asked for your credit card number, which you supply, and then you place your pizza order.

Computers are as dumb as a bag of rocks. Not very smart rocks at that. So, imagine that this phone operator isn’t smart enough to say “what, another pizza? You just ordered one.”

That’s a rough, non-technical description of the HTTPS attack. There’s another subtle variation, in which the caller states his pizza order, then says “oh, and ignore my attempt to order a pizza in a few seconds”. The computer is dumb enough to accept that, too.

For a more technical description, go see Eric Rescorla’s summary at Understanding the TLS Renegotiation Attack, or Marsh Ray’s original report.

Let’s call these the HTTPS client-auth attack and the HTTPS request-splitting attack. That’s a basic description of what they do.

HTTPS client-authentication attack

The client-authentication attack is getting the biggest press, because it allows the attacker one go (per try) at persuading the server to perform an action in the context of the authenticated user. From ordering a pizza to pretty any activity that can be caused in a single request to a web site can be achieved with this attack.

Preventing the attack at the server.

Servers have been poorly designed in this respect – but out of some necessity. Eric Rescorla explains this in the SSL and TLS bible, “SSL and TLS” [Subtitle: Designing and Building Secure Systems] on page 322, section 9.18.

“The commonly used approach is for the server to negotiate an ordinary SSL connection for all clients. Then, once the request has been received, the server determines whether client authentication is required… If it is required, the server requests a rehandshake using HelloRequest. In this second handshake, the server requests client authentication.”

How does HTTP handle other authentication, such as Forms, Digest, Basic, Windows Integrated, etc? Is it different from the above description?

A client can provide credentials along with its original request using the WWW-Authenticate header, or the server can refuse an unauthorised (anonymous) request with a 401 error code indicating that authentication is necessary (and listing WWW-Authenticate headers containing appropriate challenges). In the latter case, the client resends the request with the appropriate WWW-Authenticate header.

HTTPS Mutual Authentication (another term for client authentication) doesn’t do this. Why on earth not? I’m not sure, but I think it’s probably because SSL already has a mostly unwarranted reputation for being slow, and this would add another turnaround to the process.

Whatever the reason, a sudden dose of unexpected ‘401’ errors would lead to clients failing, because they aren’t coded to re-request the page with mutual auth in place.

So, we can’t redesign from scratch to fix this immediately – how do we fix what’s in place?

The best way is to realise what the attack can do, and make sure that the effects are as limited as possible. The attack can make the client engage in one action – the first action it performs after authenticating – using the credentials sent immediately after requesting the action to be performed.

A change of application design is warranted, then, to ensure that the first thing your secure application does on authenticating with a client certificate is to display a welcome screen, and not to perform an action. Reject any action requested prior to authentication having been received.

Sadly, while this is technically possible using SSL if you’ve written your own server to go along with the application, or can tie into information about the underlying SSL connection, it’s likely that most HTTPS servers operate on the principle that HTTP is stateless, and the app should have no knowledge of the SSL state beyond “have I been authenticated or not”.

Doubtless web server vendors are going to be coming out with workarounds, advice and fixes – and you should, of course, be looking to their advice on how to fix this behaviour.

The best defence against the client-authentication attack, of course, is to not use client authentication.

Preventing the attack at the client

Not much you can do here, I’m afraid – the client can’t tell if the server has already received a request. Perhaps it would work to not provide client certificates to a server unless you already have an existing SSL connection, but that would kill functionality to perfectly good web sites that are operating properly. Assuming that most web sites operate in the mode of “accept a no-client-auth connection before requesting authentication”, you could rework your client to insist on this happening all the time. Prepare for failures to be reported.

Again, the best defence is not to use client authentication right now. Perhaps split your time between browsers – one with client certificates built in for those few occasions when you need them, and the other without client certs, for your main browsing. That will, at least, limit your exposure.

HTTPS Request-splitting attack

Preventing the attack at the server

The HTTPS Request-splitting attack is technically a little easier to block at the server, if you write the server’s SSL interface – there should be absolutely no reason for an HTTP Request to be split across an SSL renegotiation. So, an HTTPS server should be able to discard any connection state, including headers already sent, when renegotiation happens. Again, consult with your web server developer / vendor for their recommendations.

Preventing the attack at the client?

Again, you’re pretty much out of luck here – even sending a double carriage return to terminate any previous request would cause the attacker’s request to succeed.

The long term approach – fix the protocol

As you can imagine, there are some changes that can be made to TLS to fix all of this. The basic thought is to have client and server add a little information in the renegotiation handshake that checks that client and server both agree about what has already come before in their communication. This allows client and server both to tell when an interloper has added his own communication before the renegotiation has taken place.

Details of the current plan can be found at draft-rescorla-tls-renegotiate.txt

Final thoughts

Yeah, this is a significant attack against SSL, or particularly HTTPS. There are few, if any, options for protecting yourself as a client, and not very many for protecting yourself as a server.

Considering how long it’s taken some places to get around to ditching SSLv2 after its own security flaws were found and patched 14 years ago with the development of SSLv3 and TLS, it seems like we’ll be trying to cope with these issues for many years to come.

Like it or not, though, the long-term approach of revising TLS is our best protection, and it’s important as users that we consider keeping our software up-to-date with changes in the security / threat landscape.

Update: read Part 2 of this discussion for answers to a number of questions.

Update: read Part 3 for some details on FTPS and the potential for attacks.

When “All” isn’t everything you need – Terminal Services Gateway certificates.

Setting up Terminal Services Gateway on Windows Server 2008 the other day.

It’s an excellent technology, and one I’ve been waiting for for some time – after all, it’s fairly logical to want to have one “bounce point” into which you connect, and have your connection request forwarded to the terminal server of your choice. Before this, if you were tied to Terminal Services, you had to deal with the fact that your terminal connection was taking up far more traffic than it should, and that the connection optimisation settings couldn’t reliably tell that your incoming connection was at WAN speeds, rather than LAN speeds.

image But to get TS Gateway working properly, it needs a valid server certificate that matches the name you provide for the gateway, and that certificate needs to be trusted by the client. Not usually a problem, even for a small business operating on the cheap – if you can’t afford a third-party trusted certificate, there are numerous ways to deploy a self-signed certificate so that your client computers will trust it.

I have a handily-created certificate that’s just right for the job.

I ran into a slight problem when I tried to install the certificate, however.


The certificate isn’t there! In this machine, it isn’t even possible for me to “Browse Certificates” to find the certificate I’m looking for. On another machine, the option is present:


That’s promising, but my certificate doesn’t appear in the list of certificates available for browsing:


I checked in the Local Computer’s Personal Certificates store, which is where this certificate should be, and sure enough, on both machines, it’s right there, ready to be used by TSG.


So, why isn’t TSG offering this certificate to me to select? The clue is in the title.

The certificate that doesn’t show up is the one with “Intended purposes: <All>” – the cert that shows up has only “Server Authentication” enabled. Opening the certificate’s properties, I see this:


Simply selecting the radio-button “Enable only the following purposes”, I click “OK”:


And now, back over in the TSG properties, when I Browse Certficates, the Install Certificate dialog shows me exactly the certificates I expected to see:


This isn’t a solution I would have expected, and if that one certificate hadn’t shown up there, I wouldn’t have had the one clue that let me solve this issue.

Hopefully my little story will help someone solve this issue on their system.