FTP – Page 2 – Tales from the Crypto

FTP

Vista’s Secret Windows Firewall hole

First, the good news – it’s not a flaw in the operation of Windows Firewall on Windows Vista. It’s a design feature, it makes sense, and it fits in with the principle that the firewall should keep out unsolicited traffic. It’s not really a hole, but I thought I’d grab your attention.


The symptom first came up in a Usenet posting (thanks, Jesper, for bringing me in) about Vista and a third-party FTP client:



When I do a directory listing, and a PORT command is issued, and the
server attempts to connect, it works, but at the same time a dialogue
appears telling me it’s blocked, and I can keep blocking or unblock.
I choose keep blocking but it doesn’t actually block it once.


Here’s how it looks.

First, if you haven’t got a third-party FTP client let’s fake it, by copying Microsoft’s command-line FTP client from the Windows System32 directory to another directory:


C:\users\MyMe> copy %windir%\system32\ftp.exe
1 file(s) copied.


The FTP client will not display prompts to you, but that’s a minor issue – if it upsets you, try downloading a third-party client and trying it.

Anyway, here we go – let’s try the issue in question:


  • Type ftp ftp.microsoft.com

  • After you see the “220” greeting message, enter ftp as the user – press enter.

  • Now you’re prompted for a password – enter anything and press enter.

  • Once you’re logged on, enter dir – again, press enter.

  • You’ll see the directory listing succeed, but you’ll also see a warning that a connection is being blocked:

image-0063


Wow – that’s freaky – at the same time you’re being told that the connection used for the file listing will be blocked, it allows the connection through!

What’s more, even if you specify Keep Blocking, and then go issue another dir command, that one succeeds.

Huh? And why on earth did you make me use a copy of FTP?

Let’s go look at the Windows Advanced Firewall Rules for Inbound, and see if this sheds any light:

[That means click the Start button, type Firewall into the search box, and right-click on Windows Firewall with Advanced Security – select Run as Administrator

and accept the elevation prompt from UAC. If you don’t have an elevation prompt, then you should really re-enable UAC. Now select Inbound Rules in the left-hand pane]


Me, I’ve got a few rules labeled File Transfer Program:


Image-0064


That first (and fourth) rule is set to block any listening ports opened by the File Transfer Program in C:\users\myme\ftp.exe, the second two seem to be allowing any listening ports created by the one in C:\windows\system32\ftp.exe.


Obviously, that’s why I asked you to copy ftp.exe to a new directory, so that any previous allowance by the firewall rules wouldn’t get in the way.


So what’s happening here? Is the “Allow” rule somehow overriding the “Block” rule, even though it’s not dealing with the same executable?


We can test that simply by deleting both sets of rules – go ahead and do that, I’ll wait for you.


Didn’t make a bit of difference, did it? It still allowed the traffic, then prompted you if you wanted to block it. Even if you selected to “Keep Blocking“, the next and subsequent transfers still worked, right?


Okay – let’s consult the Big Book of Knowledge (alright, what I can vaguely remember after mumbleteen years in the networking world). Some routers and firewalls use an Application Layer Gateway (ALG) to translate FTP commands, and open ports. Is that what’s going on here?


Let’s take a peek at the services on this machine (as an administrator, run services.msc):


Image-0065


Bingo – there it is, the Application Layer Gateway Service. And when you have Internet Connection Sharing running, that’s what translates IP addresses in FTP commands for you, and what opens up port mappings and holes in the NAT that ICS hosts.


Oh, but wait a moment – what’s that in the “Status” column?


That’s right, nothing. This service isn’t running.


Something must be happening to open this port up – it’s not just a case of “port left open”, nor is it an outbound port. Those ports are closed tight until the FTP client starts listening for incoming data connections, and then they’re opened up.


Here’s where I go into MVP-mode, and start searching in all the nooks and crannies of the web and whatever documentation it holds.


Net result – Windows Firewall in Windows Vista includes something called a “connection inspection engine”.


Sounds like something from “Schoolhouse Rock“.


No, seriously, there’s a “connection inspection engine” for FTP – if you connect to port 21, the firewall monitors your communications on that channel, looking for PORT commands. When it finds one, it opens up a hole in the firewall for the incoming data connection.


So why the scary dialog warning that something’s going to block traffic?


Probably because the dialog pops up whenever an application starts listening, whereas the connection inspection engine only opens a hole when it sees a PORT command. And an FTP client can’t actually give the PORT command until it’s started listening.


So, the process goes something like this:



  • Start the FTP client.

  • Connect to the FTP server on port 21, waking up the connection inspection engine.

  • Log on, then type dir

  • The FTP client knows that it needs to open a data connection.

  • To start the data connection, the FTP client binds to port 0, and starts listening.

  • The firewall says “Oh no, an unknown program has started listening – better warn them that they won’t get any traffic.”

  • The FTP client checks what port it actually got, and sends a matching PORT command.

  • The connection inspection engine says “PORT command? That’s my cue!” and opens a hole in the firewall to incoming data connections.

Well, that’s easy, but what if I don’t ever want to do an FTP connection? How do I stop this from becoming a potential hacker tool?


Okay, apart from the obvious – that if a hacker could connect out to a server on port 21, nothing’s stopping that hacker from transferring data in – you might want to cripple this functionality.


No problem – just set the following DWORD registry value to 1:


HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Services \ SharedAccess \ Parameters \ FirewallPolicy \ DisableStatefulFTP


The default setting for this value on Windows Vista is 0. [It remains to be seen what value will be the default on Windows Server 2008]


How could Microsoft make this better?



  • I’d really like to see this documented. Just so that it’s not a surprise to anyone.

  • I’d like to know how many other connection inspection engines there are (at least one, judging from the DisableStatefulPPTP value – but I don’t know enough about PPTP to know how that affects operation).

  • I’d like to know if I can add my own connection inspection engine to the firewall.

  • Above all, I’d like to do away with the rather confusing and clumsy “We’re going to block your incoming … wait, what just happened?” dialog. If the connection inspection engine is monitoring a command channel, and the process that owns the socket for that command channel starts listening, perhaps we could wait a quarter of a second for a PORT command before calling this a blocked connection?

Finally, is this a vulnerability, a hole, or anything outside the correct operation of a firewall?


No, because the firewall is documented as blocking unsolicited incoming connections – and by any reasonable definition, the data connection requested by a PORT command is solicited.

What should I do now I can compete?

My departure from Microsoft is very nearly reaching its first anniversary.

As befits someone approaching that milestone, my thoughts drift to … the non-compete clause.

That’s the niggling part of the contract every Microsoft employee signs, and which restricts them, to lesser or greater extent, from engaging in any activity considered to be competitive to Microsoft, using knowledge gained while working at Microsoft.

Now, in my case, the non-compete clause is weak to begin with – a condition I made of my employment was that I could continue my WFTPD work, which was, in some ways, directly competitive to Microsoft’s FTP server in IIS.

It’s further weakened, I’d say, by the fact that Microsoft sent me to exactly one class while I was there – the mandatory coding security class – and bought me one book – the mandatory “Writing Secure Code”.

But, weakened or not, I have chosen to observe it steadfastly – I have not added a single feature to WFTPD or WFTPD Pro or WFTPD Explorer that was based on anything I learned at Microsoft, or even on anything I had hoped to add to Microsoft’s FTP server during my stay there. I have even steered clear of adding features that I was planning to add to WFTPD and WFTPD Pro before Microsoft offered me the job, just to avoid the appearance of competing with my ex-employer.

So now, I’m giddy with anticipation, as this mostly self-imposed deadline is about to expire.

What should I start to code into my software?

Well, there’s an open comments section below – what do you think I should do, now that I can compete?

I wish Larry hadn’t written that…

Oh, Larry, Larry, Larry…


Articles 1 and 2 were great – really necessary reading to a lot of would-be network programmers.


But article 3… where to start with the corrections?


I’m not going to. It’s an article you shouldn’t read, because you’re not going to use the right terms for the right things, and when you go asking for help from networking experts, they’ll look at you in much the same way as security experts look at Steve Gibson. [The look is “how the hell do you get anything done, knowing so little about the field?”]


I’ve met Larry, and he’s a nice guy, so I really thought twice about making this post – and I apologise if I hurt Larry’s feelings by saying it… but I have been on a fifteen-year march to persuade people to stop writing crap networking apps, by getting them to understand what they’re doing, and I can’t stop now.


At the risk of opening myself up to abuse similar to that which I’m heaping on Larry now, I’ll point you to my earlier article, where I describe the interaction between delayed ACKs and Nagle – it’s much, much simpler than Larry stated, and I think I’ve got it more accurately described.


Finally, in the case that perhaps I don’t have it correct, I’m going to retro-edit it if mistakes are pointed out, because the worst thing you can do is have someone search for the answer, and the first text they come back with is wrong.

DELAY or NODELAY – Riffing on Larry, who’s riffing on Raymond…

[Why is this under “Programmer Hubris”? Because it’s about developers who find “an easy fix” and apply it, without trying to figure out why it made things appear to work better.]


I like to read Larry Osterman and Raymond Chen’s blogs, because they’ve seen most things, and learned most of the lessons right.  Today, Larry posted on network optimisation – reminding me once again:


Trying to fix network speed problems by enabling TCP_NODELAY is almost always wrong.  Setting TCP_NODELAY disables the Nagle algorithm


When you do this, it’s an indication that either your program is broken, or your protocol is broken, or quite possibly both.


“But what about FTP?” people say.


Yes, the protocol is broken.  It requires a send / send / recv exchange, and you have to enable TCP_NODELAY (disabling the Nagle algorithm in the process) to make it work properly, so that you can have more than five files transfer per second.


So what does Nagle do, exactly? Well, John Nagle spends his time pushing dead bodies down staircases.  The Nagle algorithm, named after John despite his humble protestations, does a couple of simple things every time you try to send data:

If there is unacknowledged data in the queue, then don’t send until:

  • all data is acknowledged, or
  • you have a segment’s worth to send.
  • Uh… that’s it.

The idea is to take a program that, rather stupidly, sends one character at a time, resulting in a 1:40 data:framing ratio, and turn it into one that sends several characters at a time, using the network bandwidth more efficiently and not becoming a network hog.


Way back when, this was a perfect idea, and could have remained perfect, if it weren’t for the delayed ACK algorithm, whose author is not remembered so fondly as to name the algorithm after him.  Delayed Ack states that you should not send an unaccompanied ack until either:

  • you receive two segments
  • 200ms expires since the first piece of data was received.

[An ACK is included in every TCP segment, so it’s not an overhead of any kind except when you’re sending nothing but an ACK.]

In most environments, this is still good, because most protocols are “client sends command, server sends response” over and over again, so each side is doing one send, then one recv. Model this behaviour in your head, and you’ll see that the Nagle algorithm won’t stop any outgoing traffic, and nor will delayed ACK hold up any acknowledgements.


If one side hiccups and calls send() twice (on short data) before receiving, however, things come to (in networking terms) a screeching halt. The second send queues up because the Nagle algorithm doesn’t get an acknowledgement from the first send until the delayed ACK algorithm has exhausted its timer.


The answer is always “don’t send / send / recv”.  Always group logically-associated data together in one call to send, unless you’re sending large amounts of data, in which case you can happily send / send / send until you’ve exhausted your data.


Setting TCP_NODELAY will look like it makes your program perform at top speed, but it’s now become the network hog that you programmed it to be, and that Nagle was helpfully preventing you from being.  Fixing the program will make your program perform faster than it would with TCP_NODELAY alone, and you will find that the TCP_NODELAY setting will have no further effect, on or off.  Your program is now working smoothly, a good network citizen, and the Nagle net-cop allows it to go about its business unimpeded.


So, my final piece of advice – if TCP_NODELAY looks like it makes your program perform better, fix your damn program! There’s too much crappy networking software out there already, and you don’t want to add to it.

"FTPS" document finally makes it to RFC status.

News I’ve been waiting for for years – the document formally known as draft-murray-auth-ftp-ssl-16.txt has finally been released by the RFC editor as RFC 4217 – “Securing FTP with TLS


What exactly does this mean?  Technically, not very much – FTPS has been implemented by several FTP clients, servers and wrappers for several years.  I added FTPS support to WFTPD Pro back in 2001, after first expressing interest in doing so in 1997, but being held back by the lack of crypto support in Windows.


I nearly had it ready in 2000, but spent some time trying to debug an issue that turned out to be caused by a corrupted certificate issued by the Windows 2000 Server CA that I was testing against.  Let that be a lesson to you crypto developers – sometimes the code is right, and it’s the certs that are wrong!


A few minor things have changed since then in the document that is now RFC 4217, but almost nothing significant to the compatibility of FTPS offerings.


I will end with a brief FAQ for you – please let me know if there are any other questions you’d like to see answered:


1. What’s TLS, and what is its relation to SSL?


TLS is Transport Layer Security, and is the name of the protocol that grew from Netscape’s SSL and Microsoft’s PCT.  Most people still use the term “SSL”, but TLS is where all ongoing work is carried out by the IETF.


2. Is FTPS the official term?


No – the RFC is “Securing FTP with TLS”, and perhaps the official term should be “AUTH TLS”.  However, with the general public already familiar with the concept of “https” being the secured equivalent of “http”, the term “ftps” has sprung up in general use to describe an FTP transfer, or session, encrypted and/or authenticated with SSL or TLS.


3. How different is FTPS from HTTPS?


Quite significantly – HTTPS uses a separate port for incoming SSL connections (usually port 443), compared to the port for unprotected HTTP connections (usually port 80).  Because FTP is (and has always been) a session-based protocol, it allows the client to “negotiate up” to SSL or TLS security through the use of the AUTH command described in RFC 2228.


Note also that FTP uses two channels – a control channel and a data channel, and that these channels can be secured – or left unsecured – almost independently.  HTTPS is secured from the moment you connect to the HTTPS port, until you close down the connection.  FTP is secured on the control channel from the moment you send an “AUTH TLS” or “AUTH SSL” command, until you log out; the data channel is not necessarily secured by default, and security on the data channel can be turned on or off using the PROT command, with parameters “C” for “Clear” or “P” for “Private”.


FTPS always authenticates the server through its certificate, and can be configured to authenticate the client by certificate, or by USER / PASS commands supplying username and password.  HTTP and HTTPS have several other methods of authentication (none of which bear much examination at the moment) – NTLM CHAP, Basic, Digest, etc, etc.


4. What about SFTP?  What’s that?


I get to answer this question a lot.  With all these acronyms getting thrown around, it’s easy to get confused.  Many people automatically assume that any acronym including the letters “FTP” refer to protocols based on FTP.  Obviously, that’s why “FTPS” was chosen as an informal description of “Securing FTP with TLS”.  Unfortunately, others may create confusing acronyms by including the FTP letters, either by accident or on purpose.  One such confusion was always “TFTP – Trivial File Transfer Protocol”.  This is about as far from FTP as you can get, and still be associated with transferring files from one machine to the other.


The same is true of “SFTP” – it’s a file transfer extension to “SSH”.  As that sentence implies, to do an SFTP file transfer, you need to have an SSH connection in place.  This isn’t always practical.