DNS Server Reserves 2500 Ports.

After applying the patch for MS08-037KB 953230 (the multi-OS DNS flaw found by Dan Kaminski), you may notice your Windows Server 2003 machine gets a little greedy. At least, mine sucks up 2500 – yes, that’s two thousand five hundred – UDP sockets sitting there apparently waiting for incoming packets.


Output of 'netstat -bona -p udp' command, showing ports bound by DNS.EXE


This is, apparently, one of those behaviours sure to be listed in the knowledge base as “this behavior is by design” – a description that graces some of the more entertaining elements of the Microsoft KB.


Why does this happen? I can only guess. But here’s my best guess.


The fix to DNS, implemented across multiple platforms, was to decrease the chance of an attacker faking a DNS response, by increasing the randomness in the DNS requests that has to be copied back in a response.


I don’t know how this was implemented on other platforms, but I do know that it’s already been reported that BIND’s implementation is slower than it used to be (hardly a surprise, making random numbers is always slower than simply counting up) – and maybe that’s what Microsoft tried to forestall in the way that they create the random sockets.


Instead of creating a socket and binding it to a random source port at the time of the request, Microsoft’s patched DNS creates 2500 sockets, each bound to a random source port, at the time that the DNS service is started up. This way, perhaps they’re avoiding the performance hit that BIND has been criticised for.


There are, of course, other services that also use a UDP port. ActiveSync’s connection to Exchange, IPsec, IAS, etc, etc. Are they affected?


Sometimes.


Randomly, and without warning or predictability. Because hey, the DNS server is picking ports randomly and unpredictably.


[Workaround: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ReservedPorts is a registry setting that lists multiple port ranges that will not be used when binding an ephemeral socket. The DNS server will obey these reservations, and not bind a socket to ports specified in this list. More explanation in the blog linked above, or at http://support.microsoft.com/kb/812873]


DNS, you see, is a fundamental underpinning of TCP/IP services, and as such needs to start up before most other TCP/IP based services. So if it picks the port you want, it gets first pick, and it holds onto that port, preventing your application from binding to it.


This just doesn’t seem like a fix written by someone who ‘gets’ TCP/IP. Perhaps I’m missing something that explains why the DNS server in Windows Server 2003 works this way, but I would be inclined to take the performance hit of binding and rebinding in order to find an unused random port number, rather than binding before everyone else in an attempt to pre-empt other applications’ need for a port.


There are a couple of reasons I say this:


  1. Seriously, how many Windows Server 2003 users out there have such a high-capacity DNS server that they will notice the performance hit?
  2. Most Windows Server 2003-based DNS servers are small caching servers for businesses, rather than Internet infrastructure servers responsible for huge numbers of requests per second – even if you implement this port-stealing method, it shouldn’t be the default, because the majority of users just don’t need that performance.
  3. If you do need the performance, get another server to handle incoming requests. Because the cost of having your DNS server’s cache poisoned is considerably greater than the cost of increasing the number of servers in your pool, if you’re providing major DNS service to that many customers.
  4. A major DNS service provider will be running fewer services that would pre-empt a DNS server request to bind to a random port, whereas systems running several UDP-based services are going to need less performance on their outgoing DNS requests.

I’d love to know if I’m missing something here, but I really hope that Microsoft produces a new version of the DNS patch soon, that doesn’t fill your netstat -a output with so many bound and idle sockets, each of which takes up a small piece of nonpaged pool memory (that means real memory, not virtual memory).

3 thoughts on “DNS Server Reserves 2500 Ports.”

  1. It’s not much better if you’re using BIND as your DNS server, apparently (mostly in the Linux world).

    BIND can’t tell when it’s selecting a port that’s already bound by another application, so it randomly, unpredictably, and without warning will send out a request on a port that causes the response to come back to an application other than BIND’s “named”.

    [For discussion of this problem, see: this Usenet thread]

    That’s all very well for applications that are used to being exposed to the Internet – after all, they receive random junk packets all the time, so they should be resilient.

    But what about apps that aren’t used to Internet exposure? Remember – the DNS server sends out a packet first, so it will cause most firewalls to open up a hole for the return packet.

    Any UDP-based service on your DNS server will receive random UDP data, whether you’ve blocked it at the firewall or not.

    Just like in Windows with the ReservedPorts option, there’s a way to work around this – there’s a new setting for named called “avoid-v4-udp-ports” that you can use to list ports that named shouldn’t bind to for its outgoing requests.

  2. You’re absolutely correct. Sadly.

    Remember that MSDNS still needs to pick random numbers at some point here, we still need to randomize requests among the various available ports, so we’re not saving any CPU time at all by doing the randomization first, although we are probably reducing DNS query time by a couple MS. We could achieve the same performance improvement by building a list, but not pre-binding.

    Worse, by pre-binding to 2500 ports, we might even be helping an attacker map out which ports are and are not used by DNS. Sure, sending out 2500 times more spoofed packets then before sounds like a lot, but with zombie farms ranging in the millions, getting an important DNS poison (PayPal anyone?) cached is probably not outside the scope.

    If we build a random map out in advance, but don’t bind until needed, an attacker would still need to flood some ~65000 ports. Oh well, 2500 is better then 1.

    The other performance improvement is that by pre-binding, rather then binding on-demand, we don’t need to confirm that a binding is successful before sending out a DNS query. Good, also saves a ms or two. A better fix might be to have the server pre-bind a small number of ports in advance, only using a single port once (or at least only sending it to a single destination IP once, you can probably reuse a remote-DNS-IP:local-port pair pretty safely, since this whole random port scheme does nothing for MITM attacks anyway.

    Anyway, an effective pre-binding algorithm could avoid clobbering ports other applications need, plus avoid the performance impact of binding on demand. The numbers may need to be tweaked, but start with this: For the first three minutes of operation, you’d only pre-bind five ports. These ports could be anywhere in a large range, maintaining some level of randomness, but don’t forget a comprehensive list of known-services to avoid, maybe Windows’ own “%systemroot%\system32\drivers\etc\services” list would be a good place to start.

    After three minutes, larger numbers of ports can be safely allocated without difficulty, since services have now had an opportunity to start and do their initial bindings. At this stage, MSDNS could decide how many ports to pre-bind allowing for MSDNS to stay 30-60 seconds ahead — Enough time to allow for a bump in activity, but not enough time for an attacker to build a map.

    Thoughts?

  3. What about some active evasion, as well – detect when you might be being flooded with requests, and either refuse to serve such requests (discarding any responses that you might already have cached from forged responses), or serve them using a more protected connection, such as a TCP connection to the remote DNS server?
    I’m sure that the DNS folks at Microsoft have spent a lot of time talking with other DNS server authors and coming to some consensus as to what’s a good approach – and it’s always possible that I’m missing something huge and important in my analysis.
    What I’d like to see is a new Winsock socket option, to say “when I call bind() on this socket, I want you to give me a pseudo-random (not N+1) port assignment, that isn’t currently in use by another application” – FTP would benefit from this, because PASV responses are supposed to bind to an unpredictable port to improve security.
    I do like the idea of leaving DNS less responsive (or even non-responsive to external queries) until a significant time after booting, though.
    Another angle that I don’t think has been considered is to tailor the approach to two scenarios – one of a high-traffic DNS server, which is probably almost exclusively running DNS as its only service; the other of a DNS server that runs multiple other services. I think the threats can be handled differently, because the level of service to be provided is different.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>