[nylug-talk] thoughts on distributed DNS?

[Chris Knadle]

On Tuesday 06 May 2008, Joachim Rosenfeld wrote:
> On Mon, May 05, 2008 at 08:28:40PM -0400, Chris Knadle wrote:
> > > they will call you up (or more likely, your CTO) and say, "Hey, did
> > > you know your DNS guys are dropping 20% of your queries?")
> >
> > I wonder how they pound the DNS servers to get those numbers.  :-/ I
> > see what you mean.
>
> Read and laugh/weep:
>
>     http://readlist.com/lists/trapdoor.merit.edu/nanog/4/24667.html
>     http://readlist.com/lists/trapdoor.merit.edu/nanog/4/24668.html

   Depending on how the DNS servers are tested, it is perahps BY DESIGN that 
20% of DNS queries are dropped.  i.e. if you limit the number of simultaneous 
queries per IP address, which is suggested in terms of security, then at some 
point that DNS server will drop queries when they go over that limit.  In 
fact that's exactly what you would WANT to happen in order to be resiliant 
against DDOS attacks.
   So that 20% number means squat by itself unless it's properly qualified as 
to the testing conditions.

> > > They only have about 15 domains and maybe 300 records total, but
> > > their 3 most popular domains get about 90 million queries per month.
> >
> > that's an average 35 DNS queries per second.  Looking at it that way
> > that doesn't sound so bad.  Seems like one master and a couple of
> > slaves should be able to take care of that.
>
> Right, while they are a fairly high traffic site, they know they aren't
> doing a crazy amount of DNS traffic. The real fear is that that if they
> ever get DDOS'd again, will the DNS servers be able to handle it? The
> only thing that seems to give them peace of mind is the big guys.
>
> But there must be a way to inexpensively design a DNS infrastructure
> that is resilient enough to withstand a DDOS attack -- at least for less
> than $10k/month.

   It depends.  From what I've read of DDOS attacks, the "solution" for the 
problem often seems to involve filtering traffic *upstream* at the NOC rather 
than anything you can put at your local DMZ.  Because even in the optimal 
case of being able to filter out the DDOS right at your DMZ, that DDOS 
traffic is still flooding and overwhelming your incoming pipe.  AFAIK there 
just isn't a good localized solution for the situation of haivng 40,000 
zombie PCs flood pinging your DMZ, even if they're all on 56k modems.

> > If you haven't set up your own master DNS server before, then I also
> > suggest looking into how to secure it.  Specifically:
>
> Thanks Chris, I think what you listed are all good solutions, but I'm
> looking to set up DNS servers for the following very specific case:
>
>     - single datacenter
>     - single pipe (I think)
>     - site gets DDOS'd
>     - DNS should stay up
>
> Right now, I don't have any better answers than to outsource DNS to
> someone with geographically disparate severs.

   Yep, I think you're right.  As I see it you have two three setups you could 
choose:
     a) master off-site, slaves off-site
     b) master on-site which isn't listed as an NS to be queried, slaves
        off-site that are queried
     c) master on-site and slaves off-site, all of which are queried

   IMHO, maximum control + maximum benefit is probably choice b) but I've seen 
all of the setups above.  The thing I dislike the most slave DNS services is 
that they typically don't limit zone transfers.  :-/  And likewise these 
other big guys may not allow you to dictate the other desirable security 
precautions I previously mentioned.

   But the first place I'd ask concerning pricing of hosting slave DNS servers 
would be your ISP and then get quotes from other big guys that your company 
would be comfortable using.  

   Cheers and good luck, Joe.
   -- Chris

-- 

Chris Knadle
Chris.Knadle at coredump.us