I think my DNS servers were attacked, but can't figure out why anyone would want to. Can somebody help me understand what happened?
I run about a dozen very low traffic web sites, complete with eMail and authoritative DNS servers for them. I have a commercial comcast account which is filtered through a Linux firewall. I have a single physical server which has been optimized for low power consumption, not for server speed. The server is broken into a dozen virtual machines for my hosting, and generally works very well. I'm not amazon; each server may get a few hundred hits per day.
About a week ago, I noticed that my Internet speed had fallen and was erratic. A friend was setting up a business survey on one of my sites, and he was having a very difficult time reliably uploading his survey materials. (In fact, I thought we were going to have to use a commercial hosting site since mine was so bad!)
Why was my system responding so poorly?
Some investigation showed:
- Cable Modem lights were flashing regularly, but not crazy
- Firewall-to-DMZ Ethernet switch lights were flashing, but not crazy
- Htop on the servers showed normal CPU usage
- iftop on the firewall showed some odd IP addresses repeating, but nothing that looked like a smoking gun.
- No strange processes running on the server
- Shutting down the virtual machines had no effect
- Shutting down the entire server mostly fixed the Internet speed problem, but the modem lights still flashed
I eventually concluded that server had been rooted, since I couldn't think what else might have happened.
I backed up all the virtual disks, then did a clean install of Debian (changing all my root passwords).
My server provides DNS services for my entire network (internal & external) through BIND9, so that was the first thing I got going again. You guessed it: the problem recurred as soon as I started offering DNS services again. (Other than DNS, the server was bare!)
More digging with wireshark showed that most of the traffic was coming from DNS queries from the outside Internet for domains that I am not authoritative for. Since I had set up my DNS server to forward queries (so it could service my internal network also), it was dutifully answering these external queries. The same external IP addresses were making the same repeated DNS queries and this was stealing overall bandwidth, and more importantly to me, bogging down my server and slowing down any legitimate DNS queries.
So I changed BIND9 to reject external forwarding queries, and my bandwidth utilization dropped 90% almost immediately. I'm still getting the queries, which are now responded to as "refused" by BIND.
I can't have my firewall reject all DNS queries from the Internet, because I run the authoritative DNS server for my sites and there are legitimate DNS queries from the Internet in there.
So my final questions for this august, learned group are:
- Was this a Denial of Service attack?
- How do I set BIND to efficiently ignore these fraudulent requests?
- What the heck was the motivation for the attack?
- Wow. What should I do now?
Thanks to all in advance for comments and suggestions!
--
Casey Bralla
Chief Nerd in Residence
The NerdWorld Organisation
www.NerdWorld.org