How can I reject all incoming UDP packets except for DNS lookups?

If you are following the standard security practices, then your default firewall policy will be to block everything. All you should have to do is write a rule to permit tcp and udp traffic to port 53 if you want to permit incoming DNS requests.

The traffic you are talking about is UDP. UDP is stateless. This means that people interested in saturating your connection can send the packets to your address even if you just drop them. Still you may be able to do something semi-useful with the iptables recent match, to only allow a limited ammount of traffic to actually be accepted and processed by the system. Evan has a example of the usage of this for SSH here. We might have to see your entire firewall rule set to tell you what rules would have to be added.

If you have a serious DoS against your system, you would almost certainly need your ISP to help you, trying to deal with a flood with a host-based firewall on a VPS will really not be very useful.

If you don't have it already, you should consider setting up a few secondary DNS servers for your zones on a completely different network.

In Iptables, Accept incoming UDP traffic to port 53 & reject everything in the port range for ephemeral ports.

The highest limit should not be too high otherwise, your server will be unable to resolve external domains (for instance when you do a "ping google.com") from inside your server. On a linux OS, 32768 is the first ephemeral port (aka dynamic ports) for sockets up to 61000. Thus, 32767 is the highest port for static allocated ports. This is only true if don't use your server as DNS resolver aka DNS cache aka server with an /etc/resolv.conf pointing to nameserver 127.0.0.1 or ::1

Here is a tcpdump example:

 23:10:13.315832 IP b.b.b.b.34507 > a.a.a.a.53: 23674% [1au] A? whitehouse.gov. (38)
 23:10:13.377619 IP a.a.a.a.53 > b.b.b.b.34507: 23674*- 1/2/3 A 172.230.122.69(122)

1. b.b.b.b requests nameserver a.a.a.a from port 34507 to give A record for whitehouse.gov on port 53
2. a.a.a.a from port 53 answers b.b.b.b to port 34507

Normally, to find your local dynamic (also called ephemeral or private) port ranges on your linux for UDP & TCP:

 cat /proc/sys/net/ipv4/ip_local_port_range

However, it only works for server that don't host the DNS resolver (for instance, when you point your /etc/resolv.conf to 8.8.8.8).

Server is not a DNS resolver:

 -A INPUT -p udp -m udp --dport 53 -j ACCEPT
 -A INPUT -p udp -m udp --dport 0:32767 -j DROP

server is a DNS resolver:

-A INPUT -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -p udp -m udp --dport 0:1023 -j DROP

This should be taken into account if you want to host your own DNS resolver, to resolve all domain names.

The best would be to check it yourself:

You can monitor sending ports using

tcpdump udp and port 53 and not dst host *yourserveripaddress*

then look at sending ports and try to find the lowest the number. This lowest number should not be lower than the port number xxxx in --dport 0:xxxx otherwise you block or slow down your DNS requests.

Do you actually need to allow strangers on the internet to run DNS queries against your server? I suspect you just need to make sure that your firewall allows your server to make outgoing DNS requests.

With firewalling you start by blocking everything and then being very precise/detailed about opening up specific combinations of port/service/protocol and limit it by IP address (range).

How can I reject all incoming UDP packets except for DNS lookups?

This can be stated more broad: how can I reject all traffic I didn't initiate with Linux netfilter?

Answer is simply 2 lines:

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
iptables -A INPUT -j DROP

(Taking into account you don't want to filter loopback traffic to save some CPU cycles, you can add exception for it.)

If OTOH you have DNS server on same computer and your q-n could be more precisely stated as "How can I reject all incoming UDP traffic except external queries to DNS-server?" you would use same 2 lines base and add another one which would explicitly allow DNS traffic:

iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

iptables -A INPUT -p udp --dport 53 -j ACCEPT -m comment --comment "we serve DNS"
iptables -A INPUT -p tcp --dport 53 -j ACCEPT -m comment --comment "DNS uses TCP too sometimes"

iptables -A INPUT -j DROP

Notice that you typically don't need to specify any -m state things for anything but the first line there, cause the first line is a short-cut allowing any legitimate traffic to continue had its pilot packets gotten permission to enter.

Other notices

@Zoredache's advice for recent isn't any applicable at all. recent has very limited use case scope cause it doesn't use efficient data structures but list and hashes, no trees, nope. By default it can remember only 100 IPs per list which can be extended but hashing isn't very effective for searching through it anyways.