Home / server / Questions / 489010
In Process
aSteve
Asked: 2013-03-19 12:03:54 +0800 CST

Bind FORMERR errors in syslog

  • 772

I've recently switched out a failed router... and, a while later, I discovered a lot (at least an order or magnitude more than the number of queries) of errors reported in /var/log/syslog - of the form:

Mar 18 19:53:20 kenneth named[4022]: DNS format error from 192.112.36.4#53 resolving ./NS: non-improving referral
Mar 18 19:53:20 kenneth named[4022]: error (FORMERR) resolving './NS/IN': 192.112.36.4#53

It might be relevant that I've got the following in bind.conf:

dnssec-enable no;
dnssec-validation no;

Is this likely an issue with the new router corrupting UDP datagrams, or something else? The new router is an (inexpensive) Netgear WNR854T - it has the latest firmware applied.

Can anyone suggest how best to diagnose this fault if it's not obvious from the above?

-- Additional details -- This is a typical response from dig for an address I'm sure should resolve.

$ dig A barclays.co.uk                                         ~

; <<>> DiG 9.8.1-P1 <<>> A barclays.co.uk
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22161
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;barclays.co.uk.                        IN      A

;; Query time: 84 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Wed Mar 20 23:08:40 2013
;; MSG SIZE  rcvd: 32
$
domain-name-system

3 Answers

  1. If you suspect a network element (such as your router) is truncating or corrupting UDP DNS traffic you can try the following:

    1. Perform the same query, again, with dig and see if you are also getting FORMERR as the response code. Dig (with the right options) will perform similarly to your recursing server but give you more visibility into the process.
    2. If you are getting FORMERRs with dig, try dig +tcp and see if the error persists (to rule out UDP issues)
    3. Use Wireshark or another sniffer to capture what is actually being received by your server when it recurses to satisfy a query.

    Are all of the FORMERR errors in your log complaining about non-improving referrals? What does dig say is in the "additional" section of the queries that generate these error messages?

    Finally, do you have stub zones or forward-first or forward-only zones set up that you haven't mentioned?

    • 2

  2. I am visiting in Indonesia and I get these messages because the operator/government sabotages DNS. DNSSEC fails to verify and direct access to root servers causes a FORMERR message to log (bind9, 13 messages per every DNS query). Address in your example is DNS root server G.

    • 0

  3. Your DNS traffic may be transparently intercepted, and routed to a caching DNS server and you request answers that cannot be completely answered with brief answers and the caching DNS server gives truncated, rather than minimal answers ("minimal-responses yes;"). For example, www.nvidia.com.edgekey.net is resolved via CNAMES and a long list of nameservers, and it complete answer does not fit in a 500~ish byte response. Here are the steps:

    1. Your BIND makes a non-recursive request to a root server, and expects only authoritative answers (in this case referrals to NS servers responsible for the zone).
    2. The caching DNS server which answers the intercepted request responds with a non-authoritative but incomplete answer, eg, the exact A record being sought, but with a truncated authority section which does not include the authoritative server for that record. Example (note there is no NS referal for akamaiedge.net):
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id:  59893
;; flags: qr rd ra; QUESTION: 1, ANSWER: 2, AUTHORITY: 13, ADDITIONAL: 2
;; QUESTION SECTION:
;www.nvidia.com.edgekey.net.    IN      A

;; ANSWER SECTION:
www.nvidia.com.edgekey.net. 21559 IN    CNAME   e14462.a.akamaiedge.net.
e14462.a.akamaiedge.net. 19     IN      A       184.85.26.247

;; AUTHORITY SECTION:
edgekey.net.            13382   IN      NS      a18-65.akam.net.
edgekey.net.            13382   IN      NS      ns5-66.akam.net.
edgekey.net.            13382   IN      NS      a6-65.akam.net.
edgekey.net.            13382   IN      NS      ns7-65.akam.net.
edgekey.net.            13382   IN      NS      a5-65.akam.net.
edgekey.net.            13382   IN      NS      a13-65.akam.net.
edgekey.net.            13382   IN      NS      ns1-66.akam.net.
edgekey.net.            13382   IN      NS      ns4-66.akam.net.
edgekey.net.            13382   IN      NS      a28-65.akam.net.
edgekey.net.            13382   IN      NS      adns1.akam.net.
edgekey.net.            13382   IN      NS      a12-65.akam.net.
edgekey.net.            13382   IN      NS      usw6.akam.net.
edgekey.net.            13382   IN      NS      a16-65.akam.net.

;; ADDITIONAL SECTION:
a13-65.akam.net.        11507   IN      A       2.22.230.65
adns1.akam.net.         13382   IN      A       96.7.50.66
    1. Your BIND sees the non-authoritative answer arriving from a supposedly authoritative server (the answer itself includes the NS referrals, so BIND knows the responding server is not authoritative for the A record), and discards the entire response with FORMERR:
    DNS format error from 192.58.128.30#53 resolving www.nvidia.com.edgekey.net/A for client 127.0.0.1#53636: unrelated A e14462.a.akamaiedge.net in edgekey.net authority section

    The OP question includes an equivalent description of the problem (non-improving NS record == unrelated record in authority section):

    Mar 18 19:53:20 kenneth named[4022]: DNS format error from 192.112.36.4#53 resolving ./NS: non-improving referral
    1. Your BIND then attempts the same non-recursive query to the next root server, and the cycle repeats until there are no more root servers to try, and BIND responds with SERVFAIL to the original recursive query.

    Arguably the problem is one or several bugs in BIND itself:

    • It checks the consistency of the AUTHORITY section in answers to non-recursive questions, eg, when forwarding a query to an upstream cache. It should not reject inconsistent answers, as they may be caused by valid truncation of long AUTHORITY sections
    • When caching, it truncates answers naively. It should truncate more smartly and give AUTHORITY sections that are more useful when truncated, or give minimal answers (no AUTHORITY section) if the truncated AUTHORITY section would not be useful.
    • It should default to "minimal-responses yes;" in caching mode.
    • It should LOG/report if its authoritative queries are answered by non authoritative server (ie, RD flag unset in sent query, but set in received response), ie, detect and report intercepted DNS traffic.

    I should note that queries that can be answered completely in a small packet do not trigger these bugs, and therefore, most DNS queries are resolved correctly.

    Possible solutions:

    • ask your ISP to add "minimal-responses yes;" to their cache configuration.
    • get the ISP to not intercept your DNS, maybe by switching ISPs
    • run a different BIND version without the bugs (I don't know if such a version exists) or a different software.
    • route BIND's upstream queries through a VPN, and around the DNS intercept
    • 0