We're experiencing a frustrating problem on our LAN. Periodically, DNS queries to our ISP nameservers timeout forcing a 5 second delay. Even if I bypass /etc/resolv.conf
by using a direct dig to one of our DNS servers, I still encounter the problem. Here's an example:
mv-m-dmouratis:~ dmourati$ time dig www.google.com @209.81.9.1
; <<>> DiG 9.8.3-P1 <<>> www.google.com @209.81.9.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14473
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 4, ADDITIONAL: 4
;; QUESTION SECTION:
;www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 174 IN A 74.125.239.148
www.google.com. 174 IN A 74.125.239.147
www.google.com. 174 IN A 74.125.239.146
www.google.com. 174 IN A 74.125.239.144
www.google.com. 174 IN A 74.125.239.145
;; AUTHORITY SECTION:
google.com. 34512 IN NS ns2.google.com.
google.com. 34512 IN NS ns1.google.com.
google.com. 34512 IN NS ns3.google.com.
google.com. 34512 IN NS ns4.google.com.
;; ADDITIONAL SECTION:
ns2.google.com. 212097 IN A 216.239.34.10
ns3.google.com. 207312 IN A 216.239.36.10
ns4.google.com. 212097 IN A 216.239.38.10
ns1.google.com. 212096 IN A 216.239.32.10
;; Query time: 8 msec
;; SERVER: 209.81.9.1#53(209.81.9.1)
;; WHEN: Fri Jul 26 14:44:25 2013
;; MSG SIZE rcvd: 248
real 0m5.015s
user 0m0.004s
sys 0m0.002s
Other times, the queries respond instantly, as in under 20 ms or so. I've done a packet trace and discovered something interesting. The DNS server is responding but the client ignores the initial response, then sends a second identical query which is immediately responded to.
See packet trace. Note the identical source ports to the queries (62076).
Question: what is causing the first DNS query to fail?
UPDATE
Resources:
Packet trace:
http://www.cloudshark.org/captures/8b1c32d9d015
Dtruss (strace for mac):
https://gist.github.com/dmourati/6115180
Mountain Lion firewall is randomly delaying DNS requests from apple.stackexchange.com:
UPDATE 2
System Software Overview:
System Version: OS X 10.8.4 (12E55)
Kernel Version: Darwin 12.4.0
Boot Volume: Macintosh HD
Boot Mode: Normal
Computer Name: mv-m-dmouratis
User Name: Demetri Mouratis (dmourati)
Secure Virtual Memory: Enabled
Time since boot: 43 minutes
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro10,1
Processor Name: Intel Core i7
Processor Speed: 2.7 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 6 MB
Memory: 16 GB
Firewall Settings:
Mode: Limit incoming connections to specific services and applications
Services:
Apple Remote Desktop: Allow all connections
Screen Sharing: Allow all connections
Applications:
com.apple.java.VisualVM.launcher: Block all connections
com.getdropbox.dropbox: Allow all connections
com.jetbrains.intellij.ce: Allow all connections
com.skype.skype: Allow all connections
com.yourcompany.Bitcoin-Qt: Allow all connections
org.m0k.transmission: Allow all connections
org.python.python: Allow all connections
Firewall Logging: Yes
Stealth Mode: No
This appears to be a bug in Lion's firewall. Is it enabled on your system?
In this MacRumors thread (DNS problems after updating to Mountain Lion (10.8)), a possible workaround is discussed:
Could you check whether reducing the MTU size mitigates your problem?
I had a similar issue recently and found that the Cisco ASA firewall wasn't configured to support EDNS0, the spec that allows DNS UDP packets larger then 512 bytes. Once my fw admin allowed up to 4096 bytes the issue was resolved. Great info here:
http://www.petenetlive.com/KB/Article/0000312.htm