I have DNSMasq set up on a relatively idle Pentium D 3.4Ghz Debian Linux machine. When I run dig queries locally, the second result is always 0 ms. When I run dig queries on any other machine on my network, the cached response time is a constant 35ms. This is in spite of the fact that I get LAN ping times back -- under 1ms.
Using DNS Benchmark, I ran a test that shows I can hit my ISP's DNS servers faster than my own for cached queries.
How am I accruing 35ms on cached DNS responses for remote queries but <1ms for local queries at the server's command prompt?
I suspect this due to a low amount of memory on your local DNS server and it reading the cache DB entry for the name from the disk causing a slower response time. It could also be a poor quality network card causing a large amount of CPU usage... I would monitor the system resources when you do a look up from the host.
Check to see how much memory is free using the "free" command, you can do "free -h" for a more human readable output.
I ran into the same issue with my local dnsmasq instance being slower than opendns and many other ISP provided DNS servers. The following finally worked for me resulting in a massive improvement and my dnsmasq being 3X faster than any remote DNS server.
Adjust the cache-size parameter in /etc/dnsmasq.conf. I have set it to 10000.