Dunner1991's questions

I am implementing a solution to load balance DNS queries across multiple bind recursive DNS servers to increase QPS limit

Each centos VM has a namespace gi set up with the loopback of the ns set to asingle DNS Public IP

Each DNS server advertises the same DNS IP to my network across bgp peerings configured on my quagga router

all incoming queries are load-balanced via the network core using the bgp maximum-paths feature

However only 1 Bind DNS server will query the DNS IP, the other will just return servfail (this is not static, if i kill the bgp peerings to Server1, queries are succesful, the same happens if i kill the peerings to Server2) however they will not work in tandem.

One thing i have noticed is that if i do a

ip netns exec gi dig @DNSIP +trace

ip netns exec gi dig @DNSIP  cloudflare.com +trace

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @DNSIP cloudflare.com +trace
; (1 server found)
;; global options: +cmd
.           509520  IN  NS  e.root-servers.net.
.           509520  IN  NS  c.root-servers.net.
.           509520  IN  NS  f.root-servers.net.
.           509520  IN  NS  j.root-servers.net.
.           509520  IN  NS  b.root-servers.net.
.           509520  IN  NS  i.root-servers.net.
.           509520  IN  NS  h.root-servers.net.
.           509520  IN  NS  m.root-servers.net.
.           509520  IN  NS  k.root-servers.net.
.           509520  IN  NS  a.root-servers.net.
.           509520  IN  NS  l.root-servers.net.
.           509520  IN  NS  d.root-servers.net.
.           509520  IN  NS  g.root-servers.net.
.           509520  IN  RRSIG   NS 8 0 (didn't include the key)

whereas Server2 does not return an RRSIG even though both named.conf files have dnssec-enable yes and dnssec-validation yes

ip netns exec gi dig @DNSIP  cloudflare.com +trace

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-16.P2.el7_8.6 <<>> @DNSIP cloudflare.com +trace
; (1 server found)
;; global options: +cmd
.           518400  IN  NS  c.root-servers.net.
.           518400  IN  NS  k.root-servers.net.
.           518400  IN  NS  g.root-servers.net.
.           518400  IN  NS  d.root-servers.net.
.           518400  IN  NS  a.root-servers.net.
.           518400  IN  NS  j.root-servers.net.
.           518400  IN  NS  e.root-servers.net.
.           518400  IN  NS  h.root-servers.net.
.           518400  IN  NS  f.root-servers.net.
.           518400  IN  NS  i.root-servers.net.
.           518400  IN  NS  m.root-servers.net.
.           518400  IN  NS  b.root-servers.net.
.           518400  IN  NS  l.root-servers.net.

My dnssec configuration is as follows:

    dnssec-enable no;
    dnssec-validation no;

    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";

    managed-keys-directory "/var/named/dynamic";

If i disable dnssec in my named.conf file thr DNS servers work in tandem and I can achieve my target goal of 20,000 QPS, however with dnssec enabled it does not work.

Has anyone encountered a problem like this before, is it a limiation of BIND behind a single PublicIP? or is as I suspect an issue with DNSSEC setup

I have been benchmark testing 2 DNS servers over the last few days and I am running into the following issue

1 DNS server has a qps per seconf limit of ~6500 qps while the other DNS server gets only ~1500 qps when i use dnsperf

Statistics:
Queries sent:         77511
  Queries completed:    77511 (100.00%)
  Queries lost:         0 (0.00%)

  Response codes:       NOERROR 69982 (90.29%), NXDOMAIN 7529 (9.71%)
  Average packet size:  request 48, response 281
  Run time (s):         11.473787
  Queries per second:   6755.485351

  Average Latency (s):  1.190392 (min 0.000293, max 1.753760)
  Latency StdDev (s):   0.443875



Statistics:

  Queries sent:         34372
  Queries completed:    14676 (42.70%)
  Queries lost:         19696 (57.30%)

  Response codes:       NOERROR 13246 (90.26%), NXDOMAIN 1430 (9.74%)
  Average packet size:  request 47, response 247
  Run time (s):         10.132536
  Queries per second:   1448.403440

  Average Latency (s):  0.111563 (min 0.000680, max 1.615017)
  Latency StdDev (s):   0.050619

Both DNS servers are the same centos and bind versions they both have 16 vCPUs and 16 GB RAM They are located in different datacentres and each I have dnsperf (benchmark server) built locally for each.

Is there a way to increase this qps limit or a way to troubleshoot it?

I am in the process of rolling out external DNS servers to resolve all end user queries

Before we can move the solution into production I would like get as much useful logging in place as possible

here is the named.conf file (called gi-named.conf file for namespace)

options {
        listen-on port 53 { Public IP; };
        #listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };
        allow-query-on  { PublicIP; };

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
        allow-query-cache { Internal Range; };
        allow-query-cache-on  { PublicIP; };



        query-source address Public IP ;

        dnssec-enable yes;
        dnssec-validation yes;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";
};


logging
{
/*      If you want to enable debugging, eg. using the 'rndc trace' command,
 *      named will try to write the 'named.run' file in the $directory (/var/named).
 *      By default, SELinux policy does not allow named to modify the /var/named directory,
 *      so put the default debug log file in data/ :
 */
        /*channel default_debug {
                print-time yes;
                print-category yes;
                print-severity yes;
                file "data/named.run";
                severity dynamic;
        };*/
        channel queries_log {
                file "/var/log/queries" versions 1 size 20m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity debug 3;
        };
        channel default_log {
           file "/var/named/log/default" versions 3 size 20m;
           print-time yes;
           print-category yes;
           print-severity yes;
           severity info;
    };
       channel query-errors_log {
           file "/var/named/log/query-errors" versions 5 size 20m;
           print-time yes;
           print-category yes;
           print-severity yes;
           severity dynamic;
    };

        category queries { queries_log; };
        category client { queries_log;  };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

The issue I have is that while I see logs populating the /var/named/log/queries file i do not see any logs in either the /var/named/log/query-error log file or the /var/named/log/default log file and I am not sure where exectly I am going wrong or if maybe i have left out some configuration

Anyopne ever experienced this before?

I have a cluster of DNS servers which sit behind a public IP.

These servers resolve some of the time but other times they just return a ServFail error code for any queries

The setup I have is not typical (this was inherited).

Basically on the server there is a namespace called gi, here is where named service is been used by a new service call srv-gi '''

#!/bin/sh

start_service() {
        ip netns exec gi /usr/sbin/zebra -d -A 127.0.0.1 -f /etc/quagga/zebra.conf
        ip netns exec gi /usr/sbin/bgpd -d -A 127.0.0.1 -f /etc/quagga/bgpd.conf 
        #DNS service
        ip netns exec gi  /usr/sbin/named -u named -c /etc/gi-named.conf
}

start_service

'''

The named.conf file has also been renamed to gi-named.conf file.

// // named.conf // // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS // server as a caching only nameserver (as a localhost DNS resolver only). // // See /usr/share/doc/bind*/sample/ for example named configuration files. // // See the BIND Administrator's Reference Manual (ARM) for details about the // configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html

options {
        listen-on port 53 { Public IP; };
        #listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };
        allow-query-on  { PublicIP; };

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
        allow-query-cache { Internal Range; };
        allow-query-cache-on  { PublicIP; };



        query-source address Public IP ;

        dnssec-enable yes;
        dnssec-validation yes;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";
};


logging
{
/*      If you want to enable debugging, eg. using the 'rndc trace' command,
 *      named will try to write the 'named.run' file in the $directory (/var/named).
 *      By default, SELinux policy does not allow named to modify the /var/named directory,
 *      so put the default debug log file in data/ :
 */
        /*channel default_debug {
                print-time yes;
                print-category yes;
                print-severity yes;
                file "data/named.run";
                severity dynamic;
        };*/
        channel queries_log {
                file "/var/log/queries" versions 1 size 20m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity debug 3;
        };

        category queries { queries_log; };
        category client { queries_log;  };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

Also to note i have a quagga riuter configured to allow DNS resolution via Public IP

/etc/quagga/bgpd.conf

!
! Zebra configuration saved from vty
!   2019/10/11 10:11:45
!
!
router bgp AS
 bgp router-id PublicIP
 network PublicIP/32
 network CoreIP/32
 neighbor DUB1-WGW peer-group
 neighbor DUB1-WGW remote-as AS
 neighbor DUB1-WGW soft-reconfiguration inbound
 neighbor DUB1-WGW route-map XXXXX out
 neighbor CoreBGPIP peer-group DUB1-WGW
 neighbor CoreBGPIP peer-group DUB1-WGW
!
ip prefix-list XXXX seq 5 permit PublicIP/32
ip prefix-list XXXX seq 10 permit PrivateIP/32
!
route-map DNS_TO_GI permit 10
 match ip address prefix-list XXXXX
!
line vty
!

/etc/quagga/zebra.conf

!
! Zebra configuration saved from vty
!   2019/10/11 10:11:45
!
hostname hostname
!
interface ens160
 ipv6 nd suppress-ra
!
interface ens192
 ipv6 nd suppress-ra
!
interface ens192.890
 ipv6 nd suppress-ra
!
interface ens192.892
 ipv6 nd suppress-ra
!
interface XX
 ipv6 nd suppress-ra
!
interface lo
!
ip prefix-list XX seq 5 permit PublicIP3/32
ip prefix-list XX seq 10 permit PrivateIP/32
!
route-map XXXX permit 10
 match ip address prefix-list XXX
!
!
!
line vty
!

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, A - Babel,
       > - selected route, * - FIB route

B>* 0.0.0.0/0 [20/10] via neighbor IP, ens192.892, 00:02:18
C>* 127.0.0.0/8 is directly connected, lo
C>* Public IP/32 is directly connected, lo
C>* NeighborSubnet/30 is directly connected, ens192.890
C>* NeighborIP/30 is directly connected, ens192.892
C>* LocalIP/32 is directly connected, lo

I am testing resolution using a test APN and while I can get resolution one one APN as sson as I introduce a second APN i just encounter the following errors below from a tcpdump:

11:29:38.065284 IP PublicIP.domain > internal IP.p2pcommunity: 30622 ServFail 0/0/0 (44)
11:29:38.265736 IP PublicIP.domain > internal IP.32209: 12606 ServFail 0/0/0 (37)
11:29:38.266037 IP PublicIP.domain > internal IP.10793: 26678 ServFail 0/0/0 (37)
11:29:38.295727 IP PublicIP.domain > internal IP.ibm_wrless_lan: 23483 ServFail 0/0/0 (33)
11:29:38.296038 IP PublicIP.domain > internal IP.22097: 8347 ServFail 0/0/0 (33)
11:29:38.297532 IP PublicIP.domain > internal IP.31026: 23400 ServFail 0/0/0 (38)
11:29:38.298117 IP PublicIP.domain > internal IP.23707: 26481 ServFail 0/0/0 (38)

and from /var/log/queries

22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): error
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): send
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): sendto
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): error
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): senddone
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): next
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): endrequest
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): send
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): sendto
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): senddone
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): next
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): endrequest

I am really unsure of how to resolve this issue, any pointers ort advice would be greatly appreciated

Outputs of dig command

dig facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> facebook.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7204
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;facebook.com.          IN  A

;; ANSWER SECTION:
facebook.com.       93  IN  A   31.13.86.36

;; Query time: 2 msec
;; SERVER: internal DNS#53(Internal DNS)
;; WHEN: Tue Sep 22 19:38:58 UTC 2020
;; MSG SIZE  rcvd: 57


dig @PublicIP facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @PublicIP facebook.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

dig @208.67.222.222 facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @208.67.222.222 facebook.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

ip netns exec gi tcpdump -n -f 'port 53' -i any
09:55:35.676645 IP PublicIP.domain > InternalIP.46571: 36451 ServFail 0/0/0 (32)
09:55:35.676939 IP PublicIP.domain > InternalIP.37817: 52592 ServFail 0/0/0 (32)
09:55:35.677865 IP PublicIP.domain > InternalIP41737: 52624 ServFail 0/0/0 (32)
09:55:35.713870 IP PublicIP.34042 > 193.0.14.129.domain: 11264 [1au] A? mtalk.google.com. (45)
09:55:35.713914 IP PublicIP.11218 > 193.0.14.129.domain: 3623 [1au] NS? . (28)
09:55:35.768649 IP 193.0.14.129.domain > PublicIP.11218: 3623*-| 0/0/1 (28)
09:55:35.784456 IP 193.0.14.129.domain > PublicIP.34042: 11264-| 0/0/1 (45)
09:55:36.045130 IP PublicIP.wcbackup > 192.112.36.4.domain: 28368 A? update.googleapis.com. (39)
09:55:36.063323 IP InternalIP.49382 > PublicIP.domain: 57145+ A? accounts.google.com. (37)
09:55:36.064459 IP PublicIP.48169 > 193.0.14.129.domain: 15825 [1au] A? accounts.google.com. (48)
09:55:36.065883 IP APNIP.54312 > PublicIP.domain: 53585+ A? accounts.google.com. (37)
09:55:36.080202 IP 192.112.36.4.domain > PublicIP.wcbackup: 28368- 0/13/14 (499)
09:55:36.120905 IP 193.0.14.129.domain > PublicIP.48169: 15825- 0/15/27 (1182)
09:55:36.170289 IP InternalIP.59759 > PublicIP.domain: 52061+ A? www.google.com. (32)
09:55:36.224316 IP PublicIP.5346 > 192.112.36.4.domain: 40438 A? www.facebook.com. (34)
09:55:36.257993 IP 192.112.36.4.domain > PublicIP.5346: 40438- 0/13/14 (494)
09:55:36.441576 IP PublicIP.domain > InternalIP.65408: 45517 ServFail 0/0/0 (39)
09:55:36.441666 IP PublicIP.domain > InternalIP.60664: 54663 ServFail 0/0/0 (39)
09:55:36.442994 IP PublicIP.domain > InternalIP.48634: 56799 ServFail 0/0/0 (39)
09:55:36.443474 IP PublicIP.domain > InternalIP.36045: 34980 ServFail 0/0/0 (39)

I have been building DNS servers on centos 7 (i inherited the these but need to make changes to repurpose them)

When I run the any rndc command it does not work i.e. when i try rndc reload i get no confirmation if it was successful, do not even get command failed

Please also note that I am using linux namespaces so the typical rndc command has a prefic of 'ip entns exec gi'

If i check /var/log/messages simultaneously i do not get any output as compared to another DNS server.

I have posted the output of the necessary services below:

/usr/lib/systemd/system/named.service

[Unit]
Description=Berkeley Internet Name Domain (DNS)
Wants=nss-lookup.target
Wants=named-setup-rndc.service
Before=nss-lookup.target
After=network.target
After=named-setup-rndc.service

[Service]
Type=forking
Environment=NAMEDCONF=/etc/named.conf
EnvironmentFile=-/etc/sysconfig/named
Environment=KRB5_KTNAME=/etc/named.keytab
PIDFile=/run/named/named.pid

ExecStartPre=/bin/bash -c 'if [ ! "$DISABLE_ZONE_CHECKING" == "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else echo "Checking of zone files is disabled"; fi'

ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF} $OPTIONS

ExecReload=/bin/sh -c '/usr/sbin/rndc reload > /dev/null 2>&1 || /bin/kill -HUP $MAINPID'

ExecStop=/bin/sh -c '/usr/sbin/rndc stop > /dev/null 2>&1 || /bin/kill -TERM $MAINPID'

PrivateTmp=true

[Install]
WantedBy=multi-user.target

/usr/lib/systemd/system/named-setup-rndc.service

[Unit]
Description=Generate rndc key for BIND (DNS)

[Service]
Type=oneshot

ExecStart=/usr/libexec/generate-rndc-key.sh

I have tried to delete the rndc.key file and recreate using the named service but this has not worked. I am really stumped here. Anyone have any experience with this

after named restart the logs

-- Unit named-setup-rndc.service has begun starting up.
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 systemd[1]: Started Generate rndc key for BIND (DNS).
-- Subject: Unit named-setup-rndc.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit named-setup-rndc.service has finished starting up.
-- 
-- The start-up result is done.
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 systemd[1]: Starting Berkeley Internet Name Domain (DNS)...
-- Subject: Unit named.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit named.service has begun starting up.
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: starting BIND 9.9.4-RedHat-9.9.4-74.el7_6.1 -u named -c /etc/named.conf -4
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: built with '--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu' '-
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: ----------------------------------------------------
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: BIND 9 is maintained by Internet Systems Consortium,
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: corporation.  Support and training for BIND 9 are
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: available at https://www.isc.org/support
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: ----------------------------------------------------
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: adjusted limit on open files from 4096 to 1048576
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: found 8 CPUs, using 8 worker threads
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: using 8 UDP listeners per interface
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: using up to 21000 sockets
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: loading configuration from '/etc/named.conf'
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: reading built-in trusted keys from file '/etc/named.root.key'
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: initializing GeoIP Country (IPv4) (type 1) DB
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: GEO-106FREE 20180327 Build 1 Copyright (c) 2018 MaxMind Inc All Rights Reserved
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: initializing GeoIP Country (IPv6) (type 12) DB
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: GEO-106FREE 20180605 Build 1 Copyright (c) 2018 MaxMind Inc All Rights Reserved
Sep 24 14:01:40 DUB1-00-VM-GI_DNS1 named[12637]: GeoIP City (IPv4) (type 2) DB not available

Also i believe the debug logs are setup to be written to /var/named/data/named.run posted output below

tail -f /var/named/data/named.run
all zones loaded
running
shutting down
stopping command channel on 127.0.0.1#953
stopping command channel on ::1#953
no longer listening on 127.0.0.1#53
exiting
managed-keys-zone: loaded serial 1021
all zones loaded
running

I have been wracking my brain trying to resolve an issue using bind for DNS resolution in centos

The setup I have is not typical (this was inherited).

Basically on the server there is a namespace called gi, here is where named service is been used by a new service call srv-gi '''

#!/bin/sh

start_service() {
        ip netns exec gi /usr/sbin/zebra -d -A 127.0.0.1 -f /etc/quagga/zebra.conf
        ip netns exec gi /usr/sbin/bgpd -d -A 127.0.0.1 -f /etc/quagga/bgpd.conf 
        #DNS service
        ip netns exec gi  /usr/sbin/named -u named -c /etc/gi-named.conf
}

start_service

'''

The named.conf file has also been renamed to gi-named.conf file.

// // named.conf // // Provided by Red Hat bind package to configure the ISC BIND named(8) DNS // server as a caching only nameserver (as a localhost DNS resolver only). // // See /usr/share/doc/bind*/sample/ for example named configuration files. // // See the BIND Administrator's Reference Manual (ARM) for details about the // configuration located in /usr/share/doc/bind-{version}/Bv9ARM.html

options {
        listen-on port 53 { Public IP; };
        #listen-on-v6 port 53 { ::1; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        recursing-file  "/var/named/data/named.recursing";
        secroots-file   "/var/named/data/named.secroots";
        allow-query     { any; };
        allow-query-on  { PublicIP; };

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;
        allow-query-cache { Internal Range; };
        allow-query-cache-on  { PublicIP; };



        query-source address Public IP ;

        dnssec-enable yes;
        dnssec-validation yes;

        /* Path to ISC DLV key */
        bindkeys-file "/etc/named.iscdlv.key";

        managed-keys-directory "/var/named/dynamic";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";
};


logging
{
/*      If you want to enable debugging, eg. using the 'rndc trace' command,
 *      named will try to write the 'named.run' file in the $directory (/var/named).
 *      By default, SELinux policy does not allow named to modify the /var/named directory,
 *      so put the default debug log file in data/ :
 */
        /*channel default_debug {
                print-time yes;
                print-category yes;
                print-severity yes;
                file "data/named.run";
                severity dynamic;
        };*/
        channel queries_log {
                file "/var/log/queries" versions 1 size 20m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity debug 3;
        };

        category queries { queries_log; };
        category client { queries_log;  };
};

zone "." IN {
        type hint;
        file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

Also to note i have a quagga riuter configured to allow DNS resolution via Public IP

/etc/quagga/bgpd.conf

!
! Zebra configuration saved from vty
!   2019/10/11 10:11:45
!
!
router bgp AS
 bgp router-id PublicIP
 network PublicIP/32
 network CoreIP/32
 neighbor DUB1-WGW peer-group
 neighbor DUB1-WGW remote-as AS
 neighbor DUB1-WGW soft-reconfiguration inbound
 neighbor DUB1-WGW route-map XXXXX out
 neighbor CoreBGPIP peer-group DUB1-WGW
 neighbor CoreBGPIP peer-group DUB1-WGW
!
ip prefix-list XXXX seq 5 permit PublicIP/32
ip prefix-list XXXX seq 10 permit PrivateIP/32
!
route-map DNS_TO_GI permit 10
 match ip address prefix-list XXXXX
!
line vty
!

/etc/quagga/zebra.conf

!
! Zebra configuration saved from vty
!   2019/10/11 10:11:45
!
hostname hostname
!
interface ens160
 ipv6 nd suppress-ra
!
interface ens192
 ipv6 nd suppress-ra
!
interface ens192.890
 ipv6 nd suppress-ra
!
interface ens192.892
 ipv6 nd suppress-ra
!
interface XX
 ipv6 nd suppress-ra
!
interface lo
!
ip prefix-list XX seq 5 permit PublicIP3/32
ip prefix-list XX seq 10 permit PrivateIP/32
!
route-map XXXX permit 10
 match ip address prefix-list XXX
!
!
!
line vty
!

# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, A - Babel,
       > - selected route, * - FIB route

B>* 0.0.0.0/0 [20/10] via neighbor IP, ens192.892, 00:02:18
C>* 127.0.0.0/8 is directly connected, lo
C>* Public IP/32 is directly connected, lo
C>* NeighborSubnet/30 is directly connected, ens192.890
C>* NeighborIP/30 is directly connected, ens192.892
C>* LocalIP/32 is directly connected, lo

I am testing resolution using a test APN and while I can get resolution one one APN as sson as I introduce a second APN i just encounter the following errors below from a tcpdump:

11:29:38.065284 IP PublicIP.domain > internal IP.p2pcommunity: 30622 ServFail 0/0/0 (44)
11:29:38.265736 IP PublicIP.domain > internal IP.32209: 12606 ServFail 0/0/0 (37)
11:29:38.266037 IP PublicIP.domain > internal IP.10793: 26678 ServFail 0/0/0 (37)
11:29:38.295727 IP PublicIP.domain > internal IP.ibm_wrless_lan: 23483 ServFail 0/0/0 (33)
11:29:38.296038 IP PublicIP.domain > internal IP.22097: 8347 ServFail 0/0/0 (33)
11:29:38.297532 IP PublicIP.domain > internal IP.31026: 23400 ServFail 0/0/0 (38)
11:29:38.298117 IP PublicIP.domain > internal IP.23707: 26481 ServFail 0/0/0 (38)

and from /var/log/queries

22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): error
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): send
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): sendto
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): error
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): senddone
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): next
22-Sep-2020 11:31:07.552 client: debug 3: client InternalIP#61793 (www.facebook.com): endrequest
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): send
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): sendto
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): senddone
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): next
22-Sep-2020 11:31:07.553 client: debug 3: client InternalIP#48008 (2.android.pool.ntp.org): endrequest

I am really unsure of how to resolve this issue, any pointers ort advice would be greatly appreciated

Outputs of dig command

dig facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> facebook.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 7204
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;facebook.com.          IN  A

;; ANSWER SECTION:
facebook.com.       93  IN  A   31.13.86.36

;; Query time: 2 msec
;; SERVER: internal DNS#53(Internal DNS)
;; WHEN: Tue Sep 22 19:38:58 UTC 2020
;; MSG SIZE  rcvd: 57


dig @PublicIP facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @PublicIP facebook.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

dig @208.67.222.222 facebook.com

; <<>> DiG 9.9.4-RedHat-9.9.4-74.el7_6.1 <<>> @208.67.222.222 facebook.com
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

ip netns exec gi tcpdump -n -f 'port 53' -i any
09:55:35.676645 IP PublicIP.domain > InternalIP.46571: 36451 ServFail 0/0/0 (32)
09:55:35.676939 IP PublicIP.domain > InternalIP.37817: 52592 ServFail 0/0/0 (32)
09:55:35.677865 IP PublicIP.domain > InternalIP41737: 52624 ServFail 0/0/0 (32)
09:55:35.713870 IP PublicIP.34042 > 193.0.14.129.domain: 11264 [1au] A? mtalk.google.com. (45)
09:55:35.713914 IP PublicIP.11218 > 193.0.14.129.domain: 3623 [1au] NS? . (28)
09:55:35.768649 IP 193.0.14.129.domain > PublicIP.11218: 3623*-| 0/0/1 (28)
09:55:35.784456 IP 193.0.14.129.domain > PublicIP.34042: 11264-| 0/0/1 (45)
09:55:36.045130 IP PublicIP.wcbackup > 192.112.36.4.domain: 28368 A? update.googleapis.com. (39)
09:55:36.063323 IP InternalIP.49382 > PublicIP.domain: 57145+ A? accounts.google.com. (37)
09:55:36.064459 IP PublicIP.48169 > 193.0.14.129.domain: 15825 [1au] A? accounts.google.com. (48)
09:55:36.065883 IP APNIP.54312 > PublicIP.domain: 53585+ A? accounts.google.com. (37)
09:55:36.080202 IP 192.112.36.4.domain > PublicIP.wcbackup: 28368- 0/13/14 (499)
09:55:36.120905 IP 193.0.14.129.domain > PublicIP.48169: 15825- 0/15/27 (1182)
09:55:36.170289 IP InternalIP.59759 > PublicIP.domain: 52061+ A? www.google.com. (32)
09:55:36.224316 IP PublicIP.5346 > 192.112.36.4.domain: 40438 A? www.facebook.com. (34)
09:55:36.257993 IP 192.112.36.4.domain > PublicIP.5346: 40438- 0/13/14 (494)
09:55:36.441576 IP PublicIP.domain > InternalIP.65408: 45517 ServFail 0/0/0 (39)
09:55:36.441666 IP PublicIP.domain > InternalIP.60664: 54663 ServFail 0/0/0 (39)
09:55:36.442994 IP PublicIP.domain > InternalIP.48634: 56799 ServFail 0/0/0 (39)
09:55:36.443474 IP PublicIP.domain > InternalIP.36045: 34980 ServFail 0/0/0 (39)