Andrew B's questions

Here's an odd one. I noticed that one of my nameservers was complaining about some outbound congestion, and that it seemed to correlate incrementing interface discards on my outbound path. Open and shut, right? When I looked closer though, I noticed that the system-wide counter was incrementing but the per-interface counters were not:

# snmpbulkwalk -v2c -cexample localhost 1.3.6.1.2.1.4.31 | grep Discards
IP-MIB::ipSystemStatsInDiscards.ipv4 = Counter32: 0
IP-MIB::ipSystemStatsInDiscards.ipv6 = Counter32: 0
IP-MIB::ipSystemStatsOutDiscards.ipv4 = Counter32: 14856
IP-MIB::ipSystemStatsOutDiscards.ipv6 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.1 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.2 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.3 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.4 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.5 = Counter32: 0
IP-MIB::ipIfStatsInDiscards.ipv6.6 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.1 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.2 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.3 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.4 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.5 = Counter32: 0
IP-MIB::ipIfStatsOutDiscards.ipv6.6 = Counter32: 0

My understanding of these counters was that the SystemStats counters reflect system-wide statistics (aggregate), but that doesn't appear to be the case here. A glance at the MIB seems to confirm my memory on the subject:

ipSystemStatsTable OBJECT-TYPE
    SYNTAX     SEQUENCE OF IpSystemStatsEntry
    MAX-ACCESS not-accessible
    STATUS     current
    DESCRIPTION
           "The table containing system wide, IP version specific
            traffic statistics.  This table and the ipIfStatsTable
            contain similar objects whose difference is in their
            granularity.  Where this table contains system wide traffic
            statistics, the ipIfStatsTable contains the same statistics
            but counted on a per-interface basis."
    ::= { ipTrafficStats 1 }

Has anyone run into this before? Trying to determine whether this should be chalked up to the Broadcom NICs, or if there's something new that I could be learning here. For reference, eth2 and eth3 are bonded, but all three interfaces have their own counters in the table above.

• Chassis: Dell r620
• OS: RHEL6
• Kernel: 2.6.32-573.22.1.el6.x86_64

NIC info:

# dmesg | grep Ethernet
tg3 0000:01:00.0: eth0: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0000:01:00.1: eth1: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0000:02:00.0: eth2: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
tg3 0000:02:00.1: eth3: attached PHY is 5720C (10/100/1000Base-T Ethernet) (WireSpeed[1], EEE[1])
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
# ethtool -i eth3
driver: tg3
version: 3.137
firmware-version: FFV7.8.53 bc 5720-v1.32
bus-info: 0000:02:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

It's extremely common for RFCs to be cited in support of opinions (including Serverfault Q&A's), but the average IT employee has a very poor understanding in regards to which RFCs define standards and which ones are purely informative. This should be no surprise: system administrators of all experience levels typically avoid glazing their eyes at RFCs unless they have no choice but to.

On a site like ours, it is extremely important that we don't perpetuate common misunderstandings in our upvoted answers. Random users cruising in from search engines are going to assume that upvotes with no disputing comments are sufficient indicators of vetting. Recently I stumbled across an answer from 2011 making it apparent that this is definitely not getting caught in some cases as we upvote and probably warrants some efforts to inform our community and the internet at large.

So without further ado, how does one differentiate between a RFC that is quotable as an internet standard and one that is purely informative?

This is a Canonical Question about rate limiting in the ISC BIND nameserver software.

Recently I've heard of BIND's rate limiting features. DNS RRL in particular seems to be becoming more commonplace.

I'm a little bit confused. Some of the documentation seems to say that these features should not be used with recursive DNS servers, but other documents very specifically address recursive DNS infrastructure. Who is right?

This is a Canonical Question about DNS geo-redundancy.

It's extremely common knowledge that geo-redundant DNS servers located at separate physical locations are highly desirable when providing resilient web services. This is covered in-depth by document BCP 16, but some of the most frequently mentioned reasons include:

• Protection against datacenter disasters. Earthquakes happen. Fires happen in racks and take out nearby servers and network equipment. Multiple DNS servers won't do you much good if physical problems at the datacenter knock out both DNS servers at once, even if they're not in the same row.

• Protection against upstream peer problems. Multiple DNS servers won't prevent problems if a shared upstream network peer takes a dirt nap. Whether the upstream problem completely takes you offline, or simply isolates all of your DNS servers from a fraction of your userbase, the end result is that people can't access your domain even if the services themselves are located in a completely different datacenter.

That's all well and good, but are redundant DNS servers really necessary if I'm running all of my services off of the same IP address? I can't see how having a second DNS server would provide me any benefit if no one can get to anything provided by my domain anyway.

I understand that this is considered a best practice, but this really seems pointless!

I work for a company with a split DNS configuration on the AD domain. I know that this is less than ideal, but I'm not in a position to drive change in this area. I own authoritative DNS (internal and external) where Active Directory is not involved, and another team owns the domain controllers.

Background:

• We have a split domain called example.com which lives on all the domain controllers.
• The DCs are configured to use forwarders for all domains that they are not authoritative for.
• There is a subdomain of that (sub.example.com) that is delegated to public IP addresses in a DMZ using NS records. I have a need to eliminate these IP addresses use internal IP addresses that are outside the DMZ.
• The new IP addresses are reachable from the forwarders, but not the domain controllers.

To represent this visually:

    example.com. (DCs are authoritative)
sub.example.com. (subdomain not managed by the DCs)

I would like to have the sub.example.com. NS records converted to a conditional forwarder that sends traffic along to the standard forwarders, but I am being told by our domain admins that Windows DNS will not allow a forwarder within a forward lookup zone.

Is it true that this is an unsupported configuration? Other DNS products have no problem with a forwarder that is beneath an authoritative zone, so I want to make sure I'm working with the correct information before I move on to a different strategy, such as firewall holes for every DC that bypass the forwarders. (argh)

I've already reviewed Forward requests for subdomain to another DNS server in Windows 2k3 and the accepted answer that recommends a NS delegation, which doesn't answer this question.

(Disclaimer: I am not a Windows DNS admin. I do have a decent amount of DNS experience under my belt though, and this is not making any sense. I am working closely with the admins responsible for these devices and can get tests performed as needed.)

We have run into an issue where we cannot add conditional forwarders that point to BIND nameservers under Windows Server 2012. Adding the IP address of the server results in a validation error: An unknown error occurred while validating the server.

Looking at the query log on the BIND server, we found something rather interesting: the Windows DNS server was querying for . IN SOA, i.e. the SOA record for the root nameservers. No query for example.com. IN SOA at all. It attempts to query for root authority and doesn't continue when it receives a response of REFUSED.

client 192.168.203.20#59067 (.): query: . IN SOA - (192.168.208.201)
client 192.168.203.20#50553 (.): query: . IN SOA - (192.168.208.201)
client 192.168.203.20#55468 (.): query: . IN SOA - (192.168.208.201)

Madness. To humor it, we reproduced this problem in the lab. I downloaded a copy of the root zone and configured a . zone (commenting out my root hints), and lo and behold, this error no longer occurs.

I really don't get this. I'm providing an authoritative nameserver that shouldn't have to provide answers to . SOA, and as matters stand I'm going to have to add this zone to all of our production servers just to play nicely with Windows 2012. In my experience, a forwarder should only be concerned with whether or not the target nameserver is authoritative for the zone in question.

Why is this happening?

If we try to ignore the error (click on OK anyway), we get the following error dialog:

The query log still shows that the upstream server is only asking for . IN SOA. There is never an attempt to see if the server is authoritative for example.com..

This is a Canonical Question about DNS server administration.

I have one hundred or so domains. All of these domains need to be configured identically, but it seems like a huge waste of time to have to configure a new zone and/or zonefile for every one of these domains. There has to be a better way to automate this!

I think I'm on to something...if I create a zone called ., or use some other feature in my DNS software to always return a specific IP when an A record is requested, this seems to get me pretty close to my desired end result. My server is responding authoritatively to the requests and it's so much easier to manage!

This was working great until nameserver validation software started checking these domains. I figured out that I can make most of the errors go away by adding NS records, but my software won't let me put more than one SOA record in the same zone file.

How do I work around this multiple SOA record problem?

I have read and understood Can you help me with my capacity planning?, but I'm not sure I understand what my next steps are in a DNS server scenario. I think my CPU loads are high or that I might be starting to drop queries, but I'd like to better understand the load of my server before I take action on it. This is particularly concerning to me because it's common knowledge that scaling your infrastructure to DDoS loads is losing battle.

What should I be analyzing in order to understand my environment?

This is a Canonical Question about securing public DNS resolvers

Open DNS servers seem pretty neat and convenient, as they provide IP addresses that we can use consistently across our company regardless of where they are located. Google and OpenDNS provide this functionality, but I'm not sure that I want these companies to have access to our DNS queries.

I want to set up something like this for use by our company, but I hear a lot about this being dangerous practice (particularly in regards to amplification attacks) and I want to make sure that we do this right. What things do I need to keep in mind when building this type of environment?

This is a Canonical Question about CNAMEs at the apices (or roots) of zones

It's relatively common knowledge that CNAME records at the apex of a domain are a taboo practice.

Example: example.com. IN CNAME ithurts.example.net.

In a best case scenario nameserver software might refuse to load the configuration, and in the worst case it might accept this configuration and invalidate the configuration for example.com.

Recently I had a webhosting company pass instructions to a business unit that we needed to CNAME the apex of our domain to a new record. Knowing that this would be a suicide config when fed to BIND, I advised them that we would not be able to comply and that this was bunk advice in general. The webhosting company took the stance that it is not outright forbidden by standard defining RFCs and that their software supports it. If we could not CNAME the apex, their advice was to have no apex record at all and they would not provide a redirecting webserver. ...What?

Most of us know that RFC1912 insists that A CNAME record is not allowed to coexist with any other data., but let's be honest with ourselves here, that RFC is only Informational. The closest I know to verbiage that forbids the practice is from RFC1034:

If a CNAME RR is present at a node, no other data should be present; this ensures that the data for a canonical name and its aliases cannot be different.

Unfortunately I've been in the industry long enough to know that "should not" is not the same as "must not", and that's enough rope for most software designers to hang themselves with. Knowing that anything short of a concise link to a slam dunk would be a waste of my time, I ended up letting the company get away with a scolding for recommending configurations that could break commonly used software without proper disclosure.

This brings us to the Q&A. For once I'd like us to get really technical about the insanity of apex CNAMEs, and not skirt around the issue like we usually do when someone posts on the subject. RFC1912 is off limits, as are any other Informational RFC applicable here that I didn't think of. Let's shut this baby down.

The requirements surrounding reverse DNS are confusing! People frequently talk about everything breaking if reverse DNS is not present, and that sounds scary. Even in cases where applications don't require reverse DNS, RFCs are frequently cited in support of mandatory PTR record creation.

Some of these RFCs include:

RFC1912: Common DNS Operational and Configuration Errors

Every Internet-reachable host should have a name. ... Make sure your PTR and A records match. For every IP address, there should be a matching PTR record in the in-addr.arpa domain. If a host is multi-homed, (more than one IP address) make sure that all IP addresses have a corresponding PTR record (not just the first one). Failure to have matching PTR and A records can cause loss of Internet services similar to not being registered in the DNS at all.

RFC1033: DOMAIN ADMINISTRATORS OPERATIONS GUIDE

Adding a host.

 To add a new host to your zone files:

    Edit the appropriate zone file for the domain the host is in.

    Add an entry for each address of the host.

    Optionally add CNAME, HINFO, WKS, and MX records.

    Add the reverse IN-ADDR entry for each host address in the
    appropriate zone files for each network the host in on.

Despite all of that, some people still insist on saying that PTR record creation is not required by the standards governing DNS administration. What are the actual requirements?

My team has a server pointing at the DNS supplied by Active Directory to ensure that it is able to reach any hosts managed by the domain. Unfortunately, my team also needs to run dig +trace frequently and we sporadically get strange results. I am a DNS admin but not a domain admin, but the team responsible for these servers isn't sure what is going on here either.

The problem seems to have shifted around between OS upgrades, but it's hard to say whether that's a characteristic of the OS version or other settings being changed during the upgrade process.

• When the upstream servers were Windows Server 2003, the first step of dig +trace (request . IN NS from the first entry in /etc/resolv.conf) would occasionally return 0 byte responses.
• When the upstream servers were upgraded to Windows Server 2012, the zero byte response problem went away but was replaced with an issue where we would sporadically get the list of forwarders configured on the DNS server.

Example of the second problem:

$ dig +trace -x 1.2.3.4      

; <<>> DiG 9.8.2 <<>> +trace -x 1.2.3.4
;; global options: +cmd
.                       3600    IN      NS      dns2.ad.example.com.
.                       3600    IN      NS      dns1.ad.example.com.
;; Received 102 bytes from 192.0.2.11#53(192.0.2.11) in 22 ms

1.in-addr.arpa.         84981   IN      NS      ns1.apnic.net.
1.in-addr.arpa.         84981   IN      NS      tinnie.arin.net.
1.in-addr.arpa.         84981   IN      NS      sec1.authdns.ripe.net.
1.in-addr.arpa.         84981   IN      NS      ns2.lacnic.net.
1.in-addr.arpa.         84981   IN      NS      ns3.apnic.net.
1.in-addr.arpa.         84981   IN      NS      apnic1.dnsnode.net.
1.in-addr.arpa.         84981   IN      NS      ns4.apnic.net.
;; Received 507 bytes from 192.0.2.228#53(192.0.2.228) in 45 ms

1.in-addr.arpa.         172800  IN      SOA     ns1.apnic.net. read-txt-record-of-zone-first-dns-admin.apnic.net.
4827 7200 1800 604800 172800
;; Received 127 bytes from 202.12.28.131#53(202.12.28.131) in 167 ms

In most cases this isn't a problem, but it will cause dig +trace to follow the wrong path if we are tracing within a domain that AD has an internal view for.

Why is dig +trace losing its mind? And why do we seem to be the only ones complaining?

$ORIGIN example.com. ; not necessary, using this to self-document

$TTL 3600
@        IN     SOA   ns1.example.com. admin.example.com. (
                      1970010100 7200 1800 1209600 300)

@        IN     NS   ns1.example.com.
@        IN     NS   ns2.example.com.

@        IN     A    198.51.100.1
ns1      IN     A    198.51.100.2
ns2      IN     A    198.51.100.3

sub1     IN     NS   ns1.example.edu.

sub2     IN     NS   ns1.sub2
ns1.sub2 IN     A    203.0.113.1 ; inline glue record

The role of a NS record beneath the apex of a domain is well-understood; they exist to delegate authority for a subdomain to another nameserver. Examples of this above would include the NS records for sub1 and sub2. These allow the nameserver to hand out referrals for portions of the domain that it does not consider itself authoritative for.

The purpose of the NS records at the apex of a domain, ns1 and ns2 in this case, seem to be less understood by the internet at large. My understanding (which may not be holistic) is as follows:

1. They are not used by caching DNS servers in order to determine the authoritative servers for the domain. This is handled by nameserver glue, which is defined at the registrar level. The registrar never uses this information to generate the glue records.
2. They are not used to delegate authority for the entire domain to other nameservers. Trying to do so with software such as ISC BIND will not result in the "expected" referral behavior at all, as the nameserver will continue to consider itself authoritative for the zone.
3. They are not used by the nameserver to determine whether it should return authoritative responses (AA flag set) or not; that behavior is defined by whether the software is told to be a master or a slave for the zone. Most nameserver software will quite happily serve apex NS records that disagree with the information contained by the upstream glue records, which will in turn cause well-known DNS validation websites to generate warnings for the domain.

With this being the case, what are we left with? Why are we defining this information if it doesn't appear to be consumed by caching DNS servers on the internet at large?

I'm finding instances of syslog-ng writing out garbage followed by a blank kernel.emerg line in one of our production environments. Example of one:

Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <Q▒b
+\c 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <;E0
Dec 21 00:14:56 someserver [syslog-ng.err] Error processing log message: <▒"▒l
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒▒▒e▒F
Dec 21 00:14:57 someserver [syslog-ng.err] Error processing log message: <▒▒
Dec 21 00:14:58 someserver [kernel.emerg]

The kernel.emerg line is of particular concern to me. According to man 3 syslog:

       LOG_KERN
              kernel messages (these can’t be generated from user processes)

This seems to suggest that the kernel facility cannot be spoofed to. I can see where the system calls themselves might prevent such spoofing, but am I correct in thinking that there is nothing to stop a process from writing directly to /dev/log and spoofing a facility of kernel? I want to say that the only thing that could truly stop such spoofing would be the syslog daemon differentiating between whether it obtained a message from /proc/kmsg vs. other sources.

• Distro is RHEL5.5. Kernel version is 2.6.18-194.8.1.el5. These are not factors I have control over yet; consider me scolded.
• The syslog daemon is a company-built syslog-ng 3.1.4 package (32-bit and running on a 64-bit kernel, but I wouldn't expact that to be related).
• There are some mailing list posts I found via Google suggesting that changes to kmsg in kernel 3.5 can create output errors like this, but that's definitely not the case here.

The messages are not coming in from the network. This is the only source defined:

source s_syslog {
# message generated by Syslog-NG
internal();
# standard Linux log source (this is the default place for the syslog()
# function to send logs to)
unix-stream("/dev/log");
# messages from the kernel
file("/proc/kmsg" program_override("kernel: "));
};

Server: Poweredge r620
OS: RHEL 6.4
Kernel: 2.6.32-358.18.1.el6.x86_64

I'm experiencing application alarms in my production environment. Critical CPU hungry processes are being starved of resources and causing a processing backlog. The problem is happening on all the 12th Generation Dell servers (r620s) in a recently deployed cluster. As near as I can tell, instances of this happening are matching up to peak CPU utilization, accompanied by massive amounts of "power limit notification" spam in dmesg. An excerpt of one of these events:

Nov  7 10:15:15 someserver [.crit] CPU12: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU6: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU14: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU0: Package power limit notification (total events = 11)
Nov  7 10:15:15 someserver [.crit] CPU6: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU14: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU18: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU20: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU2: Package power limit notification (total events = 12)
Nov  7 10:15:15 someserver [.crit] CPU10: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Core power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU4: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU16: Package power limit notification (total events = 13)
Nov  7 10:15:15 someserver [.crit] CPU20: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU8: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU10: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU22: Package power limit notification (total events = 14)
Nov  7 10:15:15 someserver [.crit] CPU15: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU3: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU1: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU5: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU15: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU3: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU1: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU5: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU19: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU17: Package power limit notification (total events = 377)
Nov  7 10:15:15 someserver [.crit] CPU9: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU21: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU23: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU11: Core power limit notification (total events = 369)
Nov  7 10:15:15 someserver [.crit] CPU13: Package power limit notification (total events = 376)
Nov  7 10:15:15 someserver [.crit] CPU7: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU19: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU9: Package power limit notification (total events = 374)
Nov  7 10:15:15 someserver [.crit] CPU21: Package power limit notification (total events = 375)
Nov  7 10:15:15 someserver [.crit] CPU23: Package power limit notification (total events = 374)

A little Google Fu reveals that this is typically associated with the CPU running hot, or voltage regulation kicking in. I don't think that's what is happening though. Temperature sensors for all servers in the cluster are running fine, Power Cap Policy is disabled in the iDRAC, and my System Profile is set to "Performance" on all of these servers:

# omreport chassis biossetup | grep -A10 'System Profile'
System Profile Settings
------------------------------------------
System Profile                                    : Performance
CPU Power Management                              : Maximum Performance
Memory Frequency                                  : Maximum Performance
Turbo Boost                                       : Enabled
C1E                                               : Disabled
C States                                          : Disabled
Monitor/Mwait                                     : Enabled
Memory Patrol Scrub                               : Standard
Memory Refresh Rate                               : 1x
Memory Operating Voltage                          : Auto
Collaborative CPU Performance Control             : Disabled

• A Dell mailing list post describes the symptoms almost perfectly. Dell suggested that the author try using the Performance profile, but that didn't help. He ended up applying some settings in Dell's guide for configuring a server for low latency environments and one of those settings (or a combination thereof) seems to have fixed the problem.
• Kernel.org bug #36182 notes that power-limit interrupt debugging was enabled by default, which is causing performance degradation in scenarios where CPU voltage regulation is kicking in.
• A RHN KB article (RHN login required) mentions a problem impacting PE r620 and r720 servers not running the Performance profile, and recommends an update to a kernel released two weeks ago. ...Except we are running the Performance profile...

Everything I can find online is running me in circles here. What's the heck is going on?

BIND 9.4.1 implemented automatic empty reverse zones for private address space in order to prevent reverse DNS leakage. This saves an administrator the trouble of having to define these empty zones, and reduces the number of "out of the box" configurations that will leak reverse DNS lookups to IANA's blackholing servers.

What is the default format of the synthesized replies from an automatic empty reverse zone? The format isn't completely described within the documentation. (just the defaults of a few options)

This question is similar, but doesn't elaborate on the confusing case of a why a NS record cannot be obtained.

One of our caching DNS environments (RHEL 5.8, BIND 9.3.6-20.P1.el5_8.4) has ceased to return any useful data at all for a zone. Usually this sort of problem ends up being a stale NS or glue record, but in this particular case I can't seem to even get the cache to report a NS record for the zone.

• dig @mycache somedomain NS returns SERVFAIL. There are no nameserver records cached at all.
• dig +trace shows a healthy delegation path, with the final nameserver returning a response. Manually running the dig query against the final nameserver returns a valid NS record, the corresponding A record exists and agrees with the glue, etc.

What gives? Why is there no NS record for me to obtain from the DNS cache, not even a bad one?

When the accuracy of a DNS cache is in question, dig +trace tends to be the recommended way of determining the authoritative answer for an internet facing DNS record. This seems to be particularly useful when also paired with +additional, which also shows the glue records.

Occasionally there seems to be some disagreement on this point -- some people say that it relies on the local resolver to look up the IP addresses of the intermediate nameservers, but the command output offers no indication that this is happening beyond the initial list of root nameservers. It seems logical to assume that this wouldn't be the case if the purpose of +trace is to start at the root servers and trace your way down. (at least if you have the right list of root nameservers)

Does dig +trace really use the local resolver for anything past the root nameservers?

Background:

I've inherited a high volume caching nameserver environment (Redhat Enterprise Linux 5.8, IBM System x3550) that has inconsistent ring buffer settings: 1020 for eth0 and 255 for eth1. eth0 is connected to switch 1 of its local datacenter, eth1 is connected to switch 2 of the same. Every server in the cluster alternates between whether eth0 or eth1 is the active interface, and every cluster is located in a different region. The ring buffers obviously need to be made consistent.

Here's where things get trickier: I discovered the problem above when researching why a number of the nameservers are frequently logging "error sending response: unset" errors, which the ISC knowledgebase suggests is related to outbound congestion. Servers with the higher ring buffer setting (1020) drop fewer packets on ifconfig (as one would expect), but tend to log the above error with great frequency, ~20k times a day in one of my highest load groups. We'll call this ''Group 1''. The servers with the lower ring buffer (255) setting drop significantly more inbound packets per day (again, expected), but have far fewer instances of the BIND error, typically 0-150 in that same load group.

Not a huge mystery here either. Caching DNS is a recursive service: if something isn't cached, the server has to make multiple queries on behalf of that one question until it can finally return an answer. It's a (one in)->(many out) query relationship. Fixing the RX ring buffers should cause this number to equalize to a new value across the board, and from there it would probably be a good idea to tune the kernel's outbound network queue in proc (wmem_max/wmem_default).

I like being able to gauge the influence of configuration changes on a performance problem, so I wrote a report to gather some data before I started making production changes. Here's an example of the output for the first two servers in Group 1:

group1-01
    RX: 7166.27/sec av.
    TX: 7432.57/sec av.
    RXDROP: 7.43/sec av.
    unset_err: 27633
group1-02
    RX: 7137.37/sec av.
    TX: 7398.50/sec av.
    RXDROP: 9.94/sec av.
    unset_err: 107

These are the formulas. Note that this is a local script, and there are no reliance on shell scripts that have to be maintained per-server.

    RXPACK=$(ssh $server "sar -n DEV -f /var/log/sa/sa$(date --date=yesterday '+%d') | grep \"Average: .*\$(awk '{if (\$2 == "00000000") { print \$1 }}' /proc/net/route)\" | awk '{print \$3}'" 2>/dev/null)
    TXPACK=$(ssh $server "sar -n DEV -f /var/log/sa/sa$(date --date=yesterday '+%d') | grep \"Average: .*\$(awk '{if (\$2 == "00000000") { print \$1 }}' /proc/net/route)\" | awk '{print \$4}'" 2>/dev/null)
    RXDROP=$(ssh $server "sar -n EDEV -f /var/log/sa/sa$(date --date=yesterday '+%d') | grep \"Average: .*\$(awk '{if (\$2 == "00000000") { print \$1 }}' /proc/net/route)\" | awk '{print \$6}'" 2>/dev/null)
    TXDROP=$(ssh $server "sudo grep 'error sending response: unset' /var/log/dns_named.1" 2>/dev/null | wc -l)

Once I start running this report across all of my caching DNS environments, I notice that another group with a near identical packet load, which we'll call Group 2, has no problems at all:

group2-01
    RX: 7066.44/sec av.
    TX: 7345.95/sec av.
    RXDROP: 0.00/sec av.
    unset_err: 0
group2-02
    RX: 7019.18/sec av.
    TX: 7312.47/sec av.
    RXDROP: 0.00/sec av.
    unset_err: 0

The question:

Why does group2 behave this way without requiring further tuning of RX ring buffers or net.core.wmem_default/net.core.wmem_max? I'm going to need to normalize the ring buffers no matter what, but I would like to understand what else is going on here before I start playing with wmem values in /proc.

The only thing I can think of is that the queue is getting emptied faster by the application, but network stack tuning is not something I have a great deal of hands-on experience with and I'd like to get second opinions. (my eyes glaze over at some of the ethtool counter names, I won't deny it)

I have eliminated the following as possibilities. Proofs follow after the divider.

• The ring buffer layout is the same. (first server of group1 and group2 configured the same, second server of group1 and group2 configured the same)
• The default gateway layout is the same.
• The network cards are the same. (Broadcom BCM5708)
• The firmware version reported by ethtool is the same. (bc 4.0.3 ipms 1.6.0)
• sysctl -a output matches between the first servers of both groups and the second servers of both groups. (excluding kernel and fs sections)
• The total number of servers in Group 1 and Group 2 are the same. (10)

For confidentiality reasons I cannot show the raw named.conf, or the grep filter I'm using to exclude information. You will have to take my word for it that the following configuration parameters are constant between all four servers:

    notify no;
    allow-transfer { none; };
    allow-recursion { any; };
    allow-query { any; };
    allow-query-cache { any; };
    recursive-clients 100000;
    max-cache-size 2G;
    max-ncache-ttl 900;

Below is a great deal of system information. The "hosthash" is just to demonstrate that each iteration of the loop is in fact hitting a different server without revealing the actual hostname.

Host hashes:

group1-1: dc78abcb154b74c87feecb3f35222263d40c028c
group1-2: 9fe491d58fd1e7d4e21e5bf10c164e4cf66e884b
group2-1: fc76bb3ee1ff580c6aba0d685713bb4145bd5fe3
group2-2: b7550c65d37622a131b1e47f066773defbb4d817

for server in $group1_1 $group1_2 $group2_1 $group2_2
do
    echo ____________________
    ssh $server "echo -en hosthash: \$(echo \$HOSTNAME | sha1sum)\\\n\\\n &&
         SARFILE=/var/log/sa/sa\$(date --date=yesterday '+%d') &&
         uname -srvmpio &&
         sudo /usr/sbin/dmidecode -s system-product-name
         dmesg | grep Broadcom &&
         head /proc/cpuinfo &&
         GWIF=\$(awk '{if (\$2 == 00000000) { print \$1 }}' /proc/net/route) &&
         sar -n DEV -f \$SARFILE | egrep '(IFACE|Average)' &&
         sar -n EDEV -f \$SARFILE | egrep '(IFACE|Average)' &&
         sudo /sbin/ethtool \$GWIF &&
         sudo /sbin/ethtool -i \$GWIF &&
         sudo /sbin/ethtool -g \$GWIF &&
         sudo /sbin/ethtool -c \$GWIF &&
         sudo /sbin/ethtool -S \$GWIF &&
         echo sysctl linecount: \$(sudo /sbin/sysctl -a | egrep -v '^(fs|kernel)' | wc -l) &&
         echo sysctl hash: \$(sudo /sbin/sysctl -a | egrep -v '^(fs|kernel)' | sha1sum)"
done

Output:

____________________
hosthash: dc78abcb154b74c87feecb3f35222263d40c028c -

Linux 2.6.18-308.16.1.el5 #1 SMP Tue Sep 18 07:21:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
IBM System x3550 -[7978AC1]-
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.11 (July 20, 2011)
eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem c8000000, IRQ 90, node addr 001a649db00e
eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem ce000000, IRQ 177, node addr 001a649db010
cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.7 (July 20, 2011)
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Aug 04, 2011)
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping    : 6
cpu MHz     : 2493.750
cache size  : 6144 KB
physical id : 0
siblings    : 4
12:00:01 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo   1269.15   1269.15 206600.39 206600.39      0.00      0.00      0.00
Average:         eth0   7166.27   7432.57 704051.80 2419779.42      0.00      0.00      0.94
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01 AM     IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      7.43      0.00      0.00      0.00      0.00      0.00
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
driver: bnx2
version: 2.1.11
firmware-version: bc 4.0.3 ipms 1.6.0
bus-info: 0000:04:00.0
Ring parameters for eth0:
Pre-set maximums:
RX:     2040
RX Mini:    0
RX Jumbo:   8160
TX:     255
Current hardware settings:
RX:     1020
RX Mini:    0
RX Jumbo:   0
TX:     255

Coalesce parameters for eth0:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 12
rx-usecs-irq: 18
rx-frames-irq: 2

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 18
tx-frames-irq: 2

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

NIC statistics:
     rx_bytes: 1505439501410
     rx_error_bytes: 0
     tx_bytes: 4672574845104
     tx_error_bytes: 0
     rx_ucast_packets: 15315548049
     rx_mcast_packets: 2035415
     rx_bcast_packets: 1101989
     tx_ucast_packets: 15505474251
     tx_mcast_packets: 40018
     tx_bcast_packets: 36019
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 92309552
     rx_65_to_127_byte_packets: 1243637891
     rx_128_to_255_byte_packets: 790117566
     rx_256_to_511_byte_packets: 127197337
     rx_512_to_1023_byte_packets: 168929387
     rx_1024_to_1522_byte_packets: 11591832
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 60586118
     tx_65_to_127_byte_packets: 1976738758
     tx_128_to_255_byte_packets: 2830395753
     tx_256_to_511_byte_packets: 157607989
     tx_512_to_1023_byte_packets: 1483716940
     tx_1024_to_1522_byte_packets: 406821340
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 0
     rx_xoff_frames: 0
     tx_xon_frames: 116422
     tx_xoff_frames: 134780
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 0
     rx_ftq_discards: 0
     rx_discards: 0
     rx_fw_discards: 14015105
sysctl linecount: 504
sysctl hash: dd6aab90d0fd9ae90742c5f812a78734e2f2ff1c -
____________________
hosthash: 9fe491d58fd1e7d4e21e5bf10c164e4cf66e884b -

Linux 2.6.18-308.16.1.el5 #1 SMP Tue Sep 18 07:21:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
IBM System x3550 -[7978EHU]-
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.11 (July 20, 2011)
eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem c8000000, IRQ 90, node addr 001a6479655c
eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem ce000000, IRQ 177, node addr 001a6479655e
cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.7 (July 20, 2011)
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Aug 04, 2011)
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping    : 6
cpu MHz     : 2493.746
cache size  : 6144 KB
physical id : 0
siblings    : 4
12:00:01 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo   1261.04   1261.04 205548.08 205548.08      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1   7137.37   7398.50 702340.35 2409580.71      0.00      0.00      0.97
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01 AM     IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1      0.00      0.00      0.00      9.94      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
driver: bnx2
version: 2.1.11
firmware-version: bc 4.0.3 ipms 1.6.0
bus-info: 0000:06:00.0
Ring parameters for eth1:
Pre-set maximums:
RX:     2040
RX Mini:    0
RX Jumbo:   8160
TX:     255
Current hardware settings:
RX:     255
RX Mini:    0
RX Jumbo:   0
TX:     255

Coalesce parameters for eth1:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 12
rx-usecs-irq: 18
rx-frames-irq: 2

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 18
tx-frames-irq: 2

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

NIC statistics:
     rx_bytes: 1501719289640
     rx_error_bytes: 0
     tx_bytes: 4654179094291
     tx_error_bytes: 0
     rx_ucast_packets: 15253610508
     rx_mcast_packets: 2108112
     rx_bcast_packets: 1136240
     tx_ucast_packets: 15438361249
     tx_mcast_packets: 40135
     tx_bcast_packets: 1721
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 92376678
     rx_65_to_127_byte_packets: 1183040190
     rx_128_to_255_byte_packets: 788176623
     rx_256_to_511_byte_packets: 126838328
     rx_512_to_1023_byte_packets: 168170816
     rx_1024_to_1522_byte_packets: 13350337
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 60806588
     tx_65_to_127_byte_packets: 1955234150
     tx_128_to_255_byte_packets: 2806601346
     tx_256_to_511_byte_packets: 154015585
     tx_512_to_1023_byte_packets: 1466206531
     tx_1024_to_1522_byte_packets: 405928513
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 0
     rx_xoff_frames: 0
     tx_xon_frames: 150648
     tx_xoff_frames: 173552
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 1
     rx_ftq_discards: 0
     rx_discards: 0
     rx_fw_discards: 19605427
sysctl linecount: 504
sysctl hash: 4626e3788c72e091487afe1e3a7cfd32278ab07d -
____________________
hosthash: fc76bb3ee1ff580c6aba0d685713bb4145bd5fe3 -

Linux 2.6.18-308.16.1.el5 #1 SMP Tue Sep 18 07:21:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
IBM System x3550 -[7978AC1]-
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.11 (July 20, 2011)
eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem c8000000, IRQ 90, node addr 001a649dc68a
eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem ce000000, IRQ 177, node addr 001a649dc68c
cnic: Broadcom NetXtreme II CNIC Driver cnic v2.5.7 (July 20, 2011)
Broadcom NetXtreme II iSCSI Driver bnx2i v2.7.0.3 (Aug 04, 2011)
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping    : 6
cpu MHz     : 2493.750
cache size  : 6144 KB
physical id : 0
siblings    : 4
12:00:01 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo   1891.67   1891.67 266593.77 266593.77      0.00      0.00      0.00
Average:         eth0   7066.44   7345.95 730519.41 2215508.99      0.00      0.00      4.37
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01 AM     IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
driver: bnx2
version: 2.1.11
firmware-version: bc 4.0.3 ipms 1.6.0
bus-info: 0000:04:00.0
Ring parameters for eth0:
Pre-set maximums:
RX:     2040
RX Mini:    0
RX Jumbo:   8160
TX:     255
Current hardware settings:
RX:     1020
RX Mini:    0
RX Jumbo:   0
TX:     255

Coalesce parameters for eth0:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 12
rx-usecs-irq: 18
rx-frames-irq: 2

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 18
tx-frames-irq: 2

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

NIC statistics:
     rx_bytes: 4640887074833
     rx_error_bytes: 0
     tx_bytes: 12640942400790
     tx_error_bytes: 0
     rx_ucast_packets: 46405845860
     rx_mcast_packets: 14487857
     rx_bcast_packets: 3476467
     tx_ucast_packets: 47159091638
     tx_mcast_packets: 118147
     tx_bcast_packets: 5504
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 136463411
     rx_65_to_127_byte_packets: 4245502343
     rx_128_to_255_byte_packets: 2357984838
     rx_256_to_511_byte_packets: 355610202
     rx_512_to_1023_byte_packets: 608223572
     rx_1024_to_1522_byte_packets: 65320154
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 112166114
     tx_65_to_127_byte_packets: 3010346100
     tx_128_to_255_byte_packets: 4087240164
     tx_256_to_511_byte_packets: 1625596725
     tx_512_to_1023_byte_packets: 3037109096
     tx_1024_to_1522_byte_packets: 927187571
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 0
     rx_xoff_frames: 0
     tx_xon_frames: 79164
     tx_xoff_frames: 89685
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 1
     rx_ftq_discards: 0
     rx_discards: 0
     rx_fw_discards: 6857729
sysctl linecount: 504
sysctl hash: dd6aab90d0fd9ae90742c5f812a78734e2f2ff1c -
____________________
hosthash: b7550c65d37622a131b1e47f066773defbb4d817 -

Linux 2.6.18-308.16.1.el5 #1 SMP Tue Sep 18 07:21:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux
IBM System x3550 -[7978EHU]-
bnx2: Broadcom NetXtreme II Gigabit Ethernet Driver bnx2 v2.1.11 (July 20, 2011)
eth0: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem c8000000, IRQ 90, node addr 00215e3f1ec4
eth1: Broadcom NetXtreme II BCM5708 1000Base-T (B2) PCI-X 64-bit 133MHz found at mem ce000000, IRQ 177, node addr 00215e3f1ec6
processor   : 0
vendor_id   : GenuineIntel
cpu family  : 6
model       : 23
model name  : Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz
stepping    : 6
cpu MHz     : 2493.753
cache size  : 6144 KB
physical id : 1
siblings    : 4
12:00:01 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
Average:           lo   1883.04   1883.04 263726.79 263726.79      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1   7019.18   7312.47 720911.92 2214861.10      0.00      0.00      1.02
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:00:01 AM     IFACE   rxerr/s   txerr/s    coll/s  rxdrop/s  txdrop/s  txcarr/s  rxfram/s  rxfifo/s  txfifo/s
Average:           lo      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         eth1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:         sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
driver: bnx2
version: 2.1.11
firmware-version: bc 4.0.3 ipms 1.6.0
bus-info: 0000:06:00.0
Ring parameters for eth1:
Pre-set maximums:
RX:     2040
RX Mini:    0
RX Jumbo:   8160
TX:     255
Current hardware settings:
RX:     255
RX Mini:    0
RX Jumbo:   0
TX:     255

Coalesce parameters for eth1:
Adaptive RX: off  TX: off
stats-block-usecs: 999936
sample-interval: 0
pkt-rate-low: 0
pkt-rate-high: 0

rx-usecs: 18
rx-frames: 12
rx-usecs-irq: 18
rx-frames-irq: 2

tx-usecs: 80
tx-frames: 20
tx-usecs-irq: 18
tx-frames-irq: 2

rx-usecs-low: 0
rx-frame-low: 0
tx-usecs-low: 0
tx-frame-low: 0

rx-usecs-high: 0
rx-frame-high: 0
tx-usecs-high: 0
tx-frame-high: 0

NIC statistics:
     rx_bytes: 4621548539323
     rx_error_bytes: 0
     tx_bytes: 12598031299743
     tx_error_bytes: 0
     rx_ucast_packets: 46260356368
     rx_mcast_packets: 5352446
     rx_bcast_packets: 3474589
     tx_ucast_packets: 47008853953
     tx_mcast_packets: 118164
     tx_bcast_packets: 5471
     tx_mac_errors: 0
     tx_carrier_errors: 0
     rx_crc_errors: 0
     rx_align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     tx_deferred: 0
     tx_excess_collisions: 0
     tx_late_collisions: 0
     tx_total_collisions: 0
     rx_fragments: 0
     rx_jabbers: 0
     rx_undersize_packets: 0
     rx_oversize_packets: 0
     rx_64_byte_packets: 126851062
     rx_65_to_127_byte_packets: 4117708205
     rx_128_to_255_byte_packets: 2346047550
     rx_256_to_511_byte_packets: 356266112
     rx_512_to_1023_byte_packets: 604666332
     rx_1024_to_1522_byte_packets: 62938478
     rx_1523_to_9022_byte_packets: 0
     tx_64_byte_packets: 111216848
     tx_65_to_127_byte_packets: 2984505931
     tx_128_to_255_byte_packets: 4027485330
     tx_256_to_511_byte_packets: 1577669672
     tx_512_to_1023_byte_packets: 3015060448
     tx_1024_to_1522_byte_packets: 933575954
     tx_1523_to_9022_byte_packets: 0
     rx_xon_frames: 0
     rx_xoff_frames: 0
     tx_xon_frames: 129873
     tx_xoff_frames: 145090
     rx_mac_ctrl_frames: 0
     rx_filtered_packets: 1
     rx_ftq_discards: 0
     rx_discards: 0
     rx_fw_discards: 6752713
sysctl linecount: 504
sysctl hash: 4626e3788c72e091487afe1e3a7cfd32278ab07d -