Matt's questions

All machines are running CentOS 6.5. We have about 85 client machines doing a Kerberized NFSv4 mount (sec=krb5p) to a server. This weekend, the server was changed (serverA changed to serverB).

Everything appears to work, except, on one a few client servers only, and only for one a few special users, it looks like idmapd stops working after around 30 to 45 minutes.

Simple test: on the client machine in question, I do something like this:

while [ 1 ]; do touch test.`date +%H%M%S`.txt ; sleep 1m ; done

And then watch the files as they are being created. They start out with the proper user and group ID. But after 35 minutes or so, they suddenly switch to being owned by nfsnobody:nfsnobody.

The idmapd process is still running. Other users and other machines are apparently not affected. (Of course we didn't test all users and all machines, but spot testing other users and other machines found no problems.)

Edit: Forgot to post some important details:

• Initially, the new NFS server did not have the correct /etc/idmapd.conf. It had the default. That has since been rectified, and the idmapd service restarted on both server and client.
• The /etc/idmapd.conf files are the same on both client and server.
• The /etc/passwd and /etc/group files are the same on both client and server.

Edit2: Upon further review, we observed the following:

• The /etc/idmapd.conf on the server is different from the client in that it also has some static mappings. These static mappings are for a few key users who need to run cronjobs from Kerberized NFSv4 shares. This link describes exactly the type of special config on the server's /etc/idmapd.conf file.
• The problem would actually come and go (not simply get bad and stay bad). In my example "touch" test above, the files would be created for a while with the right user and group ownership. Then after ~45 minutes, they'd start getting created with nfsnobody ownership. Then after some time they'd be created with the right ownership. Off and on, with no discernible pattern.
• This happened only for the users with the "special" /etc/idmapd.conf mappings described above, and only on the machines where those users had cron jobs.

Is there any way to make a seasoned Linux syadmin productive without giving him full root access?

This question comes from a perspective of protecting intellectual property (IP), which in my case, is entirely code and/or configuration files (i.e. small digital files that are easily copied). Our secret sauce has made us more successful than our smallish size would suggest. Likewise, we are once-bitten, twice shy from a few former unscrupulous employees (not sysadmins) who tried to steal IP. Top management's position is basically, "We trust people, but out of self-interest, cannot afford the risk of giving any one person more access than they absolutely need to do their job."

On the developer side, it's relatively easy to partition workflows and access levels such that people can be productive but only see only what they need to see. Only the top people (actual company owners) have the ability to combine all the ingredients and create the special sauce.

But I haven't been able to come up with a good way to maintain this IP secrecy on the Linux admin side. We make extensive use of GPG for code and sensitive text files... but what's to stop an admin from (for example) su'ing to a user and hopping on their tmux or GNU Screen session and seeing what they're doing?

(We also have Internet access disabled everywhere that could possibly come into contact with sensitive information. But, nothing is perfect, and there could be holes open to clever sysadmins or mistakes on the network admin side. Or even good old USB. There are of course numerous other measures in place, but those are beyond the scope of this question.)

The best I can come up with is basically using personalized accounts with sudo, similar to what is described in Multiple Linux sysadmins working as root. Specifically: no one except the company owners would actually have direct root access. Other admins would have a personalized account and the ability to sudo into root. Furthermore, remote logging would be instituted, and the logs would go to a server only the company owners could access. Seeing logging turned off would set off some kind of alerts.

A clever sysadmin could probably still find some holes in this scheme. And that aside, it's still reactive rather than proactive. The problem with our IP is such that competitors could make use of it very quickly, and cause a lot of damage in very short order.

So still better would be a mechanism that limits what the admin can do. But I recognize that this is a delicate balance (particularly in the light of troubleshooting and fixing production issues that need to be resolved right now).

I can't help but wonder how other organizations with very sensitive data manage this issue? For example, military sysadmins: how do they manage servers and data without being able to see confidential information?

Edit: In the initial posting, I meant to preemptively address the "hiring practices" comments that are starting to surface. One, this is supposed to be a technical question, and hiring practices IMO tend more towards social questions. But, two, I'll say this: I believe we do everything that's reasonable for hiring people: interview with multiple people at the firm; background and reference checks; all employees sign numerous legal documents, including one that says they've read and understood our handbook which details IP concerns in detail. Now, it's out of the scope of this question/site, but if someone can propose "perfect" hiring practices that filter out 100% of the bad actors, I'm all ears. Facts are: (1) I don't believe there is such a perfect hiring process; (2) people change - today's angel could be tomorrow's devil; (3) attempted code theft appears to be somewhat routine in this industry.

We have an in-house "compute farm" with about 100 CentOS (free re-distribution of RHEL) 5.7 and 6.5 x86_64 servers. (We are in the process of upgrading all the 5.7 boxes to 6.5.) All these machines do two NFSv4 mounts (with sec=krb5p) to two CentOS 6.5 servers. One NFS server is for user home directories, the other contains various data for user processes.

Randomly, one of the client machines will get into a bad state such that any access to the NFSv4 mount hangs ("ls" for example). This means no one (except root) can login, and all user processes that require access to the shares get stuck. In other words, so far this is non-deterministic and cannot be replicated.

I have very verbose NFS logging enabled in both the clients and servers, but never get any errors. However, when this state is triggered, I do get these kernel trace errors on the client machines:

Mar 25 00:49:48 servername kernel: INFO: task ProcessName:8230 blocked for more than 120 seconds.
Mar 25 00:49:48 servername kernel:      Not tainted 2.6.32-431.el6.x86_64 #1
Mar 25 00:49:48 servername kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Mar 25 00:49:48 servername kernel: ProcessName D 0000000000000000     0  8230   8229 0x00000000
Mar 25 00:49:48 servername kernel: ffff8804792cdb68 0000000000000046 ffff8804792cdae8 ffffffffa0251940
Mar 25 00:49:48 servername kernel: ffff88010cdc8080 ffff8804792cdb18 ffff88010cdc8130 ffff88010ea5c208
Mar 25 00:49:48 servername kernel: ffff88047b011058 ffff8804792cdfd8 000000000000fbc8 ffff88047b011058
Mar 25 00:49:48 servername kernel: Call Trace:
Mar 25 00:49:48 servername kernel: [<ffffffffa0251940>] ? rpc_execute+0x50/0xa0 [sunrpc]
Mar 25 00:49:48 servername kernel: [<ffffffff810a70a1>] ? ktime_get_ts+0xb1/0xf0
Mar 25 00:49:48 servername kernel: [<ffffffff8111f930>] ? sync_page+0x0/0x50
Mar 25 00:49:48 servername kernel: [<ffffffff815280a3>] io_schedule+0x73/0xc0
Mar 25 00:49:48 servername kernel: [<ffffffff8111f96d>] sync_page+0x3d/0x50
Mar 25 00:49:48 servername kernel: [<ffffffff81528b6f>] __wait_on_bit+0x5f/0x90
Mar 25 00:49:48 servername kernel: [<ffffffff8111fba3>] wait_on_page_bit+0x73/0x80
Mar 25 00:49:48 servername kernel: [<ffffffff8109b320>] ? wake_bit_function+0x0/0x50
Mar 25 00:49:48 servername kernel: [<ffffffff81135bf5>] ? pagevec_lookup_tag+0x25/0x40
Mar 25 00:49:48 servername kernel: [<ffffffff8111ffcb>] wait_on_page_writeback_range+0xfb/0x190
Mar 25 00:49:48 servername kernel: [<ffffffff81120198>] filemap_write_and_wait_range+0x78/0x90
Mar 25 00:49:48 servername kernel: [<ffffffff811baa3e>] vfs_fsync_range+0x7e/0x100
Mar 25 00:49:48 servername kernel: [<ffffffff811bab2d>] vfs_fsync+0x1d/0x20
Mar 25 00:49:48 servername kernel: [<ffffffffa02cf8b0>] nfs_file_flush+0x70/0xa0 [nfs]
Mar 25 00:49:48 servername kernel: [<ffffffff81185b6c>] filp_close+0x3c/0x90
Mar 25 00:49:48 servername kernel: [<ffffffff81074e0f>] put_files_struct+0x7f/0xf0
Mar 25 00:49:48 servername kernel: [<ffffffff81074ed3>] exit_files+0x53/0x70
Mar 25 00:49:48 servername kernel: [<ffffffff81076f4d>] do_exit+0x18d/0x870
Mar 25 00:49:48 servername kernel: [<ffffffff81077688>] do_group_exit+0x58/0xd0
Mar 25 00:49:48 servername kernel: [<ffffffff81077717>] sys_exit_group+0x17/0x20
Mar 25 00:49:48 servername kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

At this point, the only reliable way to make the machine usable again is to reboot it. (And even that requires a hard power cycle, since the software reboot hangs when it tries to unmount the NFS filesystems.)

It seems like this problem is correlated with a process that malfunctions and starts writing data like crazy. For example, a segfault that generates a huge core file, or a bug with a tight print loop.

However, I've tried to duplicate this problem in a lab environment with multiple "dd" processes hammering away at the NFS server, but all machines chug along happily.

Host system is CentOS 6.5 x86_64. I have setup an Ubuntu guest by following this wiki. The guest works, but: if I reboot the host, it takes a very long time (about 30 minutes on the last reboot) before the guests actually start. If, during this time, I start virsh and do a "list", it just hangs.

Here's a snippet from /var/log/libvirt/libvirt.log:

2014-10-14 16:47:11.509+0000: 4668: info : virDomainLoadAllConfigs:14696 : Scanning for configs in /var/run/libvirt/qemu
2014-10-14 16:47:11.527+0000: 4668: info : virDomainLoadAllConfigs:14696 : Scanning for configs in /etc/libvirt/qemu
2014-10-14 16:47:11.527+0000: 4668: info : virDomainLoadAllConfigs:14718 : Loading config file 'mythbuntu.xml'
2014-10-14 16:47:11.529+0000: 4668: info : qemuDomainSnapshotLoad:484 : Scanning for snapshots for domain mythbuntu in /var/lib/libvirt/qemu/snapshot/mythbuntu
2014-10-14 17:14:41.751+0000: 4668: info : virNetDevProbeVnetHdr:94 : Enabling IFF_VNET_HDR
2014-10-14 17:14:41.805+0000: 4668: info : virSecurityDACSetOwnership:296 : Setting DAC user and group on '/home/kvm/mythbuntu.img' to '107:107'
2014-10-14 17:14:41.806+0000: 4668: info : virSecurityDACSetOwnership:296 : Setting DAC user and group on '/mnt/mythtv1/mythbackend_recordings' to '107:107'
2014-10-14 17:14:42.084+0000: 4668: info : lxcSecurityInit:1380 : lxcSecurityInit (null)

Notice that huge jump in time, from 16:47:11 to 17:14:41? The latter time is when the guest actually started, and at this point, "virsh list" was once again usable.

It seems like libvirt/kvm/qemu is getting hung up on something, but eventually is able to proceed. Anyone have any idea what it might be hung up on?

Edit: I am not using snapshots. And in fact:

# ls /var/lib/libvirt/qemu/snapshot/mythbuntu
ls: cannot access /var/lib/libvirt/qemu/snapshot/mythbuntu: No such file or directory

The parent looks like this:

# ls -R /var/lib/libvirt/qemu/
/var/lib/libvirt/qemu/:
dump  mythbuntu.monitor  save  snapshot

/var/lib/libvirt/qemu/dump:

/var/lib/libvirt/qemu/save:

/var/lib/libvirt/qemu/snapshot:

Using two physical Linux servers on different sub-nets, I'd like to have master and slave DNS servers. These DNS servers will only resolve internal names and addresses. Neither the DNS servers nor the peer hosts have Internet access, so a query for something like "google.com" should fail.

I can't quite get my head around what alchemy of configuration directives bind/named needs to do this. For example: recursion. This website suggests that what I want is an "Authoritative Only DNS Server". But it achieves this by disabling recursion. I get the impression that I really don't want to disable recursion, as that is the only type of query some programs/APIs make. I know with dig you can specify recursion/no recursion. But what about nslookup, or gethostbyname()? Everything that needs DNS should "just work".

But on the other hand, that same website shows examples with recursion enabled. But that requires a "." zone to be defined, with the accompanying root servers file. But the whole root servers thing is pointless in my config, as there is only one domain (our internal domain). As far as this network is concerned, our internal domain is the only domain.

For what it's worth, this is a relatively small network (about 200 servers and 50 PCs). There will also be a third DNS slave, running Windows Server.

We have a network of Windows 7 PCs that are managed as part of a domain. What we want is for the domain admin to be unable to view the PC's local drive (C:) unless he is physically at the PC. In other words, no remote desktop and no ability to use UNC. In other words, the domain admin should not be allowed to put \\user_pc\c$ in Windows Explorer and see all the files on that computer, unless he is physically present at the PC itself.

Edit: to clarify some of the questions/comments that have come up. Yes, I am an admin---but a complete Windows novice. And yes, for the sake of this and my similar questions, it is fair to assume that I am working for someone who is paranoid.

I understand the arguments about this being a "social problem versus a technical problem", and "you should be able to trust your admins", etc. But this is the situation in which I find myself. I'm basically new to Windows system administration, but am tasked with creating an environment that is secure by the company owner's definition---and this definition is clearly very different from what most people expect.

In short, I understand that this is an unusual request. But I'm hoping there is enough expertise in the ServerFault community to point me in the right direction.

This is admittedly a duplicate of this question: Secure Exchange mailboxes

In short, how can Microsoft Exchange 2010 be configured such that email is only visible to the owner of that mailbox, and perhaps the owner/CEO of the organization?

The answers to the linked question dealt with admin trust and the Microsoft "model" for Exchange, that is, it is assumed that admins will have all the power.

But let's assume that simply isn't an option: as someone who is new to Exchange, and Microsoft products in general, I have been tasked with setting up Exchange in such a way. No one, except the owner of the company, should be able to see anyone's email but his own. We realize that this will limit what the "admin" can do (e.g. fix mailbox corruption), but that is acceptable.

Furthermore, as a follow-up to the general theme of the answers to the duplicate question I linked: is it really true, in large organizations that use Microsoft Exchange, that the Enterprise Admins can potentially read anyone's email? For example, at Microsoft itself, there is someone (possibly many people) who can potentially read Steve Balmer's email? Or read sensitive HR documents about peoples' compensation, or maybe an employee asking about something like EAP (employee assistance program)? Or emails with the legal team, or conversations with the SEC, or perhaps an upcoming acquisition?

We have Microsoft Exchange 2010 running under Windows Server 2008 R2 Standard. We would like to migrate to Windows Server Enterprise so that we can set up a DAG (the Exchange version will remain the same (2010 Standard)). Additionally, the hardware for this server is getting old.

What we would like to do as a first step is replicate the current Exchange setup/configuration on new hardware (already purchased) running 2008-R2 Enterprise. For the mailbox portion, we plan to use Exchange's "Mailbox Move" feature. But what we are struggling with is how to copy all the settings and configuration from the old server to the new?

Over the years, we've made various tweaks and and such to filters, policies, etc. How can we ensure that the new server retains all of our customizations? The configuration is the sum of many menus, sub-menus, dialogs, tabs, buttons, etc etc... if we try to replicate these settings by hand, it seems likely that we will miss something.

I can't imagine we are the first people to face this problem. Is there a built-in facility in Exchange that we've overlooked that supports what we want to do? Or maybe a 3rd party application we could buy?

I have a server that is doing an NFSv4 export for user's home directories. There are roughly 25 users (mostly developers/analysts) and about 40 servers mounting the home directory export. Performance is miserable, with users often seeing multi-second lags for simple commands (like ls, or writing a small text file). Sometimes the home directory mount completely hangs for minutes, with users getting "permission denied" errors.

The hardware is a Dell R510 with dual E5620 CPUs and 8 GB RAM. There are eight 15k 2.5” 600 GB drives (Seagate ST3600057SS) configured in hardware RAID-6 with a single hot spare. RAID controller is a Dell PERC H700 w/512MB cache (Linux sees this as a LSI MegaSAS 9260). OS is CentOS 5.6, home directory partition is ext3, with options “rw,data=journal,usrquota”.

I have the HW RAID configured to present two virtual disks to the OS: /dev/sda for the OS (boot, root and swap partitions), and /dev/sdb for the home directories.

What I find curious, and suspicious, is that the sda device often has very high utilization, even though it only contains the OS. I would expect this virtual drive to be idle almost all the time. The system is not swapping, according to "free" and "vmstat". Why would there be major load on this device?

Here is a 30-second snapshot from iostat:

Time: 09:37:28 AM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await svctm  %util
sda               0.00    44.09  0.03 107.76     0.13   607.40    11.27     0.89    8.27   7.27  78.35
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00    44.09  0.03 107.76     0.13   607.40    11.27     0.89    8.27   7.27  78.35
sdb               0.00  2616.53  0.67 157.88     2.80 11098.83   140.04     8.57   54.08   4.21  66.68
sdb1              0.00  2616.53  0.67 157.88     2.80 11098.83   140.04     8.57   54.08   4.21  66.68
dm-0              0.00     0.00  0.03 151.82     0.13   607.26     8.00     1.25    8.23   5.16  78.35
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00  0.67 2774.84     2.80 11099.37     8.00   474.30  170.89   0.24  66.84
dm-3              0.00     0.00  0.67 2774.84     2.80 11099.37     8.00   474.30  170.89   0.24  66.84

Looks like iotop is the ideal tool to use to sniff out these kinds of issues. But I'm on CentOS 5.6, which doesn't have a new enough kernel to support that program.

I looked at Determining which process is causing heavy disk I/O?, and besides iotop, one of the suggestions said to do a "echo 1 > /proc/sys/vm/block_dump". I did that (after directing kernel messages to tempfs). In about 13 minutes I had about 700k reads or writes, roughly half from kjournald and the other half from nfsd:

# egrep " kernel: .*(READ|WRITE)" messages | wc -l
768439
# egrep " kernel: kjournald.*(READ|WRITE)" messages | wc -l
403615
# egrep " kernel: nfsd.*(READ|WRITE)" messages | wc -l
314028

For what it's worth, for the last hour, utilization has constantly been over 90% for the home directory drive. My 30-second iostat keeps showing output like this:

Time: 09:36:30 PM
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sda               0.00     6.46  0.20 11.33     0.80    71.71    12.58     0.24   20.53  14.37  16.56
sda1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sda2              0.00     6.46  0.20 11.33     0.80    71.71    12.58     0.24   20.53  14.37  16.56
sdb             137.29     7.00 549.92  3.80 22817.19    43.19    82.57     3.02    5.45   1.74  96.32
sdb1            137.29     7.00 549.92  3.80 22817.19    43.19    82.57     3.02    5.45   1.74  96.32
dm-0              0.00     0.00  0.20 17.76     0.80    71.04     8.00     0.38   21.21   9.22  16.57
dm-1              0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
dm-2              0.00     0.00 687.47 10.80 22817.19    43.19    65.48     4.62    6.61   1.43  99.81
dm-3              0.00     0.00 687.47 10.80 22817.19    43.19    65.48     4.62    6.61   1.43  99.82

I'm experiencing some significant performance issues on an NFS server. I've been reading up a bit on partition alignment, and I think I have my partitions mis-aligned. I can't find anything that tells me how to actually quantify the effects of mis-aligned partitions. Some of the general information I found suggests the performance penalty can be quite high (upwards of 60%) and others say it's negligible. What I want to do is determine if partition alignment is a factor in this server's performance problems or not; and if so, to what degree?

So I'll put my info out here, and hopefully the community can confirm if my partitions are indeed mis-aligned, and if so, help me put a number to what the performance cost is.

Server is a Dell R510 with dual E5620 CPUs and 8 GB RAM. There are eight 15k 2.5” 600 GB drives (Seagate ST3600057SS) configured in hardware RAID-6 with a single hot spare. RAID controller is a Dell PERC H700 w/512MB cache (Linux sees this as a LSI MegaSAS 9260). OS is CentOS 5.6, home directory partition is ext3, with options “rw,data=journal,usrquota”.

I have the HW RAID configured to present two virtual disks to the OS: /dev/sda for the OS (boot, root and swap partitions), and /dev/sdb for a big NFS share:

[root@lnxutil1 ~]# parted -s /dev/sda unit s print

Model: DELL PERC H700 (scsi)
Disk /dev/sda: 134217599s
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start    End         Size        Type     File system  Flags
 1      63s      465884s     465822s     primary  ext2         boot
 2      465885s  134207009s  133741125s  primary               lvm

[root@lnxutil1 ~]# parted -s /dev/sdb unit s print

Model: DELL PERC H700 (scsi)
Disk /dev/sdb: 5720768639s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start  End          Size         File system  Name  Flags
 1      34s    5720768606s  5720768573s                     lvm

Edit 1 Using the cfq IO scheduler (default for CentOS 5.6):

# cat /sys/block/sd{a,b}/queue/scheduler
noop anticipatory deadline [cfq]
noop anticipatory deadline [cfq]

Chunk size is the same as strip size, right? If so, then 64kB:

# /opt/MegaCli -LDInfo -Lall -aALL -NoLog
Adapter #0

Number of Virtual Disks: 2
Virtual Disk: 0 (target id: 0)
Name:os
RAID Level: Primary-6, Secondary-0, RAID Level Qualifier-3
Size:65535MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:7
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Number of Spans: 1
Span: 0 - Number of PDs: 7

... physical disk info removed for brevity ...

Virtual Disk: 1 (target id: 1)
Name:share
RAID Level: Primary-6, Secondary-0, RAID Level Qualifier-3
Size:2793344MB
State: Optimal
Stripe Size: 64kB
Number Of Drives:7
Span Depth:1
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU
Access Policy: Read/Write
Disk Cache Policy: Disk's Default
Number of Spans: 1
Span: 0 - Number of PDs: 7

If it's not obvious, virtual disk 0 corresponds to /dev/sda, for the OS; virtual disk 1 is /dev/sdb (the exported home directory tree).

For an Microsoft Exchange server, is it possible to encrypt the email database such that the sysadmin cannot see the emails?

In other words, the admin would be responsible for all aspects of running the Windows server and Exchange process, but would not be able to see the contents of any one email (except those sent to him, obviously). Only another individual (e.g. company owner) would be able to see all emails contained in the database.

For a Linux NFSv3 server, is there any way to see statistics on actual file access? In other words, I would like to see a timestamped log of every file read or written.

Motivation: I have several NFS servers that are slated to be replaced by some kind of "big iron" storage system. In my particular case, the NFS client load is virtually all reads of large files (averaging around 700 MB). One vendor's solution uses a large RAM disk as a cache to provide the kind of massive random read throughput that I need.

What I want is a way to study what the NFS file access pattern looks like so that I can "right size" the cache.

My goal is to configure our CentOS ("free" RHEL) 5.x servers for custom low-latency network programs. I would like to experiment with binding ethernet NIC interrupt handling to the same CPU on which the program runs (to hopefully improve cache utilization). The first step in this process is to determine the NIC's IRQ.

Here is the contents of /proc/interrupts on one server (note that I deleted CPUs 2 through 14 for brevity):

           CPU0       CPU1       CPU15
  0:  600299726          0          0    IO-APIC-edge  timer
  1:          3          0          0    IO-APIC-edge  i8042
  8:          1          0          0    IO-APIC-edge  rtc
  9:          0          0          0   IO-APIC-level  acpi
 12:          4          0          0    IO-APIC-edge  i8042
 50:          0          0          0   IO-APIC-level  uhci_hcd:usb6, uhci_hcd:usb8
 58:       6644      25103          0   IO-APIC-level  ioc0
 66:          0          0          0   IO-APIC-level  ata_piix
 74:        221     533830          0   IO-APIC-level  ata_piix
 98:         35          0    2902361       PCI-MSI-X  eth1-0
106:         61         11       3841       PCI-MSI-X  eth1-1
114:         28          0      61452       PCI-MSI-X  eth1-2
122:         24       1586         22       PCI-MSI-X  eth1-3
130:       2912          0        337       PCI-MSI-X  eth1-4
138:         21          0         28       PCI-MSI-X  eth1-5
146:         21          0         56       PCI-MSI-X  eth1-6
154:         34          1          1       PCI-MSI-X  eth1-7
209:         23          0          0   IO-APIC-level  ehci_hcd:usb1
217:          0          0          0   IO-APIC-level  ehci_hcd:usb2, uhci_hcd:usb5, uhci_hcd:usb7
225:          0          0          0   IO-APIC-level  uhci_hcd:usb3
233:          0          0          0   IO-APIC-level  uhci_hcd:usb4
NMI:       7615       2989       2931
LOC:  600328144  600328099  600327086
ERR:          0
MIS:          0

Why are there multiple entries for "eth1" in the form of "eth1-X"?

Furthermore, the contents of "/sys/class/net/eth1/device/irq" is "90". But there's no 90 in the interrupt list above.

So let's say I look at just "eth1-0", which is IRQ 98. The contents of /proc/irq/98/smp_affinity is:

00000000,00000000,00000000,00000000,00000000,00000000,00000000,00008000

That's a list of numbers, rather than just one number.

So how do I set eth1's smp_affinity?

None of the online examples and documentation I could find mentioned any cases like this; they always have exactly one "ethX" entry in /proc/interrupts; the indicated interrupt matches /sys/class/net/ethX/device/irq; and there is only one number in /proc/irq/N/smp_affinity.

FWIW, I'll add that this application is extremely latency sensitive. To the point where we disable C-states and processor frequency scaling (because those features induce too much latency). Micro seconds make a difference here.

Edit: I stumbled across the following web page http://www.kernel.org/doc/man-pages/online/pages/man7/cpuset.7.html that, although it is about cpuset, it has a section titled "Mask Format", which I assume is the same as what I am seeing in the /proc/irq//smp_affinity file. Quoting:

This format displays each 32-bit word in hexadecimal (using ASCII characters "0" - "9" and "a" - "f"); words are filled with leading zeros, if required. For masks longer than one word, a comma separator is used between words. Words are displayed in big-endian order, which has the most significant bit first. The hex digits within a word are also in big-endian order.

The number of 32-bit words displayed is the minimum number needed to display all bits of the bitmask, based on the size of the bitmask.

Examples of the Mask Format:

   00000001                        # just bit 0 set
   40000000,00000000,00000000      # just bit 94 set
   00000001,00000000,00000000      # just bit 64 set
   000000ff,00000000               # bits 32-39 set
   00000000,000E3862               # 1,5,6,11-13,17-19 set

A mask with bits 0, 1, 2, 4, 8, 16, 32, and 64 set displays as:

   00000001,00000001,00010117

The first "1" is for bit 64, the second for bit 32, the third for bit 16, the fourth for bit 8, the fifth for bit 4, and the "7" is for bits 2, 1, and 0.