grifferz's questions

Given this sequence of commands…

$ pwd
/data/backup/rsnapshot
$ sudo du -csk daily.{2,3}/ruminant
1195040 daily.2/ruminant
36712   daily.3/ruminant
1231752 total
$ ls -isk /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmunix \
          /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmlinux2
13344429 6728 /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmlinux2
16476851 6728 /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmunix
$ sudo ln -f /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmunix \
             /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmlinux2
$ ls -isk /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmunix \
          /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmlinux2
16476851 6728 /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmlinux2
16476851 6728 /data/backup/rsnapshot/daily.3/ruminant/home/andy/vmunix
$ sudo du -csk daily.{2,3}/ruminant
1195040 daily.2/ruminant
36712   daily.3/ruminant
1231752 total

…why does the usage of these two directories remain at 1,231,752k after the two clearly different files are hard linked together? I would have expected /data/backup/rsnapshot/daily.3/ruminant to now be 6,728k smaller.

ext3 filesystem mounted at /data/backup, Linux Debian squeeze host.

I've checked with lsof | grep deleted that neither file is being held open by some process.

I have a Debian jessie server with two Intel DC S3610 SSDs in it, in RAID-10. It's reasonably busy for IO and for the last several weeks I've been graphing the IOPS:

As you can see, most of the time it was happily doing about 1k average read ops peaking at about 5.5k, until at midnight UTC on Friday that abruptly appears to stop and read ops drop down to almost nothing.

I've actually only noticed this retroactively because the thing is, the server is still performing as it should. That is, I believe it is the monitoring that is broken, not the amount of IOPS that the setup can do. If the actual IOPS had dropped to the level displayed, I would know because everything else would very noticeably break.

On further investigation the kilobytes read/written graphs are also broken at the same point. The request latency graphs are okay though.

In an attempt to rule out the particular graphing solution in use here (cacti and SNMP) I had a look at iostat. Its output matches what is being displayed on the graphs.

As far as I am aware iostat gets its info from /proc/diskstats. According to https://www.kernel.org/doc/Documentation/iostats.txt there will be the major, minor, device name and then a set of fields, of which the first one is the number of reads completed. So:

$ for i in {1..10}; do awk '/sda / { print $4 }' /proc/diskstats; sleep 1; done
3752035479
3752035484
3752035484
3752035486
3752035486
3752035519
3752035594
3752035631
3752036016
3752036374

It's just not credible that such a low number of reads were completed in that 10 second span.

But if /proc/diskstats is lying to me then what could the issue be and how can I hope to fix it?

Also of interest is the fact that whatever changed, changed at exactly midnight, which is rather a coincidence.

The server has quite a lot of block devices. 187 of them are LVM LVs, and another 18 are the usual partitions and md devices.

I've been regularly adding more LVs so it is possible that on Thursday I reached some sort of limit, though I did not add any anywhere near midnight so it is still strange that whatever has gone wrong has done so at midnight.

I know that /proc/diskstats can overflow, but when it does the numbers are typically erroneously huge.

Looking at the graph a bit harder we can see that it seems more spiky on Thursday than previously in the week (and weeks) before it. Zooming in on the results just for that period we see:

Those gaps of zero or close to zero reads are abnormal and I do not believe they reflect reality. Perhaps the number of requests has been exceeding some threshold as I added more load such that it started to manifest on Thursday and by Friday most of the readings are now zero?

Does anyone have any ideas as to what is going on here?

Kernel version 3.16.7-ckt11-1+deb8u3.

On trying to resolve dnsviz.net from a host using an Unbound resolver that is configured to use DNSSEC validation, the result is "no servers could be reached":

$ dig -t soa dnsviz.net
; <<>> DiG 9.6-ESV-R4 <<>> -t soa dnsviz.net
;; global options: +cmd
;; connection timed out; no servers could be reached

Nothing is logged by Unbound to suggest why this is the case.

Here is the /etc/unbound/unbound.conf:

server:
    verbosity: 1
    interface: 192.168.0.8
    interface: 127.0.0.1
    interface: ::0
    access-control: 0.0.0.0/0      refuse
    access-control: ::0/0          refuse
    access-control: 127.0.0.0/8    allow_snoop
    access-control: 192.168.0.0/16 allow_snoop
    chroot: ""
    auto-trust-anchor-file: "/etc/unbound/root.key"
    val-log-level: 2
python:
remote-control:
    control-enable: yes

If I add:

module-config: "iterator"

(thus disabling DNSSEC validation) then I am able to resolve this host normally.

The domain and its DNSSEC check out fine according to http://dnscheck.iis.se/ so there must be something wrong with my resolver configuration.

What is it and how do I go about debugging that?

Update:

Someone suggested that I use unbound-host in debug mode to get more info. Here we go:

$ /usr/local/sbin/unbound-host -d -4 -v -C /etc/unbound/unbound.conf -t a dnsviz.net
[1341735286] libunbound[27690:0] notice: init module 0: validator
[1341735286] libunbound[27690:0] notice: init module 1: iterator
[1341735286] libunbound[27690:0] info: resolving dnsviz.net. A IN
[1341735286] libunbound[27690:0] info: priming . IN NS
[1341735288] libunbound[27690:0] info: response for . NS IN
[1341735288] libunbound[27690:0] info: reply from <.> 192.5.5.241#53
[1341735288] libunbound[27690:0] info: query response was ANSWER
[1341735288] libunbound[27690:0] info: priming successful for . NS IN
[1341735288] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735288] libunbound[27690:0] info: reply from <.> 128.8.10.90#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735288] libunbound[27690:0] info: reply from <net.> 192.42.93.30#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: resolving ns8.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: resolving ns9.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: resolving ns2.ca.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <.> 199.7.83.42#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <.> 192.58.128.30#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for ns2.ca.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <.> 192.112.36.4#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <gov.> 209.112.123.30#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <gov.> 209.112.123.30#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735288] libunbound[27690:0] info: response for ns2.ca.sandia.gov. A IN
[1341735288] libunbound[27690:0] info: reply from <gov.> 209.112.123.30#53
[1341735288] libunbound[27690:0] info: query response was REFERRAL
[1341735300] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.29 port 53
[1341735300] libunbound[27690:0] info: response for ns2.ca.sandia.gov. A IN
[1341735300] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.29#53
[1341735300] libunbound[27690:0] info: query response was ANSWER
[1341735300] libunbound[27690:0] info: resolving ns1.ca.sandia.gov. A IN
[1341735301] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.66 port 53
[1341735301] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735301] libunbound[27690:0] info: reply from <dnsviz.net.> 198.206.219.66#53
[1341735301] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735310] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.65 port 53
[1341735310] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735310] libunbound[27690:0] info: reply from <sandia.gov.> 198.206.219.65#53
[1341735310] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735310] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.65 port 53
[1341735310] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735310] libunbound[27690:0] info: reply from <sandia.gov.> 198.206.219.65#53
[1341735310] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735310] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.28 port 53
[1341735310] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735310] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.28#53
[1341735310] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735310] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.28 port 53
[1341735310] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735310] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.28#53
[1341735310] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735310] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.29 port 53
[1341735310] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735310] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.29#53
[1341735310] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735311] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.66 port 53
[1341735311] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735311] libunbound[27690:0] info: reply from <sandia.gov.> 198.206.219.66#53
[1341735311] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735315] libunbound[27690:0] info: resolving ns2.ca.sandia.gov. A IN
[1341735315] libunbound[27690:0] info: response for ns2.ca.sandia.gov. A IN
[1341735315] libunbound[27690:0] info: reply from <gov.> 69.36.157.30#53
[1341735315] libunbound[27690:0] info: query response was REFERRAL
[1341735328] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.28 port 53
[1341735328] libunbound[27690:0] info: response for ns1.ca.sandia.gov. A IN
[1341735328] libunbound[27690:0] info: reply from <ca.sandia.gov.> 198.102.153.28#53
[1341735328] libunbound[27690:0] info: query response was ANSWER
[1341735328] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.65 port 53
[1341735328] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735328] libunbound[27690:0] info: reply from <dnsviz.net.> 198.206.219.65#53
[1341735328] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735332] libunbound[27690:0] info: response for ns2.ca.sandia.gov. A IN
[1341735332] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.28#53
[1341735332] libunbound[27690:0] info: query response was ANSWER
[1341735332] libunbound[27690:0] info: resolving ns1.ca.sandia.gov. A IN
[1341735332] libunbound[27690:0] info: response for ns1.ca.sandia.gov. A IN
[1341735332] libunbound[27690:0] info: reply from <gov.> 69.36.157.30#53
[1341735332] libunbound[27690:0] info: query response was REFERRAL
[1341735332] libunbound[27690:0] info: response for ns1.ca.sandia.gov. A IN
[1341735332] libunbound[27690:0] info: reply from <sandia.gov.> 198.102.153.28#53
[1341735332] libunbound[27690:0] info: query response was ANSWER
[1341735333] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735333] libunbound[27690:0] info: reply from <dnsviz.net.> 198.102.153.28#53
[1341735333] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735333] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.102.153.29 port 53
[1341735333] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735333] libunbound[27690:0] info: reply from <dnsviz.net.> 198.102.153.29#53
[1341735333] libunbound[27690:0] info: query response was DNSSEC LAME
[1341735333] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735333] libunbound[27690:0] info: reply from <dnsviz.net.> 198.102.153.28#53
[1341735333] libunbound[27690:0] info: query response was ANSWER
[1341735333] libunbound[27690:0] info: prime trust anchor
[1341735333] libunbound[27690:0] info: resolving . DNSKEY IN
[1341735333] libunbound[27690:0] info: response for . DNSKEY IN
[1341735333] libunbound[27690:0] info: reply from <.> 192.5.5.241#53
[1341735333] libunbound[27690:0] info: query response was ANSWER
[1341735333] libunbound[27690:0] error: Could not open autotrust file for writing, /etc/unbound/root.key: Permission denied
[1341735333] libunbound[27690:0] info: validate keys with anchor(DS): sec_status_secure
[1341735333] libunbound[27690:0] info: Successfully primed trust anchor . DNSKEY IN
[1341735333] libunbound[27690:0] info: validated DS net. DS IN
[1341735333] libunbound[27690:0] info: resolving net. DNSKEY IN
[1341735333] libunbound[27690:0] info: response for net. DNSKEY IN
[1341735333] libunbound[27690:0] info: reply from <net.> 192.48.79.30#53
[1341735333] libunbound[27690:0] info: query response was ANSWER
[1341735333] libunbound[27690:0] info: validated DNSKEY net. DNSKEY IN
[1341735333] libunbound[27690:0] info: validated DS dnsviz.net. DS IN
[1341735333] libunbound[27690:0] info: resolving dnsviz.net. DNSKEY IN
[1341735333] libunbound[27690:0] info: response for dnsviz.net. DNSKEY IN
[1341735333] libunbound[27690:0] info: reply from <dnsviz.net.> 198.102.153.29#53
[1341735333] libunbound[27690:0] info: query response was ANSWER
[1341735333] libunbound[27690:0] info: validated DNSKEY dnsviz.net. DNSKEY IN
[1341735333] libunbound[27690:0] info: Could not establish validation of INSECURE status of unsigned response.
[1341735333] libunbound[27690:0] info: resolving dnsviz.net. A IN
[1341735358] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.66 port 53
[1341735358] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735358] libunbound[27690:0] info: reply from <dnsviz.net.> 198.206.219.66#53
[1341735358] libunbound[27690:0] info: query response was ANSWER
[1341735358] libunbound[27690:0] info: Could not establish validation of INSECURE status of unsigned response.
[1341735358] libunbound[27690:0] info: resolving dnsviz.net. A IN
[1341735358] libunbound[27690:0] info: timeouts, concluded that connection to host drops EDNS packets 198.206.219.65 port 53
[1341735358] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735358] libunbound[27690:0] info: reply from <dnsviz.net.> 198.206.219.65#53
[1341735358] libunbound[27690:0] info: query response was ANSWER
[1341735358] libunbound[27690:0] info: Could not establish validation of INSECURE status of unsigned response.
[1341735358] libunbound[27690:0] info: resolving dnsviz.net. A IN
[1341735374] libunbound[27690:0] info: resolving dnsviz.net. A IN
[1341735375] libunbound[27690:0] info: response for dnsviz.net. A IN
[1341735375] libunbound[27690:0] info: reply from <net.> 192.54.112.30#53
[1341735375] libunbound[27690:0] info: query response was REFERRAL
[1341735375] libunbound[27690:0] info: resolving ns9.sandia.gov. A IN
[1341735375] libunbound[27690:0] info: response for ns9.sandia.gov. A IN
[1341735375] libunbound[27690:0] info: reply from <gov.> 69.36.157.30#53
[1341735375] libunbound[27690:0] info: query response was REFERRAL
[1341735375] libunbound[27690:0] info: resolving ns8.sandia.gov. A IN
[1341735375] libunbound[27690:0] info: response for ns8.sandia.gov. A IN
[1341735375] libunbound[27690:0] info: reply from <gov.> 69.36.157.30#53
[1341735375] libunbound[27690:0] info: query response was REFERRAL
Host dnsviz.net not found: 2(SERVFAIL). (insecure)

I haven't had chance to pick through this properly yet, but the concluded that connection to host drops EDNS packets bit jumps out at me.

Update:

This has nothing to do with Unbound - my firewall host is not forwarding some UDP packets.

eth0 is the Internet side of the firewall, eth1 is LAN side. tcpdump of both interfaces while issuing dig +norec +dnssec @198.102.153.29 sandia.gov on a machine on the LAN (the DNS server of this question):

# tcpdump -vpni eth0 'host 198.102.153.29'
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
09:37:57.234085 IP (tos 0x0, ttl  63, id 32258, offset 0, flags [none], length: 67) 82.69.129.108.37722 > 198.102.153.29.53: [udp sum ok]  24755 [1au] A? sandia.gov. (39)
09:37:57.387165 IP (tos 0x4, ttl  47, id 48355, offset 0, flags [+], length: 1196) 198.102.153.29.53 > 82.69.129.108.37722:  24755*- 2/5/13 sandia.gov. A 132.175.81.4, sandia.gov. (1168)
09:37:57.387502 IP (tos 0x4, ttl  47, id 48355, offset 1176, flags [none], length: 1498) 198.102.153.29 > 82.69.129.108: udp
09:38:02.234014 IP (tos 0x0, ttl  63, id 32259, offset 0, flags [none], length: 67) 82.69.129.108.37722 > 198.102.153.29.53: [udp sum ok]  24755 [1au] A? sandia.gov. (39)
09:38:02.386762 IP (tos 0x4, ttl  47, id 48356, offset 0, flags [+], length: 1196) 198.102.153.29.53 > 82.69.129.108.37722:  24755*- 2/5/13 sandia.gov. A 132.175.81.4, sandia.gov. (1168)
09:38:02.387101 IP (tos 0x4, ttl  47, id 48356, offset 1176, flags [none], length: 1498) 198.102.153.29 > 82.69.129.108: udp
09:38:07.260492 IP (tos 0x0, ttl  63, id 32260, offset 0, flags [none], length: 67) 82.69.129.108.37722 > 198.102.153.29.53: [udp sum ok]  24755 [1au] A? sandia.gov. (39)
09:38:07.433906 IP (tos 0x4, ttl  47, id 48357, offset 0, flags [+], length: 1196) 198.102.153.29.53 > 82.69.129.108.37722:  24755*- 2/5/13 sandia.gov. A 132.175.81.4, sandia.gov. (1168)
09:38:07.434244 IP (tos 0x4, ttl  47, id 48357, offset 1176, flags [none], length: 1498) 198.102.153.29 > 82.69.129.108: udp

9 packets captured
9 packets received by filter
0 packets dropped by kernel
# tcpdump -vpni eth1 'host 198.102.153.29'                                                                                                          
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
09:38:20.646202 IP (tos 0x0, ttl  64, id 32261, offset 0, flags [none], length: 67) 192.168.0.8.54056 > 198.102.153.29.53: [udp sum ok]  31422 [1au] A? sandia.gov. (39)
09:38:25.645589 IP (tos 0x0, ttl  64, id 32262, offset 0, flags [none], length: 67) 192.168.0.8.54056 > 198.102.153.29.53: [udp sum ok]  31422 [1au] A? sandia.gov. (39)
09:38:30.645640 IP (tos 0x0, ttl  64, id 32263, offset 0, flags [none], length: 67) 192.168.0.8.54056 > 198.102.153.29.53: [udp sum ok]  31422 [1au] A? sandia.gov. (39)

Note that eth0 gets a bunch of UDP packets that aren't being forwarded.

The firewall rules are quite simple, being basically "NAT everything to/from 192.168.0.8 to 82.69.129.108, NAT everything else to 82.69.129.105, block all traffic after allowing a few sensible ports/protocols".

Here's a rules list:

# iptables -vnL
Chain INPUT (policy DROP 87 packets, 5073 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1010  216K ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  eth0   *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
   58  4408 ACCEPT     udp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           udp dpt:123 
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:123 
    0     0 ACCEPT     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            0.0.0.0/0           tcp dpt:22 
   87  5073 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 1/sec burst 5 LOG flags 0 level 4 prefix `INPUT: ' 

Chain FORWARD (policy DROP 6 packets, 300 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    2  1383 LOG        tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:!0x16/0x02 state NEW LOG flags 0 level 4 prefix `New but not syn: ' 
    2  1383 DROP       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           tcp flags:!0x16/0x02 state NEW 
78595   75M ACCEPT     all  --  eth1   *       0.0.0.0/0            0.0.0.0/0           
58873   13M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
    9   576 ACCEPT     icmp --  eth0   *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.0.8         tcp dpt:22 
    4   240 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.0.8         tcp dpt:80 
    0     0 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.0.8         tcp dpt:443 
    2   120 ACCEPT     tcp  --  eth0   *       0.0.0.0/0            192.168.0.8         tcp dpt:25 
    0     0 ACCEPT     udp  --  eth0   *       192.168.2.1          192.168.0.8         udp dpt:514 
    2   152 ACCEPT     udp  --  eth0   *       192.168.2.1          192.168.0.8         udp dpt:123 
    0     0 ACCEPT     all  --  eth0   *       192.168.1.1          0.0.0.0/0           
    6   300 LOG        all  --  *      *       0.0.0.0/0            0.0.0.0/0           limit: avg 1/sec burst 5 LOG flags 0 level 4 prefix `FORWARD: ' 

Chain OUTPUT (policy ACCEPT 460 packets, 67812 bytes)
 pkts bytes target     prot opt in     out     source               destination

# iptables -t nat -vnL
Chain PREROUTING (policy ACCEPT 2696K packets, 192M bytes)
 pkts bytes target     prot opt in     out     source               destination         
   21  1236 DNAT       all  --  eth0   *       0.0.0.0/0            82.69.129.108       to:192.168.0.8 

Chain POSTROUTING (policy ACCEPT 108K packets, 10M bytes)
 pkts bytes target     prot opt in     out     source               destination         
 1549  115K SNAT       all  --  *      eth0    192.168.0.8          0.0.0.0/0           to:82.69.129.108 
  709 42396 SNAT       all  --  *      eth0    0.0.0.0/0            0.0.0.0/0           to:82.69.129.105 

Chain OUTPUT (policy ACCEPT 19719 packets, 3998K bytes)
 pkts bytes target     prot opt in     out     source               destination

Nothing useful is being logged by those LOG rules.

The firewall is a Linux install but it's running on a Soekris device read-only from a CF card; as such I treat it more like an appliance and haven't upgraded it since it was installed. It's therefore a really old Debian etch install with a 2.6.12 kernel. Could this be a kernel bug related to UDP fragmentation or connection tracking?

Anyway I'm going to remove the DNSSEC and Unbound tags from this and add iptables etc.

I have several Entropy Keys with egd in front of them and then all load balanced by haproxy. I then have many client machines using the haproxy service IP as a network source of entropy. I have no idea how much entropy they are requesting.

The Entropy Keys can produce a limited amount of usable entropy. The specs say about 30 kilobit/sec minimum. As far as I can see, the Entropy Key doesn't have a way to query how much it is being asked for. The EGD protocol seems quite hard to sniff to find this information. Clients can request a variable amount of entropy, and they might not get back the same amount.

Has anyone found a simple way to measure how much is being requested from an Entropy Key?

It would be good to know, in order to be able to plan when additional keys are required, and to spot haywire clients.

Say I've got an LVM logical volume that I'm going to pass to a virtual machine (Xen, but I don't think this matters) to be used as a whole disk. The virtual machine is going to see that as /dev/xvda and expect a partition table on it.

Tools like fdisk like to suggest a starting sector of 63 for the first partition, but given the possibility of 4KiB sectors on the host, I think this could lead to future alignment problems causing performance issues.

What is a safe way to determine the starting sector of the first partition? How much space at the start should be left for a bootloader?

I think if I stuck to multiples of 8 for a starting sector (thus working for storage with 512B or 4,096B sectors) that would cover alignment. Since fdisk seems to like sector 63, perhaps it would be best to start at sector 64?