Crash Override's questions

How do I configure a new mdadm software RAID-10 to use a specific order for its drive mirroring?

Say I have 4 hard drives: A, B, C, and D.

For my own reliability and performance reasons, I want drive A to be a mirror of drive C, and drive B to be a mirror of drive D. Then each mirror striped, like this

 |----- RAID-0 ----- |
  RAID-1      RAID-1
|---------| |---------|
| drive A | | drive B |
| drive C | | drive D |
|---------| |---------|

I don't see a way to configure this. Not sure about the drive order this command produces:

mdadm --create /dev/md0 --level=10 --raid-devices=4 /dev/sd[a-d]1

Is there a way to manually order the drives in a mdadm array?

If not, is there a way to check the order after the array is created so I could physically move the drives into the configuration I want? (I think I can create the array members using /dev/disk/by-id/ata* so I can physically move them without worrying about /dev/sd* re-ordering.)

Bonus question. I also will want a hot spare, drive E, to be available to either mirror set.

Say I have two hard drives that max at 200 MBytes/sec writes per hard drive. Now let's say both drives are in a USB enclosure that maxes at 300 MBytes/sec writes for the entire enclosure.

When setting up software RAID-1 (mirroring) of both drives using mdadm on Linux, would it improve write performance by putting each of the two drives into different USB enclosures?

Another way to ask this question, does mdadm software RAID-1 double the write bandwidth needed to USB enclosure(s)?

My exim relays all its incoming email to third-party smtp relays. They use smtp authentication (username/password).

It seems I can only define ONE username/password in the begin authenticators section of the Exim config. I want to route certain email through one relay (with its own username/password auth), and other email through a second relay (its username/password auth is different than the first).

Here's much of my config. The begin routers relays domains in the +local_domains list to smtp-relay-1 and the rest relays to smtp-relay-2.

begin routers

my_domains_relay:
  debug_print = "R: my_domains_relay for $local_part@$domain"
  driver = manualroute
  domains = +local_domains
  transport = remote_smtp_smarthost
  route_list = * "<+ smtp-relay-1.example.com:465"
  host_find_failed = defer
  no_more

smart_host_relay:
  debug_print = "R: smart_host_relay for $local_part@$domain"
  driver = manualroute
  transport = remote_smtp_smarthost
  route_list = * "<+ smtp-relay-2.example.net:465"
  host_find_failed = defer
  no_more

begin transports

remote_smtp_smarthost:
  debug_print = "T: remote_smtp_smarthost for $local_part@$domain"
  driver = smtp
  port = 465
  hosts_require_tls = *
  hosts_require_auth = *
  protocol = smtps

begin authenticators

login:
  driver = plaintext
  public_name = PLAIN
  client_send = ^my-username^top-secret-password

What I want is to define separate username/password auth in the begin authenticators section, and assign each to a single router/transport. Right now, the username/password auth is global and used for all relays.

The exim documentation says it matches the public_name with an advertised authentication by the server. So if both my smtp relays advertise as AUTH PLAIN then they both use that one authenticators username/password in the config. I'm hoping there's a setting that allows me to link an authenticators instance with a specific router/transport, but I don't see how.

I have a perl and shell script that process a bunch of data and output results as they happen, like:

5 processed in 0.58 seconds.
10 processed in 0.79 seconds.
...
150 processed in 0.65 seconds. DONE!

etc.

However, when I visit the URL with Chrome, it shows nothing, waits until the script finishes, then shows the entire output at once. Is there a way to configure nginx to show each line output as it happens? I used to be able to do this with Apache.

I put gzip off in the nginx config, thinking this could nginx waiting to compress the text before sending it to the client, but that did not help.

I also tried buffer-flushing techniques in the scripts themselves, like these lines at the top of my Perl scripts:

use IO::Handle;
$| ++;
STDERR->autoflush(1);
STDOUT->autoflush(1);
print "Content-type: text/plain\n\n";

Relevant parts of the nginx config look like this

  ssl on;
  root /my/path/cgi ;
  proxy_read_timeout 900s ;
  fastcgi_read_timeout 900s ;
  fastcgi_request_buffering off ;
  gzip off ;
  location ~ \.pl|cgi$ {
    try_files $uri =404;
    gzip off;
    fastcgi_pass  unix:/var/run/fcgiwrap.socket;
    include /etc/nginx/fastcgi_params;
    fastcgi_param SCRIPT_FILENAME  $document_root$fastcgi_script_name;
  }

Any ideas?

Old server:

• 3ware 9650se PCIe hardware RAID controller with 2 sets of RAID-1 (4 total drives, 2 mirrored) with a 5th drive as a hot-spare.

New server:

• Software RAID is the only option. Plan to run linux mdadm with the same RAID-1 configurations.

These are not boot drives, storing just data.

Can I transfer the drives from hardware RAID to software RAID? Will the data be intact?

If I set my DNS zone's SOA serial number to "1" and never change it, would domain name servers around the internet ignore or be slow in seeing zone updates?

Since all my primary DNS servers for my domain are masters, I don't have a case where slaves transfer zone data when the serial changes. Each of my DNS maintain identical master zone files.

Say I have an "A" record. I assume each time the TTL expires, resolvers re-query the domain's masters for a new "A" record and will return the IP in the master zone at each time of query. So does the SOA serial matter?

Some additional context. I see that AWS Route53 defaults new DNS zones to serial "1" and says that incrementing the serial is "optional" but does not say why it's optional https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/SOA-NSrecords.html

Anyone know a good way keep a copy of outgoing email that passes through an Exim mail/smtp server? I know incoming mail can be copied and piped to a script, but can outgoing mail also be piped to a script? Any know if it's possible to keep a log of outgoing mail and its contents before mail is sent to its destination?

I have a VPN, and my server frequently sends data to a private IP address that routes over the VPN. When the OpenVPN gets established or dies, it enables/disables the routes.

I want to null-route that private IP range from going out over the main Internet interface (eth0). Is there an easy way to do that without interfering with the route commands coming from the VPN software?

Iptables won't do it. I tried

iptables -A OUTPUT -i eth0 -p tcp -d 192.168.0.0/16 -j REJECT

But iptables does not work when specifying an interface in the output chain.

Anyone know if there is a way to add a dummy route onto a specific interface (eth0) only, without interfering with other interfaces that may be using that route?

Ps- I am aware 192.168.0.0/16 is not INTERNET routeable, but for security reasons, want to ensure no data gets out in the rare chance another local server or network device starts listening on the private subnet.

Does Windows 10 Pro file sharing (aka Samba) throttle its speed? Mine seems to cap out at exactly 25 megabytes per second.

My Windows 10 Pro machine is directly connected (via crossover cable) to a CentOS samba server with 10gbe (10-gigabit) network cards.

I get at least 7 gigabits per second between the machines. I tested this by doing an FTP from the Windows box's RAM drive to the CentOS server's RAM drive, I get 899 Megabytes/second transfer (which is about 7.5 gigabits per second). So I know they can communicate at that speed.

When I connect to the same drives via Windows File Sharing (Samba), it maxes out at exactly 25 Megabytes/second. That's a rounded number, which makes me think I'm hitting some throttle somewhere, either on the Windows side or the Samba server side.

Samba version: samba-3.6.23-35.el6_8.x86_64

I should also note, Samba appears to use far more server CPU than the FTP server does.

I just installed a Mellanox ConnectX-2 10gbe PCIe x8 Card into my server running CentOS 6.7. On boot, dmesg shows the mlx4_core driver being loaded automatically however I see no eth1 device corresponding to the card. ifconfig -a only shows eth0, which is my onboard/motherboard Gigabit network interface.

So, I deleted /etc/udev/rules.d/70-persistent-net.rules and rebooted. It re-created 70-persistent-net.rules but only with my onboard/motherboard gigabit NIC, no entry for the Mellanox 10gbe card.

Output of /etc/udev/rules.d/70-persistent-net.rules

# PCI device 0x10ec:0x8168 (r8169)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:30:67:aa:bb:cc", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"

Good news is, lspci shows the Mellanox card. Output of lspci | grep -i ether

02:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 03)

Relevant output from dmesg:

mlx4_core: Mellanox ConnectX core driver v2.2-1 (Feb, 2014)
mlx4_core: Initializing 0000:02:00.0
mlx4_core 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mlx4_core 0000:02:00.0: setting latency timer to 64
  alloc irq_desc for 26 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 26 for MSI/MSI-X
  alloc irq_desc for 27 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 27 for MSI/MSI-X
  alloc irq_desc for 28 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 28 for MSI/MSI-X
  alloc irq_desc for 29 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 29 for MSI/MSI-X
  alloc irq_desc for 30 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 30 for MSI/MSI-X
  alloc irq_desc for 31 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 31 for MSI/MSI-X
  alloc irq_desc for 32 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 32 for MSI/MSI-X
  alloc irq_desc for 33 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 33 for MSI/MSI-X
  alloc irq_desc for 34 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 34 for MSI/MSI-X
  alloc irq_desc for 35 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 35 for MSI/MSI-X
  alloc irq_desc for 36 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 36 for MSI/MSI-X
  alloc irq_desc for 37 on node -1
  alloc kstat_irqs on node -1
mlx4_core 0000:02:00.0: irq 37 for MSI/MSI-X

Any ideas why my Mellanox ConnectX-2 is not showing up as an eth1 device?

I encrypted a bunch of hard drives with cryptsetup luks encryption on CentOS 5. Everything was fine, until I upgraded my system to CentOS 6. Now I cannot mount the disks anymore.

To mount with my keyphrase:

sudo cryptsetup luksOpen /dev/sdc1 d2

I get this error:

device-mapper: reload ioctl on  failed: Invalid argument
Failed to setup dm-crypt key mapping for device /dev/sdc1.
Check that kernel supports aes-xts-plain64 cipher (check syslog for more info).
Failed to read from key storage.

In /var/log/messages:

Feb  3 23:43:23 data kernel: device-mapper: table: 253:0: crypt: Device lookup failed
Feb  3 23:43:23 data kernel: device-mapper: ioctl: error adding target to table

Any thoughts on how I can mount??

Anyone know the maximum length of a LUKS passphrase?

The passphrase will be randomly generated, and I can create one with any length.

I am using CentOS 6 with Linux kernel 2.6.32-573.12.1.el6.x86_64, and cryptsetup-luks version 1.2.0-11.el6.x86_64

I just spent about 6 hours trying to figure this out, and I now believe CentOS/Linux is unable to bind to a specific virtual IPv4 address when connecting to a hostname that has an IPv6 address. This is a problem on servers that have multiple IP addresses.

I am using Centos 6 (Linux kernel 2.6.32-573.12.1.el6.x86_64)

To reproduce this big:

1. Find a Linux machine with at least a /29 IPv4 public address space, and an IPv6 public IP.
2. Alias at least one additional IPv4 to the main interface (eth0 or otherwise). For this example, I will say 30.0.0.1 as the machine's primary eth0 IPv4 address, and 30.0.0.2 is an alias bound to eth0:2 on a network of 30.0.0.0/29.
3. Find a hostname that has both IPv4 and IPv6 addresses. For example, www.microsoft.com.
4. telnet -b 30.0.0.2 www.microsoft.com 80 (This tests making an outbound connection using a specific ipv4 address)
5. The IPv4 request connects successfully after unsuccessfully trying the hostname's IPv6 addresses, but the TCP connection actually initiates from the machine's main IP (30.0.0.1), not the IP you want (30.0.0.2). Netstat says otherwise, but it is WRONG. If you connect to a hostname that you own and can view its logs, the connection comes from the main IP (30.0.0.1) not the IP you want (30.0.0.2).
6. Try another domain that does not have an IPv6 address, like: telnet -b 30.0.0.2 serverfault.com. It works. It makes the connection from the IP you want.

This is a problem, because certain programs, like mail (exim), need to use certain IPs when making outbound TCP requests, which is not necessarily the main machine's IP. Certain clients/programs on the machine rely on ACLs or reverse DNS to match up correctly when making outbound TCP connections.

So if anyone else is noticing the same weird problem where their program cannot bind to the correct interface when making outgoing connections, this is probably why.

This problem only effects IPv4 connections. IPv6 connections correctly bind to whatever outgoing IP you have on the machine.

This is not an issue with telnet. I tested this problem using my mail server (exim) too, and got similar results. It makes IPv4 connections from the wrong IP if the target hostname has an IPv6 address.

Maybe someone has a solution to this strange problem, but at this time, I think it may be a Linux networking bug.

Ps- If someone wonders why not just make an IPv6 connection if the hostname resolves to an IPv6 address...sometimes the IPv6 address is down or the connection cannot establish, then it reverts to its IPv4 address.

Strange problem. I have server "ALPHA" in Dallas and server "BETA" in Hong Kong, about 195ms ping times apart from each other. ALPHA is connected at 1gbps full, and BETA at 10mbps full. Occasionally when they communicate to each other over TCP, the connection hangs. Eventually ALPHA gives up and closes the connection, but BETA still sees the connection as established and waits for a long time until it eventually times out.

Example.

On BETA, wget -O /dev/null ALPHA:50001/1mb.test

--2013-05-19 02:45:54--  http://ALPHA:50001/1mb.test Resolving dfw... ip.address.redacted Connecting to ALPHA|ip.address.redacted|:50001... connected. HTTP request sent, awaiting response... 200 OK Length: 1000000 (977K)
[application/octet-stream] Saving to: `/dev/null'

and it freezes at 0 bytes received. Note that it did get a 200 OK from the server, so they did seem to handshake successfully. To eliminate wget as part of the problem, I did a telnet:

telnet ALPHA 50001
Trying ip.address.redacted...
Connected to ALPHA.
Escape character is '^]'.
GET /1mb.test HTTP/1.0

HTTP/1.1 200 OK
Server: nginx
Date: Sun, 19 May 2013 22:43:48 GMT
Content-Type: application/octet-stream
Content-Length: 1000000
Last-Modified: Sat, 21 Jan 2012 21:47:29 GMT
Connection: close
ETag: "4f1b3271-f4240"
Accept-Ranges: bytes

It also freezes at this point.

Both servers do have iptables rules of -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

Here is a tcpdump that was running on ALPHA during the above wget freeze. http://pastebin.com/axX7ap6T

This problem happens only occasionally. Sometimes it works just fine. Actually, if I run about 3-5 wget's, usually everything starts working on about the fourth or fifth try.

BETA can communicate with GAMMA, DELTA, and my other servers without problem. Only seems to happen when talking to ALPHA. ICMP pings show no packet losses, but even if there was packet loss, it would slow it down, not disconnect. Both machines have the same CentOS 6.4 64-bit OS, kernel 2.6.32-358.6.2.el6.x86_64. Nothing useful logged in /var/log files. ALPHA is listening on 50001 using nginx-1.4.1-1.el6.ngx.x86_64. The nginx access log file shows a 200 (request OK) 43440 bytes delivered during the frozen connection.

The problem does not seem to be nginx related, since I have had an ssh connection also freeze in the middle of communicating on the same day. Other servers connect to that nginx and never had any problems.

In summary, this appears to be what is happening in this particular communication:

BETA -> ALPHA "SYN!"
ALPHA -> BETA "SYN Acknowledgement"
BETA -> ALPHA "ACK"
ALPHA -> BETA "Established"
BETA -> ALPHA "GET /file HTTP/1.0"
ALPHA -> BETA "200 OK, here's a bunch of data"
BETA freezes, gets no data.

Anyone seen a situation like this?

Strange. My bind is not validating dnssec even though I configured it to. Version according to named -V is BIND 9.8.2rc1-RedHat-9.8.2-0.10.rc1.el6_3.2 which has a built-in DLV key.

Under options in named.conf

dnssec-enable yes;
dnssec-validation yes;
dnssec-lookaside auto;

But when I query a known bad zone, like doing dig www.dnssec-failed.org @localhost I get IP addresses- not a failure like I was expecting. Any thoughts?

I have a strange problem with one of my servers. ntpd and ntpdate fail to work, but debugging shows no errors at all. At first I thought maybe a local or network firewall was blocking UDP port 123, but that is not the case- this server can talk UDP port 123 (the ntp protocol) to the Internet and get answers.

Let me demonstrate the issue.

date -s "30 DEC 2012 02:30:00" - works, so I can successfully set the clock without error.

ntpq -pn pool.ntp.org - works, I do get detailed time data from the timeserver, and proves UDP packets are functioning.

ntpdate -d pool.ntp.org - debug mode works, shows a ton of debug data and shows the current time offset: 30 Dec 02:38:56 ntpdate[19267]: step time server 208.97.140.69 offset 228.234554 sec

All looks normal, until: ntpdate pool.ntp.org - after a 4.7 second pause, it returns: 30 Dec 02:41:29 ntpdate[19274]: no server suitable for synchronization found

Similar problem running ntpd, it does not update the clock.

After ntpd is started, ntpq -pn results in all refid's forever stuck on .INIT. which means they cannot sync.

/var/lib/ntp/drift is the driftfile setting in ntp.conf, which is chmod 644 and owned by ntp:ntp, same as all my other systems.

I tried a dozen other ntp time servers, disabled iptables firewall, and confirmed the datacenter is not filtering udp traffic. Any ideas what is stopping ntpd and ntpdate from syncing my clock?

This is CentOS 6.3 x64 on a dedicated server with Intel CPU.

I have a domain, call it DOMAIN.NET, which is an Internet service provider. DOMAIN.NET has Glue Records that I put in via the existing registrar, which enable the client domains like FOO.COM, BAR.COM, BAZ.COM, etc. to use NS1.DOMAIN.NET and NS2.DOMAIN.NET as their DNS servers. For anyone who doesn't know, Glue Records are essential for the functionality of NS1, NS2, etc., not going to explain it here, but these explain it

• http://faq.domainmonster.com/dns/glue_record/
• What is a glue record?

I want to transfer DOMAIN.NET to another registrar. But, do the Glue Records get transferred? My guess is no, because I would think registrars all send and manage Glue Records themselves, and send the Glue IPs directly to the Internet root servers for .Net, so how would the new registrar know about it during a domain transfer?

I need to know before I transfer, because if the Glue records disappear, all my client domains that have NS1.DOMAIN.NET and NS2.DOMAIN.NET as their dns servers will likely start failing for a period of time until I get the Glue records re-added.

Is there a magical way to transfer Glue Records to the new registrar when transferring a domain?

EDIT Jan 2, 2013 I am happy to report that the Glue Records did get transferred with my domain. Before transferring my live domain, I transferred a "test" domain which had Glue Records. Transferred from Network Solutions into Godaddy. After transfer, when I query the root servers, it still shows my glue records are there. When I look at the Godaddy Domain Control Panel, the domain shows all the Glue Records under "Host Summary", meaning they did transfer into the new registrar with the domain. For others resting this, I would be safer to use a different DNS during the transfer of that domain, just in case the Glue Records were lost to avoid loops.

I imagine others may be in similar positions as me at some point, and I hope this is very useful information, since I couldn't find the answer anywhere.