mike's questions

When copying large files (50+GB) from an NVMe disk to a SATA 7200rpm HDD disk I see the following error in the logs on a fully patched Ubuntu 20.04:

Aug 08 00:45:59 host kernel: ata6.00: exception Emask 0x20 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 08 00:45:59 host kernel: ata6.00: irq_stat 0x20000000, host bus error
Aug 08 00:45:59 host kernel: ata6.00: failed command: WRITE DMA EXT
Aug 08 00:45:59 host kernel: ata6.00: cmd 35/00:08:30:a2:e0/00:00:e8:00:00/e0 tag 23 dma 4096 out
                                    res 50/00:00:00:00:00/00:00:00:00:00/00 Emask 0x20 (host bus error)
Aug 08 00:45:59 host kernel: ata6.00: status: { DRDY }
Aug 08 00:45:59 host kernel: ata6: hard resetting link
Aug 08 00:46:00 host kernel: ata6: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Aug 08 00:46:00 host kernel: ata6.00: configured for UDMA/133
Aug 08 00:46:00 host kernel: ata6: EH complete

ata6.00 is the disk which is being written to.
The issue is intermittent. Sometimes does not appear for 24 hours, sometimes a couple times per hour. Often times the disk recovers, but sometimes the filesystem just becomes corrupt, needs to be unmounted, repaired (if possible) and remounted.

What I tried:

1. I tried 3 different brands of HDD. All have the same issue.
2. I suspected hardware issue. I replaced the motherboard and SATA cables. None of this helped.
3. I have another server with an identical configuration. The issue does not occur there. Same workload.
4. I have yet another server with a completely different configuration (Intel vs. AMD). The issue occurs there. Same workload.
5. I disabled NCQ via echo 1 > /sys/block/sda/device/queue_depth. Did not help.

I ran out of ideas...
These are all data center grade components. Given the steps I've taken, I suppose it's not a hardware manufacturing defect.
Could this be software/OS/BIOS related?
Any ideas what else should I try?

We're running Docker containers with devicemapper on CentOS 7 (3.10.0-514.10.2.el7.x86_64).
For some containers we can easily throttle reads through blkio.throttle.read_bps_device by specifying the parent block device, but for some containers it just doesn't work.

Here's our lsblk:

NAME                                  MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                     8:0    0   32G  0 disk
└─sda1                                  8:1    0   32G  0 part /
sdb                                     8:16   0  100G  0 disk
├─data-docker_thin_tmeta              253:0    0  512M  0 lvm
│ └─data-docker_thin                  253:2    0   90G  0 lvm
│   └─docker-8:1-151197496-c619...6ed 253:4    0   10G  0 dm
├─data-docker_thin_tdata              253:1    0   90G  0 lvm
│ └─data-docker_thin                  253:2    0   90G  0 lvm
│   └─docker-8:1-151197496-c619...6ed 253:4    0   10G  0 dm
└─data-data                           253:3    0    9G  0 lvm  /data

When we specify in a cgroup the following, it properly throttles the reads for some containers:
echo "8:16 1000000" > blkio.throttle.read_bps_device

But for other containers, we need to specify the exact device such as:
echo "253:4 1000000" > blkio.throttle.read_bps_device

Any ideas what may be the reason why the throttling with parent device at 8:16 does not work sometimes?

The journald documentation says that adding a user to 'systemd-journal' group or 'adm' group allows the user to access system-wide journal.

I'm running the latest CentOS 7 and I seem to have problem accessing the journal as a non-root user.

Here's my config:

$ id
uid=1000(centos) gid=1000(centos) groups=1000(centos),4(adm),10(wheel),190(systemd-journal) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

$ cat /etc/systemd/journald.conf
[Journal]
Storage=persistent

$ journalctl
-- Logs begin at Sat 2015-08-29 16:35:52 UTC, end at Sat 2015-08-29 17:28:47 UTC. --
Aug 29 16:35:52 hostname ... <log continues>

There are no system logs in journalctl output. Here's my permissions config:

$ ll -a /var/log/journal/f9afeb75a5a382dce8269887a67fbf58/
total 24592
drwxr-xr-x. 2 root root     4096 Aug 29 16:35 .
drwxr-xr-x. 3 root root     4096 Aug 29 17:28 ..
-rw-r-----. 1 root root 16777216 Aug 29 17:27 system.journal
-rw-r-----+ 1 root root  8388608 Aug 29 17:33 user-1000.journal

If I change the ownership group of system.journal to systemd-journal everything works fine. However this does not seem right, since the documentation does not say anything about it.

Is there anything I'm missing or is it actually required to manually change the group of the system.journal file?

Thank you

I'm running Phusion Passenger 4.0.40 with nginx.
The server is used to host several websites and each has its own error.log file.

However if an application crashes during startup, the error message is not written to the website's error.log, rather to global error.log. Since global error.log is not accessible to the websites, it's hard to debug the issue.

Is there any way to redirect startup errors along with console.log() from global error.log to website's error.log?

Here's the relevant part of nginx config:

error_log logs/error.log;

http {
  passenger_friendly_error_pages off;
}

server {
  passenger_enabled on;
  error_log /var/www/subdomain/log/error.log;
}

Any help is appreciated. Thanks