Ján Lalinský's questions

We have a centos8 web server hosting hundreds of websites, using 3 services: apache, mariadb and php-fpm. Because apache and mariadb do not support per-account resource limitation, all websites have the same access to many system resources (storage, database) and a single website can overload the machine (often that happens through database queries) and bring down all websites hosted there.

We would like to use Linux cgroups and possibly other container features to introduce limits to resources one website can use, mostly the number of php processes and I/O operations and database queries per second.

One (not necessarily the most performant) way to achieve this is to run one dedicated group of services (apache+mariadb+php-fpm) for each website or website group, with specific configuration of resource limits. Thus, to have hundreds of simultaneously running service groups.

What is the more straightforward way to set this up from existing setup of working 3 services? Performance is secondary and we do not want to go into creating and running hundreds of container images. Instead, we would like to have everything defined in an easily runtime-modifiable form in the root filesystem and control all services through systemd. Thus, no docker containers.

I would like to stop Apache (2.2) from doing reverse lookups for client IP addresses. I have

HostnameLookups Off

and

LogFormat "%h %l %u %t %V \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" port:%p" combined_vhost
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common

in httpd.conf but Apache still in many cases often does the lookups and writes down domain names instead of IP addresses into accesslogs.

I need the IPs from accesslogs to block them on firewall.

If it is not possible to set this up in Apache config., maybe it can be done in the bind config. On the machine I need to fix, bind runs as local recursive resolver.

The domain names are useless because they can't be resolved back to the original IP address (seems there is intentional lack of usual DNS A/AAAA record for these domain names).

I would like to overwrite a very large hard drive (18TB) with random bytes, to then check smart data for reallocated sectors or other errors.

Since badblocks has some limitations on number of blocks it will work with in a single run, I have tried the "cryptsetup method" described on archlinux wiki:

https://wiki.archlinux.org/title/Badblocks#Finding_bad_sectors

I set up an encrypted logical device eld on the whole drive and then used the command "shred" to write zeroes to the opened eld device:

cryptsetup open /dev/device eld --type plain --cipher aes-xts-plain64
shred -v -n 0 -z /dev/mapper/eld

It went on to print lines such as

shred: /dev/mapper/eld: pass 1/1 (000000)...870MiB/17TiB 0%
shred: /dev/mapper/eld: pass 1/1 (000000)...1.7GiB/17TiB 0%
...
shred: /dev/mapper/eld: pass 1/1 (000000)...4.1TiB/17TiB 24%

but then it stopped at 4.1TiB/17TiB written. I've verified this with hexdump, zeroes were not written beyond byte address 0x428249b0000 (4570459340800 ~ 4.156 TiB):

hexdump -C  --skip 0x428249a0000 /dev/mapper/eld | head
428249a0000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
428249b0000  b3 cd d0 34 72 15 f2 2c  f6 32 90 fb 69 24 1f ec  |...4r..,.2..i$..|
428249b0010  a0 f4 88 a5 56 e7 13 82  94 e5 e0 f5 37 da c3 59  |....V.......7..Y|
428249b0020  9b 55 9f d8 39 a1 41 dc  52 ca 7b 3a 95 f5 59 e2  |.U..9.A.R.{:..Y.|

Many standard commands seem to have problems with high capacity disks because the numbers involved are too big for 32 bit data types. Which read/write tools on Linux are able to read/write beyond these 2TiB,4TiB imaginary boundaries reliably?

We have set up centralized logging of auditd messages for two machines:

• machine (www22.domain.com) is the source (centos8)
• machine (cls.domain.com) is the centralized log server (centos7)

This was done in the standard way using auditd+audisp plugin sending to auditd server listening on port 60, e.g. like described here:

https://luppeng.wordpress.com/2016/08/06/setting-up-centralized-logging-with-auditd/

But then when I observe the audit log on the centralized log server after restarting auditd client on the source, the only thing that appears are the lines

node=cls.domain.com type=DAEMON_CLOSE msg=audit(1632773977.760:3884): addr=::ffff:x.y.z.152 port=42652 res=success
node=cls.domain.com type=DAEMON_ACCEPT msg=audit(1632773988.330:3885): addr=::ffff:x.y.z.152 port=44282 res=success

where ::ffff:x.y.z.152 is obviously due to some packet(s) from IP address x.y.x.152 (address of www22.domain.com). So the TCP connection between client-server gets established and it seems further message logging should work.

But then the only new lines that ever appear in the log file are those that originate on cls.domain.com. There are never audit messages from www22.domain.com.

I've checked what happens if auditd www22.domain.com is set up to write also to local audit log file; then the local file gets lots of messages from audit. But still nothing is sent over the network.

How to make sure the auditd client sends the same messages over the network?

Every once in a while, our Linux LAMP server (using PHP-FPM, XFS on thin LVM on HW RAID, Centos8) becomes inaccessible and stops responding to HTTP(S) requests.

Via centralized logging we found out that in those cases, load average quickly shoots up to hundreds, while more and more processes (systemd-journald, php processes, kernel xfs/dm threads...) get into a D state. According to iostat and pidstat, CPU and disk are not loaded much at all while load average hovers around 170, which is quite strange. From htop/ps output, there is no single or group of rogue processes that would explain this behaviour. It's just standard processes that seem to encounter some kind of "road block".

The only other strange thing with disk monitoring is that during those overload events, iostat intermitently reports quite high w_await for partition /var (2500-5000ms, while other partitions like /var/log, /var/lib/mysql mostly do not get over 10ms). This partition should be quiet most of the time, so it is not clear why iostat reports such large w_await times there.

The only solution then is to power cycle the server.

This happens on two servers of the same kind, never on others. It seems to be some kind of FS/block layer/controller/disk malfunction; lots of processes suddenly start waiting for disk or something else in the kernel, but according to iotop/iostat, disk is not doing much.

Is there a way to query Linux kernel FS/block layer/controller driver what exactly they are doing with storage and on behalf of which process? Standard tools like iotop/iostat tell me only names of I/O active processes and disk partition activity, but not which processes access which disk partition and what exactly they are doing there.

On a shared hosting server (centos8), thousands of websites are hosted. They all use single DNS service running as local recursive resolver (BIND 9).

We observe random overload events with no clear culprit.

One hypothesis is that some website (unix user) is attacking local DNS resolver.

In the context of locally running bind daemon, what is the best way to make it log not only the DNS query, but also the unix user that sent it? Is it even possible?

I am trying to improve performance (responsivity) of a webserver during rare events of high memory utilization (90%...) where it starts to swap a lot, in order to help manual intervening faster and less error prone. I can't add more RAM to the machine (32G RAM + 6G swap partition).

One way to improve performance in such a situation is using compressed swap caching in RAM, so that the system uses RAM more efficiently and the impact of I/O waits for slow nonvolatile memory decreases.

Older research from 1999 [1] indicates that the faster the CPU compared to I/O operations, the greater the benefits of compressed swap cache and this should get even better as CPUs get faster. It is recommended to use swap compression if using rotational disks.

I wonder whether this advice to increase performance still holds true for servers that have only SSD drives. Here the I/O operations are much faster, so the benefit is lower than with rotational disks.

An implementation of this functionality was introduced in Linux 3.11 via zswap in 2012 [2]. Does anybody have experience with it on a server, is there also a benefit when using purely SSD drives?

[1] https://www.usenix.org/legacy/event/usenix99/full_papers/wilson/wilson.pdf

[2] https://events.static.linuxfound.org/sites/events/files/slides/tmc_sjennings_linuxcon2013.pdf

I'm researching for ways to build and run a huge storage server (must be running Linux) where for all data arrays I can run consistency check and fix, while the usual applications using the arrays (reads and writes) keep on working as usual.

Say you have a many-TB of data on a single traditional Linux filesystem (EXT4, XFS) that is used by hundreds of users and suddenly the system reports consistency/corruption problem with it, or you know that the machine went down recently in a dirty way and filesystem corruption is very likely.

Taking the filesystem offline and running the filesystem check can easily take many hours/days of downtime, since neither EXT4 nor XFS can run check & repair while in normal operation; the filesystem needs to be taken offline first.

How to avoid this weakness of EXT4/XFS with Linux? How can I build a large storage server without ever needing to get it offline for hours for maintenance?

I've read a lot about ZFS and its reliability due to its use of data/metadata consistency checks. Is it possible to run consistency check and fix ZFS filesystem without taking it offline? Would some other new filesystem or some other organization of the data on disk be better?

One other option I'm thinking about is to divide the data array into ridiculously many (hundreds) of partitions, each having its own independent filesystem, and fix applications to know to use all those partitions. Then, when the need to check one of them arises, only that one will need to be taken offline. Not a perfect solution, but better than nothing.

Is there a perfect solution to this problem?

I need to connect several servers on various public IPs together to form a virtual encrypted network, where ports and services on the servers are accessible only to those servers in the network and where all communication is authenticated and encrypted.

One obvious solution is VPN made of IPs in one of the private ranges, say 10.1.0.0/16. But this requires assigning each server additional IP address, which I'd like to avoid.

Is it possible to use e.g. OpenVPN or other software to create such network of public IPs, without assigning new IPs from private range?