sastorsl's questions

In one of our environments Linux servers are set up with sssd / OpenLDAP for OS login. To support older servers our OpenLDAP server has to support TLSv1.0 and TLSv1.1 still.

RedHat 8 does no longer support TLS levels below TLSv1.2, and thus the standardized /etc/sssd/sssd.conf failed to connect to the LDAP server.

Error message:

sssd_be[1236697]: Could not start TLS encryption. error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

It seems(?) that the ldap protcol - or the server - prefers the weaker TLS protocols first, and thus connecting from RHEL8 fails.

Of course the LDAP server has to remove support for older protocols, but how can the client side be forced to use TLSv1.2.

Is it possible to somehow add arbitrary HTTP response headers to the "stats" pages in HAProxy? haproxy version 1.7.x.

Regular stats setup:

listen view
    bind *:10002
    stats enable
    stats uri /
    stats hide-version

Customizing and trying to add headers makes no difference.

This does not work, i.e. no X-Frame-Options header is added to the response:

listen view
    bind *:10002
    rspadd X-Frame-Options:\ SAMEORIGIN
    stats enable
    stats uri /
    stats hide-version

Nor does this work:

listen view
    bind *:10002
    http-response set-header X-Frame-Options SAMEORIGIN
    stats enable
    stats uri /
    stats hide-version

The exact same parameters in a "regular" listen block works like a charm.

We have defined a "[email protected]" file which handles our application server starts. On some servers there are a lot of application servers and we hit issues where one or two servers won't start.

This is a RedHat Enterprise Linux 7 server.

Is there a way to limit the number of concurrent / simultaneous processes for this one service?

The service file:

[Unit]
Description=Appserver Instance %i
After=network.target

[Service]
Type=forking
ExecStart=/custom/server start %i
ExecStop=/custom/server stop %i
PIDFile=/custom/servers/.pid/%i.pid
User=%i
SuccessExitStatus=1
TimeoutStartSec=600
TimeoutStopSec=60

[Install]
WantedBy=multi-user.target

Target is that all servers should start, but at boot time I don't want all of them to start at the same time. They should all be able to run at the same time.

Background

RedHat 7 (and 6) servers with multiple application servers (shared binaries).

We set up separate user accounts when adding applications servers with $HOME = appserver_root. The installation however demands that the target directory does not exist.

So we:

• Create the user without creating the home directory
• Install the server into (and creating) the home directory
• Reset the permissions of the home directory

This process will not set up the skeleton files, i.e. copy /etc/skel/

Question

We can simply copy /etc/skel/.*, but is there a tooled way - command - which will do this?

I'd prefer using a standard tool instead of making a guess, or parsing /etc/default/useradd for the correct SKEL path.

We have a filesystem, on LVM, created as ext2 - and in daily use running 8-10 webservers (software, tempfiles, logfiles). The discs are "local" to the OS, but are really SAN disks provided to the VmWare ESX server. Our "server" is a virtual server running RedHat 6.

The disk size is 56GB, about 26GB usage (varies). The log files are the most active, and ranges from some 10kb up to 2GB. Lifetime writes (since august 2012) are about 23 TB (yes, Tera) according to tune2fs.

With regards to the filesystem (not the process) - What would be the pros and cons with upgrading from ext2 to ext4 vs creating av new filesystem as ext4, copying and switching to the new disk?

We will go to ext4, and will probably create a new fs and copy - since disk is not an issue, but will an upgrade give us the same - and will files and directories created before the upgrade be changed?

If there will be a difference between the old and the new files: What will de difference be?

An ext2 filesystem can be resized - as long as the underlying block device can be resized - but not ONLINE.

However, as another posting of mine has showed, if the ext2 filesystem is mounted as ext4 then resize2fs will not give an errormessage.

System information:

• Red Hat Enterprise Linux Server release 6.4 (Santiago)
• e2fsprogs-1.41.12-14.el6_4.2.x86_64

Recreate (example with an LVM block device ("disk"):

Create a block device and an ext2 filesystem (default for mke2fs in RedHat 6)

[root@myServer ~]# lvcreate -L 8M -n test3 vg_myServer
  Logical volume "test3" created
[root@myServer ~]# mke2fs /dev/mapper/vg_myServer-test3
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=4 blocks, Stripe width=0 blocks
2048 inodes, 8192 blocks
409 blocks (4.99%) reserved for the super user
First data block=1
Maximum filesystem blocks=8388608
1 block group
8192 blocks per group, 8192 fragments per group
2048 inodes per group

Writing inode tables: done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 38 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

    # verify by checking the block ID
[root@myServer ~]# blkid /dev/mapper/vg_myServer-test3 | grep ext
/dev/mapper/vg_myServer-test3: UUID="97a46d2c-e157-46ec-a942-2df0d07eb39f" TYPE="ext2"

Mount the filesystem

    # Create a mount point and mount the file system
[root@myServer ~]# mkdir -p /mnt/test3
[root@myServer ~]# mount -t ext4 /dev/mapper/vg_myServer-test3 /mnt/test3/

    # See that mount percieves the filesystem as "ext4"
[root@myServer ~]# mount | grep test3
/dev/mapper/vg_myServer-test3 on /mnt/test3 type ext4 (rw)

Fill the filesystem to capacity

[root@myServer ~]# dd if=/dev/zero of=/mnt/test3/largefile bs=1024 count=8000
dd: writing `/mnt/test3/largefile': No space left on device
7849+0 records in
7848+0 records out
8036352 bytes (8.0 MB) copied, 2.22541 s, 3.6 MB/s

    # Verify
[root@myServer ~]# df -P /dev/mapper/vg_myServer-test3
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/vg_myServer-test3     7931     7926      0      100% /mnt/test3

Extend the block device / LVM, and resize

[root@myServer ~]# lvextend -L +8M /dev/mapper/vg_myServer-test3
  Extending logical volume test3 to 16.00 MiB
  Logical volume test3 successfully resized

    # Perform a resize, _without_ umount
[root@myServer ~]# resize2fs /dev/mapper/vg_myServer-test3
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_myServer-test3 is mounted on /mnt/test3; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
Performing an on-line resize of /dev/mapper/vg_myServer-test3 to 16384 (1k) blocks.
The filesystem on /dev/mapper/vg_myServer-test3 is now 16384 blocks long.

Verify - the filesystem thinks the filesystem is extended

[root@myServer ~]# df /dev/mapper/vg_myServer-test3
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg_myServer-test3   15863      7957      7089  53% /mnt/test3

Create a second "large" file

[root@myServer ~]# dd if=/dev/zero of=/mnt/test3/largefile2 bs=1024 count=8000
dd: writing `/mnt/test3/largefile2': No space left on device
5230+0 records in
5229+0 records out
5354496 bytes (5.4 MB) copied, 0.280078 s, 19.1 MB/s

    # Verify that the OS sees the FS as 88% full - but unable to write any more
[root@myServer ~]# df -P /dev/mapper/vg_myServer-test3
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/vg_myServer-test3     15863     13227      1819      88% /mnt/test3

umount/mount and do an fsck on the filesystem

[root@myServer ~]# umount /dev/mapper/vg_myServer-test3
[root@myServer ~]# mount -t ext4 /dev/mapper/vg_myServer-test3 /mnt/test3/

    # FS suddenly has less usage...
[root@myServer ~]# df -P /dev/mapper/vg_myServer-test3
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/vg_myServer-test3     15863      7963      7083      53% /mnt/test3

    # Perfor an fsck after an umount, then mount again
[root@myServer ~]# umount /dev/mapper/vg_myServer-test3
[root@myServer ~]# e2fsck -f /dev/mapper/vg_myServer-test3
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/mapper/vg_myServer-test3: 13/4096 files (15.4% non-contiguous), 8484/16384 blocks
[root@myServer ~]# mount -t ext4 /dev/mapper/vg_myServer-test3 /mnt/test3/

What size do we have now?

    # Same as before the fsck
[root@myServer ~]# df -P /dev/mapper/vg_myServer-test3
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/vg_myServer-test3     15863      7963      7083      53% /mnt/test3

    # A bit strange, the 8MB file and an 5MB file...?
[root@myServer ~]# ls -ld /mnt/test3/large*
-rw-r--r--. 1 root root 8036352 Jun 18 09:54 /mnt/test3/largefile
-rw-r--r--. 1 root root 5354496 Jun 18 09:55 /mnt/test3/largefile2

Try to create a second large file

    # Fill up again
[root@myServer ~]# dd if=/dev/zero of=/mnt/test3/largefile2 bs=1024 count=8000
dd: writing `/mnt/test3/largefile2': No space left on device
7870+0 records in
7869+0 records out
8057856 bytes (8.1 MB) copied, 2.43879 s, 3.3 MB/s

    # "largefile2" has increased in size - OK
[root@myServer ~]# ls -ld /mnt/test3/large*
-rw-r--r--. 1 root root 8036352 Jun 18 09:54 /mnt/test3/largefile
-rw-r--r--. 1 root root 8057856 Jun 18 10:12 /mnt/test3/largefile2

    # We are again at capacity
[root@myServer ~]# df -P /dev/mapper/vg_myServer-test3
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/vg_myServer-test3     15863     15858         0     100% /mnt/test3

And the FS is (of course) still an ext2 filesystem

    # Still an "ext2" filesystem
[root@myServer ~]# blkid /dev/mapper/vg_myServer-test3
/dev/mapper/vg_myServer-test3: UUID="97a46d2c-e157-46ec-a942-2df0d07eb39f" TYPE="ext2"

    # But mounted as "ext4"
[root@myServer ~]# mount | grep /dev/mapper/vg_myServer-test3
/dev/mapper/vg_myServer-test3 on /mnt/test3 type ext4 (rw)

If the filesystem is mounted as ext2 you get an error message (correctly) when trying to resize

[root@myServer ~]# umount /dev/mapper/vg_myServer-test3
[root@myServer ~]# mount -t ext2 /dev/mapper/vg_myServer-test3 /mnt/test3/
[root@myServer ~]# lvextend -L +8M /dev/mapper/vg_myServer-test3
  Extending logical volume test3 to 24.00 MiB
  Logical volume test3 successfully resized
[root@myServer ~]# resize2fs /dev/mapper/vg_myServer-test3
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_myServer-test3 is mounted on /mnt/test3; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 1
resize2fs: Kernel does not support online resizing

I know that the ext4 filesystem driver can mount ext2 - which is fine if you are on a system without the ext2 driver.

Question: Is this a bug or a feature - meaning that resize2fs does not give an error message? If so, is it in resize2fs?

On a RedHat 6 server, we ran into an issue with online resizing of an ext4 filesystem.

With only /dev/sda we had 13GB available in the volume group, but needed 20GB more on one logical volume which was 36GB. Added /dev/sdb to the volume group, and the file system was extended (lvextend) and resized (resize2fs) to 56GB. No error messages during the resize, and the OS reported the new size.

The logical volume in question hosts an installation of IBM HTTP Server (apache 2.2), config and log files for some 8 different web servers.

This morning the file system usage grew beyond 36GB. What happened first was that the webservers stopped logging (discovered after), while the web servers kept on running without issues. 2,5 hours later, in relation to log rotation and some other writes to the file system things started to freeze up. Meaning: the webservers stopped taking traffic, allthough the processes stayed up, trying to "tail" a log file would hang, and could not be interupted. The load of the server went from 0.10 to 4000 (yes...) - mostly related to iowait (it would seem).

The sollution was to shut down the webserver - kill -9 was the only way, and reboot the server. Umount the filesystem, did an fsck (no errors), and start things up again. No issues since.

We can excactly time the error with logging stopping to the time the disk (lv) usage grew above it's previous size of 36GB.

Services on other file systems seemed to work fine - amongs others the operating system.

In /var/log/messages we saw i.e.:

kernel: INFO: task httpd:<pid> blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: httpd         D 0000000000000001     0  6889   6865 0x00000080
kernel: ffff88023aa99c88 0000000000000086 0000000000000000 0000000000006102
kernel: ffff88010aebaa80 ffff880105dd0ae0 000000003aa99c08 ffff880105dd0ae0
kernel: ffff880105dd1098 ffff88023aa99fd8 000000000000fb88 ffff880105dd1098
kernel: Call Trace:
kernel: [<ffffffff8150efbe>] __mutex_lock_slowpath+0x13e/0x180
kernel: [<ffffffff8150ee5b>] mutex_lock+0x2b/0x50
kernel: [<ffffffff8111c461>] generic_file_aio_write+0x71/0x100
kernel: [<ffffffffa0097fb1>] ext4_file_write+0x61/0x1e0 [ext4]
kernel: [<ffffffff81180d7a>] do_sync_write+0xfa/0x140
kernel: [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff8121bc06>] ? security_file_permission+0x16/0x20
kernel: [<ffffffff81181078>] vfs_write+0xb8/0x1a0
kernel: [<ffffffff81181971>] sys_write+0x51/0x90
kernel: [<ffffffff810dc645>] ? __audit_syscall_exit+0x265/0x290
kernel: [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b

Versions:

Kernel: 2.6.32-358.2.1.el6.x86_64
lvm2-2.02.98-9.el6.x86_64
e2fsprogs-1.41.12-14.el6.x86_64

There were found no issues with the underlying hardware.