Mikko Rantalainen's questions

I have a system with 3 SSD devices (/dev/sda, /dev/sdb, /dev/sdc) which contain a single LVM logical volume that span over all the devices. I have a single ext4 partition on the logical volume.

I think that one of the SSD devices (/dev/sdb) might be somewhat faulty and have reduced performance compared to other devices.

Is there a command to get the list of files that are backed by that device?

I know that I can get list of logical segments with sudo pvdisplay -m and the output looks like following:

  --- Physical volume ---
  PV Name               /dev/sda
  VG Name               storage
  PV Size               <1,82 TiB / not usable <1,09 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              476932
  Free PE               0
  Allocated PE          476932
  PV UUID               h3x3O1-1KWj-3pY6-kZ24-MVV4-54UE-ltEdfA
   
  --- Physical Segments ---
  Physical extent 0 to 476931:
    Logical volume  /dev/storage/vm
    Logical extents 0 to 476931
   
  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               storage
  PV Size               <3,64 TiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              953861
  Free PE               0
  Allocated PE          953861
  PV UUID               MsNlhh-W2It-CbX4-IxJn-lXJN-hlcd-EpBh9Q
   
  --- Physical Segments ---
  Physical extent 0 to 953860:
    Logical volume  /dev/storage/vm
    Logical extents 476932 to 1430792
   
  --- Physical volume ---
  PV Name               /dev/sdc
  VG Name               storage
  PV Size               <3,64 TiB / not usable <3,84 MiB
  Allocatable           yes (but full)
  PE Size               4,00 MiB
  Total PE              953861
  Free PE               0
  Allocated PE          953861
  PV UUID               sklK6w-XZd6-DqIp-ZT1g-O9rj-1ufw-UaC0z4
   
  --- Physical Segments ---
  Physical extent 0 to 953860:
    Logical volume  /dev/storage/vm
    Logical extents 1430793 to 2384653
   

So I know that logical extends 476932 to 1430792 are the potentially problematic area. How to map this logical segment range to actual files on the filesystem (ext4) on top of the LVM?

Basically I'm trying to figure out if the device is actually faulty or if the usage pattern for these files might be unlucky enough that I'm hitting usage pattern that's problematic for the hardware and the performance is worse than expected. No device is showing any errors and all the data looks good, but the performance of this single device appears to be worse than expected.

The system is in use so I'd prefer to diagnose this online without overwriting any data. I know that if I could simply take the potentially problematic storage device offline and overwrite its contents, I could use fio to benchmark it to see if it's working below spec or not.

$ lsblk -s
...
storage-vm 253:0    0   9,1T  0 lvm  /mnt/storage
├─sda        8:0    0   1,8T  0 disk 
├─sdb        8:16   0   3,7T  0 disk 
└─sdc        8:32   0   3,7T  0 disk 

I'm basically asking how to get list of files backed by a single storage device when the filesystem spans over multiple storage devices.

Or if you can provide instructions how to figure where a given file is actually stored, that would be fine, too. I would then run that routine for every file to figure out which files are backed by the device I'm interested in. I'm aware that it could be that a single big file is backed by all devices if the file is fragmented over a big range of locigal segments so the answer could be that a single file is backed by all the devices but I have currently no idea how to do this either.

$ sudo vgdisplay 
  --- Volume group ---
  VG Name               storage
  System ID             
  Format                lvm2
  Metadata Areas        3
  Metadata Sequence No  6
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                3
  Act PV                3
  VG Size               <9,10 TiB
  PE Size               4,00 MiB
  Total PE              2384654
  Alloc PE / Size       2384654 / <9,10 TiB
  Free  PE / Size       0 / 0   
  VG UUID               MOrTMY-5Dly-48uQ-9Fa8-JNvf-tont-9in7ol

$ sudo lvdisplay 
  --- Logical volume ---
  LV Path                /dev/storage/vm
  LV Name                vm
  VG Name                storage
  LV UUID                RDkaLH-mh6C-cXxT-6ojc-DxkB-o4jD-3CMHdl
  LV Write Access        read/write
  LV Creation host, time staging, 2021-01-21 09:57:06 +0200
  LV Status              available
  # open                 1
  LV Size                <9,10 TiB
  Current LE             2384654
  Segments               3
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0

I have a software listening on port 1234 that I would like to keep hidden from outside world. I know that I can say e.g.

sudo iptables -A INPUT ! -s 1.2.3.4/32 -p tcp -m tcp --dport 9000 -j 1234 --reject-with icmp-port-unreachable

to make the port 1234 accessible only for the remote host 1.2.3.4.

However, other hosts can still run nmap -sS myhost, nmap -sW myhost or some other advanced test to see output such as

PORT     STATE    SERVICE
1234/tcp filtered someservice-guessed-from-port-number

For basic nmap scan (as normal user instead of root) the port 1234 appears to be closed.

Note that this output is identical regardless if any software is actually running on port 1234 or not.

Is it possible to use iptables to make the port 1234 appear totally identical to normal closed port for all hosts except the selected ones? I know that I could modify all ports to appear as filtered but that seems like a workaround instead of true solution.

We have a web service behind a HAProxy server running in caching reverse proxy configuration. The backend servers send Cache-Control headers correctly for all responses so HAProxy can cache all responses according to HTTP spec.

However, when the end user hits the Shift+Reload button in e.g. Google Chrome, the client (Chrome) sends Pragma: no-cache and Cache-Control: no-cache which forces HAProxy to always fetch the request from the backend server. Obviously, DDoS attacks can use this same trick to easily cause more load on the backend servers.

As we know that the cache headers are correct, how can we configure HAProxy to ignore client submitted Pragma: no-cache and avoid calling backend when the request could be directly fulfilled from HAProxy cache?

I know that ignoring this header would not be okay for a generic proxy use, but in this case we control both the reverse proxy and the backend so we know this is fine.

Here's an example of a response from backend server that will be re-done from the backend when the client sends cache-control: no-cache and pragma: no-cache:

cache-control: public, max-age=31536000, s-maxage=31536000
content-length: 463
content-type: image/svg+xml
date: Thu, 24 Jun 2021 14:14:19 GMT
etag: "338"
expires: Fri, 24 Jun 2022 14:14:19 GMT
server: Apache
x-content-type-options: nosniff

It's obviously totally pointless to fetch this from backend servers again because its valid for one year for any user using the given URL. Also worth noting is that NGINX does not honor the [client] Pragma header by default.

We have a setup where front facing HTTP2 server is HAProxy and multiple backend servers run Apache+PHP. We have set HAProxy configuration option http-buffer-request. End users upload files (using basic file upload using <form> with method POST) and small files are handled correctly. However, when end users upload e.g. 50 MB file, it seems that the HAProxy is not buffering the uploaded file before connecting the backend server.

We have a pretty short timeout for the backend server because the expected use case is to just transfer the file from HAProxy to backend server using 1 Gbps connection in local network. The HAProxy->backend connection is currently limited to 35 seconds (timeout server 35s). However, when end user upload is big and his or her upload bandwidth is low, the upload fails unless the upload completes within timeout server limit. In case of Google Chrome, the browser automatically retries and submits the file another time. And third time, before finally displaying error ERR_HTTP2_SERVER_REFUSED_STREAM.

If I've understood correctly, HAProxy buffering reverse proxy buffering is configured with option tune.buffers.limit which defaults to zero meaning unlimited. Is there some other limit that prevents big uploads from buffering at the HAProxy level? The HAProxy server is only running HAProxy and does TLS termination and has 16 GB of RAM for this so I'm okay with buffering in RAM if buffering uploads to disk is not supported. Maybe limiting buffering up to 14 GB would be good, though.

We want to allow uploads up to 250 MB and preferably the whole upload should be buffered by HAProxy before HAProxy even connects to Apache to process the upload. In practice, HAProxy should just accept incoming HTTP2 connection, read all the request headers and whole body up to e.g. 260 MB and then connect to backend server. Currently, it seems that it connects to backend server much sooner and then spends timeout server to wait bytes from the end user and connection is terminated by HAProxy.

I'm re-building a server that needs to do awfully lot of I/O to complete the task. Is there a way to tell Linux to temporarily ignore all sync(), syncfs() and fdatasync() from any processes? I think the rebuilding would complete faster if I never wait for storage to flush caches until once when I finally unmount the storage.

I'm hoping something like

mount -o remount,nosync /path/to/mount/point

but obviously nosync does not exist (only sync and async).

I know about some hacks with LD_LIBRARY_PATH to override those syscalls but is there anything system wide that I can activate when processes have already been started?

I understand that using such a feature would force me to restart from the start if power is interrupted or the system crashes. I would happily take my changes during the rebuild process.

I'm running a Linux workstation without swap and I have installed earlyoom daemon to automatically kill some processes if I'm running out of RAM. The earlyoom works by monitoring kernel MemAvailable value and if the available memory gets low enough, it kills less important processes.

This has worked fine for a long time but suddenly I'm now running into situation where MemAvailable is suddenly really low compared to the rest of the system. For example:

$ grep -E '^(MemTotal|MemFree|MemAvailable|Buffers|Cached):' /proc/meminfo 
MemTotal:       32362500 kB
MemFree:         5983300 kB
MemAvailable:    2141000 kB
Buffers:          665208 kB
Cached:          4228632 kB

Note how MemAvailable is much lower than MemFree+Buffers+Cached.

Are there any tools I can run to further investigate why this happens? I feel that the system performance is a bit worse than normally and I had to stop the earlyoom service because its logic will not work unless MemAvailable is stable (that is, it correctly describes the available memory to user mode processes).

According to https://superuser.com/a/980821/100154 MemAvailable is an estimate of how much memory is available for starting new applications, without swapping. As I have no swap, what is this supposed to mean? Is this supposed to mean the amount of memory a new process can acquire before OOM Killer is triggered (because that would logically hit "the swap is full" situation)?

I had assumed that MemAvailable >= MemFree would be always true. Not here.

Additional info:

Searching around the internet suggests that the cause may be open files that are not backed by the filesystem and as a result, cannot be freed from the memory. The command sudo lsof | wc -l outputs 653100 so I definitely cannot manually go through that list.

The top of the sudo slabtop says

 Active / Total Objects (% used)    : 10323895 / 10898372 (94.7%)
 Active / Total Slabs (% used)      : 404046 / 404046 (100.0%)
 Active / Total Caches (% used)     : 104 / 136 (76.5%)
 Active / Total Size (% used)       : 6213407.66K / 6293208.07K (98.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.58K / 23.88K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
4593690 4593656  99%    1.06K 153123       30   4899936K ext4_inode_cache
3833235 3828157  99%    0.19K 182535       21    730140K dentry
860224 551785  64%    0.06K  13441       64     53764K kmalloc-64
515688 510872  99%    0.66K  21487       24    343792K proc_inode_cache
168140 123577  73%    0.20K   8407       20     33628K vm_area_struct
136832 108023  78%    0.06K   2138       64      8552K pid
...

which looks normal to me.

Creating a rough summary of lsof

$ sudo lsof | awk '{ print $2 }' | sort | uniq -c | sort -h | tail
   6516 1118
   7194 2603
   7884 18727
   8673 19951
  25193 28026
  29637 31798
  38631 15482
  41067 3684
  46800 3626
  75744 17776

points to me PID 17776 which is a VirtualBox instance. (Other processes with lots of open files are Chrome, Opera and Thunderbird.) So I wouldn't be overly surprised to later figure out that the major cause of this problem is VirtualBox because that's the only thing that really messes with the kernel.

However, the problem does not go away even if I shutdown virtualbox and kill Chrome, Opera and Thunderbird.

Given a server with Intel E3-1275 v6 CPU, is it possible to instruct Linux kernel to stop using the integrated graphics and restore video RAM back to OS? I'd prefer using a kernel command line flag and disable the integrated graphics during linux boot process so that I can still use graphics adapter for debugging if needed (that is, graphics adapter were still available for grub). The server is otherwise headless and does not require a display for normal use. I'm okay with a solution that requires full power cycle to restore ability to enable the graphics adapter. I'm running Ubuntu 16.04 LTS 64 bit with HWE stack in case it makes a difference (roughly equivalent to vanilla linux 4.8.17).