xogoxec344's questions

I have a remote linux system that became super slow yesterday. Since the remote luks unlocking I've setup doesn't seem to work reliable and I won't be able to physically access the machine within the next 10 days I'm trying to debug this instead of rebooting.

The system status tools I'm used to are htop and dstat and since I had dstat running in an ssh session I can see that since yesterday 2021-09-09 08:51:42 one cpu core is always fully used by "sys" - which I guess means the kernel?

I can't see any culprit process or thread in htop.

I've stopped all user services and unmounted everything non essential which made the system respond a bit better again but still not nearly as fast as it should (got an Intel i7 CPU with an SSD).

I've found https://tanelpoder.com/posts/high-system-load-low-cpu-utilization-on-linux/ and installed the referenced https://0x.tools/ to get this result for psn -G syscall,wchan:

=== Active Threads ========================================================================================

 samples | avg_threads | comm              | state                  | syscall   | wchan                    
-----------------------------------------------------------------------------------------------------------
     100 |        1.00 | (btrfs-cleaner)   | Running (ON CPU)       | [running] | 0                        
     100 |        1.00 | (dpkg)            | Disk (Uninterruptible) | fsync     | btrfs_commit_transaction 
     100 |        1.00 | (systemd-journal) | Disk (Uninterruptible) | ftruncate | wait_current_trans       
       1 |        0.01 | (sshd)            | Running (ON CPU)       | [running] | 0                        
       1 |        0.01 | (thermald)        | Disk (Uninterruptible) | [running] | ec_guard                 
       1 |        0.01 | (thermald)        | Running (ON CPU)       | [running] | 0    

The dpkg process can be explained by me trying to run apt upgrade which run's around at a 1/1000th of the speed you'd normally expect (just a feeling, didn't measure it).

Maybe there's a problem with my btrfs root file system...? I can't find the btrfs-cleaner in htop, I guess I'm gonna research some more on what that is..

I did run a btrfs scrub last night which completed super fast and didn't find any problems:

# btrfs scrub status /
UUID:             2f38e0ad-7f16-4a36-8096-b7981d47b4ff
Scrub started:    Thu Sep  9 23:59:00 2021
Status:           finished
Duration:         0:00:24
Total to scrub:   53.09GiB
Rate:             1.78GiB/s
Error summary:    no errors found

But when I used nano to modify a config file on the root partition loading and saving it was super slow just now.

I just stumbled upon this: https://www.reddit.com/r/btrfs/comments/fmucrq/btrfs_snapshots_make_entire_system_lag_cpu_usage/ which has a comment that sounds similar to my problem:

every time on boot and after a snapshot btrfs-transacti and btrfs-cleaner would use up a core completely causing immense lag

only that this says it just lasts a few minutes on boot and snapshot creation, but I've disabled my btrbk backup setup on this system a few days ago when one of the attached disks started to show problems.

I'm not sure if my btrfs root filesystem was using qgroups, but I just ran btrfs quota disable / which took around 10 seconds and didn't give any feedback.

Anybody got any other hint's for me how to debug / solve this problem?

I'm trying to use tcpdump to explore what on my computer talks with whom on the world wide web.

I've came this far as of yet:

lan_hosts="(hosts || to || exclude)"
local_hosts="(127.0.0.1 || ips_of_my_nics || localhost || local_hostname)"
excused_local_ports="(ssh || https || domain || $(netstat -ap | egrep -h '/(processes|I_dont|want|to_see|traffic_of)' | tr -s ' ' | cut -d ' ' -f 4 | rev | cut -d ':' -sf 1 | rev | sort | sed ':a; N; $!ba; s/\n/ \|\| /g'))"
tcpdump -vi any "ip && ! icmp && ! arp && (src ! $local_hosts || dst ! $local_hosts) && (src $local_hosts || dst $local_hosts) && (src ! $local_hosts || src port ! $excused_local_ports) && (dst ! $local_hosts || dst port ! $excused_local_ports) && host ! $lan_hosts"

Now the reason I'm posting this here as question: I don't understand why / how these packets still show up in the output:

tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
14:49:30.660109 IP (tos 0x0, ttl 64, id 23067, offset 0, flags [DF], proto UDP (17), length 71)
    localhost.38976 > localhost.domain: 26002+ [1au] A? lan_host.lan. (43)
14:49:30.686174 IP (tos 0x0, ttl 64, id 23110, offset 0, flags [DF], proto UDP (17), length 80)
    localhost.47181 > localhost.domain: 45895+ [1au] PTR? some_ip.in-addr.arpa. (52)
14:49:30.686219 IP (tos 0x0, ttl 64, id 2440, offset 0, flags [DF], proto UDP (17), length 103)
    localhost.domain > localhost.47181: 45895 1/0/1 some_ip.in-addr.arpa. PTR localhost. (75)

I thought that the segment (src ! $local_hosts || dst ! $local_hosts) of my filter declaration, which is only concatenated with && logic to other filter parts should exclude those? And even if I didn't have that part in the filter, they should again be excluded because of the inclusion of domain in my excused_local_ports variable.

Let's unpack the filter declaration a bit and try to explain what I wanted to achieve with each part:

• ip -> only look at ipv4 packets, since this machine doesn't have an ipv6 path to the internet I am not concerned with ipv6 here.
• ! icmp -> don't care about ping requests / replies and other icmp routing metadata.
• ! arp -> filter out adress resolution protocol packets
• (src ! $local_hosts || dst ! $local_hosts) -> hide packets of the localhost talking to itself
• (src $local_hosts || dst $local_hosts) -> hide broadcasts from other hosts
• (src ! $local_hosts || src port ! $excused_local_ports) -> hide packets my host has sent from one of the ports of the programs I don't want to look at right now.
• (dst ! $local_hosts || dst port ! $excused_local_ports) -> hide packets my host has received on one of the ports of the programs I don't want to look at right now.
• (host ! $lan_hosts) -> hide communication with the defined hosts