I have LUKS encrypted drives and ZFS on top of that. I unlock the drives on boot using a script. I made a systemd service for that, but the script take some time and the zfs-import-cache.service starts before my script finished.

• zfs-import-cache.service Journalctl

# journalctl -e -u cryptdisks-unlock.service -u zfs-import-cache.service
Sep 25 15:53:45 server systemd[1]: Starting Unlock LUKS drives...
Sep 25 15:53:45 server boot[1216]: Decrypting evo850m21-crypt ...
Sep 25 15:53:47 server systemd[1]: Starting Import ZFS pools by cache file...
Sep 25 15:53:47 server boot[1216]: Device evo850m21-crypt decrypted.
Sep 25 15:53:47 server boot[1216]: Decrypting evo850m22-crypt ...
Sep 25 15:53:49 server boot[1216]: Device evo850m22-crypt decrypted.
Sep 25 15:53:49 server boot[1216]: Decrypting st30001-crypt ...
Sep 25 15:53:50 server zpool[1390]: cannot import 'data': no such pool or dataset
Sep 25 15:53:51 server zpool[1390]:         Destroy and re-create the pool from
Sep 25 15:53:51 server zpool[1390]:         a backup source.
Sep 25 15:53:51 server systemd[1]: zfs-import-cache.service: Main process exited, code=exited, status=1/FAILURE
Sep 25 15:53:51 server systemd[1]: zfs-import-cache.service: Failed with result 'exit-code'.
Sep 25 15:53:51 server systemd[1]: Failed to start Import ZFS pools by cache file.
Sep 25 15:53:52 server boot[1216]: Device st30001-crypt decrypted.
Sep 25 15:53:52 server boot[1216]: Decrypting st30002-crypt ...
Sep 25 15:53:54 server boot[1216]: Device st30002-crypt decrypted.
Sep 25 15:53:54 server systemd[1]: cryptdisks-unlock.service: Succeeded.
Sep 25 15:53:54 server systemd[1]: Finished Unlock LUKS drives.

cryptdisks-unlock.service

[Unit]
Description=Unlock LUKS drives
Before=zfs-import.target

[Service]
Type=oneshot
ExecStart=/usr/local/cryptdisks-tools/boot

[Install]
WantedBy=zfs-import-cache.service

Running on Ubuntu server, a CentOS 7 VM is eating the memory :

Host:

# free -m
              total        used        free      shared  buff/cache   available
Mem:          80140       71676        5346           7        3117        7655
Swap:          4095         265        3830

# ps_mem
 Private  +   Shared  =  RAM used   Program

 59.7 MiB +  67.4 MiB = 127.1 MiB   fail2ban-server (4)
189.2 MiB + 189.6 MiB = 378.8 MiB   mysqld
202.2 MiB + 206.9 MiB = 409.1 MiB   lxd (5)
128.5 MiB + 558.5 MiB = 687.0 MiB   systemd-journald (5)
 30.1 GiB +  30.3 GiB =  60.4 GiB   qemu-system-x86_64
---------------------------------
                         62.8 GiB
=================================

# virsh dommemstat ispmanager-master
actual 33554432
swap_in 1214268
swap_out 6718144
major_fault 429115
minor_fault 27183560666
unused 569424
available 32779288
usable 19841256
last_update 1597820230
rss 33239928

Guest:

# free -m
              total        used        free      shared  buff/cache   available
Mem:          32011       11235         548         978       20226       19403
Swap:         10235        2568        7667

# lsmod | grep virt
virtio_balloon         18015  0 
virtio_net             28114  0 
net_failover           18147  1 virtio_net
virtio_blk             18323  4 
virtio_console         28076  1 
virtio_pci             22985  0 
virtio_ring            22952  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_console
virtio                 14959  5 virtio_blk,virtio_net,virtio_pci,virtio_balloon,virtio_consol

# virsh dommemstat ispmanager-master
actual 33554432
swap_in 0
swap_out 168
major_fault 24275
minor_fault 796131035
unused 395204
available 32779288
usable 25097908
last_update 1597885620
rss 33535884

The guest config:

<domain type="kvm">
  <name>ispmanager-master</name>
  <uuid>353f2014-53d7-413e-b635-60eb4e78751b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://centos.org/centos/7.0"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">33554432</memory>
  <currentMemory unit="KiB">33554432</currentMemory>
  <vcpu placement="static">10</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-bionic">hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
  </features>
  <cpu mode="custom" match="exact" check="partial">
    <model fallback="allow">Skylake-Server-IBRS</model>
  </cpu>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm-spice</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/zvol/pool1/vm/isp-master"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/zvol/pool1/vm/isp-master-backups"/>
      <target dev="vdb" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/zvol/ssd1/vm/isp-master"/>
      <target dev="vdc" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="hda" bus="ide"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="ich9-ehci1">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x7"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci1">
      <master startport="0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0" multifunction="on"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci2">
      <master startport="2"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x1"/>
    </controller>
    <controller type="usb" index="0" model="ich9-uhci3">
      <master startport="4"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="ide" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </controller>
    <interface type="bridge">
      <mac address="52:54:00:83:c1:a8"/>
      <source bridge="br0"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="2"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
    </graphics>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </video>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </memballoon>
  </devices>
</domain>

What is wrong ? How can I fix it ?

Since the beginning of the month, automatic reboot after kernel change takes ... 3 hours.

It is a Lenovo Thinksystem server in a datacenter.

journalctl -b -1 ends with:

juil. 23 03:02:25 host systemd-shutdown[1]: Sending SIGTERM to remaining processes...
juil. 23 03:02:25 host systemd-journald[658]: Journal stopped

and journalctl -b starts with:

juil. 23 06:16:26 host kernel: microcode: microcode updated early to revision 0x2006906, date = 2020-04-24
juil. 23 06:16:26 host kernel: Linux version 4.15.0-112-generic (buildd@lcy01-amd64-027) (gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)) #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 (U
juil. 23 06:16:26 host kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.0-112-generic root=UUID=7b9c74b1-d80e-457e-957a-32be0fca891e ro

I have a GRUB_TIMEOUT=0 so I have no idea of what could be the issue.

Edit:

One weird thing is that lshw used to show the server model:

product: ThinkSystem SR590 -[7X99CTO1WW]- (7X99CTO1WW)
vendor: Lenovo
version: 04

And now:

    description: Rack Mount Chassis
    product: -[none]- (none)
    vendor: Lenovo
    version: none
    serial: none
    width: 64 bits
    capabilities: smbios-3.2 dmi-3.2 smp vsyscall32
    configuration: boot=normal chassis=rackmount family=ThinkSystem sku=none
  *-core
       description: Motherboard
       product: -[none]-
       vendor: Lenovo
       physical id: 0
       version: none
       serial: none
       slot: none
     *-firmware
          description: BIOS
          vendor: Lenovo
          physical id: 0
          version: -[TEE128O-1.51]-
          date: 10/30/2018
          size: 64KiB
          capacity: 15MiB
          capabilities: pci pnp upgrade shadowing cdboot bootselect edd int14serial acpi usb biosbootspecification uefi
     *-cpu:0
          description: CPU
          product: Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz

Edit 2:

It seems to be stuck on that screen for hours:

I manage three Ubuntu servers running latest LTS, all up 2 date.

The two first are behind a fast fiber networks, with a bridge interface (for containers) Both have an OPNsense router running in libvirt using a dedicated interface via PCI passthrough

The third is hosted by my ISP on a fast network, also with a bridge interface for containers.

ALL THREE servers upload speed is topping at 3 to 4 Mbit/s only! The OPNsense routers can upload at full speed. If I plug a device directly on the modem (WAN), I get the full speed too.

So I suspect that it is related to the network bridges.

Server 1:

/etc/network/interfaces

auto br0
    iface br0 inet static
        address 10.0.4.4
            netmask 255.255.254.0
            network 10.0.4.0
            broadcast 10.0.5.255
            # gateway 10.0.4.1
            dns-nameserver 10.0.4.1
        bridge_ports enp3s0
            bridge_stp on
            bridge_maxwait 0

Interface speed

# ethtool enp3s0 | grep Speed
        Speed: 1000Mb/s

ip a

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 60:a4:4c:cb:4f:5d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::62a4:4cff:fecb:4f5d/64 scope link 
       valid_lft forever preferred_lft forever
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 0a:91:86:76:2e:08 brd ff:ff:ff:ff:ff:ff
    inet 10.0.4.4/23 brd 10.0.5.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::891:86ff:fe76:2e08/64 scope link 
       valid_lft forever preferred_lft forever

Server 2:

/etc/netplan/01-netcfg.yaml

 bridges:
    br0:
      dhcp4: no
      dhcp6: no
      addresses:
        - 10.0.14.2/24
      gateway4: 10.0.14.1
      nameservers:
        addresses: 
        - 10.0.14.6
      interfaces:
        - eno1

Interface speed

# ethtool eno1 | grep Speed
        Speed: 1000Mb/s

Server 3:

/etc/netplan/01-netcfg.yaml

network:
    ethernets:
        eno1:
            dhcp4: false
            dhcp6: false
        eno2:
            dhcp4: false
            dhcp6: false
            optional: true
        enp0s20f0u1u6:
            dhcp4: true
            dhcp6: false
            optional: true
    bridges:
        br0:
            dhcp4: false
            dhcp6: false
            addresses:
            - [redacted]/24
            - [redacted]/64
            gateway4: [redacted]
            gateway6: [redacted]
            nameservers:
                addresses: 
                - 1.1.1.1
                - 1.0.0.1
                - 8.8.8.8
                - 8.8.8.4
                - 2606:4700:4700::1111
                - 2606:4700:4700::1001
                - 2001:4860:4860::8888
                - 2001:4860:4860::8844
            interfaces:
                - eno1
            parameters:
                stp: false
                forward-delay: 0
    version: 2

Interface speed

# ethtool eno1 | grep Speed
        Speed: 1000Mb/s

Typical servers speed tests:

Ping: 2.987 ms
Download: 279.34 Mbit/s
Upload: 4.10 Mbit/s

Speedtest from a computer plugged into the fiber router OR behind the OPNsense routers

Ping: 2.939 ms
Download: 283.25 Mbit/s
Upload: 180.77 Mbit/s

What I am doing wrong ?

This issue is driving me crazy. I run a fresh install of Ubuntu 18.04, with:

• ufw to manage the firewall
• a br0 bridge
• lxd and libvirt (KVM)

I tried stock docker.io package and packages form docker's own deb repository.

I want o be able to deploy docker containers choosing the ip to bind its port (eg. -p 10.58.26.6:98800:98800) and then open the port with UFW.

But docker seems to create iptables rules that pertubates the br0 bridge (eg. host cannot ping libvirt guests)

I have looked all around and cannot find good, security aware solution.

Manually doing iptables -I FORWARD -i br0 -o br0 -j ACCEPTseems to makes everything work.

Also setting "iptables": false for the docker daemon allows the bridge to behave normally, but breaks docker's containers egress network.

I have found this solution that seemed simple, by editing a single UFW's file https://stackoverflow.com/a/51741599/1091772, but it doesn't work at all.

What would be the best practice and secure way of solving this permanently, surviving to reboots ?

EDIT: I ended up adding -A ufw-before-forward -i br0 -o br0 -j ACCEPT at the end of /etc/ufw/before.rules before the COMMIT. Can I consider this as a fix or doesn't it raise some issues ?

I have a pool of IP addresses and a single ethernet cable. I have set aliases:

auto enp2s0:0
iface enp2s0:0 inet static
  address x.x.x.x
  netmask x.x.x.x

I want to dedicate some of the aliases to a libvirt kvm guest, so that the ip is set from inside the guest.

How can I achieve it ?

I am new to exim4. I have this configuration:

.ifdef SA_ENABLE
        warn
            !authenticated = *
            hosts = !127.0.0.1/24
            condition = ${if < {$message_size}{1K}}
            spam       = SA_SPAMD_USER:true
            add_header = X-Spam_score: $spam_score\n\
                   X-Spam_score_int: $spam_score_int\n\
                   X-Spam_bar: $spam_bar\n\
                   X-Spam_report: $spam_report

        warn
            !authenticated = *
            hosts = !+relay_from_hosts
            spam       = SA_SPAMD_USER:true/defer_ok
            add_header = X-Spam_score: $spam_score\n\
                    X-Spam_score_int: $spam_score_int\n\
                    X-Spam_bar: $spam_bar\n\
                    X-Spam_report: $spam_report
            set acl_m4 = $spam_score_int
            condition = ${if and{{<{$message_size}{100K}}{<{$acl_m4}{SA_SCORE_REJECT}}} {yes}{no}}
            logwrite = From $sender_address to $recipients X-Spam_score: $acl_m4.

        deny
            condition = ${if and{{>{$message_size}{10K}}{!eq{$acl_m4}{}}{>{$acl_m4}{SA_SCORE_REJECT}}} {yes}{no}}
            message = Content analisis tool detect spam (from $sender_address to $recipients). Contact SA_ABUSE_ADDR.
.endif

How can I tweak for adding an X-Spam-Flag (yes or no) if the score is over/under a value, and reject if over an other value. (Currently SA_SCORE_REJECT = 50)

I wrote a small script to handle my zfs backups. When i run it by hand, it works well:

/root/bin/zfs-backup -k 7 -p zfs-auto-snap_daily -r vm/containers tank/backups

But running it in cron like so, throws an error "The file system 'vm/containers' doesn't exists."

30 0 * * * root /root/bin/zfs-backup -k 7 -p zfs-auto-snap_daily -r vm/containers tank/backups

Sometimes my Apache server get high load. I have a lot of VirtualHost, each one having its own log file.

The structure is like this /var/www/httpd-logs/virtualhost.com.access.log

What command could I use to find the website that got a high load in the past X minutes ?

I'll then analyze the corresponding website logs to find the issue.

I have two raid1 arrays and an LVM partition across both raid arrays.

A disk died and I tried to copy back the partition table of the other member to it using sgdisk -R

But I mixed the arguments, so after a reboot, the good disk seems as unusable as the bad one.

Can I fix it? I mean it was running well before I rebooted ...