rlpowell's questions

I have several servers with the same basic setup: primary drive with VFAT /boot/efi, XFS /boot, and XFS / on LVM; and secondary drive with exactly the same thing, each partition rsynced nightly.

I also have a script that makes sure (theoretically) that grub and EFI and so on are set up so that when the server is rebooted, it boots from the primary drive. Normally this works, but on this particular server, it seems to always prefer the secondary drive, even when I swap drives. That is: it seems to consistently boot the drive in the second slot (although it's worth noting that since I cabled the drives myself, it's entirely possible that the second slot actually enumerates first on the motherboard, but I can't see why that would even matter).

Here's the LVM situation:

rlpowell@stodi> sudo pvs
  PV         VG                   Fmt  Attr PSize   PFree
  /dev/sda3  stodi_orange_2021_10 lvm2 a--  930.55g    0
  /dev/sdb3  stodi_pink_2020_10   lvm2 a--  930.55g    0
rlpowell@stodi> sudo lvs
  LV   VG                   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root stodi_orange_2021_10 -wi-ao---- 930.55g
  root stodi_pink_2020_10   -wi-ao---- 930.55g

(On this host, orange is currently primary and pink is currently secondary.)

When it's up and running we have:

/dev/mapper/stodi_orange_2021_10-root  931G  686G  245G  74% /
/dev/sda2                              483M  256M  228M  53% /boot
/dev/sda1                              488M  6.4M  482M   2% /boot/efi

rlpowell@stodi> cat /proc/partitions
major minor  #blocks  name

   8        0  976762584 sda
   8        1     499712 sda1
   8        2     499712 sda2
   8        3  975762119 sda3
  11        0    1048575 sr0
 253        0  975757312 dm-0
   8       16  976762584 sdb
   8       17     499712 sdb1
   8       18     499712 sdb2
   8       19  975762119 sdb3
 253        1  975757312 dm-1

Here's the EFI situation:

rlpowell@stodi> sudo efibootmgr -v
BootCurrent: 0013
Timeout: 30 seconds
BootOrder: 0001,0015,000C,0000,0013,0002,0003,0004,0005,0006,0007,0008,0009,000A,000B,000C,000D,000E,000F,0010,0011,0012,0013,0014,0016
Boot0000  Startup Menu  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)....ISPH
Boot00000013* Samsung SSD 870 EVO 1TB   PciRoot(0x0)/Pci(0x11,0x4)/Sata(0,0,0)N.....YM....R,Y.....ISPH
Boot0001* stodi_orange_2021_10  HD(1,GPT,cb8227c2-dc52-fc4f-bd8f-b46f71104428,0x800,0xf4000)/File(\EFI\fedora\shimx64.efi)
Boot0002  Bios Setup    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0003  3rd Party Option ROM Management       FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0004  System Diagnostics    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0005  System Diagnostics    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0006  System Diagnostics    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0007  System Diagnostics    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0008  Boot Menu     FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0009  USB:          BBS(65535,,0x0)/PciRoot(0x0)/Pci(0x1d,0x0)......ISPH
Boot000A  Network Boot  FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot000B* Samsung SSD 870 EVO 1TB       BBS(HD,Harddisk1,0x400)/PciRoot(0x0)/Pci(0x11,0x4)......ISPH
Boot000C* IBA GE Slot 00C8 v1550        BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x19,0x0)......ISPH
Boot000D* IBA GE Slot 0500 v1550        BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)......ISPH
Boot000E* IBA GE Slot 0500 v1550        BBS(Network,Network1,0x0)/PciRoot(0x0)/Pci(0x1c,0x0)/Pci(0x0,0x0)......ISPH
Boot000F  USB:          PciRoot(0x0)/Pci(0x1d,0x0)N.....YM....R,Y.....ISPH
Boot0010* hp DVDRW DU8A6SH      PciRoot(0x0)/Pci(0x1f,0x2)/Sata(0,0,0)N.....YM....R,Y.....ISPH
Boot0011* hp DVDRW DU8A6SH      BBS(CDROM,CDROM1,0x400)/PciRoot(0x0)/Pci(0x1f,0x2)......ISPH
Boot0012  System Information    FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0013* Fedora        HD(1,GPT,cb8227c2-dc52-fc4f-bd8f-b46f71104428,0x800,0xf4000)/File(\EFI\fedora\shimx64.efi)....ISPH
Boot0014  HP Recovery   FvVol(cdbb7b35-6833-4ed6-9ab2-57d2acddf6f0)/FvFile(9d8243e8-8381-453d-aceb-c350ee7757ca)......ISPH
Boot0015* stodi_pink_2020_10    HD(1,GPT,3dc251c1-c845-1f4b-9a0b-ebc15792efd6,0x800,0xf4000)/File(\EFI\fedora\shimx64.efi)
Boot0016* Samsung SSD 860 EVO 1TB       PciRoot(0x0)/Pci(0x11,0x4)/Sata(1,0,0)N.....YM....R,Y.....ISPH

And here's grub:

rlpowell@stodi> sudo grub2-editenv /boot/grub2/grubenv list
saved_entry=a7afe39309446d92f17c60bc5467f619-5.14.9-200.fc34.x86_64
kernelopts=root=/dev/stodi_orange_2021_10/root ro rd.auto enforcing=0 crashkernel=auto video=DP-1:1280x1024-32@60e
boot_success=1
boot_indeterminate=0

rlpowell@stodi> sudo cat /boot/loader/entries/a7afe39309446d92f17c60bc5467f619-5.14.9-200.fc34.x86_64.conf
title Fedora (5.14.9-200.fc34.x86_64) 34 (Thirty Four)
version 5.14.9-200.fc34.x86_64
linux /vmlinuz-5.14.9-200.fc34.x86_64
initrd /initramfs-5.14.9-200.fc34.x86_64.img
options root=/dev/mapper/stodi_orange_2021_10-root ro rd.auto enforcing=0 crashkernel=auto video=DP-1:1280x1024-32@60e
grub_users $grub_users
grub_arg --unrestricted
grub_class kernel

But if I reboot, as I said, it'll reliably end up running on the pink LV instead (i.e. on the secondary drive). This is true even when I previously booted from orange with the pink drive removed.

I can't see what part of this makes that possible.

Maybe there's something going on in the BIOS that efibootmgr can't affect?

EDIT: UUID info:

rlpowell@stodi> sudo blkid
/dev/sda1: SEC_TYPE="msdos" UUID="E6B6-7A41" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="/boot/efi - EFI System Partition" PARTUUID="cb8227c2-dc52-fc4f-bd8f-b46f71104428"
/dev/sda2: UUID="544b6c29-f436-4b2f-ab73-f2cf643638b7" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="/boot Partition" PARTUUID="fe677923-076f-2c47-bf0d-98b4d7e3e86f"
/dev/sda3: UUID="ur5eed-EJqk-y2op-F1dY-RpUC-SQGW-MCUk48" TYPE="LVM2_member" PARTLABEL="LVM: root on stodi_orange_2021_10" PARTUUID="b4754fd9-f637-2b4c-9fec-3e120521956f"
/dev/mapper/stodi_orange_2021_10-root: UUID="011f7198-26d2-4c34-bc39-85fc45c55422" BLOCK_SIZE="512" TYPE="xfs"
/dev/sdb1: SEC_TYPE="msdos" UUID="6CBC-8CFB" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="/boot/efi - EFI System Partition" PARTUUID="3dc251c1-c845-1f4b-9a0b-ebc15792efd6"
/dev/sdb2: UUID="0719024b-8442-4b64-a279-842a6533b511" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="/boot Partition" PARTUUID="8b0e313c-37bc-d242-8b23-b4fc6dbaf519"
/dev/sdb3: UUID="82fp7k-I0Th-92EP-gJ0O-MD5A-efK0-YxZ0bP" TYPE="LVM2_member" PARTLABEL="/ Partition" PARTUUID="b64d042d-443e-6445-afbf-0673810b5a2a"
/dev/mapper/stodi_pink_2020_10-root: UUID="ac593057-5586-416c-b141-c8173cdb4d61" BLOCK_SIZE="512" TYPE="xfs"

I have a complex Apache config that mostly just does ProxyPassReverse, and that seems pretty silly so I've been converting it to haproxy. Mostly this has led to things being much, much simpler and clearer; yay. The exception is complex RewriteRule and RedirectMatch sorts of things. In particular, I have these:

RewriteRule ^(?:/l?mw)+/(.*\.php.*)$ https://mw-live.lojban.org/$1 [R=301,L]
RewriteRule ^(?:/l?mw)+/(.*)$ https://mw-live.lojban.org/papri/$1 [R=301,L]

The intent here is to redirect requests to, for example, /lmw/mw/lmw/foo.php to /foo.php, and /mw/lmw/mw/MyPage with /papri/MyPage (yes, those URLs are weird; an old version of the site had some redirect problems).

As far as I can tell, the right way to do this is really not to do it in haproxy, and use the back-end web service itself. I'm asking about it in case I'm missing something, but it really seems like doing this sort of thing on the back-end web service works way better than doing it in haproxy.

I'm having 2 problems with trying to do this in haproxy:

1. I can't find a way to get haproxy to record the redirects that it itself performs. I've told haproxy to show me the Location response header, but that only shows redirects from the back-end server:

    capture response header Location len 128
   

2. The ways of doing this in haproxy are ... clunky. I've found two ways that seem to work:

   # Old URL formats; trim leading "/mw" and "/lmw" when the target is a php file
   http-request redirect code 301 location http://%[hdr(host)]%[url,regsub(^/lmw,,g)] if { path_beg /lmw } { path_sub .php }
   http-request redirect code 301 location http://%[hdr(host)]%[url,regsub(^/mw,,g)] if { path_beg /mw } { path_sub .php }
   # Old URL formats; trim leading "/mw" and "/lmw"
   http-request redirect code 301 location http://%[hdr(host)]/papri%[url,regsub(^/lmw,,g)] if { path_beg /lmw }
   http-request redirect code 301 location http://%[hdr(host)]/papri%[url,regsub(^/mw,,g)] if { path_beg /mw }
   http-request redirect code 301 location http://%[hdr(host)]/papri%[url,regsub(^/papri/lmw,,g)] if { path_beg /papri/lmw }
   http-request redirect code 301 location http://%[hdr(host)]/papri%[url,regsub(^/papri/mw,,g)] if { path_beg /papri/mw }
   

The problem with this method, besides it being many lines long, is that it produces a series of redirects, repeatedly redirecting each URL to one with one less /mw or whatever.

The other is having this on the front end:

use_backend mw-live-back-old-with-php if { hdr_beg(host) -i mw-live. } { path_reg ^/l?mw.*\.php }
use_backend mw-live-back-old-without-php if { hdr_beg(host) -i mw-live. } { path_reg ^/l?mw }

and then these special backends to go with it:

backend mw-live-back-old-with-php
    http-request replace-path ^(?:/l?mw)+/(.*\.php.*)$ /\1
    http-request redirect prefix / code 301

backend mw-live-back-old-without-php
    http-request replace-path ^(?:/l?mw)+/(.*) /papri/\1
    http-request redirect prefix / code 301

The problem with this method is that it's also super long, but also it seems silly to create backends just for this.

The thing you'd think would work, which I stole from https://fromanegg.com/post/2014/12/05/how-to-rewrite-and-redirect-with-haproxy/ , is having these lines as part of the generic backend.

http-request replace-path ^(?:/l?mw)+/(.*\.php.*)$ /\1 if { path_reg ^/l?mw.*\.php }
http-request redirect prefix / code 301 if { path_reg ^/l?mw.*\.php }
http-request replace-path ^(?:/l?mw)+/(.*)$ /papri/\1 if { path_reg ^/l?mw }
http-request redirect prefix / code 301 if { path_reg ^/l?mw }

This fails because the redirect never fires, because by the time we get to the redirect line, the path_reg no longer matches.

So.

Am I missing something, or should I really just move this sort of complexity to the back-end web service?

My old DD-WRT router was dying, so I grabbed a UniFi EdgeRouter, which is, stylistically, much more like a "real router". I have various public IPs that I have 1-to-1 NAT setup for, i.e. the internal system has an IP of 192.168.123.134 , and any external requests to 173.13.139.236 get translated to the internal IP and back.

With the new router, I had to additionally setup hairpin NAT rules, so that internal systems could reach 173.13.139.236 , by having their requests translated to the internal IP but with a source IP on the router (i.e. masqueraded NAT).

(The specific issue, as I understand it, is that if you just NAT a packet that is from an internal address so that its destination is also to an internal address, then the reply just goes back directly to the requester, but the requester sent its packet to the router, so when it sees the reply come back from not-the-router, it drops the packet as invalid.)

With my DD-WRT router, I didn't have to do the hairpin NAT rule, and I don't understand why.

Specifically, the entire config for that IP on my DD-WRT system, as far as I know!, was:

iptables -t nat -I PREROUTING -d 173.13.139.236 -j DNAT --to 192.168.123.134
iptables -t nat -I POSTROUTING -s 192.168.123.134 -j SNAT --to 173.13.139.236
iptables -I FORWARD -d 192.168.123.134 -j ACCEPT

This did, in fact, work. Indeed, it worked for years. Why?

It's totally possible that there's some other config in the DD-WRT setup that handles this, but if there is, I have no idea where it would be; I reviewed the entire config when I changed routers, and I saw nothing else related to those IPs.

I have a number of Fedora 27 systems. I am reasonably comfortable with SELinux. I run it on all my systems, with the "unconfined" module disabled.

On this particular system, SELinux is blocking all "systemctl --user" commands:

$ systemctl --user status
Failed to read server status: Access denied

This worked until recently. I don't know what changed. I did upgrade from Fedora 26 to Fedora 27 recently, but the timing is not the same as this problem, I don't think.

The part that's weirding me out, and making it hard to know what to do next, is that there's nothing about it in auditd, at all.

In syslog I get:

Dec 25 09:48:07 jukni systemd[669]: selinux: avc:  denied  { status } for auid=n/a uid=1086 gid=1086 cmdline="" scontext=staff_u:staff_r:staff_t:s0-s0:c0.c1023 tcontext=staff_u:staff_r:staff_t:s0-s0:c0.c1023 tclass=system permissive=0

Further, this:

$ systemctl --user restart lojban_mediawiki_web
Failed to restart lojban_mediawiki_web.service: Access denied
See user logs and 'systemctl --user status lojban_mediawiki_web.service' for details.

Gives this in syslog:

Dec 25 09:49:06 jukni systemd[669]: selinux: avc:  denied  { start } for auid=n/a uid=1086 gid=1086 path="/home/sampre_mw/.config/systemd/user/lojban_mediawiki_web.service" cmdline="" scontext=staff_u:staff_r:staff_t:s0-s0:c0.c1023 tcontext=staff_u:object_r:systemd_unit_file_t:s0 tclass=service permissive=0

I can't find anything in sesearch about self:system, and all I can find in https://github.com/TresysTechnology/refpolicy.git or https://github.com/TresysTechnology/refpolicy-contrib.git is:

policy/modules/kernel/kernel.te
481:    allow can_load_kernmodule self:system module_load;

policy/modules/system/init.te
225:    allow init_t self:system { status reboot halt reload };

My other similar systems don't have this problem. A relabel reboot did not help. The contexts of the user unit files are:

$ ls -lZ ~/.config/systemd/user/
total 8
drwxr-xr-x. 2 sampre sampre staff_u:object_r:user_home_t:s0  66 Feb  6  2017 default.target.wants
-rw-rw-r--. 1 sampre sampre staff_u:object_r:user_home_t:s0 417 Jul 14 00:32 jbotcan_database.service
-rw-rw-r--. 1 sampre sampre staff_u:object_r:user_home_t:s0 419 Jul 14 00:32 jbotcan_site.service

, which I have no idea if that's correct or not.

Changing this user's login to sysadm_u or user_u didn't help either.

I have no idea where to even go from here. I mean, I could make a custom module to implement the AVCs that are going to syslog, but that seems like The Wrong Thing (tm), since clearly the OS didn't ship without this working.

Added:

$ cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values:
#     targeted - Targeted processes are protected,
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

Added #2:

I've now rebooted one of my other Fedora 27 boxes and:

Jan  5 23:31:38 vrici systemd[5992]: selinux: avc:  denied  { status } for auid=n/a uid=1000 gid=1000 cmdline="" scontext=staff_u:staff_r:staff_t:s0-s0:c0.c1023 tcontext=staff_u:staff_r:staff_t:s0-s0:c0.c1023 tcla

ss=system permissive=0

which makes it feel a lot more like a bug with systemd or selinux.