Home / server / Questions / 948959
Accepted
Huygens
Asked: 2019-01-15 02:30:27 +0800 CST

SELinux relabeling on boot is stuck

  • 772

I have problem restoring SELinux on a server and would like some insight.

Context

On our server, I recently changed SELinux from enforcing to permissive, and because it did not solve our problem which was an odd one (unexpected permissions denied), I even set it to disabled... Anyway, SELinux wasn't the problem. We figured it out and it is now solved ... but SELinux is now disabled!

Now I want to restore it. So I was simply changing it to enforcing again and I was expecting the system to detect automatically it needs to relabel (so no needs to touch /.autorelabel) on next boot, that-worked-in-the-past(tm). Sadly it got stuck during boot and after 10 minutes we went back to disabled to restore services. The documentation states it can take a long long time, so I've scheduled the reboot over the weekend. Well after the weekend it was still stuck in boot.

Strangely the server's HDDs do not show any disk activity.

When I do the above on a VM, it boots to the same point and then prints a message that SELinux relabeling needs to be performed and does it. On the server we do not see that message.

Remarks

The server has some CIFS mounts, would that be the reason that SELinux tries to relabel the CIFS filesystems? That could explain the longer time and the no disk activities, but that do not explain the absence of messages on the console.

Technical info

  • CentOS 7.6 (1810) x86_64
  • Baremetal, Dell server, 6x HDDs RAID6 (hardware)
  • No logs is written to disk, last started unit is Update UTMP about System Boot/Shutdown.

Note: There is a RHEL knowledge base article which recommends going from disabled to enforcing via permissive first. And with each reboot, forcing the relabeling by touching /.autorelabel. That's going to be my next try on the weekend. But any insight to what's going on is welcome.

Update 01

I have now done further testing. All of the following boot mode failed:

  1. booting with selinux=1, enforcing=0, autorelabel=0
  2. booting with selinux=1, enforcing=0, autorelabel=1
  3. booting with selinux=1, enforcing=1, autorelabel=0
  4. booting with selinux=1, enforcing=1, autorelabel=1

Only booting with selinux=0 is working. When I mean failed, it means there is no error messages on the console while booting nor a message about SELinux relabeling, the system hangs for many hours, no disk activity is visible.

I have now commented out all cifs volumes from my /etc/fstab and rebooted in permissive with autorelabeling. The boot took about 20min but I could see it was doing something (it printed on the console that relabeling was started, showed that FS like sysfs are read-only and be ignored, disk activity was visible, etc.). So SELinux is now enabled in permissive mode. I will now try enforcing mode this weekend.

Obviously it seems that those CIFS mounts were the problem, but I do not understand why. To me it was my understanding that CIFS mounts would be ignored by relabeling, just like sysfs. Have I misunderstood something here?

centos7

1 Answers

  1. Best Answer

    Is there any specific reason to use /.autorelabel? My understanding is that it triggers a restorecon.

    To figure out why that hangs, I'd suggest to switch to permissive mode, and boot the machine without a /.autorelabel file. When the system is up and running try restorecon -rFv /. Eventually do not start with / but with e.g. /var or a smaller portion of the whole system.

    The -v flag of restorecon shows you all the ongoing modifications on the filesystem. It should give you a good idea where it "hangs" during a boot with a /.autorelabel. I'd expect that there is some area in the filesystem where lots of tiny files are stored. Or eventually network mounted storage.

    Once this is done switching to enforcing should not require another /.autorelabel run.

    [edit]

    I've just verified above statement regarding /.autorelabel and restorecon on a Fedora 29 machine. /lib/systemd/system/selinux-autorelabel.service starts /usr/libexec/selinux/selinux-autorelabel which is a bash script. That script runs /sbin/fixfiles with the restore parameter. /sbin/fixfiles is another bash script which actually runs restorecon.

    • 4