Questions[unmount](server)

1. Create juicefs Network mount(like NFS mount) in host machine

juicefs mount -d redis://10.111.100.91:6379/0 /mnt/jfs-mount
2021/12/10 20:05:25.914969 juicefs[33027] <INFO>: Meta address: redis://10.111.100.91:6379/0
2021/12/10 20:05:25.916720 juicefs[33027] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
2021/12/10 20:05:25.917140 juicefs[33027] <INFO>: Ping redis: 329.765µs
2021/12/10 20:05:25.917626 juicefs[33027] <INFO>: Data use minio://10.102.8.247:9000/test/minio/
2021/12/10 20:05:25.917812 juicefs[33027] <INFO>: Disk cache (/var/jfsCache/3680a8cc-a3f7-40a9-ac6f-fc79505bb728/): capacity (1024 MB), free ratio (10%), max pending pages (15)
2021/12/10 20:05:26.419836 juicefs[33027] <INFO>: OK, minio is ready at /mnt/jfs-mount
[root@kube-node-1 juice]#
[root@kube-node-1 juice]# df -h|grep jfs-mount
JuiceFS:minio   1.0P  8.4M  1.0P    1% /mnt/jfs-mount

2. Try to bind mount to an existing directory, and make it shared mode

[root@kube-node-1 juice]# mkdir /mnt/jfs-bind
[root@kube-node-1 juice]# mount --bind --make-shared /mnt/jfs-mount /mnt/jfs-bind
[root@kube-node-1 juice]# cat /proc/self/mountinfo |grep jfs | sed 's/ - .*//'
152 40 0:219 / /mnt/jfs-mount rw,relatime shared:117
155 40 0:219 / /mnt/jfs-bind rw,relatime shared:117
[root@kube-node-1 juice]#

3. Kill the fuse process, and make mount point not work

[root@kube-node-1 juice]# ps -ef |grep juicefs
root      33043      1  0 20:05 ?        00:00:00 juicefs mount -d redis://10.111.100.91:6379/0 /mnt/jfs-mount
root      34338 129878  0 20:06 pts/1    00:00:00 grep --color=auto juicefs
[root@kube-node-1 juice]# kill -9 33043
[root@kube-node-1 juice]# ls /mnt/jfs-mount
ls: cannot access /mnt/jfs-mount: Transport endpoint is not connected
[root@kube-node-1 juice]# ls /mnt/jfs-bind
ls: cannot access /mnt/jfs-bind: Transport endpoint is not connected

4. Recover the source mount point of bind(/mnt/jfs-mount), and check the target mount point

[root@kube-node-1 juice]# umount /mnt/jfs-mount
[root@kube-node-1 juice]# juicefs mount -d redis://10.111.100.91:6379/0 /mnt/jfs-mount
2021/12/10 20:07:19.357752 juicefs[35185] <INFO>: Meta address: redis://10.111.100.91:6379/0
2021/12/10 20:07:19.359160 juicefs[35185] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
2021/12/10 20:07:19.359528 juicefs[35185] <INFO>: Ping redis: 340.317µs
2021/12/10 20:07:19.360107 juicefs[35185] <INFO>: Data use minio://10.102.8.247:9000/test/minio/
2021/12/10 20:07:19.360264 juicefs[35185] <INFO>: Disk cache (/var/jfsCache/3680a8cc-a3f7-40a9-ac6f-fc79505bb728/): capacity (1024 MB), free ratio (10%), max pending pages (15)
2021/12/10 20:07:19.862758 juicefs[35185] <INFO>: OK, minio is ready at /mnt/jfs-mount
[root@kube-node-1 juice]#ls  /mnt/jfs-bind
ls: cannot access /mnt/jfs-bind: Transport endpoint is not connected

I thought /mnt/jfs-bind can be recovered automatically, because the remount action can be propagated to bind target mount. Seems it's not the same behavior as https://www.kernel.org/doc/Documentation/filesystems/sharedsubtree.txt . I wonder why?

And further question, if I want to make bind mount automatically recover when the source mount point of juice is recovered. Is there any way to do so?

I have a problem where every time I reboot a Linux system (Ubuntu-based), the root filesystem is not unmounted successfully.

Doing a fsck after a reboot results in output like:

Recovering journal [...] clearing orphaned inode [...]

The message displayed on reboot is:

Failed to remount '/' as read-only: Device or resource busy

Is there a way to interrupt the reboot and get a command prompt inside the ramfs root filesystem used during reboot so I can debug things there? Like see what processes are in memory, inspect file handles etc.

Or have any other recommendations for getting to the root cause?

Thanks

How to umount this cgroup?

Also i have no idea is it important or not. It seem cgroup is comefrom docker but im still not sure.

I was trying install gns3-remote

And it give me cgroup like this.

Then i deleted gns3-remote because it's for fresh vps not my private vps. I dont need it anymore. Also im still not sure if i clearly remove it

Is it okay keep this cgroup? Im just annoying seeing this filesystem too much. I just want look my vps fresh like just born

I have a mount point in Linux that does not show any open files in lsof or fuser. The mount point was exported in NFS, but is no longer exported, and there are no more files or directories in the mount point. We have checked to make sure no one is in the directory, going so far as to count and check the bash processes as well. At this point, we are absolutely stumped as to why we are unable to unmount the mount points.

[root@head-02 proc]# umount /PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))

We do not have the directory exported in NFS:

[root@head-02 proc]# showmount -e localhost
Export list for localhost:
/PoolB/LegacyExchange 172.18.10.0/24
/PoolB/Customers      172.18.10.0/24
/PoolB/Import         172.18.10.0/24

Output from FUSER:

[root@head-02 proc]# fuser -vm /PoolB/daily.2020.12.13
                 USER        PID ACCESS COMMAND
/PoolB/daily.2020.12.13:
                 root     kernel mount /PoolB/daily.2020.12.13

Output from LSOF:

[root@head-02 proc]# lsof +D /PoolB/daily.2020.12.13
[root@head-02 proc]# 

The system is a ZFS On Linux system, so I could try to change the mount point there or delete it. The ultimate goal is to delete the filesystem from the ZPOOL:

[root@head-02 proc]# zfs set mountpoint=legacy PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
        the device is found by lsof(8) or fuser(1))
cannot unmount '/PoolB/daily.2020.12.13': umount failed
[root@head-02 proc]# 

I even tried just deleting the filesystem:

[root@head-02 proc]# zfs destroy PoolB/daily.2020.12.13
umount: /PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
cannot unmount '/PoolB/daily.2020.12.13': umount failed
[root@head-02 proc]# 

Out of desperation, here is the strace output of the umount command that fails:

[root@head-02 proc]# strace umount /PoolB/daily.2020.12.13
execve("/usr/bin/umount", ["umount", "/PoolB/daily.2020.12.13"], 0x7ffcd33fef18 /* 22 vars */) = 0
brk(NULL)                               = 0x55d4721b9000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088778000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=22424, ...}) = 0
mmap(NULL, 22424, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4088772000
close(3)                                = 0
open("/lib64/libmount.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\237\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=277808, ...}) = 0
mmap(NULL, 2370944, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4088315000
mprotect(0x7f4088355000, 2097152, PROT_NONE) = 0
mmap(0x7f4088555000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x40000) = 0x7f4088555000
mmap(0x7f4088557000, 3456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4088557000
close(3)                                = 0
open("/lib64/libblkid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\207\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=265600, ...}) = 0
mmap(NULL, 2358152, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f40880d5000
mprotect(0x7f4088111000, 2093056, PROT_NONE) = 0
mmap(0x7f4088310000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3b000) = 0x7f4088310000
mmap(0x7f4088314000, 2952, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4088314000
close(3)                                = 0
open("/lib64/libuuid.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\340\24\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=20064, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088771000
mmap(NULL, 2113920, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087ed0000
mprotect(0x7f4087ed4000, 2093056, PROT_NONE) = 0
mmap(0x7f40880d3000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f40880d3000
close(3)                                = 0
open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320i\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=155784, ...}) = 0
mmap(NULL, 2255184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087ca9000
mprotect(0x7f4087ccd000, 2093056, PROT_NONE) = 0
mmap(0x7f4087ecc000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x23000) = 0x7f4087ecc000
mmap(0x7f4087ece000, 6480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087ece000
close(3)                                = 0
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20&\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2156160, ...}) = 0
mmap(NULL, 3985888, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f40878db000
mprotect(0x7f4087a9e000, 2097152, PROT_NONE) = 0
mmap(0x7f4087c9e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c3000) = 0x7f4087c9e000
mmap(0x7f4087ca4000, 16864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087ca4000
close(3)                                = 0
open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\25\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=402384, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088770000
mmap(NULL, 2494984, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087679000
mprotect(0x7f40876d9000, 2097152, PROT_NONE) = 0
mmap(0x7f40878d9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x60000) = 0x7f40878d9000
close(3)                                = 0
open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\220\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=19288, ...}) = 0
mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087475000
mprotect(0x7f4087477000, 2097152, PROT_NONE) = 0
mmap(0x7f4087677000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f4087677000
close(3)                                = 0
open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0@m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=142232, ...}) = 0
mmap(NULL, 2208904, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f4087259000
mprotect(0x7f4087270000, 2093056, PROT_NONE) = 0
mmap(0x7f408746f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16000) = 0x7f408746f000
mmap(0x7f4087471000, 13448, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f4087471000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876f000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876e000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f408876c000
arch_prctl(ARCH_SET_FS, 0x7f408876c880) = 0
mprotect(0x7f4087c9e000, 16384, PROT_READ) = 0
mprotect(0x7f408746f000, 4096, PROT_READ) = 0
mprotect(0x7f4087677000, 4096, PROT_READ) = 0
mprotect(0x7f40878d9000, 4096, PROT_READ) = 0
mprotect(0x7f4087ecc000, 4096, PROT_READ) = 0
mprotect(0x7f40880d3000, 4096, PROT_READ) = 0
mprotect(0x7f4088310000, 12288, PROT_READ) = 0
mprotect(0x7f4088555000, 4096, PROT_READ) = 0
mprotect(0x55d470fcf000, 4096, PROT_READ) = 0
mprotect(0x7f4088779000, 4096, PROT_READ) = 0
munmap(0x7f4088772000, 22424)           = 0
set_tid_address(0x7f408876cb50)         = 37241
set_robust_list(0x7f408876cb60, 24)     = 0
rt_sigaction(SIGRTMIN, {sa_handler=0x7f408725f820, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f40872685f0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {sa_handler=0x7f408725f8b0, sa_mask=[], sa_flags=SA_RESTORER|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f40872685f0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
statfs("/sys/fs/selinux", 0x7fffc7932c50) = -1 ENOENT (No such file or directory)
statfs("/selinux", 0x7fffc7932c50)      = -1 ENOENT (No such file or directory)
brk(NULL)                               = 0x55d4721b9000
brk(0x55d4721da000)                     = 0x55d4721da000
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 350
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7fffc7932830) = -1 ENOENT (No such file or directory)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
access("/etc/selinux/config", F_OK)     = 0
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=106075056, ...}) = 0
mmap(NULL, 106075056, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f4080d2f000
close(3)                                = 0
getuid()                                = 0
geteuid()                               = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
stat("/run", {st_mode=S_IFDIR|0755, st_size=960, ...}) = 0
lstat("/run/mount/utab", 0x7fffc7932b50) = -1 ENOENT (No such file or directory)
mkdir("/run/mount", 0755)               = -1 EEXIST (File exists)
stat("/run/mount/utab", 0x7fffc7932a40) = -1 ENOENT (No such file or directory)
stat("/run/mount", {st_mode=S_IFDIR|0755, st_size=40, ...}) = 0
geteuid()                               = 0
getegid()                               = 0
getuid()                                = 0
getgid()                                = 0
access("/run/mount", R_OK|W_OK)         = 0
stat("/PoolB/daily.2020.12.13", {st_mode=S_IFDIR|0755, st_size=39, ...}) = 0
readlink("/PoolB", 0x7fffc79309b0, 4096) = -1 EINVAL (Invalid argument)
readlink("/PoolB/daily.2020.12.13", 0x7fffc79309b0, 4096) = -1 EINVAL (Invalid argument)
lstat("/etc/mtab", {st_mode=S_IFLNK|0777, st_size=17, ...}) = 0
open("/proc/self/mountinfo", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "18 41 0:18 / /sys rw,nosuid,node"..., 1024) = 1024
stat("/etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned", 0x7fffc7930370) = -1 ENOENT (No such file or directory)
read(3, "rfs efivarfs rw\n29 25 0:25 / /sy"..., 1024) = 1024
read(3, "ev,noexec,relatime shared:19 - c"..., 1024) = 1024
read(3, ",xattr,posixacl\n131 128 0:42 / /"..., 1024) = 1024
read(3, " - zfs PoolB/daily.2020.12.13 rw"..., 1024) = 145
readlink("/proc", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
readlink("/proc/self", "37241", 4096)   = 5
readlink("/proc/37241", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
readlink("/proc/37241/mountinfo", 0x7fffc7930830, 4096) = -1 EINVAL (Invalid argument)
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
getuid()                                = 0
geteuid()                               = 0
getgid()                                = 0
getegid()                               = 0
prctl(PR_GET_DUMPABLE)                  = 1 (SUID_DUMP_USER)
stat("/run", {st_mode=S_IFDIR|0755, st_size=960, ...}) = 0
stat("/run/mount/utab", 0x7fffc79328c0) = -1 ENOENT (No such file or directory)
stat("/sbin/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/sbin/fs.d/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/sbin/fs/umount.zfs", 0x7fffc7931a80) = -1 ENOENT (No such file or directory)
stat("/run/mount/utab", 0x7fffc7932b30) = -1 ENOENT (No such file or directory)
umount2("/PoolB/daily.2020.12.13", 0)   = -1 EBUSY (Device or resource busy)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2502, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4088777000
read(3, "# Locale name alias data base.\n#"..., 4096) = 2502
read(3, "", 4096)                       = 0
close(3)                                = 0
munmap(0x7f4088777000, 4096)            = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/util-linux.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "umount: ", 8umount: )                 = 8
write(2, "/PoolB/daily.2020.12.13: target "..., 153/PoolB/daily.2020.12.13: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))) = 153
write(2, "\n", 1
)                       = 1
close(1)                                = 0
close(2)                                = 0
exit_group(32)                          = ?
+++ exited with 32 +++
[root@head-02 proc]# 

As you can see, the strace reveals the system call umount2 fails with EBUSY, which in the man page for that system call, just says "The Target is Busy".

We even deleted all the contents of that filesystem with rm -rf /PoolB/daily.2020.12.13/ which ran successfully. We don't have swap spaces on it, and it isn't listed in /etc/exports.

We are at a loss at this point, and for obvious reasons, rebooting the system is for all intents and purposes a non-starter. We should not have to reboot anyway just because a filesystem has to be unmounted. Any advice would be greatly appreciated.

Linux Kernel version is 3.10.0-1062.4.3.el7.x86_64 ZFS version is 0.8.2-1.el7 as is the zfs-dkms package.

Thank you for reading this far.

I am just trying to figure out if this is possible to do. I am trying to resize a number of filesystems, and to be safe (these are production boxes as well as QA) I wanted to do the resize offline.

I have gotten into the server with single user mode and am having no issues unmounting /var, and /opt. However I cannot unmount /.

Any ideas would be appreciated, using a rescue cd is not an option.