I have asked this question yesterday but it was marked as duplicate and closed because of its context, as it was thought to be an X/Y question, whereas I was just interested in the general matter of "how this could be", as my personal research (on this website as well as on the internet) returned nothing and I'd like to learn more about how to detect and deal with that kind of peculiar issue.
So, without any context, yesterday I found some files on one of our Debian servers that where visible to a regular user but not to root, despite them being owned by root.
It tried a whole lot of commands on those files, and whatever I try, user treats the files as regular files, but root reacts like those files simply don't exist in the first place (but cannot overwrite them nonetheless). Those are NOT dot files.
Here are the results of these commands :
As user
user@debian:/tmp$ groups
user cdrom floppy audio dip video plugdev netdev
user@debian:/tmp$ pwd
/tmp
user@debian:/tmp$ ls -lai
total 320
1048577 drwxrwxrwt 11 root root 4096 Sep 7 13:04 .
2 drwxr-xr-x 23 root root 4096 Sep 6 17:34 ..
5901230 -rw-r----- 1 root root 0 Sep 7 12:59 invisible_file
<other_files>
user@debian:/tmp$ touch invisible_file
touch: cannot touch 'invisible_file': Permission denied
user@debian:/tmp$ rm invisible_file
rm: remove write-protected regular empty file 'invisible_file'? y
rm: cannot remove 'invisible_file': Operation not permitted
user@debian:/tmp$ stat invisible_file
File: invisible_file
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 801h/2049d Inode: 5901230 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2021-09-07 12:59:54.859124530 +0200
Modify: 2021-09-07 12:59:54.859124530 +0200
Change: 2021-09-07 13:04:03.063441285 +0200
Birth: -
user@debian:/tmp$ install /dev/null invisible_file
install: cannot remove 'invisible_file': Operation not permitted
user@debian:/tmp$ cat invisible_file
cat: invisible_file: Permission denied
user@debian:/tmp$ find /tmp/ -iname "*invisible_file*"
/tmp/invisible_file
user@debian:/tmp$
As root
root@debian:/tmp# groups
root
root@debian:/tmp# pwd
/tmp
root@debian:/tmp# ls -lai
total 308
1048577 drwxrwxrwt 11 root root 4096 Sep 7 13:04 .
2 drwxr-xr-x 23 root root 4096 Sep 6 17:34 ..
<other_files>
root@debian:/tmp# touch invisible_file
root@debian:/tmp# ls -lai
total 308
1048577 drwxrwxrwt 11 root root 4096 Sep 7 13:04 .
2 drwxr-xr-x 23 root root 4096 Sep 6 17:34 ..
<other_files>
root@debian:/tmp# rm invisible_file
rm: cannot remove 'invisible_file': No such file or directory
root@debian:/tmp# stat invisible_file
stat: cannot stat 'invisible_file': No such file or directory
root@debian:/tmp# install /dev/null invisible_file
install: cannot create regular file 'invisible_file': No such file or directory
root@debian:/tmp# cat invisible_file
cat: invisible_file: No such file or directory
root@debian:/tmp# find /tmp/ -iname "*invisible_file*"
root@debian:/tmp#
Notice that even in the ls command the number of total used blocks is different, the difference corresponding to the invisible_file size.
The only way I am able to overwrite the file is by creating a file with another name (and even other permissions) and as root, mv it over invisible_file, but invisible_file keeps being hidden to root.
My question is : how, in the Linux world, can one make root completely ignore some regular files as if they just weren't there in the first place, like in my case? And how could I investigate that matter, make those files visible again, and be sure that there are no other invisible files to root?
EDIT :
Here's mount output, it doesn't show anything special to me:
root@debian:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=4078644k,nr_inodes=1019661,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=817960k,mode=755)
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=9463)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=817956k,mode=700,uid=1000,gid=1000)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
The output of fsck -nf is the following:
root@debian:~# fsck -nf
fsck from util-linux 2.29.2
e2fsck 1.43.4 (31-Jan-2017)
Warning! /dev/sda1 is mounted.
Warning: skipping journal recovery because doing a read-only filesystem check.
Pass 1: Checking inodes, blocks, and sizes
Deleted inode 524799 has zero dtime. Fix? no
Inodes that were part of a corrupted orphan linked list found. Fix? no
Inode 1441794 was part of the orphaned inode list. IGNORED.
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: -(11108512--11108538)
Fix? no
Free blocks count wrong (16886612, counted=16857986).
Fix? no
Inode bitmap differences: -524799 -1441794
Fix? no
Free inodes count wrong (5867140, counted=5866555).
Fix? no
/dev/sda1: ********** WARNING: Filesystem still has errors **********
/dev/sda1: 162172/6029312 files (0.3% non-contiguous), 7230636/24117248 blocks
root@Confluence:~#
I've finally been able to run a full fsck on the filesystem. It corrected the shown above errors, but to no avail as the files are still invisible.