Why am I allowed to move, but not delete, a file on Windows Server 2019?

Today I needed to remove a file, but I couldn't:

[capv@TKG-VC-ANTREA-M]: C:\Users\capv> rm 'C:\Program Files\containerd\containerd-shim-runhcs-v1.exe'
rm : Cannot remove item C:\Program Files\containerd\containerd-shim-runhcs-v1.exe: Access to the path 'C:\Program Files\containerd\containerd-shim-runhcs-v1.exe' is denied.

Oddly, I found a workaround: Just mv it instead, and it worked.

[capv@TKG-VC-ANTREA-M]: C:\Users\capv> mv 'C:\Program Files\containerd\containerd-shim-runhcs-v1.exe' a

After this I was able to accomplish my original goal of untarring the contents of a tarball into this directory..

[capv@TKG-VC-ANTREA-M]: C:\Users\capv> cmd /c tar -zxvf containerd.tar.gz -C "c:/program files/containerd" --strip-components 1
x containerd-shim-runhcs-v1.exe
x containerd-revision.txt
x containerd.exe
x hcsshim-revision.txt
x ctr.exe

Out of curiosity, after moving the file, I tried, but still couldn't delete the moved version of the file:

[capv@TKG-VC-ANTREA-M]: C:\Users\capv> rm a
rm : Cannot remove item C:\Users\capv\a: Access to the path 'C:\Users\capv\a' is denied.
At line:1 char:1

Why would Windows Server allow me to move but not delete this file?