Keep in mind, both machines used are hosted in a Proxmox Node.
SETUPS
My Samba setup is installed on Ubuntu inside an LXC. The shares are located on a RAIDZ2 dataset, and are accessible from the LAN. 4 LCores, 6 GB RAM
The Windows 10 VM is on a different physical SSD from the LXC. It is on the same network as the Ubuntu LXC and both are connected to the same virtual switch (iperf shows a bandwidth of around 15.9 Gbits/sec). 4 LCores, 8 GB RAM
SCENARIO
I open a VNC console to the Windows 10 machine and login to the Samba share through explorer. I select a file over 2 GB and it starts to transfer the file. Initially, the transfer will start at like 255 MB/s (fast), but then it will slow down to 0MB/s within 5 seconds of starting the transfer. After the initial drop, the transfer will top out at only 5 MB/s for the rest of the time, while occasionally dipping down to 0.
I suspect something can’t keep up and do not know if it is a physical hardware issue or a virtual one.
Is there anything that I should check/change to get consistent samba speeds?
Since you have already checked your network using iPerf I would assume networking isn't the problem.
As a further step in troubleshooting the issue I would recommend you try to create an NFS share on your Ubuntu virtual machine and try copying the files in a same manner from NFS share to your Windows virtual machine.
If the speed problem persists I would take a close look into storage configuration and benchmark storage performance on both virtual machines to make sure it is not the root cause of the issue.
If the NFS performs fine I would start exploring and tweaking the Samba settings.