I am tasked with trying to find a way to download zipped logs from a remote server. There are quite a bit of these logs and they are constantly created.
I do have limited ssh access to the remote server and can scp or rsync the files.
However, due to the sheer size of these logs file, I do not want to rsync all of them. The logs could get to terabytes and for rsync to compare them may take some time. I only want to get any new file that was created/last updated an hour ago.
I also am worried that I will rsync logs that are in the process of being created, so I was thinking to only rsync files that were last modified 3-5 minutes ago.
Would anyone be so kind as to help me with such a process? Thank you in advance.