I am tasked with trying to find a way to download zipped logs from a remote server. There are quite a bit of these logs and they are constantly created.
I do have limited ssh access to the remote server and can scp or rsync the files.
However, due to the sheer size of these logs file, I do not want to rsync all of them. The logs could get to terabytes and for rsync to compare them may take some time. I only want to get any new file that was created/last updated an hour ago.
I also am worried that I will rsync logs that are in the process of being created, so I was thinking to only rsync files that were last modified 3-5 minutes ago.
Would anyone be so kind as to help me with such a process? Thank you in advance.
You can use ls -t to list the latest files by date and send that to rsync, you can easily set it to only transfer yesterdays logs or do you need it from the same day aswell?
Something like http://www.commandlinefu.com/commands/view/1481/rsync-find should do; you may want to execute the one-liner on the remote server and make it connect back to your "upload" server.