I want to have limited access to a Linux server to be able to grab a certain set of logs. Ideally, I would like to be able to run rsync locally to grab the logs from the server and for security reasons I don't want to give the user full ssh login access.
Make a separate user for this purpose only. Have the user login with an SSH key only. In the authorized_keys file for the user, edit the public key to allow only a command. That command should not be a pointer to a shell script; instead insert the shell script into the key directly.
Here's an example. The setup here is that on the server, there's a cron job that moves daily logs to /var/log/logfetch. Another server, with IP 10.1.2.3, will connect and send a command. If the command is BACKUP, the client will receive a gzipped tar file of the files in the directory /var/log/logfetch. If it is instead a file name, the file with that name in /var/log/logfetch will be deleted. Any other command will be ignored. All commands will be logged. Connections are only allowed from that one IP address.
This may be overkill for your particular situation, but it's reasonably hard for a third party to abuse and should be possible to adapt to your particular needs.