Should SSH w/o password prevent rsync a folder with read permission?

Sorry for the long question. I try to add as many details as I have.

On server_1 I have a systems account user named backup which has a home directory where the cron jobs read the backup scripts.

backup@server_1~]$ ls 
script_1    script_2 ...

On server_2 I have created a standard user "backup" to hold some repositories that are currently backed up on server_1. Those backed up fine for months. Now, I have added a link to a new repo in /srv which is owned by the user & group srv.

backup@server_2~]$ ls 
drwxrwxr.. backup:backup   repo_1
drwxrwxr.. backup:backup   repo_2 
lrwxrwx... backup:backup   repo_3 -> /srv/repo_3

backup@server_2~]$ ls /srv
drwxrwxr.. srv:srv         repo_3
       ^ notice the r, indicating any user should be able to read this data too.

I then added backup on server_2 into the srv group, so that backup should be able to read all the data for a sync down onto server_1.

root@server_2~]# usermod -a -G srv backup

Then I tried to rsync:

backup@server_1~]$ rsync -avi -e "ssh -i /home/backup/.ssh/server_2_ssh_key" \
                   backup@server_2/srv/repo_3 ./

The problem is, when I run the backup script, using the passwordless login from server_1, it fails to read data, because rsync is unable to change to directory /srv/repo_3 due to "permission denied" The same happens when I tried using the symlink.

backup@server_1~]$ rsync -avi -e "ssh -i /home/backup/.ssh/server_2_ssh_key" \
                   backup@server_2/home/backup/repo_3 ./

Then I even logged in using the backup users key pair to server_1 and I am unable to even list the contents of /srv/repo_3

I happen to have another standard user account on server_2 which uses an SSH Key Login that has a password. When I login that way, "user_2" is am able to list the contents of /srv

So, I copied the second users' ssh key from server_2 into /home/backup/.ssh/ssh_key_w_password on server_1, and added the public part to backup's trusted hosts on server_2. Then I tried the backup using that key.

backup@server_1~]$ rsync -avi -e "ssh -i /home/backup/.ssh/ssh_key_w_password" \
                   backup@server_2/home/backup/repo_3 ./
Password for ssh_key_w_password: 

I enter the password and the backup ran correctly, even though user_2 is not even in the backup or srv group on server_2. It works by the symlink or by the direct location /srv/repo_3.

Some user details:

 backup@server_2~]$ cat /etc/passwd | grep backup
 backup:x:1008:1008:backup:/home/backup:/bin/bash
 backup@server_2~]$ groups
 backup srv

 user_2@server_2~]# cat /etc/passwd | grep user_2
 user_2:x:1012:1012:user_2:/home/user_2:/bin/bash
 root@server_2~]# groups user_2
 user_2

 user_2@server_2~]$ cat /etc/passwd | grep srv
 srv:x:1018:1021::/home/srv:/bin/bash
 user_2@server_2~]$ groups srv
 srv : srv mycorp 4h jndj ax

There we are. The only difference I can find from my side is that backup uses a passwordless key-pair from server_1, while the other standard user has the password on the SSH key.

Can anyone help me understand what is different or what I am missing? I must have backup on server_1 use the pwordless login to run the sync. I cannot allow server_2 to sync up to server_1.

Update: re: MadHatter's comment Login direct fails as backup from server1 because the password based login is not permitted. But using the password-less key returns output (as does the same attempt with the password key from the other user.

[backup@server_1 ~]$ ssh backup@server_2 "id -a"
backup@server_2's password:
Permission denied, please try again.

[backup@server_1 ~]$ ssh -i .ssh/backup_server_2_ssh.key backup@server_2 "id -a"
uid=1008(backup) gid=1008(backup) groups=1008(backup),1021(srv)
[backup@server_1 ~]$
[backup@server_1 ~]$ ssh -i .ssh/ssh_key_w_password user_2@server_2 "id -a"
Enter passphrase for key '.ssh/ssh_key_w_password':
uid=1012(user_2) gid=1012(user_2) groups=1012(user_2)
[backup@server_1 ~]$ 

For reference, this is the fail log messages from rsync when I just tried again using the passwordless login as root on server_1.

[root@server_1 ~]# bash /home/backup/add_srv.sh
2013-11-18 12:24:14 - ************ Backup Robot Checking In **********
.... login and mount backup destination goes ok here rsync fail is below ....
2013/11/18 12:24:20 [920] receiving file list
2013/11/18 12:24:20 [920] rsync: change_dir "/srv" failed: Permission denied (13)
2013/11/18 12:24:20 [920] sent 8 bytes  received 10 bytes  12.00 bytes/sec
2013/11/18 12:24:20 [920] total size is 0  speedup is 0.00
2013/11/18 12:24:20 [920] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1505) [receiver=3.0.6]
[root@server_1 ~]#