capistrano 3 "does not appear to be a git repository"

I have a problem when I try to deploy a rails application via capistrano. The following are my current configuration files with redacted servernames:

config/deploy.rb

  lock '3.2.1'

  set :application, "my_app"
  set :repo_url, "my.server.example.com:/path/to/my_app.git"
  set :branch, 'master'
  set :use_sudo, false
  set :deploy_to, "/srv/#{fetch(:application)}"

config/deploy/production.rb

role :app, "my.server.example.com"
role :web, "my.server.example.com"
role :db,  "my.server.example.com", :primary => true

server 'my.server.example.com',
  user: 'root',
  roles: %w{web app},
  ssh_options: {
    user: 'me', # overrides user setting above
    keys: %w(/home/me/.ssh/id_rsa),
    forward_agent: false,
    auth_methods: %w(publickey),
    verbose: :debug
}

Basically there are three computers involved:

1. localhost, my workstation
2. my.server.example.com, hosts the rails application
3. my.git.example.com, hosts the git-repositories.

When I try to deploy I get the following error:

[me@localhost my_app]$ cap production deploy
 INFO [d2263887] Running /usr/bin/env mkdir -p /tmp/my_app/ on my.server.example.com
DEBUG [d2263887] Command: /usr/bin/env mkdir -p /tmp/my_app/
 INFO [d2263887] Finished in 0.713 seconds with exit status 0 (successful).
DEBUG Uploading /tmp/my_app/git-ssh.sh 0.0%
 INFO Uploading /tmp/my_app/git-ssh.sh 100.0%
 INFO [acbfa48d] Running /usr/bin/env chmod +x /tmp/my_app/git-ssh.sh on my.server.example.com
DEBUG [acbfa48d] Command: /usr/bin/env chmod +x /tmp/my_app/git-ssh.sh
 INFO [acbfa48d] Finished in 0.005 seconds with exit status 0 (successful).
DEBUG [b2d0392c] Running /usr/bin/env git ls-remote -h my.git.example.com:/path/to/repos/my_app.git on my.server.example.com
DEBUG [b2d0392c] Command: ( GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/my_app/git-ssh.sh /usr/bin/env git ls-remote -h my.git.example.com:/path/to/repos/my_app.git )
DEBUG [b2d0392c]    fatal: '/path/to/repos/my_app.git' does not appear to be a git repository
DEBUG [b2d0392c]    fatal: The remote end hung up unexpectedly
DEBUG [b2d0392c] Finished in 1.025 seconds with exit status 128 (failed).

Further investigation lead me to believe that the line

git ls-remote -h my.git.example.com:/path/to/repos/my_app.git

produces the error, as it gives me the exact same error message when I try to execute it on my.server.example.com. It works if I modify it to:

git ls-remote -h me@my.git.example.com:/path/to/repos/my_app.git

So I have the strong feeling that my user is somehow configured wrong and would appreciate any pointers in the right direction.