Update: seems to be a bug from 13.10:

https://bugs.launchpad.net/ubuntu/+source/libpam-ssh/+bug/1247169

Anyway running the following commands the problem was fixed for me:

How to fix

I fixed this by entering the following commands:

$ ssh-agent bash

This creates a new bash process that allows you to add private keys. When adding a new private key you will be prompted for the passphrase once and only once.

And then:

$ ssh-add /home/username/.ssh/id_rsa
Enter passphrase for /home/username/.ssh/id_rsa: 
Identity added: /home/username/.ssh/id_rsa (/home/username/.ssh/id_rsa)

...where username is your username. You can do the same using $USER variable:

$ ssh-add /home/$USER/.ssh/id_rsa

Alternatively, just use ~ for your home directory.

$ ssh-add ~/.ssh/id_rsa

And the problem was fixed.

Short answer

Use AddKeysToAgent and add the following to your .ssh/config at the beginning:

AddKeysToAgent yes

and run git/ssh/... If it's not enough, check your ssh version and check that ssh-agent is loaded with these instructions:

1) Check the openssh version

Firstly check that your ssh version, it must be greater of equal to 7.2:

ssh -V

2) Edit the config file

If it's the case just add in your .ssh/config one line at the beginning:

AddKeysToAgent yes

Other options are no (the default), yes, confirm (optionally followed by a time interval), ask or a time interval.

3) Check if ssh-agent is already open

Usually distributions automatically load an ssh-agent. To check it, run

ps aux | grep -v grep | grep ssh-agent

If you don't see any line containing it, you need to load it by running:

eval $(ssh-agent)

Note that this enable the agent only on the current terminal, so to enable it everywhere, you can try to add this line in your ~/.profile file and reboot.

This Atlassian document (archive.org backup) fixed the issue for me on Ubuntu 14.04 Server Edition:

Just add this values into your .bashrc file:

SSH_ENV=$HOME/.ssh/environment
   
# start the ssh-agent
function start_agent {
    echo "Initializing new SSH agent..."
    # spawn ssh-agent
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    echo succeeded
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
    /usr/bin/ssh-add
}
   
if [ -f "${SSH_ENV}" ]; then
     . "${SSH_ENV}" > /dev/null
     ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

And after logging in, it asks for password only once and it caches. You don't need to enter it each time.

A workaround for this bug is to add the following to the bottom of ~/.bashrc

eval `gnome-keyring-daemon --start`

Users of the fish shell can use this script to do the same thing.

# content has to be in .config/fish/config.fish
# if it does not exist, create the file
setenv SSH_ENV $HOME/.ssh/environment

function start_agent                                                                                                                                                                    
    echo "Initializing new SSH agent ..."
    ssh-agent -c | sed 's/^echo/#echo/' > $SSH_ENV
    echo "succeeded"
    chmod 600 $SSH_ENV 
    . $SSH_ENV > /dev/null
    ssh-add
end

function test_identities                                                                                                                                                                
    ssh-add -l | grep "The agent has no identities" > /dev/null
    if [ $status -eq 0 ]
        ssh-add
        if [ $status -eq 2 ]
            start_agent
        end
    end
end

if [ -n "$SSH_AGENT_PID" ] 
    ps -ef | grep $SSH_AGENT_PID | grep ssh-agent > /dev/null
    if [ $status -eq 0 ]
        test_identities
    end  
else
    if [ -f $SSH_ENV ]
        . $SSH_ENV > /dev/null
    end  
    ps -ef | grep $SSH_AGENT_PID | grep -v grep | grep ssh-agent > /dev/null
    if [ $status -eq 0 ]
        test_identities
    else 
        start_agent
    end  
end

I've spend far too long to get it running on WSL2 Ubuntu 20.04. Finally, we need to start ssh-agent on spawning new console, but don't load a key then. Load key upon first usage and use AddKeysToAgent.

Add following at the end of your ~/.bashrc or ~/.zshrc:

SSH_ENV="$HOME/.ssh/agent-environment"

function start_agent {
    /usr/bin/ssh-agent | sed 's/^echo/#echo/' > "${SSH_ENV}"
    chmod 600 "${SSH_ENV}"
    . "${SSH_ENV}" > /dev/null
}

if [ -f "${SSH_ENV}" ]; then
    . "${SSH_ENV}" > /dev/null
    #ps ${SSH_AGENT_PID} doesn't work under cywgin
    ps -ef | grep ${SSH_AGENT_PID} | grep ssh-agent$ > /dev/null || {
        start_agent;
    }
else
    start_agent;
fi

Note that /usr/bin/ssh-add; is deliberately missing here, comparing to original script.

And add following at the end of ~/.ssh/config file:

Host *
   AddKeysToAgent yes

I use this:

vim ~/.profile

eval `/usr/bin/gnome-keyring-daemon --start --components=pkcs11,secrets,ssh,gpg`
export SSH_AUTH_SOCK
export GPG_AGENT_INFO

If you use azure .ppk file

Just convert it to pem and add permission 400 with simple steps:

sudo apt-get install putty
puttygen <path_to_key>/keyname.ppk -O private-openssh -o <path>/aws_key.pem
sudo chmod 400 <path>/aws_key.pem
ssh -vi aws_key.pem ubuntu@<ip_address>

On Ubuntu 18.04, the ssh-agent is started when the session X is opened, it is managed in the file /etc/X11/Xsession.options:

# cat /etc/X11/Xsession.options
# $Id: Xsession.options 189 2005-06-11 00:04:27Z branden $
#
# configuration options for /etc/X11/Xsession
# See Xsession.options(5) for an explanation of the available options.
allow-failsafe
allow-user-resources
allow-user-xsession
use-ssh-agent
use-session-dbus

Alternate solution is use keychain.
Man page.

Keychain helps you to manage SSH and GPG keys in a convenient and secure manner. It acts as a frontend to ssh-agent and ssh-add, but allows you to easily have one long running ssh-agent process per system, rather than the norm of one ssh-agent per login session.

This dramatically reduces the number of times you need to enter your passphrase.