Git access fails with "Permission denied, please try again." after a client changed their system processor.
Is there a way to check to see if SSHD is getting the proper auth / key info from GitLab's server? This seems to be the crux of the issue.
This was working fine for the user on my GitLab server for several months and then after the CPU change they were no longer able to login. No such problems happen for them on GitHub or other git servers, so they're thinking it could be server related.
It fails with a simple git ls-remote (to emulate the first step in deployment of a project update):
ssh -vv -p 43210 -A [email protected] 'git ls-remote [email protected]:projects/project_name.git'
It gets onto the server (hostname1.tld) just fine (for example, if they issue ls instead of the git ls-remote command there the remote server directory contents are returned). However, when it executes the git ls-remote command, we see the following:
...
debug1: Authentication succeeded (publickey).
Authenticated to hostname1.tld ([hostname1.tld]:43210).
debug1: setting up multiplex master socket
debug2: fd 4 setting O_NONBLOCK
debug1: channel 0: new [/tmp/[email protected]:43210]
debug1: channel 1: new [client-session]
debug2: channel 1: send open
debug1: Entering interactive session.
debug1: pledge: id
debug1: client_input_global_request: rtype [email protected] want_reply 0
debug2: channel_input_open_confirmation: channel 1: callback start
debug1: Requesting authentication agent forwarding.
debug2: channel 1: request [email protected] confirm 0
debug2: fd 3 setting TCP_NODELAY
debug2: client_session2_setup: id 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 1: request env confirm 0
debug1: Sending env LC_TERMINAL = iTerm2
debug2: channel 1: request env confirm 0
debug1: Sending env LC_TERMINAL_VERSION = 3.3
debug2: channel 1: request env confirm 0
debug1: Sending env LC_CTYPE = en_US.UTF-8
debug2: channel 1: request env confirm 0
debug1: Sending command: git ls-remote [email protected]:projects/project_name.git
debug2: channel 1: request exec confirm 1
debug2: channel_input_open_confirmation: channel 1: callback done
debug2: channel 1: open confirm rwindow 0 rmax 32768
debug2: channel 1: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 1
debug2: exec request accepted on channel 1
debug1: client_input_channel_open: ctype [email protected] rchan 2 win 65536 max 16384
debug2: fd 8 setting O_NONBLOCK
debug1: channel 2: new [authentication agent connection]
debug1: confirm [email protected]
debug2: channel 2: rcvd eof
debug2: channel 2: output open -> drain
debug2: channel 2: obuf empty
debug2: channel 2: chan_shutdown_write (i0 o1 sock 8 wfd 8 efd -1 [closed])
debug2: channel 2: output drain -> closed
debug1: channel 2: FORCE input drain
debug2: channel 2: ibuf empty
debug2: channel 2: send eof
debug2: channel 2: input drain -> closed
debug2: channel 2: send close
debug2: channel 1: rcvd ext data 38
Permission denied, please try again.
debug2: channel 1: written 38 to efd 7
debug2: channel 1: rcvd ext data 38
Permission denied, please try again.
debug2: channel 1: written 38 to efd 7
debug2: channel 1: rcvd ext data 41
Permission denied (publickey,password).
debug2: channel 1: written 41 to efd 7
debug1: client_input_channel_req: channel 1 rtype exit-status reply 0
debug1: client_input_channel_req: channel 1 rtype [email protected] reply 0
debug2: channel 1: rcvd eow
debug2: channel 1: chan_shutdown_read (i0 o0 sock -1 wfd 5 efd 7 [write])
debug2: channel 1: input open -> closed
debug2: channel 1: rcvd ext data 126
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
On the GitLab server itself (hostname2.tld) with ssh logging turned up to DEBUG, the following can be seen in the auth.log:
gitlab sshd[8053]: debug1: Forked child 8075.
gitlab sshd[8075]: debug1: Set /proc/self/oom_score_adj to 0
gitlab sshd[8075]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
gitlab sshd[8075]: debug1: inetd sockets after dupping: 3, 3
gitlab sshd[8075]: Connection from hostname1.tld port 38412 on hostname2.tld port 22
gitlab sshd[8075]: debug1: Client protocol version 2.0; client software version OpenSSH_7.4
gitlab sshd[8075]: debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
gitlab sshd[8075]: debug1: Enabling compatibility mode for protocol 2.0
gitlab sshd[8075]: debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
gitlab sshd[8075]: debug1: permanently_set_uid: 108/65534 [preauth]
gitlab sshd[8075]: debug1: list_hostkey_types: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_KEXINIT sent [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_KEXINIT received [preauth]
gitlab sshd[8075]: debug1: kex: algorithm: [email protected] [preauth]
gitlab sshd[8075]: debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
gitlab sshd[8075]: debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none [preauth]
gitlab sshd[8075]: debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none [preauth]
gitlab sshd[8075]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
gitlab sshd[8075]: debug1: rekey after 134217728 blocks [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
gitlab sshd[8075]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
gitlab sshd[8075]: debug1: SSH2_MSG_NEWKEYS received [preauth]
gitlab sshd[8075]: debug1: rekey after 134217728 blocks [preauth]
gitlab sshd[8075]: debug1: KEX done [preauth]
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method none [preauth]
gitlab sshd[8075]: debug1: attempt 0 failures 0 [preauth]
gitlab sshd[8075]: debug1: PAM: initializing for "git"
gitlab sshd[8075]: debug1: PAM: setting PAM_RHOST to "hostname1.tld"
gitlab sshd[8075]: debug1: PAM: setting PAM_TTY to "ssh"
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 1 failures 0 [preauth]
gitlab sshd[8075]: Failed none for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 2 failures 1 [preauth]
gitlab sshd[8075]: Failed password for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: debug1: userauth-request for user git service ssh-connection method password [preauth]
gitlab sshd[8075]: debug1: attempt 3 failures 2 [preauth]
gitlab sshd[8075]: Failed password for git from hostname1.tld port 38412 ssh2
gitlab sshd[8075]: Connection closed by hostname1.tld port 38412 [preauth]
gitlab sshd[8075]: debug1: do_cleanup [preauth]
gitlab sshd[8075]: debug1: monitor_read_log: child log fd closed
gitlab sshd[8075]: debug1: do_cleanup
gitlab sshd[8075]: debug1: PAM: cleanup
gitlab sshd[8075]: debug1: Killing privsep child 8076
gitlab sshd[8075]: debug1: audit_event: unhandled event 12
It appears that it's failing at this error above: Failed password for git
Any idea of what's happening here or how to troubleshoot further?
0 Answers