process zombifies while trying to connect on server then disappears

I have connection problems on my server:

• often, ssh hangs when trying to connect
• sometimes, other commands take too much time, like su - someuser
• when I restart the daemon, it quickly listens on 127.0.0.1, but it takes very much time to listen on our VPN ip address
• when I try to edit the sshd config file, it takes time to get the file to display in vi

I have tried running ssh with more verbosity, and I can see that it hangs after the following message :

debug1: Entering interactive session.

When it hangs and I already have a session opened on the server, I can look at the process list, and I can see like this :

root     19835  0.0  0.0  90308  3948 ?        Ss   10:25   0:00  \_ sshd: root [priv]   
sshd     19836  0.0  0.0      0     0 ?        Z    10:25   0:00      \_ [sshd] <defunct>

When I finally login, the defunct process has disappears and the process with [priv] has a sane child :

root     19835  0.0  0.0  91524  4132 ?        Ss   10:25   0:00  \_ sshd: root@pts/3    
root     20898  3.3  0.1  24244  5264 pts/3    Ss+  10:27   0:00      \_ -bash

Here is what I get after the message that hangs:

debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug3: Ignored env IM_CONFIG_PHASE
debug3: Ignored env GNOME_KEYRING_CONTROL
debug3: Ignored env KONSOLE_DBUS_WINDOW
debug3: Ignored env SESSIONTYPE
debug3: Ignored env XCURSOR_THEME
debug3: Ignored env GTK_MODULES
debug3: Ignored env SESSION
debug3: Ignored env DEFAULTS_PATH
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env XDG_GREETER_DATA_DIR
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env KDE_SESSION_UID
debug3: Ignored env PATH
debug3: Ignored env GNOME_KEYRING_PID
debug3: Ignored env KDE_FULL_SESSION
debug3: Ignored env CLUTTER_IM_MODULE
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env UPSTART_INSTANCE
debug3: Ignored env XDG_VTNR
debug3: Ignored env QT_PLUGIN_PATH
debug3: Ignored env SELINUX_INIT
debug3: Ignored env LANGUAGE
debug3: Ignored env GDM_LANG
debug3: Ignored env GS_LIB
debug3: Ignored env SHELL_SESSION_ID
debug3: Ignored env XAUTHORITY
debug3: Ignored env TEXTDOMAIN
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env GTK_IM_MODULE
debug3: Ignored env KDE_MULTIHEAD
debug1: Sending env LANG = fr_FR.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env SPEECHD_PORT
debug3: Ignored env KONSOLE_DBUS_SERVICE
debug3: Ignored env LOGNAME
debug3: Ignored env PWD
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env KONSOLE_PROFILE_NAME
debug3: Ignored env HOME
debug3: Ignored env GTK2_RC_FILES
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env XDG_SEAT
debug3: Ignored env UPSTART_JOB
debug3: Ignored env QT4_IM_MODULE
debug3: Ignored env XMODIFIERS
debug3: Ignored env MANDATORY_PATH
debug3: Ignored env QT_IM_MODULE
debug3: Ignored env WINDOWID
debug3: Ignored env KDE_SESSION_VERSION
debug3: Ignored env COLORFGBG
debug3: Ignored env DISPLAY
debug3: Ignored env KONSOLE_DBUS_SESSION
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env UPSTART_SESSION
debug3: Ignored env SHELL
debug3: Ignored env JOB
debug3: Ignored env TEXTDOMAINDIR
debug3: Ignored env SSH_AGENT_LAUNCHER
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env NODE_PATH
debug3: Ignored env QT_QPA_PLATFORMTHEME
debug3: Ignored env GTK_RC_FILES
debug3: Ignored env PROFILEHOME
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env TERM
debug3: Ignored env INSTANCE
debug3: Ignored env GDMSESSION
debug3: Ignored env USER
debug3: Ignored env UPSTART_EVENTS
debug3: Ignored env SHLVL
debug3: Ignored env OLDPWD
debug3: Ignored env GREP_OPTIONS
debug3: Ignored env GREP_COLOR
debug3: Ignored env PAGER
debug3: Ignored env LESS
debug1: Sending env LC_CTYPE = fr_FR.UTF-8
debug2: channel 0: request env confirm 0
debug3: Ignored env LSCOLORS
debug3: Ignored env EDITOR
debug3: Ignored env OLDHISTFILE
debug3: Ignored env OLDPATH
debug3: Ignored env rvm_prefix
debug3: Ignored env rvm_path
debug3: Ignored env __array_start
debug3: Ignored env rvm_uname
debug3: Ignored env rvm_tar
debug3: Ignored env rvm_bin_path
debug3: Ignored env escape_flag
debug3: Ignored env _first
debug3: Ignored env _second
debug3: Ignored env rvm_version
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0

What could be the problem ?

UPDATE

strace su toto outputs hangs after the following :

socket(PF_FILE, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_FILE, path="/dev/log"}, 110) = 0
sendto(3, "<86>Jul 25 11:19:39 su[28347]: S"..., 61, MSG_NOSIGNAL, NULL, 0) = 61
sendto(3, "<86>Jul 25 11:20:14 su[28347]: +"..., 53, MSG_NOSIGNAL, NULL, 0