ssh and sshfs connection through nginx reverse proxy problems

I have a small annoying problem with ssh and sshfs connection to a server that is behind an nginx reverse proxy.

I use sshfs remotely to mount some folders from the server and ssh to connect to it and both get disconnected when idle. I solved the ssh connection by adding the ServerAliveInterval and ServerAliveCountMax in .ssh/config but the problem remains with sshfs.

I have used sshfs in the past (without the reverse proxy) and adding those two options to the fstab line and all was well, except in this case where it doesn't work. The folders are mounted, I can use them, but if they remain idle for a minute they disconnect.

I cannot figure this out! Is there a way to solve this? Is there some setting in the nginx that can also solve the ssh connection problem without those options (I have some people connecting to the server too that are not too technical and they complain about disconnects!)

Here is the ssh portion of the reverse proxy:

upstream server {
  server 10.10.0.xxx:SSH_INTERNAL_PORT;
}

server {
  listen SSH_EXTERNAL_PORT;
  proxy_pass server;
}

EDIT:

I forgot to mention that I have already set this in /etc/nginx/nginx.conf:

# ssh reverse proxy conf
stream {
    include /etc/nginx/ssh_enabled/*;
}