Does it makes sense from a security perspective to remove the Server HTTP header?

RFC 7231 says about the Server header:

An origin server MAY generate a Server field in its responses.

MAY is interpreted as in RFC 2119:

5. MAY This word, or the adjective "OPTIONAL", mean that an item is truly optional. One vendor may choose to include the item because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same item. An implementation which does not include a particular option MUST be prepared to interoperate with another implementation which does include the option, though perhaps with reduced functionality. In the same vein an implementation which does include a particular option MUST be prepared to interoperate with another implementation which does not include the option (except, of course, for the feature the option provides.)

It is therefore perfectly fine to restrict or remove the Server response header. Of course it's a good idea to be aware of what you might be giving up by doing so. For that, go back to RFC 7231:

The "Server" header field contains information about the software used by the origin server to handle the request, which is often used by clients to help identify the scope of reported interoperability problems, to work around or tailor requests to avoid particular server limitations, and for analytics regarding server or operating system use. An origin server MAY generate a Server field in its responses.

Server = product *( RWS ( product / comment ) )

The Server field-value consists of one or more product identifiers, each followed by zero or more comments (Section 3.2 of [RFC7230]), which together identify the origin server software and its significant subproducts. By convention, the product identifiers are listed in decreasing order of their significance for identifying the origin server software. Each product identifier consists of a name and optional version, as defined in Section 5.5.3.

Example:

Server: CERN/3.0 libwww/2.17

An origin server SHOULD NOT generate a Server field containing needlessly fine-grained detail and SHOULD limit the addition of subproducts by third parties. Overly long and detailed Server field values increase response latency and potentially reveal internal implementation details that might make it (slightly) easier for attackers to find and exploit known security holes.

Though in practice, attackers don't really check the Server: header. They just try every security exploit they know of, whether your server gives any indication of being vulnerable or not. Removing the Server: header is a security by obscurity action, and an almost entirely ineffective one. But if it makes you feel better, or you're being told to do it by your boss or an auditor, go for it. Just don't expect it to result in any significant improvement to your security posture.

For example, nmap can identify a web server with fairly good accuracy even when it's configured to not send a Server header at all, or when the header content is completely bogus. Try it yourself with nmap -sV -P0 -p 80,443 <IP address>.

The Server HTTP header only serves one purpose - identification. It is not required anywhere for running your website properly, and by removing it, nothing is going to break.

It reveals the internal server infrastructure, and thus leaks security information that may be useful for potential intrusion.

After gaining knowledge of your web server software, via Server HTTP header, the potential intruder can search your web server's publicly known vulnerabilities. Then they can use this information in conjunction with any other information they might gain (e.g. through scanning) - to build a proper attack vector.

Thus, you may want to remove the Server header altogther, e.g. remove it in nginx.

If you have "server_tokens off" in your config (and it seems you do since there's only 'nginx' and not say 'nginx/1.13.11'), then it's OK to leave things as they are now. The problem could appear if you have a vulnerable version, and a bad person could use this info to exploit the vulnerability, but for that your nginx should be publicly accessible. So in a nutshell, use "server_tokens off;" and do no open nginx port for all IPs but for Varnish only, and you should be safe.