Is there an HTTP status code which is appropriate to use for clients which send a bad hostname (or none at all) through SNI or the HTTP Host header?
An older question address how and why such requests happen in the first place as well as how you can technically deal with them in Apache. It does however not address the choice of status code for the response.
In the past I have implemented an HTTP proxy which would send status code 502 and an html page explaining why the error message was produced. My rationale for using 502 was an expectation that I would mainly see it due to misconfigurations which meant the proxy could not find a suitable backend. In reality it turned out to be much more frequent to simply see completely bogus hostnames.
Is there another status code which is more appropriate and clearly signals to the client that the server on this IP address does not recognize the value sent through SNI and/or the Host header?
RFC 6066 doesn't specify or even recommend any particular HTTP error in the case that the hostname sent via SNI doesn't match the HTTP Host header. It does recommend that the server abort the TLS handshake if the SNI hostname is not one that it provides service for. From section 3:
Since such a malformed request can get past the TLS handshake and need to be rejected in HTTP, an HTTP response code is necessary. Of all those that exist, only one really fits the situation:
This is, in fact, the response that RFC 7230 specifies. From section 5.4 describing the Host header:
I'm going to recommend strongly against using 502 for this. Its semantics indicate that something is wrong on the server side and that the request would succeed if tried later.