Perhaps some server can handle both non-secure and secure protocols. Is there known way to determine if existing TCP connection secure or not?
(I believe it is possible just monitor traffic and check if some text is readable, but maybe there is better or maybe more detailed suggestions?)
I am assuming that you are packet-sniffing...
If you have all the packets from the beginning of the connection, you should be able to see from the first few packets what type of upper-layer protocol is being used (ssh/https/sftp).
If it is a plain TLS encrypted data stream, then you should be able to see the TLS handshake where the client and server agree on what protocol to use and exchange public keys for use in the encryption... From then on you know it is ssl encrypted, or some other data.
Your average packet sniffer such as wireshark can normally decode these initial messages, but obviously not the encrypted content...