Solaris 10 using snoop to capture packets.
When I telnet to an unused IP address, and wait for the connection attempt to time out, everything behaves as expected (SYNs sent several times until connection establishment time out exceeded, then telnet quits).
When do the same thing, but cancel the attempt before it reaches the time out, with control-c, the last packet has a RSET.
Is this done for a reason? Maybe the network stack is assuming that the SYNs were successful, even if the SYN-ACK didn't come back, so it sends a RSET to keep the remote system from being left half-open?
I had read Stevens TCP/IP Illustrated, Vol 1 a long time ago, and I don't recall this being discussed (I checked with a quick google, but didn't dig up the book).
As you suspect - it's as a just-in-case measure to notify the remote host to abort the connection attempt; for all your system knows, the host sent back a SYNACK and you didn't get it.
It's basically a "hey, remote host, if you tried to connect back to me, you can go ahead and stop".
TCP doesn't allow for a session to be in an ambiguous state, whether that session is established or in the process of being established. It needs to terminate the connection or the connection attempt gracefully in the sense that it won't just abort the connection without sending either a FIN or a RST (as far as I know).
That's not to say that a session can never be in an ambiguous state due to factors outside the purview of TCP, such as a broken link which causes a half-open on one end of the session. From the point of view of TCP, a half-open connection is simply a connection that is long idle (unless TCP keep-alive is implemented).