Can somebody give me ideas about following situation (Linux 2.6.18-348.4.1.el5 ):
- At some point tcpdump shows [FIN, ACK] packet sent from server port to local client
- strace log shows no socket activity performed on that socket handle for that port near that moment (strace log properly shows the rest of communication for that client)
- Firewall and SELinux are stopped
- Problem is 100% reproducible with complex conditions which require the Server to perform network request for Kerberos authentication of another client connection.
What are potential causes which may lead tcpdump to show packet which missing in strace? Does it look more like server problem, TCP settings problem or some firewall service problem?
Something triggers the closing of the connection on the server side. Most likely it is the application on the remote server that shuts down the connection. Possible reasons are idle disconnection feature on the server, software bugs etc.
If the remote server is behind NAT, it could also be caused by expiration of the NAT entry in the device doing the NAT.
You should run
tcpdump
on the server and see if theFIN
packet is originated from that server. If it is, then the likely culprit is the application. If not, then it is the from firewall.kasperd was right about strace not showing all threads of the process (somehow I was sure that attaching to process will monitor all threads). So auditd confirmed it is my application closing the socket (because some misterious "signal 33" happens and recv() doesn't handle EINTR properly. I will ask another question about this).