How to find other end of unix socket connection?

This answer is for Linux only.

Update for Linux 3.3: As Zulakis wrote in a separate answer (+1 that), you can use ss from iproute2 to get a pair of inode numbers for each socket connection identifying local end and peer. This appears to be based on the same machinery as sock_diag(7) with the UNIX_DIAG_PEER attribute identifying the peer. An answer by Totor over at Unix & Linux Stack Exchange links to the relevant commits in kernel and iproute2 and also mentions the need for the UNIX_DIAG kernel config setting.

Original answer for Linux pre 3.3 follows.

Based on an answer from the Unix & Linux Stack Exchange, I successfully identified the other end of a unix domain socket using in-kernel data structures, accessed using gdb and /proc/kcore. You need to enable the CONFIG_DEBUG_INFO and CONFIG_PROC_KCORE kernel options.

You can use lsof to get the kernel address of the socket, which takes the form of a pointer, e.g. 0xffff8803e256d9c0. That number is actually the address of the relevant in-kernel memory structure or type struct unix_sock. That structure has a field called peer which points at the other end of the socket. So the commands

# gdb /usr/src/linux/vmlinux /proc/kcore
(gdb) p ((struct unix_sock*)0xffff8803e256d9c0)->peer

will print the address of the other end of the connection. You can grep the output of lsof -U for that number to identify the process and file descriptor number of that other end.

Some distributions seem to provide kernel debug symbols as a separate package, which would take the place of the vmlinux file in the above command.

Quite recently I stumbled upon a similar problem. I was shocked to find out that there are cases when this might not be possible. I dug up a comment from the creator of lsof (Vic Abell) where he pointed out that this depends heavily on unix socket implementation. Sometimes so called "endpoint" information for socket is available and sometimes not. Unfortunatelly it is impossible in Linux as he points out.

On Linux, for example, where lsof must use /proc/net/unix, all UNIX domain sockets have a bound path, but no endpoint information. Often there is no bound path. That often makes it impossible to determine the other endpoint, but it is a result of the Linux /proc file system implementation.

If you look at /proc/net/unix you can see for yourself, that (at least on my system) he is absolutelly right. I'm still shocked, because I find such feature essential while tracking server problems.

Actually, ss from iproute2 (replacement for netstat, ifconfig, etc.) can show this information.

Here is an example showing an ssh-agent unix domain socket to which a ssh process has connected:

$ sudo ss -a --unix -p
Netid  State      Recv-Q Send-Q Local                             Address:Port          Peer    Address:Port
u_str  ESTAB      0      0      /tmp/ssh-XxnMh2MdLBxo/agent.27402 651026                *       651642                users:(("ssh-agent",pid=27403,fd=4)
u_str  ESTAB      0      0       *                                651642                *       651026                users:(("ssh",pid=2019,fd=4))

Unix sockets usually are assigned numbers in pairs, and are usually consecutive. So the pair for you would likely be 1013410+/-1. See which of those two exists and guess at the culprit.

I wrote a tool which uses MvG's gdb method to reliably get socket peer information, kernel debug symbols not needed.

To get the process connected to a given socket, pass it the inode number:

# socket_peer 1013410
3703 thunderbird 

To find out for all processes at once use netstat_unix, it adds a column to netstat's output:

# netstat_unix
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Peer PID/Program name  Path
unix  3      [ ]         STREAM     CONNECTED     6825     982/Xorg             1497/compiz            /tmp/.X11-unix/X0
unix  3      [ ]         STREAM     CONNECTED     6824     1497/compiz          982/Xorg                 
unix  3      [ ]         SEQPACKET  CONNECTED     207142   3770/chromium-brows  17783/UMA-Session-R       
unix  3      [ ]         STREAM     CONNECTED     204903   1523/pulseaudio      3703/thunderbird       
unix  3      [ ]         STREAM     CONNECTED     204902   3703/thunderbird     1523/pulseaudio           
unix  3      [ ]         STREAM     CONNECTED     204666   1523/pulseaudio      3703/thunderbird       
...

Try netstat_unix --dump if you need output that's easy to parse.
See https://github.com/lemonsqueeze/unix_sockets_peers for details.

For info, the inode +1/-1 hack isn't reliable. It works most of the time but will fail or (worse) return the wrong socket if you're out of luck.

Edit your system.conf

In this file you could add more stuff for debugging purpose.

File location: /etc/dbus-1/system.conf

For debugging purpose, you can edit your system.conf to allow eavesdropping:

1. replace the policy section by:

   <policy context="default">

   <!-- Allow everything to be sent -->

   <allow send_destination="*" eavesdrop="true"/>

   <!-- Allow everything to be received -->

   <allow eavesdrop="true"/>

   <!-- Allow anyone to own anything -->

   <allow own="*"/>

   <!-- XXX: Allow all users to connect -->

   <allow user="*"/> </policy>

2. Remove the includedir line: system.d

   <includedir>system.d</includedir>

Source: http://old.nabble.com/dbus-send-error-td29893862.html

Some other useful stuff regarding unix sockets

The simplest way to figure out what's happening on the bus is to run the dbus-monitor program, which comes with the D-Bus package

Also you can try to use dbus-cleanup-sockets to clean up leftover sockets.

Following command will show you which process is connected how many times to dbus sockets based on netstat output:

sudo netstat -nap | grep dbus | grep CONNECTED | awk '{print $8}' | sort | uniq -c

(tested on Ubuntu)

Hardcore way: This command will find manually the processes from /proc and show which are using the most connections (all type of sockets):

ls -lR */fd/* | grep socket | sed -r "s@([0-9{1}]+)/fd/@_\1_@g" | awk -F_ '{print $2}' | uniq -c | sort -n | awk '{print $1" "$2; print system("ps "$2"|tail -n1")}'

Example output:

(count, PID and the next line contains details about the process)

25 3732
 3732 ?        Ss     0:38 /usr/bin/wineserver
89 1970
 1970 ?        Ss     0:02 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

(tested on Ubuntu)

Have fun.

See also related articles for the reference:

• Linux API to determine sockets owned by a process
• /proc/PID/fd/X link number