In Apache prefork server running PHP application on Linux, each HTTP request might fetch some data from other server, for instance Memcached. Let's say we want to find out top 10 HTTP requests (ex. GET /blabla) that take the most bandwidth to Memcached. Is there any way to monitor how much outgoing bandwidth was taken per HTTP request without modifying the PHP application?
My approach was using tcpdump and netstat or /proc/net/tcp. With the below command, it'd collect incoming HTTP traffic and outgoing one. But I can't tell which particular HTTP request makes which Memcached connections easily. While it's Apache prefork, PID for every HTTP request can be logged, though tcpdump doesn't know which process made those packets.
tcpdump -i any src port 80 or port 11211 -w tcpdump.log
With netstat, often I find PID information is missing in its output.
netstat -tpc >netstat.log
To map PID to connected sockets, I can possibly scan /proc and get process id and local ports. So, I have a pair of PID and a list of local ports that connected to Memcached. By searching those port numbers in tcpdump.log, I get the length of the packets.
I'm not sure if scanning the entire directory every second or even more frequently would be efficient while the Apache spawns 100+ processes in my case. Thought there might be a better way.