Home / server / Questions / 535099
In Process
ddewaele
Asked: 2013-08-31 02:33:53 +0800 CST

Capacity planning : measuring network traffic per HTTP request

  • 772

I'm trying to figure out exactly how much network-data is required to perform a single HTTP request

I'm currently simulating the request using curl

curl -v  -X POST ${vehicles_endpoint}/updateMileageAndErrorCodes/abc123/9500/P0187,P0191,P0213

And capturing the response :

* About to connect() to localhost port 8088 (#0)
*   Trying ::1... connected
* Connected to localhost (::1) port 8088 (#0)
> POST /vdm/vehicles/updateMileageAndErrorCodes/abc123/15000/P0110 HTTP/1.1
> User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
> Host: localhost:8088
> Accept: */*
> 
< HTTP/1.1 204 No Content
< Server: Apache-Coyote/1.1
< Date: Wed, 28 Aug 2013 08:20:18 GMT
< 
* Connection #0 to host localhost left intact
* Closing connection #0

How can I find out the exact number of bytes that is required to do this request / response ?

I can look at the number of characters in the request (211) 

POST /vdm/vehicles/updateMileageAndErrorCodes/abc123/9500/P0187,P0191,P0213 HTTP/1.1
User-Agent: curl/7.21.4 (universal-apple-darwin11.0) libcurl/7.21.4 OpenSSL/0.9.8r zlib/1.2.5
Host: localhost:8088
Accept: */*

and the response (86) :

HTTP/1.1 204 No Content
Server: Apache-Coyote/1.1
Date: Wed, 28 Aug 2013 08:20:18 GMT

But I suppose there is some other protocol related overhead ?

If I do a tcpdump

sudo tcpdump -s 0 -X 'tcp dst port 80'

It gives me the following output :

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:41:47.442934 IP 34-188-5-188.mobileinternet.proximus.be.51905 > CentOS-63-64-minimal.http: Flags [S], seq 1277831461, win 13140, options [mss 1460,sackOK,TS val 3832792 ecr 0,nop,wscale 1], length
0
        0x0000:  4500 003c 52fd 4000 2906 b6e0 bc05 bc22  E..<R.@.)......"
        0x0010:  904c 3f6a cac1 0050 4c2a 2925 0000 0000  .L?j...PL*)%....
        0x0020:  a002 3354 1060 0000 0204 05b4 0402 080a  ..3T.`..........
        0x0030:  003a 7bd8 0000 0000 0103 0301            .:{.........
10:41:47.862952 IP 34-188-5-188.mobileinternet.proximus.be.51905 > CentOS-63-64-minimal.http: Flags [.], ack 197676124, win 6570, options [nop,nop,TS val 3832868 ecr 2984946753], length 0
        0x0000:  4500 0034 52fe 4000 2906 b6e7 bc05 bc22  E..4R.@.)......"
        0x0010:  904c 3f6a cac1 0050 4c2a 2926 0bc8 4c5c  .L?j...PL*)&..L\
        0x0020:  8010 19aa a223 0000 0101 080a 003a 7c24  .....#.......:|$
        0x0030:  b1ea ac41                                ...A
10:41:48.144006 IP 34-188-5-188.mobileinternet.proximus.be.51905 > CentOS-63-64-minimal.http: Flags [P.], seq 0:317, ack 1, win 6570, options [nop,nop,TS val 3832874 ecr 2984946753], length 317
        0x0000:  4500 0171 52ff 4000 2906 b5a9 bc05 bc22  E..qR.@.)......"
        0x0010:  904c 3f6a cac1 0050 4c2a 2926 0bc8 4c5c  .L?j...PL*)&..L\
        0x0020:  8018 19aa 63e2 0000 0101 080a 003a 7c2a  ....c........:|*
        0x0030:  b1ea ac41 504f 5354 202f 7664 6d2f 7665  ...APOST./vdm/ve
        0x0040:  6869 636c 6573 2f75 7064 6174 654d 696c  hicles/updateMil
        0x0050:  6561 6765 416e 6445 7272 6f72 436f 6465  eageAndErrorCode
        0x0060:  732f 6162 6331 3233 2f39 3530 302f 5030  s/abc123/9500/P0
        0x0070:  3138 372c 5030 3139 312c 5030 3231 3320  187,P0191,P0213.
        0x0080:  4854 5450 2f31 2e31 0d0a 5573 6572 2d41  HTTP/1.1..User-A
        0x0090:  6765 6e74 3a20 4461 6c76 696b 2f31 2e36  gent:.Dalvik/1.6
        0x00a0:  2e30 2028 4c69 6e75 783b 2055 3b20 416e  .0.(Linux;.U;.An
        0x00b0:  6472 6f69 6420 342e 333b 2047 616c 6178  droid.4.3;.Galax
        0x00c0:  7920 4e65 7875 7320 4275 696c 642f 4a57  y.Nexus.Build/JW
        0x00d0:  5236 3659 290d 0a48 6f73 743a 2076 646d  R66Y)..Host:.vdm
        0x00e0:  2d61 7069 2e65 636f 6d6d 6974 2d63 6f6e  -api.ecommit-con
        0x00f0:  7375 6c74 696e 672e 6265 0d0a 436f 6e6e  sulting.be..Conn
        0x0100:  6563 7469 6f6e 3a20 4b65 6570 2d41 6c69  ection:.Keep-Ali
        0x0110:  7665 0d0a 4163 6365 7074 2d45 6e63 6f64  ve..Accept-Encod
        0x0120:  696e 673a 2067 7a69 700d 0a43 6f6e 7465  ing:.gzip..Conte
        0x0130:  6e74 2d54 7970 653a 2061 7070 6c69 6361  nt-Type:.applica
        0x0140:  7469 6f6e 2f78 2d77 7777 2d66 6f72 6d2d  tion/x-www-form-
        0x0150:  7572 6c65 6e63 6f64 6564 0d0a 436f 6e74  urlencoded..Cont
        0x0160:  656e 742d 4c65 6e67 7468 3a20 300d 0a0d  ent-Length:.0...
        0x0170:  0a                                       .
10:41:49.603082 IP 34-188-5-188.mobileinternet.proximus.be.51905 > CentOS-63-64-minimal.http: Flags [.], ack 162, win 6490, options [nop,nop,TS val 3833096 ecr 2984948484], length 0
        0x0000:  4500 0034 5300 4000 2906 b6e5 bc05 bc22  E..4S.@.)......"
        0x0010:  904c 3f6a cac1 0050 4c2a 2a63 0bc8 4cfd  .L?j...PL**c..L.
        0x0020:  8010 195a 98ee 0000 0101 080a 003a 7d08  ...Z.........:}.
        0x0030:  b1ea b304                                ....
10:41:49.622962 IP 34-188-5-188.mobileinternet.proximus.be.51905 > CentOS-63-64-minimal.http: Flags [F.], seq 317, ack 163, win 6490, options [nop,nop,TS val 3833098 ecr 2984948484], length 0
        0x0000:  4500 0034 5301 4000 2906 b6e4 bc05 bc22  E..4S.@.)......"
        0x0010:  904c 3f6a cac1 0050 4c2a 2a63 0bc8 4cfe  .L?j...PL**c..L.
        0x0020:  8011 195a 98ea 0000 0101 080a 003a 7d0a  ...Z.........:}.
        0x0030:  b1ea b304                                ....

I noticed the full length of the request (including all http request headers) to be 317 but I also see a lot of length:0 outputs.

How can I figure out the exact number of bytes transferred for this request ? (I used my android phones data usage tool and noticed that it was around 900 bytes per request).

networking

6 Answers

  1. The big-daddy to figure out the total contents of a single request (including external files that your main page calls), check out Pingdom's Website Speed Test

    You can use curl -I http://sitename, or wget the site.

    curl -I only displays the headers (including Content-Length) but discards the file, where wget actually and saves the main site file (such as index.html). This gets you the size of the main file, but not all of the files that that page also calls.

    You may also try using the Chrome Dev tools to gather some insight.

    • 1

  2. try tcpdump -v to get full-packet length. What you get here (without "-v") is payload size (equal to packet size minus ethernet/IP/TCP headers).

    Then just sum up the total over the full communication (filtering "src host " might help narrow down stream)

    It isn't clear wether you're intrested in counting only outgoing packets or both ingoing and outgoing.

    • 1

  3. You can try the HttpAnalyzer utility as part of PcapPlusPlus project. This utility captures HTTP data and collects a lot of stats about it. Between the rest it collects the HTTP requests/responses size. You can read more here: http://seladb.github.io/PcapPlusPlus-Doc/examples.html

    • 1

  4. Use tshark/wireshark to capture entire traffic and analyze it visually - will be easier for you.

    • 0

  5. The tool iftop (man iftop) displays bandwidth usage on an interface by host, cumulative statistics of transmitted/received/total bytes and it supports to define well-known PCAP filters (like with tcpdump) to count only specific traffic. You can try to monitor http traffic with simple command like

    iftop -P -i eth0 -f "dst port http"

    By default, when iftop starts the upper part of the screen shows current bandwidth usage between client(s) and server(s) while the lower part shows the cumulative statistics. You can change this by creating a configuration file in ~/.iftoprc or by specifying it with -c option.

    • 0

  6. Using ifconfig you can see how many bytes have been sent and received on each network interface. You could run ifconfig before and after the test and subtract the number of bytes before from the number of bytes after.

    • 0