Home / server / Questions / 273814
Accepted
hfranco
Asked: 2011-05-26 13:03:28 +0800 CST

Gathering web page statistics in a Bash script

  • 772

I need to gather statistics on how long it takes to retrieve a web page once every couple of seconds.

I can do a 

time wget --spider http://www.google.com/index.html

(spider will not download the page, just check that they are there)

with this command I can see how long it took to run the command and the status of the page (200 OK, 404 NOT FOUND, etc)

The issue I'm having is that I need to keep track of the statistics. So if I hit a web page every couple of seconds and every so often I get a 404 I need to see those stats.

linux bash web

3 Answers

  1. What do you think about this?

    curl --write-out %{time_total} --head google.ca/ >> ~/stats.txt

    This will give you an output like:

    HTTP/1.1 301 Moved Permanently
Location: http://www.google.ca/
Content-Type: text/html; charset=UTF-8
Date: Wed, 25 May 2011 21:24:45 GMT
Expires: Fri, 24 Jun 2011 21:24:45 GMT
Cache-Control: public, max-age=2592000
Server: gws
Content-Length: 218
X-XSS-Protection: 1; mode=block

0.076

    Which you can then parse into a spreadsheet. It may, admittedly, not be the most elegant solution. You can clean it up with a grep chain (like grep -v Location: | grep -v Content-Type: | grep -v Date etc) or something more elegant. I'm curious to see what others come up with!

    • 2

  2. Check out Nagios for monitoring, or even if you don't want a full Nagios install check out the Nagios plugins, check_http will give you performance stats as well as a simple parse-able output. You could use a shell script to put this into a file which you could process later.

    • 1
  3. Best Answer

    You could try something like this to get you going

    #!/bin/bash
URL="http://serverfault.com/"
Result=$((time wget --spider "$URL") 2>&1  | egrep  'real|response')
NumFields=$(echo $Result | awk '{print NF}')

#16 Fields if there was a 302 redirect and the result is in $13
#9 fields for 200,404 the result in in $6

if [ $NumFields -eq 16 ]
then
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $13,$NF}')
else
   Stats=$(echo $Result | awk 'BEGIN {OFS=",";} {print $6,$NF}')
fi
# Outputs YYYYMMDDHHMMSS,URL,Response,Time Taken
# 20110526180254,http://www.google.oom/,302,0m1.000s
# 20110526180928,http://serverfault.com/,200,0m0.225s
# 20110526181041,http://www.google.com/fred/,404,0m0.089

echo $(date +"%Y%m%d%H%M%S"),"$URL","$Stats"

    If you use >> to redirect the output to a file you can then either pull it into a spreadsheet or use grep etc to pull information from it.

    • 1