i have just a few months working as sysadmin, hence i still have lots to learn, first thing id like to do is as follows:
We have an OpenBSD 4.5 box acting like firewall,dns,cache etc, the box has 2 network cards, one conected directly to the internet and the other to our switch, i used to work with sarg for the log analysis but then changed to the much faster free-sa.
I use a daily free-sa report to check the bandwidth usage and report our top 5 bandwidth consumers (3 days a week being #1 and you will be buying the pizzas :D, we are a small company ~20 so we are very familiar). this was working really good until recently, one of us required to download some stuff via torrent (~3GB) and since the pizza rule is active for non-work related downloads, he told me (verified) that his download was indeed work related so i would dismiss that 3GB off his quota, but to my surprise the log didnt showed that 3GB, since his ip consumption was only around 290MB.
More recently, since the FIFA world cup started, we know that some of the employees are watching the match's streaming, we know it and we dont care about it since, like already stated, we are a small company so we dont have restrictive policies, we all can chat, watch youtube, download anything we want BUT we are only allowed 300MB a day otherwise you'll get in the top5-pizza-board, anyway, that streaming consumption is also not showing in the free-sa reports.
So my question is, why is these data being excluded from the reports? im thinking that the free-sa reports list only certain types of things but im also thinking if are the squid logs the ones that are not erm... logging these conections.
Any help, guide, advice or clarification is appreciated.
I'm thinking that squid is logging your web connections, anything proxied through it (liked port 80 requests for web pages). Torrents are not going through there, so they're not seen by the squid server.
If your router supports it, you might be better off pulling bandwidth usage stats from there with SNMP to get an idea of what kind of usage is going through your network in general, but it won't show individual usage.
Otherwise you'll have to look for a way to proxy torrent connections or run a firewall that uses logging by IP, maybe a linux or BSD based turnkey firewall with logging ability (smoothwall, etc.).
Yeah as others say torrent downloads are bypassing squid.
You should try something else. Producing netflow and analysing that (man pflow) is the best and most versatile solution. This requires some effort. Some tool available to build your solution http://www.switch.ch/network/projects/completed/TF-NGN/floma/software.html
One can choose an easire path instead like using
ntopto make reports or installingiptraf.