Youtube as we know, is massive. It has thousand of concurrent users streaming at least 2 megabytes per video. Obviously, that gets to be a lot of traffic... far too much for any one server.
What networking technologies allow pushing 4 billion videos a day?
My question is about Virtual Machines and delivering their content over the servers connection to the internet.
I have an Ec2 windows instance, and its network connection appears to be 100mbps
If I was to be delivering content from that EC2 instance, is THAT my potential bottleneck?
How does s3 differ, I am guess their is no real potential outbound bottleneck with s3?
Note : I know s3 and their CDN would be better for static content, however I need to explore this situation for now. Our HTML pages need to access a server side page via AJAX, and because there is no bombproof work around for this at the moment our content and our server needs to be on exactly the same domain, so it rules out using S3.
Bandwidth needed : I am not sure, we could have up to 100 users downloading videos at any time, probably no more. Videos can be up to 5mb each, but they would view up to 20.
To summarise: I have a dedicated server with a few friends running a torrent client with web gui. Each user is running a client under their username on the server so downloads go in their user dir, and only they have access to their own files etc.
How can I monitor and limit the bandwidth per month on a per user basis?
I was thinking there must be a way using iptables maybe. And by monitoring the bandwidth used by all processes of user X. And if they have used more then their monthly allowed bandwidth of Y GB, they get a message saying that and networking gets blocked for their torrent client, or the client gets killed completely. I also thought about squid, but seeing as it would be using multiple torrent clients this could use a lot of server resources...
I am using debian lenny.
I'm not sure how to do this...
would this be possible at all? I am grateful for even just partial solutions to this...
We were a little surprised to see this on our Cacti graphs for June 4 web traffic:
We ran Log Parser on our IIS logs and it turns out this was a perfect storm of Yahoo and Google bots indexing us.. in that 3 hour period, we saw 287k hits from 3 different Google IPs, plus 104k from Yahoo. Ouch?
While we don't want to block Google or Yahoo, this has come up before. We have access to a Cisco PIX 515E, and we're thinking about putting that in front so we can dynamically deal with bandwidth offenders without touching our web servers directly.
But is that the best solution? I'm wondering if there is any software or hardware that can help us identify and block excessive bandwidth use, ideally in real time? Perhaps some bit of hardware or open-source software we can put in front of our web servers?
We are mostly a Windows shop but we have some Linux skills as well; we're also open to buying hardware if the PIX 515E isn't sufficient. What would you recommend?
I have Apache set up to serve several Virtual Hosts, and I would like to see how much bandwidth each site uses. I can see how much the entire server uses, but I would like more detailed reports.
Most of the things I have found out there are for limiting bandwidth to virtual hosts, but I don't want to do that; I just want to see which sites are using how much bandwidth.
This isn't for billing purposes, just for information.
Is there an apache module I should use? Or is there some other way to do this?