linkedlinked's questions

I'm building an analytics package, and project requirements state that I need to support 1 billion hits per day. Yep, "billion". In other words, no less than 12,000 hits per second sustained, and preferably some room to burst. I know I'll need multiple servers for this, but I'm trying to get maximum performance out of each node before "throwing more hardware at it".

Right now, I have the hits-tracking portion completed, and well optimized. I pretty much just save the requests straight into Redis (for later processing with Hadoop). The application is Python/Django with a gunicorn for the gateway.

My 2GB Ubuntu 10.04 Rackspace server (not a production machine) can serve about 1200 static files per second (benchmarked using Apache AB against a single static asset). To compare, if I swap out the static file link with my tracking link, I still get about 600 requests per second -- I think this means my tracker is well optimized, because it's only a factor of 2 slower than serving the same static asset repeatedly.

However, when I benchmark with millions of hits, I notice a few things --

1. No disk usage -- this is expected, because I've turned off all Nginx logs, and my custom code doesn't do anything but save the request details into Redis.
2. Non-constant memory usage -- Presumably due to Redis' memory managing, my memory usage will gradually climb up and then drop back down, but it's never once been my bottleneck.
3. System load hovers around 2-4, the system is still responsive during even my heaviest benchmarks, and I can still manually view http://mysite.com/tracking/pixel with little visible delay while my (other) server performs 600 requests per second.
4. If I run a short test, say 50,000 hits (takes about 2m), I get a steady, reliable 600 requests per second. If I run a longer test (tried up to 3.5m so far), my r/s degrades to about 250.

My questions --

a. Does it look like I'm maxing out this server yet? Is 1,200/s static files nginx performance comparable to what others have experienced?

b. Are there common nginx tunings for such high-volume applications? I have worker threads set to 64, and gunicorn worker threads set to 8, but tweaking these values doesn't seem to help or harm me much.

c. Are there any linux-level settings that could be limiting my incoming connections?

d. What could cause my performance to degrade to 250r/s on long-running tests? Again, the memory is not maxing out during these tests, and HDD use is nil.

Thanks in advance, all :)

EDIT Here is my nginx config -- http://pastie.org/1450749 -- it's mostly vanilla, with obvious fat trimmed out.

My Bash-Foo is not strong. Right now I have something like

function update_project {
  for i in server-{1,2,3,4} ; do
    echo "Updating $i"
    ssh $i "git pull"
  done
}

The number of servers is growing every day, and since each update takes about 20 seconds, I'd like to do the requests concurrently. What's the best way to do this, while still being able to see the output (e.g. failed merges)?

My friend runs a popular Youtube-to-GIF conversion site. Right now, he has converted 250,000 Youtube videos to GIFs (each video gets 6 thumbnails for 1.5m total GIF files) and serves about 80TB of bandwidth per month.

His server is IO blocking -- I'm not a guru admin, but it seems to be the harddrive seek time for non-sequential GIFs that's clogging everything up. He has a server with 100tb.com for $300/mo, and it comes with 100TB free bandwidth. At first, I advised him to get a CDN to solve his problems, because then the GIFs get served without consuming his server resources, and his main box could just handle the encoding -- We found one CDN for $600/mo that was too slow/unreliable, and the rest wanted at least $2000/mo for 80TB of bandwidth. We're trying to keep the whole project under $900/mo, right now.

So the cheapest bandwidth we can find is with 100TB, but we're outgrowing one server. We could add another server, but I don't really know how to partition the GIF storage so that the load is distributed evenly between two boxes. Our host recommended using software like Aflexi.net, but I'm sure there must be a cheaper solution.

Can anyone help? I'm a programmer by trade, not a sysadmin, but trying to learn the ropes. Thanks!

I'm trying to make a bash function as follows:

1. Will SSH into a server and chdir to ~/projects
2. If you pass extra arguments ('git pull'), these will be executed. If not, skip step 2
3. Leaves you with a bash shell

Right now, I have this:

function xyz {
   ssh -t xyz.com 'cd ~/projects; $*; bash'
}

Using this, running 'xyz' leaves me with a shell at xyz.com:~/projects, just like I want, but running 'xyz git pull' yields the following error:

/usr/bin/git: /usr/bin/git: cannot execute binary file

I'm sure I'm missing something simple, can anyone point me in the right direction?

Thanks!

I'm not normally a sysadmin, but I've got a production server under heavy load (serving some basic php pages, and some php redirect files that have some sql queries, and no images) that keeps crashing. Specifically, the load gets up to about 20 and requests time out. There's nothing in the apache access log or error log indicating unusual activity but the disk IO chart shows heavy read/write spikes that correlate with our downtime.

I know it's some combination of these pages and a few hundred thousand hits an hour, but I'm stumped, and I don't know which tools to use. I need to see A) How many hits per second/minute/hour these pages are getting and B) How long it's taking to serve each page. What's available to profile a live server under load? What's best?

The server is apache2, php5, ubuntu hardy. Any advice at all is greatly appreciated.

EDIT:

Thanks for the ideas. I could edit the PHP, but these are pages that designers are changing often, they like to copy/paste/delete things, and I was hoping to find something better than ducttape for this because it's a recurring issue on a lot of our servers.

Are there really no software packages for monitoring server load per-file on production servers? Do I have to resort to debugging tools and per-code-segment profiling? If my server's already choking on hits, wouldn't adding XDebug royally F*!#-up my S@^&?

My company does a lot of international advertising, and my boss wants a web proxy (that he can use with something like ProxySwitch firefox extension) in each of France, Spain, UK and Italy so that we can view/test our localized ads there.

We've already found VPS providers, now we're looking at implementation details; will Nginx work for this? I know it's used as a "reverse proxy" in load balancing situations, but how would I configure it to proxy the requests [from authorized hosts]?

Googling for "nginx proxy" gives me lots of ambiguous results. Any help is appreciated, thanks!