So I got a unique problem. Our app is designed to server large amount of files using a special link that is one time use. We have been playing with ELB but notice an interesting "bug", if you start to get a lot of traffic or load it looks like ELB will try to scale out and retry the connection. This in turns causes our system to invalidate the link before the download is done.
So quickly I put together a nginx proxy server to test the issue and in that setup the connection does not get invalided but I am seeing performance drop by about 2/3 to 1/2. Seeing as I cant throw more bandwidth at the instance as I am locked to what AWS gives me what alternatives do I have? Maybe there is a configuration with the instance/nginx I am not doing correctly or do I need to move to something like RR DNS and pay the price with that.
I'm not sure what the requirements are for your application, but since you're already using AWS, you might consider using a feature of S3 that allows you to give out a signed URL that expires at a specific point in the future (e.g., in two minutes). This would your user retry failed connections but would not let them pass it around for other people to use past the expiration time.
If S3 works for your app, it could also remove a huge burden of having to run the download servers yourself as S3 takes care of all the work.
If www.example.com is being load balanced to server1.example.com and server2.example.com, have your one-time link explicitly link to server1/2.example.com (or the current instance's AWS CNAME, if your instances change frequently enough that your DNS isn't necessarily up-to-date).