Questions[streaming](server)

Our site has a number of large PDF and MP3 files which we would like to cache in Varnish as static files. Currently we don't do much special - simply remove the cookies in vcl_recv and set resetp.ttl = 100w; in vcl_fetch.

The problem seems to be when one of these files is requested (maybe by older browsers) and it's not already in the Varnish cache. There is a delay while Varnish downloads the file from the backend. My understanding is that it doesn't start delivering to the client until the data is fully loaded. This may take 20 seconds or so and sometimes Adobe Acrobat or the MP3 plugin get confused.

Is there a way to both pass the content directly while download and save it in the cache for the next matching client request?

How can I block Spotify on our company network? Unfortunately it's killing our bandwidth and the effect is really serious.

Is there a command in Solaris to read a file, and when it gets to the end to stream the way tail does? I need to read the file from the start, and it is a binary file.

Information on Solaris and Linux would be appreciated.

We have a server that receives some data, acting as a TCP client, processes it in some way, and serving processed data to clients, acting as a TCP server. It also stores this data on disk and can serve it from files instead of real-time stream.

The problem is that this service has to be available in 24x7 mode, no interruptions allowed. Right now it is done by having two servers, one acting as a hot backup - clients maintain connections to both servers, and if something happens to the primary server, they just switch to the backup. While this solution works for about 15 years already, it is kind of inconvenient and puts a lot of failover logic on the clients.

Lately people started talking about using a cluster to ensure availability of this service, but no matter how hard I search I just can't find any clustering solutions that would allow transparent TCP connection failover so no-one would notice that something happened to the server. There are some research papers around, but I was unable to find any working implementations. Here is how I think it should work:

1. Both servers receive the data via TCP. Ideally it should look like a single connection to the "outside" world, to save bandwidth and, more importantly, to ensure that both servers receive identical data streams.

2. When a client connects to the cluster IP, it receives the processed data in a single connection, but both servers should see this connection and provide the data, it is just that only one of the streams actually reaches the client, the backup one goes to /dev/null, so to say.

3. When the server fails (it doesn't transmit any data for some time, say, 5 seconds), the client should continue receiving the same stream within the same connection. It needs to happen pretty fast, so the overall streaming delay doesn't exceed approximately 10 seconds.

Reliability is the most important thing here. Quick failover is the next one. Open source Linux solutions are preferred, but if commercial and/or non-Linux near-perfect solutions exist, I would like to know about them too. Solutions that impose a lot of restrictions or require modifications of the server application software are perfectly acceptable too.

I need an audio streaming setup that supports the following formats: WMA, AAC and MP3. The streams must be compatible both with desktop players like Winamp. Win Media Player and with Flash Player and devices like iPhone, Android, etc.

I'm looking for a single server solution and a single encoder solution that would meet all those requirements. Any ideas?