luxerama's questions

We have just started to move our infrastructure to Amazon's AWS (EU West). Since we started the move we have started seeing some quite substantial spikes in network latency. Our setup is quite simple, we are running two/three web servers (large instances) behind a ELB and a RDS (XL instance) instance to server the web servers.

Our response time now ranges from abysmal to OK, today's values: 427 ms [London, UK] - 1,642 ms[London, UK] (values from Pingdom). With our previous provider (same setup, traditional managed hosting) we were used to seeing around 200 - 300 ms. Is there anything we can do to improve the response time or is this something that we have no control over.

I read about the problems Amazon seemed to have in Jan 2010 (http://www.theregister.co.uk/2010/01/15/amazon_ec2_latency/) however haven't heard anything since thus assumed it was a brief period where this happened.

What do you guys think, is there something I can do or is it the penalty you have to use for Amazon's flexibility?

Thanks for your help.

I have a question re transfer rates on Amazons EC2. Basically I am evaluating a move to the Amazon stack of services, however wanted to benchmark it against our current setup first. My setup is pretty simple, I just have one large EC2 instance running accessing one small RDS instance, with both being located in the EU-West Zones. On the EC2 instance I am running the stock apache config file whereas on my current server I am running a modified apache config file. This means that I am expecting fewer concurrent connections however I was expecting the transfer rate to be similar.

The benchmark tests I am running are nothing are bog standard ab (apache benchmark) tests

ab -n 100 -c 10 http://www.example.com/

The setup on EC2 resulted in the following:

ab -n 100 -c 10 http://www.example.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.website.com (be patient).....done


Server Software:        Apache/2.2.14
Server Hostname:        www.example.com
Server Port:            80

Document Path:          /
Document Length:        99051 bytes

Concurrency Level:      10
Time taken for tests:   17.872 seconds
Complete requests:      100
Failed requests:        97
   (Connect: 0, Receive: 0, Length: 97, Exceptions: 0)
Write errors:           0
Total transferred:      9940632 bytes
HTML transferred:       9898332 bytes
Requests per second:    5.60 [#/sec] (mean)
Time per request:       1787.165 [ms] (mean)
Time per request:       178.716 [ms] (mean, across all concurrent requests)
Transfer rate:          543.19 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       22   29  11.9     26      87
Processing:   717 1740 689.6   1530    3744
Waiting:      586 1475 619.3   1302    3112
Total:        739 1769 688.7   1561    3770

Percentage of the requests served within a certain time (ms)
  50%   1561
  66%   1713
  75%   1892
  80%   1975
  90%   3300
  95%   3674
  98%   3674
  99%   3770
 100%   3770 (longest request)

whereas my server responded with this:

ab -n 100 -c 10 http://www.example.com/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.example.com (be patient).....done


Server Software:        Apache
Server Hostname:        www.example.com
Server Port:            80

Document Path:          /
Document Length:        116922 bytes

Concurrency Level:      10
Time taken for tests:   9.024 seconds
Complete requests:      100
Failed requests:        95
   (Connect: 0, Receive: 0, Length: 95, Exceptions: 0)
Write errors:           0
Total transferred:      11731386 bytes
HTML transferred:       11693186 bytes
Requests per second:    11.08 [#/sec] (mean)
Time per request:       902.427 [ms] (mean)
Time per request:       90.243 [ms] (mean, across all concurrent requests)
Transfer rate:          1269.51 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       15   21   9.5     17      52
Processing:   311  840 663.6    482    2987
Waiting:      188  362 416.1    216    1607
Total:        338  861 661.8    501    3003

Percentage of the requests served within a certain time (ms)
  50%    501
  66%    909
  75%   1063
  80%   1292
  90%   2130
  95%   2366
  98%   2912
  99%   3003
 100%   3003 (longest request)

Is anyone else experiencing these low transfer rates? Are they normal for EC2 instances? May I be doing something wrong?

If you need any more information on this please let me know.

Thanks for all your help.

I am thinking of hosting a project on Amazon's EC2.
However I am a little unclear about possible negative impacts this may have on local search results.

The projects main customer base is in the UK however the closest EC2 region is Ireland. Does anyone know whether this has any effect on the said search results, or is it something that I don't need to be concerned about?

I am a developer for a website which is heavily reliant on static content such a pictures, videos etc. Our current setup is very simple, we basically have one server acting as our cdn which in turn gets updated by the web servers through rsync. The simplicity if this setup has a couple of drawbacks, mainly the fact we also have to rsync any changes between the web servers.

We are now looking for a less disjointed architecture. One idea is to use NAS which could be mounted in each web server which meant that we wouldn't have to rely on rsync however would introduce a single point of failure.

Does anyone have any knowledge of successful, fairly high volume (approx. 10tb a month) cdn setups and architectures?

I know this one is a tricky one because the problem could potentially be caused by quite a few different things. However maybe you've come across something similar or have some hints or tips of how I may be successful in debugging the current setup.

Basically we are running a small/medium sized office network with the current setup:

1. Load balancer (Draytek Vigor 3300)
2. Inside the load balancer we have 2 Cisco managed switches
3. The load balancer is connected to 3 routes which supply the connection to the internet.

The problem is that every now and then the network(LAN and External) either slows down considerably, or the LAN is fine but external traffic comes to a complete standstill. This doesn't even occur in set intervals, we sometimes go weeks without noticing any problems. Restarting the load balancer solves the issue until the next time it happens.

Now, I appreciate that this is near impossible to solve without actually looking at the setup however I thought that maybe someone had been in a similar situation before or has any ideas what the problem might be.

If you need any more information please let me know

Does anyone know whether there is a way to speed up writing to a shared folder on the host system.

The host is running Ubuntu 10.10 and the guest is CentOS 5.5 and I have noticed a considerable decrease in speed when writing to the shared folder.

[vincent@dev8 ~]$ bonnie -s 2000
Bonnie 1.4: File './Bonnie.16947', size: 2097152000, volumes: 1
Writing with putc()...         done:  51196 kB/s  70.8 %CPU
Rewriting...                   done: 101780 kB/s  11.1 %CPU
Writing intelligently...       done:  46673 kB/s   6.9 %CPU
Reading with getc()...         done:  41970 kB/s  57.1 %CPU
Reading intelligently...       done: 108134 kB/s  12.3 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU
dev8.b 1*2000 51196 70.8 46673  6.9 101780 11.1 41970 57.1108134 12.3  362.8  3.9

[vincent@dev8 sharedfolder]$ bonnie -s 2000
Bonnie 1.4: File './Bonnie.16933', size: 2097152000, volumes: 1
Writing with putc()...         done:  30547 kB/s  66.6 %CPU
Rewriting...                   done:  32763 kB/s  12.0 %CPU
Writing intelligently...       done: 111468 kB/s  20.6 %CPU
Reading with getc()...         done:  36909 kB/s  71.4 %CPU
Reading intelligently...       done: 299752 kB/s  39.0 %CPU
Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              ---Sequential Output (nosync)--- ---Sequential Input-- --Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --04k (03)-
Machine    MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU   /sec %CPU
dev8.b 1*2000 30547 66.6111468 20.6 32763 12.0 36909 71.4299752 39.0 16816.8  140

As you can see the writing and reading to the shared folder is a lot heavier on the resources.

At first I though it might be to do with the fact that Ubuntu was using ext4 and CentOS ext3 however even when I used a ext3 partition for the shared folder on the host system it did not improve.

Has anyone seen this problem before?Is it normal? Can it be improved?

Thanks for all your help.

EDIT

Its a virtualbox shared folder, which means that the fs is vboxsf. Its mounted through fstab:

sf_label /mount/dir vboxsf rw,uid=48,gid=48,dmode=774,fmode=774 0 0

My expertise in networking are limited thus I'd be grateful for your opinion on the following.

We will soon be moving into a new office, the question now is how to best rig up the network. To illustrate I have created some v. crud diagrams of two different options.

Option one would only require one switch in the comms room however it would mean that we would have to run a network cable (approx 20m) per pc through the sealing.

Option two would require multiple switches, one in the comms room, and one for each row of tables in the office.

Does anyone know whether one of the two options has any great advantages over the other?