j0nes's questions

I have a set of servers inside Amazon EC2 in VPC. Inside this VPC I have a private subnet and a public subnet. In the public subnet I have set up a NAT machine on a t2.micro instance that basically runs this NAT script on startup, injecting rules into iptables. Downloading files from the internet from a machine inside the private subnet works fine.

However I compared the download speed of a file on an external high-bandwidth FTP server directly from my NAT machine to the download speed from a machine inside my private subnet (via the same NAT machine). There was a really significant difference: around 10MB/s from the NAT machine vs. 1MB/s when downloading from the machine inside the private subnet.

There is no CPU usage on the NAT machine, so this cannot be the bottleneck. When trying the same test with bigger machines (m3.medium with "moderate network performance" and m3.xlarge with "high network performance"), I also could not get download speeds greater than 2.5MB/s.

Is this a general NAT problem that can (and should) be tuned? Where does the performance drop come from?

Update

With some testing, I could narrow this problem down. When I am using Ubuntu 12.04 or Amazon Linux NAT machines from 2013, everything runs smoothly and I get the full download speeds, even on the smallest t2.micro instances. It does not matter whether I use PV or HVM machines. The problem seems to be kernel-related. These old machines have a Kernel version 3.4.x, whereas the newer Amazon Linux NAT machines or Ubunut 14.XX have Kernel version 3.14.XX. Is there any way to tune the newer machines?

I have nginx in front of an Apache server and a gunicorn server for different parts of my website. I am using the SSI module in nginx to display a snippet in every page. The websites include a snippet in this form:

For static pages served by nginx everything is working fine, the same goes for the Apache-generated pages - the SSI include is evaluated and the snippet is filled. However for requests to my gunicorn backend running a Python app in Django, the SSI include does not get evaluated.

Here is the relevant part of the nginx config:

location /cgi-bin/script.pl {
    ssi on;
    proxy_pass http://default_backend/cgi-bin/script.pl;
    include sites-available/aspects/proxy-default.conf;
}

location /directory/ {
    ssi on;
    limit_req zone=directory nodelay burst=3;
    proxy_pass http://django_backend/directory/;
    include sites-available/aspects/proxy-default.conf;
}

Backends:

upstream django_backend {
    server dynamic.mydomain.com:8000 max_fails=5 fail_timeout=10s;
}
upstream default_backend {
    server dynamic.mydomain.com:80;
    server dynamic2.mydomain.com:80;
}

proxy_default.conf:

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

What is the cause for this behaviour? How can I get SSI includes working for my pages generated on gunicorn? How can I debug this further?

I am using Amazon RDS to host the databases for my website. I am using the large instances for both my master and my read replica. Everything was running smoothly until suddenly the Read IOPS, Read Throughput and Queue Depth metrics in the AWS console started to increase suddenly (the blue instance is my master server, the orange one is the slave).

At the time where this peak occured, no jobs with heavy workload or code changes were deployed. The CPU utilization according to the AWS console has increased slightly since that time. I am also running munin plugins to monitor my master server, and I see no changes in access or network throughput since that time. The open connections to the server (checked with show processlist) also stayed the same.

What is happening to my RDS instance? What could lead to this effect and how can I make this stop? And why doesn't this behaviour show up in munin?

I am running Ubuntu 13.04 with the corresponding versions of monit (5.5-6) and asterisk (1.8.13.1). I have monit set up to watch my asterisk log file for disconnection to my SIP provider and restart asterisk for a new connection. Here is the relevant part of the monit watch script:

check file messages with path "/var/log/asterisk/messages" 
    start program = "/etc/init.d/asterisk start"
    stop program = "/etc/init.d/asterisk stop"
    if match "Retransmission timeout reached" then restart and noalert [email protected]
    if match "timed out, trying again \(Attempt " then restart and noalert [email protected]

The detection of errors is working fine, I get mails when the condition is met. Monit does also stop the asterisk daemon then and the corresponding PID files in /var/run get deleted correctly. However asterisk does not get started again. There is no corresponding error in the monit or asteriks log files.

What is the error in this case? Where can I look for more debugging information?

I have a website where I need to change the URL structure. The old URLs look like /olddir/part1_de.htm, the new ones will look like /newdir/sub/category/anotherpage.htm. There are a lot of URL rewrites I need to do, I assume about 500 distinct rewrites in the end.

As my website gets quite a lot of traffic, my main concern is about performance at the moment. My questions are:

• I assume that for each request, the rewrites block will be parsed and the regex will be evaluated. Am I right?
• Will there be a performance penalty if I use these rewrites? Can nginx handle this?
• Are there any "best practices" to follow when doing a lot of rewrites?

I am running several EC2 instances, and I want to know the exact work my CPU is doing. On "normal" machines I am doing this with munin and its CPU plugin which looks at the statistics provided by /proc/stat.

On my EC2 machines however, I get incorrect graphs. The machine has two cores, so the max CPU usage should be 200% - however it gets as high as 400%:

I know that I should use Amazon CloudWatch to see the total CPU usage (and this is the official and recommended from Amazon way to do this), but I am specifically looking on how the CPU usage is spend (e.g. system, user, iowait).

Is there a way to get detailed CPU usage statistics on EC2 instances?

I have set up munin to monitor EC2 instances. Now my CPU usage graph (using the unmodified cpu plugin) looks like this:

The problem is that the "iowait" parameter is returned correctly (see text below the graph), but it is never plotted. On other non-EC2 machines I have never seen this problem. Also there are no error shown in the munin log files.

How can I fix this?

I am trying to send notification mails using monit v5.1. I am using Google Apps as a mailserver, so I wanted to use their infrastructure for outgoing mails.

I have entered the following information in my /etc/monit/monitrc:

set mailserver smtp.gmail.com PORT 587 USERNAME "[email protected]" PASSWORT "mypassword" using TLSV1 with timeout 30 seconds

When monit tries to send out mails, I get the following error:

Sendmail error: 530 5.7.0 Must issue a STARTTLS command first.

How can I tell monit to use STARTTLS? How can I send alert mails via monit using the Google Apps mailserver?

I have a website that is getting about 7k requests per seconds on a nginx server. This server both handles rewrites to an Apache server as well as serving static files, images, etc. directly. Static files is the biggest part in there with about 5k requests.

With an architecture upgrade, I think about using a central file server that exports a directory containing these static files via NFS. There will be no write-access to these files, so the directory could be mounted read-only on the nginx machine. My main concern is:

Is NFS fast enough for this? Is there a limit on how many requests NFS can handle? Are there some "must-have" options when going this way?

Bonus: are there other alternatives for this setup besides NFS?

Thanks!

we are running a Dell Server with a PERC5i Raid card using a Raid1 setup. Now we are hitting a performance problem with our harddrive performance and think about switching to Raid10.

Which steps are necessary when switching from Raid1 to Raid10? Can I do this without a new installation of the system?

Thanks! Jonas

I am thinking about converting some tables from MyISAM to InnoDB in my mysql server. The tables will certainly benefit from the change because a lot of write requests come to these tables, while there are also quite a lot of read request at the same time.

However, they are often joined together with some tables that almost don't get any writes. Is there a performance penalty when joining together MyISAM and InnoDB tables or should everything work fine?

Second question: During backups at night, I am copying data from the InnoDB tables to MyISAM tables for archiving purposes. In these backups, a lot of write-requests happen, however there is almost no read from these archive tables. Would these tables also benefit from using InnoDB or is this just a waste of space and RAM?

I have run bonnie++ v1.96 on two different servers without any additional load. One server is a "physical" Dell server with 32GB RAM, the other one is a virtual instance with 14GB RAM. I have read in the bonnie manuals that I should use two times the size of RAM in my bonnie runs, so I used 64GB on the physical machine and 28GB on the virtual machine.

Now I want to compare the results, and I am wondering whether the results are comparable at all. The most interesting part is the latency part - on the physical machine, the values are about 10 times higher than on the virtual machine!

Can I take these results seriously (e.g. the virtual machine HD is much much faster) or does the different RAM size tamper the results?

Thanks! Jonas

I have set up a keybased SSH login on my server. Now when I am not in the office and don't have my key with me, there is no option for me to access my server. Going back to password-based logins is not an option for me.

What is the best practice in this case so that I can use SSH without my key occasionally?

I am currently looking at the file size of my Apache logs as they became huge. In my logrotate configuration, I have delaycompress enabled. Does Apache really need this (as the logrotate documentation says that some programs still write in the old file) or is it safe to disable delaycompress?

Here is my logrotate configuration:

/var/log/apache2/*.log {
    weekly
    missingok
    rotate 26 
    compress
    delaycompress
    notifempty
    create 640 root adm
    sharedscripts
    postrotate
            if [ -f /var/run/apache2.pid ]; then
                    /etc/init.d/apache2 restart > /dev/null
            fi
    endscript
}

my setup is as follows: I have one Apache2 webserver running different vhosts, one vhost is for the production website, the other vhost is for a staging / preview system. Both vhosts have different DocumentRoots and also different (Perl) CGI folders. The used modules for each of these vhosts should be in different directories, so I did the following:

<VirtualHost...>
ServerName production
SetEnv PERL5LIB /home/production/modules
</VirtualHost>

<VirtualHost...>
ServerName staging
SetEnv PERL5LIB /home/staging/modules
</VirtualHost>

However, I just noticed that in my Perl CGI scripts, both paths get filled into my @INC, so I can not separate the staging modules from the production modules, e.g. the SetEnv directive is not limited to a single virtual host, but seems to work globally.

How can I solve this?

Thanks! Jonas