rush's questions

I'm trying to speed up my webpage load time in distant locations.

My server is located in Germany and I installed the second one in LA, CA.

Then I configured BIND to return LA IP for clients from Americas and German IP for clients from Europe and Asia.

Though it seems there is still room to reduce response time by some dozens of ms with DNS and name servers.

At the moment both name servers which serve my domain name are located in Germany which from my understanding causes delays for clients who has no dns cache for my domain. Am I right here?

• If I will move one of name servers to LA will in reduce DNS resolution time?
• How name server to query is chosen during name resolution?
• E.g. specified two NS: ns1.domain.com which is in DE and ns2.domain.com which is in US. which one will be queried? Will it help to specify IP instead of ns1.domain.com?

I have domain.com and sub.domain.com configuraed in nginx. domain.com has ssl certificate, while sub.domain.com has not. wherever I try to open http://sub.domain.com in any modern browser (firefox, chrome even in clean browser without plugins) it redirects me to https://sub.domain.com and gives me an error as my ssl certificate is only for domain.com.

However wget doesn't redirect me:

 $ wget -O /dev/null http://sub.domain.com         
--2014-08-15 09:49:00--  http://sub.domain.com/
Resolving sub.domain.com (sub.domain.com)... X.X.X.X
Connecting to sub.domain.com (sub.domain.com)|X.X.X.X|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’
2014-08-15 09:49:00 (1.23 MB/s) - ‘/dev/null’ saved [15807]

Here is nginx configuration for domain.com

server {
       # Redirect all http to https
       listen         X.X.X.X:80;
       server_name    ^domain.com www.domain.com;
       rewrite        ^ https://www.domain.com$request_uri? permanent;
}

server {
        ## Redirect https no-www to www for domain.com only
        listen               X.X.X.X:443 ssl;

        # Note ssl-bundle should contain only domain.com & root certificate
        ssl_certificate      /home/domain/ssl/www_domain.com.bundle;
        ssl_certificate_key  /home/domain/ssl/domain.com.key;

        ### Need to change that to avoid SSL Beast
        ssl_protocols              SSLv3 TLSv1 TLSv1.1 TLSv1.2;
        ssl_prefer_server_ciphers  on;
        ssl_ciphers                "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK";

        ### Need to add this to enable HTTP Strict-Transport-Security
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        server_name          ^domain.com;
        rewrite              ^ https://www.domain.com$request_uri? permanent;
}


server {
       ### Main section
       listen X.X.X.X:443 ssl;
       server_name                www.domain.com;
       server_tokens              off;

       ssl_certificate      /home/domain/ssl/www_domain.com.bundle;
       ssl_certificate_key  /home/domain/ssl/domain.com.key;
       ### Need to change that to avoid SSL Beast
       ssl_protocols              SSLv3 TLSv1 TLSv1.1 TLSv1.2;
       ssl_prefer_server_ciphers  on;
       ssl_ciphers                "ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK";

        ### OCSP will be enabled only after nginx v1.3.5, so let's wait until it becomes the stable version 
        ### ( 1.6 is already in testing )
        # enable ocsp stapling (mechanism by which a site can convey certificate revocation information to visitors in a privacy-preserving, scalable manner)
        # http://blog.mozilla.org/security/2013/07/29/ocsp-stapling-in-firefox/
        #resolver 8.8.8.8;
        #ssl_stapling on;
        #ssl_trusted_certificate /home/domain/certs/ssl-bundle.crt;

        ### Need to add this to enable HTTP Strict-Transport-Security
        ###
        add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
        add_header X-Frame-Options SAMEORIGIN;

        root /home/domain/www/domain.com;
        index index.php index.html index.htm;


        location /promo/ {
            proxy_set_header   Host             $host;
            proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_set_header   X-Real-IP        $remote_addr;
            proxy_pass         http://127.0.0.1:10001;
            proxy_redirect     off;
    }

    location ^~ /s/promo/static/ {
            disable_symlinks off;
            expires 1y;
            root /home/domain/www/promo-static ;
            log_not_found off;
    }


    location / {
          <.various rules.>
    }
}

And here is configuration for sub.domain.com:

server {
        listen X.X.X.X:80;
        server_name sub.domain.com ;

        # Serve media and static with nginx
        location ^~ /media/ {
                root /home/domain/www/sub_domain_com/project/;
                access_log off;
        }

        location ^~ /static/ {
                root /home/domain/www/sub_domain_com/project/;
                access_log off;
        }

        # Proxy redirect to django
        location / {
                proxy_read_timeout 1200;
                proxy_set_header   Host             $host;
                proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
                proxy_set_header   X-Real-IP        $remote_addr;
                proxy_pass         http://127.0.0.1:10001;
                proxy_redirect     off;
        }
}

I'm out of ideas how to stop http to https redirection for sub.domain.com.

One more strange thing: if I completely remove section for redirect http domain.com to https domain.com wget will return HTTP request sent, awaiting response... No data received. on http://domain.com but firefox and chrome will keep open https version when I type http://domain.com! what is wrong with these browsers and how do I need to configure nginx to stop such behaviour?

I have an cvs server with pretty big binray file (about 50Mb) and its rcs file is about 2G

Today I discovered that file update returns an error:

cvs [update aborted]: unexpected EOF reading /repository/cvsroot/dir/file,v

I checked the cvs server and it showed me an completely destroyed /repository/cvsroot/dir/file,v with size about 100K.

I restored the file from backup, unfortunately I caught the same issue one more time.

After second file replace now it acts fine, but I couldn't get the reason of such behaviour.

It doesn't seem an hardware fail as all other files on the same disk are fine and smartctl doesn't show any error or suspicious warnings.

Is there a way to catch the root of this issue?

ps. cvs is running on red hat 5.5, cvs version is 1.12.6.

upd. actually after a while the file had broken again. so cvs looks like not usable with such behaviour. unfortunately due to political restrictions I couldn't migrate to another VCS like git ( at least before the end of this year ).

upd2. after some experiments I discovered, that most operations are successfully passes, but cvs rtag destroys file. I even moved the file into another project and tried to rtag it and I immediately got the broken file. the full rtag command:

cvs rtag -B -a -F -r 1.447 -b "branch" project/file

I've got a Debian + Xen server. After a system upgrade to the stable version the network doesn't come up after boot. Every time after reboot I need to bring it up manually. The network configuration was not changed during upgrade.

Here is /etc/network/interfaces:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
    address 11.22.33.44
    netmask 255.255.255.0
    gateway 11.22.33.1
    nameserver 8.8.8.8

After boot ip r shows no route and eth0 has no ip address. Manually ip and route setup goes fine and network starts working.

Messages from dmesg about network I've found (looks like nothing interesting)

[    3.894401] ACPI: Fan [FAN3] (off)
[    3.894444] ACPI: Fan [FAN4] (off)
[    4.178348] e1000e 0000:00:19.0: eth0: (PCI Express:2.5GT/s:Width x1) 00:1e:67:14:66:c9
[    4.178351] e1000e 0000:00:19.0: eth0: Intel(R) PRO/1000 Network Connection
[    4.178392] e1000e 0000:00:19.0: eth0: MAC: 10, PHY: 11, PBA No: 0100FF-0FF
[    4.178413] e1000e 0000:02:00.0: Disabling ASPM L0s L1
[    4.178432] xen: registering gsi 16 triggering 0 polarity 1
--
[    4.223667] ata5: DUMMY
[    4.223668] ata6: DUMMY
[    4.289153] e1000e 0000:02:00.0: eth1: (PCI Express:2.5GT/s:Width x1) 00:1e:67:14:66:c8
[    4.289155] e1000e 0000:02:00.0: eth1: Intel(R) PRO/1000 Network Connection
[    4.289245] e1000e 0000:02:00.0: eth1: MAC: 3, PHY: 8, PBA No: 1000FF-0FF
[    4.506908] usb 1-1: new high-speed USB device number 2 using ehci_hcd
[    4.542920] ata2: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
--
[   10.362999] EXT4-fs (dm-23): mounted filesystem with ordered data mode. Opts: (null)
[   10.419103] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[   10.988255] ADDRCONF(NETDEV_UP): eth1: link is not ready
[   13.175533] Event-channel device installed.
[   13.287555] XENBUS: Unable to read cpu state
--
[   13.288670] XENBUS: Unable to read cpu state
[   13.965939] Bridge firewalling registered
[   14.134048] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[   14.283862] ADDRCONF(NETDEV_UP): peth0: link is not ready
[   14.284543] ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[   17.800627] e1000e: peth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
[   17.801377] ADDRCONF(NETDEV_CHANGE): peth0: link becomes ready
[   18.307278] device peth0 entered promiscuous mode
[   24.538899] eth1: no IPv6 routers present
[   28.570902] peth0: no IPv6 routers present

I've upgraded two servers and I've such behaviour on two of them. How to fix this and get network starts automatically on boot?

I have a some software that uses its own encrypted file for password storage ( such as ftp, web and other passwords to login to external systems, there is no way to use certificates ).

On each server I've several instances of this software, each instance has its own password file.

At the moment number of servers is permanently growing and it's getting harder and harder to manage all passwords on all instances up to date.

Unfortunately, some servers are in cegregated network and there is no access from them to some centralized storage, but it works vice versa.

My first idea was to create a git repository, encrypt each password with gpg and store it there and deliver it within deployment system, but security team was not satisfied with this idea and as it is insecure to store passwords in repository even in encrypted view ( from their words ).

Nothing similar comes to my mind. Is there any way to implement safe and secure password storage with minimal effort to manage all passwords up-to-date?

ps. if that matters I've red hat everywhere.

I have about twenty servers with different webapps. Every 4 hours runs rsnapshot task and backs up all of them to a backup server.

Accidentally today I discovered that backup failed last 4 days due to input/output failure in file system. fsck fixed the issue, however 4 backup days are lost.

Is there any way to check if backups are ok?

Right now I use munin monitoring system, if it does matter, though it check only server health (memory, cpu, hdd, etc) without any software checks. I can integrate a script that will check a FATAL ERRORs entries in rsnapshot log, however I'm not sure will it be enough?

May be there is a system for bootstrapping environment from backup to check its integrity. Unfortunately I didn't find enough information about it.

There is problem with nginx. It closes connection before client finishes download. It looks like:

 $ wget -O /dev/null http://www.site.com/images/theme/front/clean.jpg
--2012-07-11 21:37:03--  http://www.site.com/images/theme/front/clean.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 90707 (89K) [image/jpeg]
Saving to: `/dev/null'

26% [===============>                    ] 24,291      --.-K/s   in 8.7s    

2012-07-11 21:37:12 (2.74 KB/s) - Connection closed at byte 24291. Retrying.

--2012-07-11 21:37:13--  (try: 2)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 66416 (65K) remaining [image/jpeg]
Saving to: `/dev/null'

53% [+++++++++++++++============>        ] 48,555      --.-K/s   in 8.7s    

2012-07-11 21:37:23 (2.74 KB/s) - Connection closed at byte 48555. Retrying.

--2012-07-11 21:37:25--  (try: 3)  http://www.site.com/images/theme/front/clean.jpg
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 90707 (89K), 42152 (41K) remaining [image/jpeg]
Saving to: `/dev/null'

100%[+++++++++++++++++++++++++++========>] 90,707      --.-K/s   in 0.1s    

2012-07-11 21:37:25 (311 KB/s) - `/dev/null' saved [90707/90707]

at the same time with smaller images all is ok:

 $ wget -O /dev/null http://www.site.com/images/theme/front/grease.jpg
--2012-07-11 21:41:28--  http://www.site.com/images/theme/front/grease.jpg
Resolving www.site.com (www.site.com)... 123.234.123.234
Connecting to www.site.com (www.site.com)|123.234.123.234|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21404 (21K) [image/jpeg]
Saving to: `/dev/null'

100%[====================================>] 21,404      --.-K/s   in 0.07s   

2012-07-11 21:41:29 (316 KB/s) - `/dev/null' saved [21404/21404]

This is the reason why this picture can't bee drawn fully size in browser. I can see only head of it.

Nginx is configured as front-end and apache as back-end. Direct link to apache works well, so there is problem in nginx. Am I right?

nginx config:

user satellite;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    access_log  /var/log/nginx/access.log;

    sendfile        on;
    keepalive_timeout  0;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    client_max_body_size 100m;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
    server {
            listen 123.234.123.234:80;
            server_name site.com www.site.com;
            location ~* ^/(admin/|dump/|webmail/|myadmin/|webim/) {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
            location / {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_redirect http://site.com:8080/ /;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
                    proxy_cache ino;
                    proxy_cache_valid 12h;
                    proxy_hide_header "Set-Cookie";
                    proxy_ignore_headers "Cache-Control" "Expires";
            }
            location ~* ^.+\.(jpg|swf|flv|ico|txt|jpeg|gif|png|svg|js|css|mp3|ogg|mpe?g|avi|zip|gz|bz2?|rar)$ {
                    access_log /home/satellite/logs/site.com.nginx.access.log;
                    error_page 404 = @fallback;
                    if ( $host ~* ^((.*).site.com)$ ) {
                            set $proot /home/satellite/www/$1;
                            break;
                    }
                    if ( $host = "www.site.com" ) {
                            break;
                    }
                    if ( $host = "site.com" ) {
                            break;
                    }

                    root /home/satellite/www/site.com;
            }
            location @fallback {
                    proxy_pass http://123.234.123.234:8080;
                    proxy_set_header Host $host;
                    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                    proxy_set_header X-Real-IP $remote_addr;
            }
    }

where I should dig to fix this issue?

I'm trying to set up nginx as front-end and apache as back-end on my server. All goes as usual, I've installed libapache2-mod-rpaf. But apache doesn't see real client ip and some .htaccess restrictions based on ip don't work at all, cause apache sees only local ip.

Here is nginx vhost configuration^

server {
    listen 80;
    server_name domain;
    location ~* ^/(admin/|dump/|) {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://127.0.0.1:8080;
        proxy_redirect http://domain:8080/ /;
    }
    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_pass http://127.0.0.1:8080;
        proxy_redirect http://domain:8080/ /;
    }
    location ~* ^.+\.(jpg|swf|flv|ico|txt|jpeg|gif|png|svg|js|css|mp3|ogg|mpe?g|avi|zip|gz|bz2?|rar)$ {
        access_log /var/www/logs/domain.nginx.access.log;
        error_page 404 = @fallback;
        if ( $host ~* ^((.*).domain)$ ) {
            set $proot /var/www/$1;
            break;
        }
        if ( $host = "domain" ) {
            break;
        }

        root /var/www/domain;
    }
    location @fallback {
        proxy_pass http://127.0.0.1:8080;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

And here is apache vhost configuration:

<VirtualHost *:8080>
    ServerName domain
    DocumentRoot /var/www/domain
    DirectoryIndex index.php
    CustomLog /var/www/logs/domain.access.log combined
    ErrorLog /var/www/logs/domain.error.log
    ServerAdmin info@domain
    AddDefaultCharset utf8
    php_admin_value open_basedir "/var/www/mod-tmp:.:/var/www/domain"
    php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f webmaster@domain"
    php_admin_value upload_tmp_dir "/var/www/mod-tmp"
    php_admin_value session.save_path "/var/www/mod-tmp"
    AddType application/x-hwikipd-php .php .php3 .php4 .php5 .phtml
    AddType application/x-hwikipd-php-source .phps
<Directory /var/www/domain>
    Options All
    AllowOverride All
</Directory>
</VirtualHost>

As I can see in nginx log:

xxx.yyy.98.14 - - [28/Jan/2012:02:08:29 +0400] "GET / HTTP/1.0" 403 179 "-" "w3m/0.5.3"

It sees my real ip.

But in apache log it sees only local ip:

127.0.0.1 - - [28/Jan/2012:02:08:29 +0400] "GET / HTTP/1.0" 403 390 "-" "w3m/0.5.3"

rpaf is enabled.

$ cat /etc/apache2/mods-enabled/rpaf.conf 
<IfModule mod_rpaf.c>
    RPAFenable On
    RPAFsethostname On
    RPAFproxy_ips 127.0.0.1 ::1
</IfModule>
$ cat /etc/apache2/mods-enabled/rpaf.load 
LoadModule rpaf_module /usr/lib/apache2/modules/mod_rpaf.so
$ ls -l /usr/lib/apache2/modules/mod_rpaf.so
-rw-r--r-- 1 root root 8488 Oct 17 20:47 /usr/lib/apache2/modules/mod_rpaf.so

So, what is wrong?

ps. I have restarted nginx and apache several times after saving this configs, but there are no any changes.

I can't find information about ways to count disk iops in GNU/Linux for some time period. I saw iostat, but I'm not sure it is right way. Also i need tool, that can count iops for all time since system is started, but iostat doesn't do this. I mean disk iops for whole systems and for any logical volume.

May be there are some tools to limit disk iops for logical volume or smth like that.

If it matters, I use Debian. Also I use hardware MegaRaid Raid Controller. May be it influences on smth.

I've xen server with 8 domU. Server is Xeon E31270 with 16gb ram. I think it is enough for 8 machines. Sometimes domU's crashes and i can't figure out the reason.

After crash i can connect to console and there is somthing like this:

Oct  8 22:20:49 server kernel: [30892.320780] lowmem_reserve[]: 0 0 0 0
Oct  8 22:20:49 server kernel: [30892.320790] Node 0 DMA: 10*4kB 3*8kB 13*16kB 10*32kB 7*64kB 3*128kB 2*256kB 2*512kB 1*1024kB 2*2048kB 0*4096kB = 8080kB
Oct  8 22:20:49 server kernel: [30892.320817] Node 0 DMA32: 648*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 5760kB
Oct  8 22:20:49 server kernel: [30892.320842] 1491 total pagecache pages
Oct  8 22:20:49 server kernel: [30892.320847] 0 pages in swap cache
Oct  8 22:20:49 server kernel: [30892.320852] Swap cache stats: add 0, delete 0, find 0/0
Oct  8 22:20:49 server kernel: [30892.320858] Free swap  = 0kB
Oct  8 22:20:49 server kernel: [30892.320862] Total swap = 0kB
Oct  8 22:20:49 server kernel: [30892.324024] 524288 pages RAM
Oct  8 22:20:49 server kernel: [30892.324024] 11010 pages reserved
Oct  8 22:20:49 server kernel: [30892.324024] 424467 pages shared
Oct  8 22:20:49 server kernel: [30892.324024] 503538 pages non-shared
Oct  8 22:20:49 server kernel: [30892.330308] apache2 invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0
Oct  8 22:20:49 server kernel: [30892.330322] apache2 cpuset=/ mems_allowed=0
Oct  8 22:20:49 server kernel: [30892.330330] Pid: 23938, comm: apache2 Not tainted 2.6.32-5-xen-amd64 #1
Oct  8 22:20:49 server kernel: [30892.330337] Call Trace:
Oct  8 22:20:49 server kernel: [30892.330349]  [<ffffffff810b7180>] ? oom_kill_process+0x7f/0x23f
Oct  8 22:20:49 server kernel: [30892.330358]  [<ffffffff810b76a4>] ? __out_of_memory+0x12a/0x141
Oct  8 22:20:49 server kernel: [30892.330367]  [<ffffffff810b77fb>] ? out_of_memory+0x140/0x172
Oct  8 22:20:49 server kernel: [30892.330376]  [<ffffffff810bb59c>] ? __alloc_pages_nodemask+0x4e5/0x5f5
Oct  8 22:20:49 server kernel: [30892.330385]  [<ffffffff810cc224>] ? do_wp_page+0x386/0x707
Oct  8 22:20:49 server kernel: [30892.330395]  [<ffffffff8100c3a5>] ? __raw_callee_save_xen_pud_val+0x11/0x1e
Oct  8 22:20:49 server kernel: [30892.330404]  [<ffffffff8100c369>] ? __raw_callee_save_xen_pmd_val+0x11/0x1e
Oct  8 22:20:49 server kernel: [30892.330412]  [<ffffffff810cdfc7>] ? handle_mm_fault+0x7aa/0x80f
Oct  8 22:20:49 server kernel: [30892.330422]  [<ffffffff8130f906>] ? do_page_fault+0x2e0/0x2fc
Oct  8 22:20:49 server kernel: [30892.330433]  [<ffffffff8130d7a5>] ? page_fault+0x25/0x30
Oct  8 22:20:49 server kernel: [30892.330439] Mem-Info:
Oct  8 22:20:49 server kernel: [30892.330443] Node 0 DMA per-cpu:
Oct  8 22:20:49 server kernel: [30892.330450] CPU    0: hi:    0, btch:   1 usd:   0
Oct  8 22:20:49 server kernel: [30892.330463] CPU    1: hi:    0, btch:   1 usd:   0
Oct  8 22:20:49 server kernel: [30892.330466] Node 0 DMA32 per-cpu:
Oct  8 22:20:49 server kernel: [30892.330469] CPU    0: hi:  186, btch:  31 usd:   0
Oct  8 22:20:49 server kernel: [30892.330472] CPU    1: hi:  186, btch:  31 usd:  60
Oct  8 22:20:49 server kernel: [30892.330476] active_anon:342076 inactive_anon:115398 isolated_anon:0
Oct  8 22:20:49 server kernel: [30892.330477]  active_file:268 inactive_file:481 isolated_file:0
Oct  8 22:20:49 server kernel: [30892.330477]  unevictable:1125 dirty:2 writeback:13 unstable:0
Oct  8 22:20:49 server kernel: [30892.330478]  free:3410 slab_reclaimable:1718 slab_unreclaimable:6946
Oct  8 22:20:49 server kernel: [30892.330478]  mapped:899 shmem:113 pagetables:35697 bounce:0
Oct  8 22:20:49 server kernel: [30892.330502] Node 0 DMA free:8036kB min:32kB low:40kB high:48kB active_anon:1144kB inactive_anon:1268kB active_file:8kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:11792kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:224kB kernel_stack:16kB pagetables:1228kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Oct  8 22:20:49 server kernel: [30892.330518] lowmem_reserve[]: 0 2004 2004 2004
Oct  8 22:20:49 server kernel: [30892.330523] Node 0 DMA32 free:5604kB min:5708kB low:7132kB high:8560kB active_anon:1367160kB inactive_anon:460324kB active_file:1064kB inactive_file:1916kB unevictable:4500kB isolated(anon):0kB isolated(file):0kB present:2052320kB mlocked:4500kB dirty:8kB writeback:52kB mapped:3600kB shmem:452kB slab_reclaimable:6872kB slab_unreclaimable:27560kB kernel_stack:3528kB pagetables:141560kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:992 all_unreclaimable? no
Oct  8 22:20:49 server kernel: [30892.330539] lowmem_reserve[]: 0 0 0 0
Oct  8 22:20:49 server kernel: [30892.330544] Node 0 DMA: 1*4kB 2*8kB 13*16kB 10*32kB 7*64kB 3*128kB 2*256kB 2*512kB 1*1024kB 2*2048kB 0*4096kB = 8036kB
Oct  8 22:20:49 server kernel: [30892.330579] Node 0 DMA32: 609*4kB 2*8kB 1*16kB 0*32kB 1*64kB 0*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 5604kB
Oct  8 22:20:49 server kernel: [30892.330605] 1522 total pagecache pages
Oct  8 22:20:49 server kernel: [30892.330610] 0 pages in swap cache
Oct  8 22:20:49 server kernel: [30892.330615] Swap cache stats: add 0, delete 0, find 0/0
Oct  8 22:20:49 server kernel: [30892.330621] Free swap  = 0kB
Oct  8 22:20:49 server kernel: [30892.330625] Total swap = 0kB
Oct  8 22:20:49 server kernel: [30892.333018] 524288 pages RAM
Oct  8 22:20:49 server kernel: [30892.333018] 11010 pages reserved
Oct  8 22:20:49 server kernel: [30892.333018] 424367 pages shared
Oct  8 22:20:49 server kernel: [30892.333018] 503658 pages non-shared

Seems like there isn't enough memory for this domU. But there is no any memory problems reported in munin monitoring:

memory usage http://xmages.net/storage/10/1/0/1/c/upload/d9a9f7b8.png

As you see system uses around 0.2G and 1G is available.

So my question is:

Is it xen specific problem, that real memory usage and memory usage that shows munin are different (I've never seen such problems oh real hardware machines)?

Or maybe it is just monitoring problem, that can't catch moment when there is unusual high load and domU go down?

And how I can to defeat this problem? it is really annoying to catch messages in e-mail that domU went down.

Btw, such situation was when domU had 2G memory.