Mr.Boon's questions

I run a new CentOS 7 machine. Linux runs on 2x SSD setup, and I also have 4x SAS drives setup in software RAID10. The RAID10 array is large, 4x 12TB drives, so 24TB usable.

File system is: ext4

Now I finished copying some files to it, and I'm doing a raid check (very first one).

Every 2.0s: cat /proc/mdstat                                                                                                                                                                                         Mon Oct 14 06:28:38 2019

Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty]
md127 : active raid10 sdf1[3] sdd1[1] sde1[2] sdc1[0]
      23437503488 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      [======>..............]  check = 32.6% (7649123136/23437503488) finish=3402.6min speed=77333K/sec
      bitmap: 0/175 pages [0KB], 65536KB chunk

md2 : active raid1 sdb2[1] sda2[0]
      20478912 blocks [2/2] [UU]

md3 : active raid1 sdb3[1] sda3[0]
      447318976 blocks [2/2] [UU]
      bitmap: 3/4 pages [12KB], 65536KB chunk

unused devices: <none>

It started around 250,000K/sec but it keeps getting slower, and it now it's around 75,000K/s

The drives in the RAID10 array are not being use by anything else at the moment.

I already tweaked the speed limit settings.

dev.raid.speed_limit_min = 100000

dev.raid.speed_limit_max = 1000000

CPU usage is on like 2%, I got tons of RAM free, and the 4 drives in the RAID array are reporting about 25% drive utilization per drive, so they are not being pushed hard by resync.

My question:

1. What can I do to speed this up?

2. And what could be causing it to slow down?

In my Nginx config I have some IP blocks in place, to fight off spammers & bots.

This is very effective, but as a result, my error logs get filled up super fast with error messages like these:

2015/12/16 00:56:28 [error] 27748#0: *120462 access forbidden by rule, client: 167.114.xxx.xxx, server: bla bla ....

Now I don't want to fully disable error logging, as I want to find out what is going wrong when something goes wrong. I just want to disable logging of these "forbidden by rule" messages.

Any idea how to do this?

I'm trying to setup MySQL db replication, it seems pretty straight forward.

I was using this tutorial: http://www.howtoforge.com/mysql_database_replication

Now I run a rather large MySQL database for a very large website, and in this tutorial it asks me to restart MySQL to apply the new settings in the /etc/my.cnf file.

I'm try to avoid that step at all costs, as I know that restarting MySQL can take a few minutes on my machine (due to large logs/dbs), and I don't want any downtime.

Is there a way to apply the necessary settings WITHOUT fully restarting Mysql?

I have just setup my CentOS 5.8 64 [final] server as a webserver.

Specs: 2x E5620 Intel CPU, DDR3 RAM, Hardware Adaptec RAID 10, 4x SAS drives.

I have installed myself:

 Nginx
 PHP-FPM
 MySQL [ourdelta.org version]
 Sphinx
 Vsftpd
 Fail2ban
 Citadel [ddos flood protection]
 Munin
 NTP
 Htop
 Iptraf

These are all things that I actively use to run my websites.

Now my question:

I found this article: http://www.vr.org/kb/1002/Optimize-and-disable-default-CentOS-services.html

Which talks about a whole list of services that you can disable.

This is the list they suggest:

chkconfig anacron off
chkconfig apmd off
chkconfig atd off
chkconfig autofs off
chkconfig cpuspeed off
chkconfig cups off
chkconfig cups-config-daemon off
chkconfig gpm off
chkconfig isdn off
chkconfig netfs off
chkconfig nfslock off
chkconfig openibd off
chkconfig pcmcia off
chkconfig portmap off
chkconfig rawdevices off
chkconfig readahead_early off
chkconfig rpcgssd off
chkconfig rpcidmapd off
chkconfig smartd off
chkconfig xfs off
chkconfig ip6tables off
chkconfig avahi-daemon off
chkconfig firstboot off
chkconfig yum-updatesd off 
chkconfig sendmail off
chkconfig mcstrans off
chkconfig pcscd off
chkconfig bluetooth off
chkconfig hidd off

Now I do use sendmail, and smartd, so those I will keep.

But most of the other processes I do not recognize. Is there anything I should be careful with disabling?

i just installed nginx 1.1.13 and php 5.4.0 on a centos 5.8 final 64bit machine. Nginx and PHP/Fpm are running, and I can run php scripts via ssh command line, but in the browser I keep getting 'File not found.' errors on all my PHP files.

This is how I have my nginx.conf handle PHP scripts:

      location ~ \.php$
      {
              root                    /opt/nginx/html;
              fastcgi_pass            unix:/tmp/fpm.sock;
              fastcgi_index           index.php;
              fastcgi_param           SCRIPT_FILENAME /opt/nginx/html$fastcgi_script_name;
              include                 fastcgi_params;
      }

This is a direct copy/paste from my other servers, where it works fine with this setup (but they run older versions of php/fpm).

Why am I getting those errors?

Is it possible in the nginx.conf file to limit incoming POST requests to just a few pages?

Say I only want to allow post requests on signup.php and login.php but not on all other possible pages.

Reason why I ask is that my server often gets flooded with POST requests on random pages, which causes unnecessary load.

I tried blocking it in PHP, but in this case nginx still has to process and accept the full POST data. This i'm trying to prevent.

my webserver running centos 5.7 is showing quite a lot of disk write activity which I cannot explain really.

I run a few websites on this server with the help of nginx and php-fpm and xcache. On the file-system i have added noatime.

Now this is running fine, but I'm seeing a high number of writes per second that I cannot explain. There aren't any files being added, I have disabled the nginx access log. The php-fpm error log and nginx error log are only getting a few lines added to it per minute. I've checked all other logs in /var/log and nothign accounts for the number of writes that are happening.

On average there are about 2 megabytes of data per minute being written, at about 200 IO's per second.

How could I find out what is causing these writes?

I run a rather busy site, and during peek hours I see over 10.000 open connections to my database server on my webserver when a run a netstat command. 99% of the connections are in the TIME_WAIT state.

I learned about this mysql variable: wait_timeout http://dev.mysql.com/doc/refman/5.1/en/server-system-variables.html#sysvar_wait_timeout today. Mine is still set at the default 28.800 seconds.

Is lowering this value safe?

Non of my queries usually takes over a second. So it seems silly to keep a connection open for 480 minutes.

I also heard about using mysql_pconnect instead of mysql_connect, but i've been reading nothing but horror stories about it, so I think i'll stay away from that.

I have a 24GB machine running MySQL (only InnoDB tables).

My innodb data set is 11.5GB big, and i have set my innodb_buffer_pool_size to 14GB to allow some growth. So the entire innodb data set is loaded into the ram, which is good.

But today i noticed my indexes size is over 11GB as well.

Would it be a good idea to increase my innodb_buffer_pool_size so that the index space would fit as well? Or does that have nothing to do with it?

I run a CentOS 5.7 64 machine with 24gb ram and running kernel 2.6.18-274.12.1.el5.

This machine runs only Nginx, php-fpm and Xcache as extra applications.

Since about 3 weeks my memory behavior on this machine has changed and I cannot explain why. There are no crons running which flush anything like this. There are also no large numbers of files being deleted/changed during these drops.

The 'cached' memory gets dropped about every few hours, but it's never a set gap between flushes, this indicates to me that some bottleneck gets reached instead. It also always seems to be when total memory usages gets to about 18GB, but again, not always exactly 18GB.

This is a graph of my memory usage:

As you can see in the graph the 'buffers' always stay more or less the same, it is mainly the 'cache' that gets dropped.

Running vmstat -m I have outputted the memory usage just before and just after a memory drop. The output is here: http://pastebin.com/diff.php?i=hJqZqztm 'old version' being before, 'new version' being after a drop.

About 3 weeks ago my server crashed during a heavy DDOS attack, after I rebooted the machine this odd behavior started. I have checked a bunch of logs, restarted the machine again, and cannot find any indication what changed.

During these 'cache' memory drops, my iNode usage drops at the same time.

Does anyone have any idea what might be causing this behavior? Clearly my RAM isn't full, so I am curious why this could be happening.

I run a CentOS 5.6 (64bit) machine that has Nginx (latest version) running, with php-fpm (latest version). Things run very well, but since about 2 weeks I noticed in my Munin graphs that about every 2 hours the 'cache' usages drops. Before it used be a steady fully graph, that didn't seem to reset every so often.

PHP-FPM settings:

pm.max_children = 300
daemonize = yes
pm = static
listen = /tmp/fpm.sock
pm.max_requests = 1000

I have checked the php-fpm.log, and about once per 5 seconds a child process is killed, and restarted. But this is all the time, so this does not explain the sudden drops.

I only run Nginx, PHP (via fpm), Munin and vsftpd on this machine. No crons run at exactly the time of the drops.

My question: What could be causing these drops in cache usage?

on my site I log every pageview (date,ip,referrer,page,etc) in a simple mysql table.

This table gets very little selects (3 per minute), but a lot of inserts. (about 100 per second)

Today I changed this table from an InnoDB table to a MEMORY table, this made sense to me to prevent unnecessary hard disk IO. I also prune this table once per minute, to make sure it never get's too big.

--

Performance wise, things are running fine. But I noticed that while running tuning-primer, that my Current Lock Wait ratio is quite high.

Current Lock Wait ratio = 1 : 561

My question: Should I worry about this Lock Wait Ratio? And is there something I can change in my my.cnf to improve things so that the lock wait ratio isn't so high?

I work for a rather busy internet site that is often gets very large spikes of traffic. During these spikes hundreds of pages per second are requested and this produces random 502 gateway errors.

Now we run Nginx (1.0.10) and PHP-FPM on a machine with 4x SAS 15k drives (raid10) with a 16 core CPU and 24GB of DDR3 ram. Also we make use of the latest Xcache version. The DB is located on another machine, but this machine's load is very low, and has no issues.

Under normal load everything runs perfect, system load is below 1, and PHP-FPM status report never really shows more than 10 active processes at one time. There is always about 10GB of ram still available. Under normal load the machine handles about 100 pageviews per second.

The problem arises when huge spikes of traffic arrive, and hundreds of page-views per second are requested from the machine. I notice that FPM's status report then shows up to 50 active processes, but that is still way below the 300 max connections that we have configured. During these spikes Nginx status reports up to 5000 active connections instead of the normal average of 1000.

OS Info: CentOS release 5.7 (Final)

CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GH (16 cores)

php-fpm.conf

daemonize = yes
listen = /tmp/fpm.sock
pm = static
pm.max_children = 300
pm.max_requests = 1000

I have not setup rlimit_files, because as far as I know it should use the system default if you don't.

fastcgi_params (only added values to standard file)

fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_intercept_errors on;

fastcgi_pass            unix:/tmp/fpm.sock;

nginx.conf

worker_processes        8;
worker_connections      16384;
sendfile                on;
tcp_nopush              on;
keepalive_timeout       4;

Nginx connects to FPM via Unix Socket.

sysctl.conf

net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 1
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.all.log_martians = 1
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.conf.all.rp_filter=1
net.ipv4.conf.default.rp_filter=1
net.ipv4.conf.eth0.rp_filter=1
net.ipv4.conf.lo.rp_filter=1
net.ipv4.ip_conntrack_max = 100000

limits.conf

* soft nofile 65536
* hard nofile 65536

These are the results for the following commands:

ulimit -n
65536

ulimit -Sn
65536

ulimit -Hn
65536

cat /proc/sys/fs/file-max
2390143

Question: If PHP-FPM is not running out of connections, the load is still low, and there is plenty of RAM available, what bottleneck could be causing these random 502 gateway errors during high traffic?

Note: by default this machine's ulimit's were 1024, since I changed it to 65536 I have not fully rebooted the machine, as it's a production machine and it would mean too much downtime.

I am setting up a nginx webserver with php-fpm and (d)dos deflate to ban attacks.

Now currently there is no traffic to my server at all, as i'm testing things.

With this command i can see who is connected to my server, and how many connections they have open:

netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

During testing I noticed that when I would load a test script which is basicly <?php phpinfo(); ?> it would start 3 connections. I guess 1 for the HTML an 2 for the 2 images on that page. All is fine so far...

But I noticed it took well over a minute before those 3 connections where closed. I kept running the above netstat command to see if those 3 external connections would close.

My nginx.conf has a keep alive timeout of 4.

  keepalive_timeout       4;

The connection was made via a default setup Chrome browser.

How come those connections stayed open so long, and is this normal? Also, is there a way I can close them sooner?

I work for a site that often gets attacked by bot networks. We have started to use this tool: http://deflate.medialayer.com/ which auto-bans IPs that have more open connections than the set value. By default it's set to 150, we are currently using 250.

I would like to know, how low is safe so that search bots and normal visitors do not get blocked?

today at 02:20 I noticed that suddenly Munin stopped updating its graphs. It is not showing blank data, the whole graphs are not being updated anymore.

I run CentOs 5.7 (64) and ive installed Munin via yum. All was working, till 02:20.

First thing I did was restart Munin: service munin-node restart.

It restarted without problems or errors, but it still didn't update the graphs.

So I checked out all the log files in: /var/log/munin and apart from munin-node.log non of the files were updated since 02:20.

All that was added to munin-node.log was:

2011/12/06-02:57:19 Server closing! Process Backgrounded 2011/12/06-02:57:20 Munin::Node::Server (type Net::Server::Fork) starting! pid(16603) Using default listen value of 128 Binding to TCP port 4949 on host 127.0.0.1 Setting gid to "0 0"

On every restart I tried after the same message showed up.

I am really clueless to why Munin stopped working all the sudden. Any idea what I can try?

I run a large website (500k uniques a day) and am about to move to another hoster.

My new machine is setup, and tested, all files are copied over, so I am almost ready to change the my domain's IP address at my registrar.

Now I would like to know if there is a away to minimize the amount of people hitting the old server, because their DNS information isn't updated yet.

Sometimes it can take ages to update, and people hitting the old server would bring my website out of sync.

Is there a way to force forward people to the new machine from my old machine?

I have been under constant DDOS attack the last couple of weeks.

Now it seems my servers network is being flooded till it just doesn't have space anymore to receive and send normal packages.

I run CENTOS 5.6, and i have hardened SYSCTL and iptables to fight off SYN attacks as much as possible.

I have a 100Mbit network card and connection to my hosting company.

Normal incoming traffic is around 8mbit/s. Incoming data spikes go up to 100mbit when the attacks happen.

Could it help me fight off attacks if i were to upgrade my server to a 1GBPS network card + network connection?

I'm hoping the pipe doesn't flood so quickly when it's under attack.

my website sometimes gets attacked by people. You can see such an attack at 18:00 and later on a bigger attack at 22:30. Basicly the servers network card gets flooded by incoming requests.

My machines is a dedicated quad core, 12GB ddr3, 4x SAS 15k RPM drives in RAID10 with CentOS5.6 64bit.

On this server I run Nginx as my webserver. During an attack I am unable to access my site because the whole network seems to be flooded. Once it stops, everything is back to normal, without the need to restart anything.

I have hardened my SYSCTL a little, see settings here: http://pastebin.com/eFfAcWkr My IPTABLES config, its quite basic really. Only need port 80, 443, 21 and my SSH port open : http://pastebin.com/MsHSka08

My question is: What can I further do to fight off these kind of attacks? Also, is there a way for me to find out what kind of attack it was exactly?*

Green line is incoming data, blue one is outgoing data.

I'm in need to upgrade my MySQL machine, as currently I am running 2x SATA 7200rpm drives in RAID0. This of course is quite dangerous incase one of the drives fails.

As I do not need too much storage (all my db's together are about 20 - 25gb), I have been thinking about getting 2 SSD's (80 or 120gb) and putting them in RAID1 setup, so incase one of the SSD's fails, there is a backup present.

Price wise, I can also go for a RAID10 setup with 4x SATA drives. But I think 2 SSD's in RAID1 will still perform better.

For a heavily used MySQL machine, do you think the latest generation SSD's are reliable enough yet? And would this be a recommendable setup?