Home / user-299979

Hassaan's questions

Martin Hope
Hassaan
Asked: 2018-04-20 10:45:05 +0800 CST
  • 5

I am seeing warning messages in my MariaDB server logs. I have tried many things.

I have optimize and repair all of my databases but warning are still there.

2018-04-19 22:50:14 139673358411520 [Warning] Aborted connection 473891 to db: 'admin_' user: 'admin_' host: 'localhost' (Got timeout reading communication packets)

2018-04-19 22:52:43 139672739514112 [Warning] Aborted connection 474608 to db: 'admin_' user: 'admin_' host: 'localhost' (Got timeout reading communication packets)

My my.cnf

skip-external-locking
key_buffer_size = 256M
max_allowed_packet = 500M
table_open_cache = 256
sort_buffer_size = 1M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache_size = 8
query_cache_size= 16M
thread_concurrency = 8

innodb_file_per_table

max_connections=500
wait_timeout=60
interactive_timeout=60
long_query_time=5

I am using CentOS 7 with 8 GB of RAM.

Can someone help me?

mysql
Martin Hope
Hassaan
Asked: 2017-04-11 23:20:41 +0800 CST
  • 1

I am continuously receiving following errors in my php-fpm logs. I have read and tried some similar question available on serverfault.com but Issue does not resolved.

[11-Apr-2017 08:33:27] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 09:49:26] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 09:49:26] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 09:56:11] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 09:56:11] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:09:56] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:09:56] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:22:39] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:22:39] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:35:52] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:35:52] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:38:52] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:46:33] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:46:33] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:54:21] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:54:21] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 10:59:17] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:03:16] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:03:16] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:05:50] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:05:50] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:05:50] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:05:50] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:12:03] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:32:45] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:32:45] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:48:28] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:48:28] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:53:11] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:55:15] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 11:55:15] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 12:06:17] ERROR: unable to read what child say: Bad file descriptor (9)
[11-Apr-2017 12:06:17] ERROR: unable to read what child say: Bad file descriptor (9)

My OS is CentOS7 and I am using PHP7-FPM with nginx

Server specification are below

Processor: Intel Xeon E5-1620v2

Sever Ram: 64GB DDR3

My current fpm configurations are below

pid = /run/plesk-php70-fpm.pid

error_log = /var/log/plesk-php70-fpm/error.log

log_level = error

emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s

listen.backlog = 65535

;[php-fpm-pool-settings]
pm = ondemand
pm.max_children = 2000
pm.process_idle_timeout = 10s
#pm.max_requests = 10000

#pm = dynamic
#pm.max_children = 5000
#pm.start_servers = 50
#pm.min_spare_servers = 20
#pm.max_spare_servers = 70
#pm.max_requests = 10000

I have tried pm ondemand & pm dynamic both

nginx
Martin Hope
Hassaan
Asked: 2017-04-09 14:49:59 +0800 CST
  • 17

I am stacked with following errors on my site when I test 200 hits per second.

First I received 499 errors

2017-04-09 03:22:45 Error 162.158.79.219 499 GET / HTTP/1.1 0 nginx access

2017-04-09 03:22:45 Error 162.158.79.87 499 GET / HTTP/1.1 0 nginx access

2017-04-09 03:22:45 Error 162.158.78.170 499 GET / HTTP/1.1 0 nginx access

2017-04-09 03:22:45 Error 162.158.78.68 499 GET / HTTP/1.1 0 nginx access

2nd Error start showing 502

2017-04-09 03:22:45 Error 162.158.79.135 502 GET / HTTP/1.1 166 nginx access

2017-04-09 03:22:45 Error 162.158.79.225 502 GET / HTTP/1.1 166 nginx access

2017-04-09 03:22:45 Error 162.158.78.110 502 GET / HTTP/1.1 166 nginx access

2017-04-09 03:22:45 Error 162.158.79.225 502 GET / HTTP/1.1 166 nginx access

and Finally I start receiving php-fpm.sock failed errors

2017-04-09 03:22:45 Error 162.158.79.207 20699#0: *3826365 connect() to unix:///var/www/vhosts/system/playhdpk.top/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream nginx error

2017-04-09 03:22:45 Error 162.158.79.207 20695#0: *3826367 connect() to unix:///var/www/vhosts/system/playhdpk.top/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream nginx error

2017-04-09 03:22:45 Error 162.158.79.207 20697#0: *3826369 connect() to unix:///var/www/vhosts/system/playhdpk.top/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream nginx error

My php-fpm-pool-settings are below, I believe this one is being generating errors may I am wrong

listen.backlog = 65535

;[php-fpm-pool-settings]
pm = dynamic
pm.max_children = 5000
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 70
pm.max_requests = 2000

My nginx conf are below

user  nginx;
worker_processes 8;

# number of file descriptors used for nginx
# the limit for the maximum FDs on the server is usually set by the OS.
# if you don't set FD's then OS settings will be used which is by default 2000
worker_rlimit_nofile 99999;

error_log /var/log/nginx/error.log crit;

include /etc/nginx/modules.conf.d/*.conf;

events {

    worker_connections 16192;
    use epoll;
    multi_accept on;
}


http {
    include   mime.types;
    default_type  application/octet-stream;

    open_file_cache max=2048 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 5;
    open_file_cache_errors off;


    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_requests 100000;
    reset_timedout_connection on;
    client_body_timeout 30;
    send_timeout 15;

    client_header_timeout 12;
    proxy_connect_timeout  600s;
    proxy_send_timeout  600s;
    proxy_read_timeout  600s;

    fastcgi_buffers 8 128k;
    fastcgi_buffer_size 256k;
    fastcgi_send_timeout 600s;
    fastcgi_read_timeout 600s;

    types_hash_max_size 2048;

    gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/v$
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";


    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
}

Note: Server specifications are below

OS CentOS 7.3

Processor: Intel Xeon E5-1620v2 - 4c/8t - 3.7 GHz/3.9 GH

Sever Ram: 64GB DDR3

nginx
Martin Hope
Hassaan
Asked: 2017-04-07 01:28:51 +0800 CST
  • 0

My website generate when I tried to hit 1000 requests per second on it. I am seeing the following errors in my apache logs

Apr 06 00:00:33 ns503772******100.net httpd[9585]: AH00515: WARNING: MaxRequestWorkers of 2000 would require 80 servers and
Apr 06 00:00:33 ns503772******100.net httpd[9585]: would exceed ServerLimit of 16, decreasing to 400.
Apr 06 00:00:33 ns503772******100.net httpd[9585]: To increase, please see the ServerLimit directive.

My current setting was following

<IfModule mpm_event_module>
LoadModule cgid_module modules/mod_cgid.so
MaxRequestWorkers 2000
</IfModule>

I'm am using Plesk Onyx and server specification are following.

Processor: Intel Xeon E5-1620v2 - 4c/8t - 3.7 GHz/3.9 GH

Sever Ram: 64GB DDR3

Then I have tried the following settings

<IfModule mpm_event_module>
StartServers                80
MinSpareThreads             50
MaxSpareThreads             300

ThreadLimit                 25
ThreadsPerChild             25

ServerLimit                 500
MaxRequestWorkers           2000
MaxConnectionsPerChild      0
</IfModule>

but issue remain un solved, infarct its generating more errors & log-writer started eating CPU

output of httpd -V | grep -i 'version\|mpm' is following

Server version: Apache/2.4.6 (CentOS)
Server MPM:     event
apache-2.4
Martin Hope
Hassaan
Asked: 2017-03-02 00:02:25 +0800 CST
  • 1

My website playsongs.pk got 499, 502 and php-fpm.sock failed (11: Resource temporarily unavailable) errors when I received 300-400 active users on site.

Sample/example error logs:

2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.170  502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.79.63   502 GET / HTTP/1.1          166     nginx access
2017-02-27 14:56:23 Error   162.158.78.50   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access
2017-02-27 14:56:23 Error   162.158.78.20   499 GET / HTTP/1.1          0   nginx access

and After few time I got

2017-02-27 14:56:23 Error   172.68.65.4     24026#0: *2784527 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24023#0: *2784529 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.100       24028#0: *2784539 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784541 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784545 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24021#0: *2784547 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784549 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   162.158.78.50       24028#0: *2784551 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error
2017-02-27 14:56:23 Error   172.68.65.112       24028#0: *2784552 connect() to unix:///var/www/vhosts/system/playsongs.pk/php-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream               nginx error

My Nginx configuration /etc/nginx/nginx.conf:

user  nginx;
worker_processes auto; #some last versions calculate it automatically

worker_rlimit_nofile 99999;

error_log /var/log/nginx/error.log crit;

include /etc/nginx/modules.conf.d/*.conf;

events {

    worker_connections 66536;
    use epoll;
    multi_accept on;
}

http {
    include   mime.types;
    default_type  application/octet-stream;

    open_file_cache max=2048 inactive=20s;
    open_file_cache_valid 30s;
    open_file_cache_min_uses 5;
    open_file_cache_errors off;


    access_log off;

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;

    keepalive_timeout  2;
    keepalive_requests 100000;
    reset_timedout_connection on;
    client_body_timeout 30;
    send_timeout 15;

client_header_timeout 12;
proxy_connect_timeout  60;
proxy_send_timeout  60;
proxy_read_timeout  60;


fastcgi_buffers 8 128k;
fastcgi_buffer_size 256k;

# reduce the data that needs to be sent over network
    gzip on;
    gzip_min_length 1000;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types application/x-javascript text/css application/javascript text/javascript text/plain text/xml application/json application/v$
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";


    server_tokens off;

    include /etc/nginx/conf.d/*.conf;
}

# override global parameters e.g. worker_rlimit_nofile
include /etc/nginx/*global_params;

My php-fpm pool setting are following:

;[php-fpm-pool-settings]
pm = dynamic
pm.max_children = 5000
pm.start_servers = 50
pm.min_spare_servers = 20
pm.max_spare_servers = 70
pm.max_requests = 2000

Please note

Server have 64GB of RAM with 1 GBit dedicate INTERNET PORT and average load on CPU also does not exceed from 1.somthing % or 2.somthing %

Server OS: Centos 7.2

Control Panel: Plesk Onyx

nginx Version: nginx/1.11.4

Output of #cat /proc/$(cat /run/nginx.pid)/limits is following. May it will be helpful to solve the issue.

Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             515124               515124               processes
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       515124               515124               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                   
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us

Output of tail /var/log/nginx/error.log

2017/03/01 15:16:15 [notice] 31867#0: getrlimit(RLIMIT_NOFILE): 1024:4096
2017/03/01 15:16:15 [notice] 31868#0: start worker processes
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31869
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31870
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31871
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31872
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31873
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31874
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31875
2017/03/01 15:16:15 [notice] 31868#0: start worker process 31876
nginx php-fpm centos7 plesk