chaintng's questions

In our environment, we have 2 MySQL (Master & Replica)

Today, we have really strange case.

The data in master sql get frozen for an hour! By frozen, i mean, there is no new record to, AT LEAST, 1 table (as far as we know)

But it still accessible by read.

This happened in production environment.

All application still can access, insert, update.

But nothing changes in the database itself.

After that frozen window pass, all the queries during that frozen window get executed!

We suspect it's query lock issue. But i checked mysql.slow_log and can't find any suspect queries. All queries get executed under a minute.

My question is..

1. Is it deadlock issue? Why does it not log to mysql.slow_log table?
2. How to prevent this? MySQL doesn't have query timeout as default value? How come 1 hour query block everything.
3. Can we specified parameter to log the queries that has high lock time?

Thank you so much.

P.S. I'm using CloudSQL mysql 5.7.11 in

I have 2 services in CentOS 7 - nginx - php-fpm

Those services have main configuration to include *.conf in certain directory.

However, it didn't read the symbolic link file in that certain directory.

Please see screenshot for better clarification.

When i tried hard copy file to that certain directory, it's working.

Do you have any idea? is it because of centOS issue? Any configuration? The docs in php-fpm configuration saying that...

; Include one or more files. If glob(3) exists, it is used to include a bunch of
; files from a glob(3) pattern. This directive can be used everywhere in the
; file.

I'm not sure what does it mean by glob(3).

Please advise,

This is output after i run df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        30G   19G  9.1G  68% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            487M  4.0K  487M   1% /dev
tmpfs           100M  348K  100M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            498M     0  498M   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sda         25G  6.4G   17G  28% /mnt/volume-sgp1-01

I want to identify what are 19G being used.

So i run below command

du -h --max-depth=10 --one-file-system / | sort -rh | head -25

to get following output

14G /
12G /var
8.8G    /var/lib
8.6G    /var/lib/mysql
8.1G    /var/lib/mysql/getapple_phpbb
1.6G    /var/www
916M    /usr
546M    /var/www/tskthai.com/public_html
546M    /var/www/tskthai.com
517M    /var/www/tskthai.com/public_html/wp-content
464M    /lib
431M    /var/www/ereaderok.com/public_html
431M    /var/www/ereaderok.com
396M    /home
381M    /var/www/himaparn.com/public_html
381M    /var/www/himaparn.com
378M    /var/lib/mysql/getapple_paysmile
368M    /lib/modules
306M    /var/www/tskthai.com/public_html/wp-content/uploads
295M    /usr/lib
255M    /var/www/getapple.net/public_html
255M    /var/www/getapple.net
250M    /opt
244M    /usr/share
242M    /opt/datadog-agent

you can see that the main directory being used is / which is only 14gb

So where are the remaining 5GB are being used?

Please advise.

I tried to set up redis sentinel to make it has high availability and support fail over. I begin by setup the redis replication (1 master and 2 slaves in 3 ec2-instances)

And after that, i install redis-sentinel service for each instance.

However, when i tried to access redis-sentinal remotely (from my local machine) It shows following message

DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients....

So, i tried to set requirepass in sentinel.conf. But after i run redis-cli -h REDIS-IP -a PASSWORD -p SENTINEL_PORT INFO

It shows following

NOAUTH Authentication required.

Is there anyway i could connect to redis-sentinel remotely?

Please advise.

My MySQL service sometimes shutdown itself at around 2AM of the day. I cannot find which part of this log file explain the cause.

I host it on DigitalOcean. I am not sure is it because of their backup and snapshot things or not.

Please advise.

150812  2:27:48 [Note] Plugin 'FEDERATED' is disabled.
150812  2:27:48 InnoDB: The InnoDB memory heap is disabled
150812  2:27:48 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150812  2:27:48 InnoDB: Compressed tables use zlib 1.2.8
150812  2:27:48 InnoDB: Using Linux native AIO
150812  2:27:48 InnoDB: Initializing buffer pool, size = 256.0M
InnoDB: mmap(274726912 bytes) failed; errno 12
150812  2:27:48 InnoDB: Completed initialization of buffer pool
150812  2:27:48 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150812  2:27:48 [ERROR] Plugin 'InnoDB' init function returned error.
150812  2:27:48 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150812  2:27:48 [ERROR] Unknown/unsupported storage engine: InnoDB
150812  2:27:48 [ERROR] Aborting

150812  2:27:48 [Note] /usr/sbin/mysqld: Shutdown complete

150812  2:27:49 [Note] Plugin 'FEDERATED' is disabled.
150812  2:27:49 InnoDB: The InnoDB memory heap is disabled
150812  2:27:49 InnoDB: Mutexes and rw_locks use GCC atomic builtins
150812  2:27:49 InnoDB: Compressed tables use zlib 1.2.8
150812  2:27:49 InnoDB: Using Linux native AIO
150812  2:27:49 InnoDB: Initializing buffer pool, size = 256.0M
InnoDB: mmap(274726912 bytes) failed; errno 12
150812  2:27:49 InnoDB: Completed initialization of buffer pool
150812  2:27:49 InnoDB: Fatal error: cannot allocate memory for the buffer pool
150812  2:27:49 [ERROR] Plugin 'InnoDB' init function returned error.
150812  2:27:49 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
150812  2:27:50 [ERROR] Unknown/unsupported storage engine: InnoDB
150812  2:27:50 [ERROR] Aborting

My dedicated server has a problem after i reboot itself. The cpu load average is very high as following detail after i run top command

top - 23:40:41 up 50 min,  3 users,  load average: 236.24, 146.96, 124.29
Tasks: 556 total,   1 running, 555 sleeping,   0 stopped,   0 zombie
Cpu(s):  1.2%us,  0.2%sy,  0.0%ni,  0.0%id, 98.6%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16230212k total,  2994040k used, 13236172k free,    26404k buffers
Swap:  2097144k total,        0k used,  2097144k free,  

i tried to stop httpd, it show 'OK' but after i run "service httpd status" it still show it's running.

there are many process related to httpd after i run "ps -ef | grep httpd"

 apache    7984  7209  0 23:42 ?        00:00:00 /usr/sbin/httpd -k start -DSSL
 apache    7985  7209  0 23:42 ?        00:00:00 /usr/sbin/httpd -k s

I have no idea what it is but it keep duplicate itself every second (the pid run very fast)

after i dig into the unix log (var/log/message) it show some log which may related to harddisk, i'm not quite sure, is it?

Nov 10 00:16:13 host kernel: ata1.00: exception Emask 0x0 SAct 0x1 SErr 0x0 action 0x0
Nov 10 00:16:13 host kernel: ata1.00: irq_stat 0x40000008
Nov 10 00:16:13 host kernel: ata1.00: failed command: READ FPDMA QUEUED
Nov 10 00:16:13 host kernel: ata1.00: cmd 60/08:00:f0:e1:4a/00:00:6b:00:00/40 tag 0 ncq 4096 in
Nov 10 00:16:13 host kernel:         res 41/40:08:f0:e1:4a/00:00:6b:00:00/00 Emask 0x409 (media error) <F>
Nov 10 00:16:13 host kernel: ata1.00: status: { DRDY ERR }
Nov 10 00:16:13 host kernel: ata1.00: error: { UNC }
Nov 10 00:16:13 host kernel: ata1.00: configured for UDMA/133
Nov 10 00:16:13 host kernel: ata1: EH complete

Please advice me how should i do next to have my server become normal again.

Best Regards,