Karem's questions

I have an issue with the following error:

Too many open files (24)

And after lots of hours researching, we noticed that Open_files status do show a count of files opened, but the error occured when this number was below the open_files_limit setting? How is this possible?

We are using table_open_cache setting as well and after reading how MySQL documentation about table_open_cache and how MySQL opens and closes tables, we start to think it's because that the status count open_files does not calculate the tables opened that goes into table_open_cache, but not sure.

Any clear answer on how this can be possible?

Ubuntu 14.04, HHVM 3.10.1

Running df -h I can see the following:

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G  8.0K  7.9G   1% /dev
tmpfs           1.6G  1.6G  8.0K 100% /run
/dev/xvda1      9.8G  2.1G  7.2G  23% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            7.9G     0  7.9G   0% /run/shm
none            100M     0  100M   0% /run/user

What's taking up the full space in tmpfs is actually the file /run/hhvm/hhvm.hhbc which is 1.6 GB currently. Is this healthy? It does not look like it is.

I have found the following, related to the full tmpfs: https://github.com/hhvm/packaging/issues/132

"On my Ubuntu (14.04) /var/run is a tmpfs mount that's only 400MB in size. The hhvm.hhbc file grew to a little over 350MB on one system. I think it may make more sense to default to putting this file under /var/cache/hhvm instead."

My central path is just as explained, defined to /var/run/hhvm/hhvm.hhbc - but i do not have a /var/cache/ in my filesystem, so i wonder if i really should change the central path to /var/cache/hhvm instead? Also, wouldn't this be bad / not be temporary since tmpfs is mounted to only /run?

I am debugging since my HHVM is crashing periodically, which requires that i run a reboot of hhvm service/remove the hhvm.hhbc before everything works again.

Suddenly, today i see the website down.

Checking the cpu with "top" in command, i found the mysqld_Safe at the top using all of the cpu.

After some google i found some different solutions, i made a "reboot" that restarted everything and luckily everything came back on.

Now it's time to find out why this could happen and prevent it.

I check the mysql log and find this:

Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140914 13:06:32 [Note] /usr/libexec/mysqld: Normal shutdown

140914 13:06:32 [Note] Event Scheduler: Purging the queue. 0 events
140914 13:06:32  InnoDB: Starting shutdown...
140914 13:06:36  InnoDB: Shutdown completed; log sequence number 0 37356736
140914 13:06:36 [Note] /usr/libexec/mysqld: Shutdown complete

140914 13:06:36 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140914 13:07:07 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140914 13:07:08  InnoDB: Initializing buffer pool, size = 8.0M
140914 13:07:08  InnoDB: Completed initialization of buffer pool
140914 13:07:08  InnoDB: Started; log sequence number 0 37356736
140914 13:07:08 [Note] Event Scheduler: Loaded 0 events
140914 13:07:08 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
141222 00:16:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
141222  0:16:49  InnoDB: Initializing buffer pool, size = 8.0M
141222  0:16:49  InnoDB: Completed initialization of buffer pool
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
141222  0:16:49  InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
141222  0:16:53  InnoDB: Started; log sequence number 0 254950608
141222  0:16:53 [Note] Event Scheduler: Loaded 0 events
141222  0:16:53 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

And normally it look like this:

Version: '5.1.73'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution
140913 23:44:50 [Note] /usr/libexec/mysqld: Normal shutdown

140913 23:44:50 [Note] Event Scheduler: Purging the queue. 0 events
140913 23:44:50  InnoDB: Starting shutdown...
140913 23:44:51  InnoDB: Shutdown completed; log sequence number 0 36460121
140913 23:44:51 [Note] /usr/libexec/mysqld: Shutdown complete

140913 23:44:51 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
140913 23:45:18 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
140913 23:45:18  InnoDB: Initializing buffer pool, size = 8.0M
140913 23:45:18  InnoDB: Completed initialization of buffer pool
140913 23:45:18  InnoDB: Started; log sequence number 0 36460121
140913 23:45:18 [Note] Event Scheduler: Loaded 0 events
140913 23:45:18 [Note] /usr/libexec/mysqld: ready for connections.

Which clearly says something: That it didnt shut down normally and it started a crash recovery.

Now my question is:

1. Why is it even shutting down and starting? Why is there such "event" (as far as i understand?) - can this be disabled.

2. How can i prevent this crash? If there is any clear answer to this problem.

So I just got my certificate issued from Digicert, and since there was other unused/old files in /etc/ssl/ i marked and removed, and accidently also removed localhost.crt.

Now I cannot start my web server (only without SSL). How do i recreate this localhost.crt file? And would i need to make a new key and get my certificate from digicert reissued?