I am trying to determine what happened on a server where MySQL started responding with too many connections for almost an hour before we had to restart the server because MySQL would not restart itself, nor could we kill it.
So my question is once a MySQL server/instance has reached its maximum number of connections, will it refuse connections until it's restarted or will it automatically allow more connections once the number of connections has dropped below it's max?
Although the site has some pages that connect to MySQL for an extended period of time and have queries that take sometimes more than 30 seconds to complete, I would have assumed that once complete, even with extremely high load (for this site anyway) MySQL would have continued accepting new connections once the load was reduced again.
The too many connections error page on the MySQL website doesn't mention anything about what should happen.
The max_connections option refers to the maximum number of simultaneous connections. If your server had performance issues it's possible that you had a lot of scripts that were waiting for free connections and were grabbing them as soon as they became available.
You should also monitor
sudo netstat -anp | grep mysqldto see how many connections are actually open. My favorite cause for a crawling server is thrashing of course. Combine a optimistically-configured Apache, a memory-hungry webapp and a mob of users suddenly appearing and you get a constantly swapping server. In a one-disc setup, this also slows MySQL that cannot get enough RAM to work efficiently. So it starts punching the disc too. Plus: Waiting users hit refresh -> another webapp instance -> another connection that stays busy much too long.Use
vmstat 10(ignore first line, it's an uptime history) and look for the swap entries (in KiByte/s). Swapping in production is Not Good(TM).