It's an InnoDB database. The query cache has 1.2GB in size with 300k queries in the cache. Most of the cached queries are related to 3 major tables (260MB data length plus 600MB index length altogether). Everytime I tried to insert new entry or update existing entries in these 3 tables, MySQL slowed down a lot just like the server is dead. I got "Timeout" if I open my website in browser.
I check the Munin stats, it shows 95% of the query cache are cleared after the operation so I guess the sudden overload was caused by inserting entries to heavy-read tables. And during the overload, Munin just shows a white strip on the graph so you may get an idea on how bad the situation was.
How can I optimize this?
Download mysqltuner.pl and run it against your database server:
http://blog.mysqltuner.com/download/
Something sounds horribly mis-optimized.
If your tables are all in InnoDB format, you should first tune the
innodb_buffer_pool_size
setting. Usually I tune this value to be about 80% of my system memory. Also, check the value ofinnodb_flush_log_at_trx_commit
. If it's set to the default of1
, it will flush at every transaction commit which can cause a lot of disk I/O (which appears to be what you're suffering from). If you don't require your database to be 100% ACID compliant, you should set this value to0
or2
. Here's the relevant snippet from MySQL pages:I usually set this to
2
. Good luck!