I have a system running 2 processes - MySQL and Java
I'm seeing 20-30% of the time spent in %system. I've looked around to try and find a way to identify which what the balance of the system time is between the processes, i.e. is it the MySQL processes using 90% of that system time, or is it Java?
I/O load is low, 1-2% utilisation figures in IOSTAT. We're not doing any network traffic. So I'm wondering if this is context switching. The MySQL process has about 400 threads and the java process has 3000. I want to be able to measure/prove that these thread numbers are the problem though and not just guess.
This is a batch process so we are actively trying to run as fast as possible, maximum CPU with minimum overhead.
From the pastebin, it looks like you're swapping, so the system time would be the kernel swapping pages for your Java application. If the usage is unacceptable to you, lower the amount of RAM java can eat up or increase the amount of RAM on your server.
I don't think context switching is playing a major part since while I do see
ksoftirqd
running (which usually means at one point there was a lot of interrupts), its CPU time is very low compared to your application.