I have a Linux server which I have installed Java on.
Sometimes, and only sometimes when a large number of visitors visit my website, the site hangs.
When I open the terminal and enter the "top" command to see whats going on, I can see that "Java
" process is eating CPU! Like 400%.
I have also tried ps aux
command, and can see that the command is from usr/bin/java
I have little experience in troubleshooting this kind of things, so I turn to you guys for help.
I have a java container installed (Jetty) which I must have in order to use SOLR (search engine) which is integrated into my website.
I can start and stop SOLR by:
etc/init.d/solr stop
But this didn't remove the java process from the "Top" command. Still java was eating 400% CPU.
Is there other methods to restart java only?
This has happened twice to me, and each time I have now restarted my entire servers and everything is fine.
If you need more input let me know!
If you stop the solr process and a java process is still running, then there's another java process on your server. The first step is to document all of the Java processes that are running. A good tool for this on Unix is the
ps
tool. Try this:That output should show you all of the java processes running, and the commands that are being executed. Try this before and after you stop solr.
The second question is "which of these java processes is eating up so much CPU"? You may need to stop the jetty process in addition to the solr process. Also, you should never really need to restart a Linux server just because a single process is misbehaving. You can also use the
kill
command to stop a process if you now it's process id, which you can get from eithertop
orps
.In the short term it may be a good idea to install some sort of "watchdog" script on your machine to help with these situations. For example,
monit
can be used to automatically restart a service or process when it consumes a certain amount of CPU resources.In the long term, I'm sorry to say that you have a performance issue. You need to look at reconfiguring solr and jetty at the very least. You may also need to look into garbage collection tuning and possibly adding more hardware. There's lots of information on these topics online, and I'm sorry to say that this process can be somewhat difficult.
Good luck!
Tom Purl
If the CPU is getting maxed out while response times are dropping, then the first place I'd be looking is the heap management within the java - it's probably doing lots of major collections - and the quickest way to solve the problem is to just give the container more memory to run in.
This isn't really the right place to ask program debugging question. If Solr is the only Java application running you should definitely check how often is your Lucene index being updated and how many search queries are you running per second when the server starts gets overloaded.