Oracle Application Server Performance Monitoring and Tuning (CPU load high)
i have just hired by a company and my boss give me a performance issue to solve as soon as possible. I don't have any experience with the Java EE before at the server side.
Let me begin what i learned about the system and still couldn't find the solution:
We have an Oracle Application Server (10.1.) and Oracle Database server (9.2.), the software guys wrote a kind of big J2EE project (X project) using specifically JSF 1.2 with Ajax which is only used in this project. They actively use PL/SQL in their code.
So, we started the application server (Solaris machine), everything seems OK. users start using the app starting Monday from different locations (app 200 have user accounts,i just checked and see that the connection pool is set right, the session are active only 15 minutes).
After sometime (2 days) CPU utilization gets high,%60, at night it is still same nothing changed (the online user amount is nearly 1 or 2 at this time), even it starts using the CPU allocated for other applications on the same server because they freed If we don't restart the server, the utilization becomes %90 following 2 days, application is so slow that end users starts calling.
The main problem is software engineers say that code is clear, and the System and DBA managers say that we have the correct configuration,the other applications seems OK why this problem happens only for X application.
I start copying the DB to a test platform and upgrade it to the latest version, also did in same with the application server (Weblogic) if there is a bug or not. i only tested by myself only one user and weblogic admin panel i can track the threads and dump them. ı noticed that there are some threads showing as a hogging. when i checked the manuals and control the trace i see that it directs me the line number where PL/SQL code is called from a .java file. The software eng. says that yes we have really complex PL/SQL codes but what's the relation with Application server? this is the problem of DB server, i guess they're right...
I know the question has many holes, i'd like to give more in detail but i appreciate the way you guide me.
Thanks in advance ...
Edit: The server both in CPU and Memory enough to run more complex applications
Heavy PL/SQL calls should block the thread - so the CPU usage should drop.
My first port of call for a slow application server is to check the gc logs - looking for frequent major collections (which indicate either a memory leak or that the JVM simply eeds more memory).
The systems I look after became a lot more stable after switching from the thick Oracle drivers to the lightweight jdbc drivers - although the issues had mainly manifested as the container crashing.
The logs should be a good indicator of any problems on the system - but a lot depends on what the developers choose to write in there. Slow SQL could result in the connection pool being exhausted - make sure that the pool is logging connection stats. Also make sure the ulimit is set correctly for the JVM.
Since you're running 9i at the DB tier you wont have the AWR functionality - you'll have to run the statspack (but this should already be standard practice for your sites performance management) to identify what's causing problems at the DB.
THe gradual degrading of performance is indicative of a memory leak in the application - usually this is caused by objects not being de-referenced and hence eligible for garbage collection - i.e. a programming issue. This should be apparent from most Java profiling tools.
Unless you're testing this with a realistic workload the results are pretty much useless.