We are using Fedora-8 64 bit in Amazon EC2 and our Java application is running on the IBM JRE. Now we are getting a Java hang "out of memory" error. Our memory is 7.6 GB and we have allotted heap memory -Xms3000M -Xmx3000M. We are still getting the same problem. The question is, if I increase the heap size then will this problem be rectified? Or is there any other alternative to solve this problem?
3G is sound like large enough amount of memory assigned to java heap, you may get not heap out of memory but permanent generation (PermGen). Consider increase PermSize and MaxPermSize.
To solve problem you may need monitor how fast and which part of JVM memory is growing. Use JMX console, also you may look deeper - take snapshot of running java heap and analyse it to define problematic places (http://www.eclipse.org/mat/) also thread dump and object histogram may help.
Fedora-8 is a four year old release. You should seriously considering upgrading your environment to the latest stable release of your distro and the EC2 kernels before trying to track down any mis-behaving software memory issues.
You shouldn't be using a "JRE" version of Java. You should be using a version of Java that supports the -server VM. It could be that since your running a Java that is a JRE, that it has limits on JVM size since its defaulting to a non-server VM. Its easy to investigate this: just looking in your JRE directory for the "server" directory to see if you have it. I don't know anything about the IBM JRE though, and so your milage may vary on my advice.
I would recommend trying the latest version of the Oracle JDK instead since I am 100% sure it has a -server implementation in it. Or, maybe a JDK version of the IBM vm, if you can get it.
I say all these things because your question states that your explicitly using a JRE.