I have a server with Dual Nehalem Quad Core Xeon 5520 and 12GB DDR3 RAM. The server's average memory usage is only around 10-15%, yet the Swap used shows 10%+. Is this normal, or could something be wrong that is causing this. I was under the impression that Swap was only used if there wasn't enough memory available.
I am running Apache/2.0.63 on CentOS 5.3.
There's nothing wrong with this. The kernel will page into virutal memory gradually over time. This can also occur if your server has activity "spikes" and the memory you have currently free quickly fills up. The pressure on the memory system causes page-outs to occur until the pressure is reduced to a set point (which you can examine in /proc/sys/vm). Even on a fairly idle system, I have seen gradual page-outs over time. So, unless the swap is fairly active (lots of page faults resulting in paging activity) I wouldn't worry about this.
If you're really that concerned, you can always turn swapping off, then on. This will force those swapped pages back into memory. I don't recommend doing this, but you can if you want. Just be sure that you have sufficient free memory before you do this.
You shouldn't worry about this - swap will be used if the kernel deems that the data is not going to be accessed enough to warrant having it in the physical memory.
If you'd like to monitor "bad" swap usage, you're better off monitoring the rate at which swap reads and writes are occurring, not simply if it's being used at all or not.
For example...
The first report tells you what the states (average) are since the machine was first brought up, the rest should you (in the case above) 3-second averages.
Notice the
siandsofields are mostly zero - that's what you're interested in.That is fine, as already pointed out usage spikes may cause this or the kernel may decide that some items are put into swap because it's smarter to use the RAM for something else (page cache).
As answers to other questions pointed out
sysctl vm.swappinessmay be of interest to you do avoid this.(posted as an answer, as it got far too long to add as an additional comment to Avery Payne's answer)
Also, if swap has been used during a peak period which has since passed you may find that most of the data in swap is also currently in RAM. If Linux reads pages back from swap in to RAM it doesn't immediately deallocate the swap space unless it needs it for more data, that way if it needs to swap the pages back out (and it knows that the data in them has not changed) it doesn't actually need to write the pages to disk as they are already there.
See /proc/meminfo for details. On a little server of mine currently:
So here ~66Mb of the 112Mb of swap space allocated is also currently present in RAM. There is no point it removing that 66Mb from swap as there is no demand for the space for other uses (there is plenty of completely free swap space). If swap gets full those pages will be reallocated, if the pages change in RAM they will be marked as dirty and free to be reallocated, but if they need to be swapped back out the kernel can save itself a bunch of disk writes.
If I force the disk caches and buffers to be cleared with
the result stays the same:
The "cached" reading stayed high after asking for the disk caches to be cleared as this is counts memory allocated to VMWare VMs too because of the way that is allocated. The SwapCached reading didn't alter as there is no point copying the pages back into RAM just because the RAM is now free - they might never be needed before the RAM gets allocated for something else again so those reads would be wasted.
The situation above is slightly different to yours in that this machine pretty much always has all its RAM allocated to something (VMs, other processes, I/O cache+buffers) but depending on your machine's load history since last boot it is not unlikely that a chunk of the pages in allocated space in your swap area are similarly also in RAM.