That requires some expert skills. It depends. Example:
If there's enough of memory and disks don't seem too busy, it may be CPU-bound. Look at CPU usage and if its bordering at 100% it's CPU bound. If it's not there's an artificial bottleneck in the implementation. E.g. on a dual-core CPU a single threaded process will not go above 50% CPU usage.
If CPU and memory are available, but disks are very busy, or IO latency seems high, its likely that its IO bound. See if adding more disks (RAID?) helps.
None of the above? Check memory available.
Enough memory? There may be an artificial bottleneck in the process itself i.e. maybe someone forgot to remove a sleep(1)? Naah its not that easy usually. ;)
There's a reason why we have a whole lab for performance engineers in most companies dealing with performance sensitive products!
Use tools like sar, vmstat, iostat, oprofile, lockstat, dtrace, product specific perf monitoring tools etc, to debug perf problems.
A tool that can be useful for real-time checking a number of process statistics (memory, cpu-usage, I/O, etc.) is htop. It doesn't replace the more specialised tools named by Sudhanshu, but might be a good start.
As well as the other tools mentioned, run ps l PID, inserting the relevant process id, or look at the STATE and WCHAN columns in top or htop.
If it's in D (for disk) state, then it's doing file IO. This could be because it's either reading a lot of files, or because it's using a lot of memory and swapping. The WCHAN column will tell you what kernel function it's inside; googling for them or asking here may give you some indication what they mean.
If it's in R (run) state, it's using the CPU in user space, in other words it's CPU bound at that moment.
If it's in S (sleep) state, it's inside an interruptible system call, which may mean it's either actually sleeping, or it's doing something like waiting for network traffic or a lock. Again, looking at the specific wchan will tell you more.
sudo apt install iotop htop # Ubuntu/debian based systems
or
sudo yum install iotop htop # Redhat/RPM based systems
Then have a quick look at htop - is the system swapping (fully filled memory bar)? If so, you want to fix that first. Swapping is super expensive in terms of performance.
If the system is not swapping, check out the cpu bars. Are they near 100% most of the time? Well, then your CPU is busy :)
Next, quit htop and start sudo iotop and check the top right numbers (total and actual disk write). How high are they?
If they about match the maximum reasonable throughput of your disk which is being read from / written too (and usually write operations are the bottleneck as disk write speeds are almost always lower then read speeds), then your IO is busy :)
Note I am referring to the actual reasonable throughput on your disk (what you were to approximately get if you copied a set of files manually when the server was otherwise idle), NOT the "label" performance, which is often impossible to achieve to start with.
Thus, doing a few simple check, you worked out if your system is memory, cpu or disk bound. If you want more detailed numbers or analysis, checkout some of the other tools, but this gives you a basic idea, which is often sufficient to help you scale etc.
That requires some expert skills. It depends. Example:
If there's enough of memory and disks don't seem too busy, it may be CPU-bound. Look at CPU usage and if its bordering at 100% it's CPU bound. If it's not there's an artificial bottleneck in the implementation. E.g. on a dual-core CPU a single threaded process will not go above 50% CPU usage.
If CPU and memory are available, but disks are very busy, or IO latency seems high, its likely that its IO bound. See if adding more disks (RAID?) helps.
None of the above? Check memory available.
Enough memory? There may be an artificial bottleneck in the process itself i.e. maybe someone forgot to remove a sleep(1)? Naah its not that easy usually. ;)
There's a reason why we have a whole lab for performance engineers in most companies dealing with performance sensitive products!
Use tools like sar, vmstat, iostat, oprofile, lockstat, dtrace, product specific perf monitoring tools etc, to debug perf problems.
check out
iotop, can be usefulA tool that can be useful for real-time checking a number of process statistics (memory, cpu-usage, I/O, etc.) is htop. It doesn't replace the more specialised tools named by Sudhanshu, but might be a good start.
As well as the other tools mentioned, run
ps l PID, inserting the relevant process id, or look at the STATE and WCHAN columns in top or htop.If it's in D (for disk) state, then it's doing file IO. This could be because it's either reading a lot of files, or because it's using a lot of memory and swapping. The WCHAN column will tell you what kernel function it's inside; googling for them or asking here may give you some indication what they mean.
If it's in R (run) state, it's using the CPU in user space, in other words it's CPU bound at that moment.
If it's in S (sleep) state, it's inside an interruptible system call, which may mean it's either actually sleeping, or it's doing something like waiting for network traffic or a lock. Again, looking at the specific wchan will tell you more.
See also What is the "Waiting Channel" of a process?
Run
topand look at the cpu usage line. A high user % indicates that it is cpu bound. A high wait % indicates that it is IO bound.I usually go:
sudo apt install iotop htop# Ubuntu/debian based systemsor
sudo yum install iotop htop# Redhat/RPM based systemsThen have a quick look at
htop- is the system swapping (fully filled memory bar)? If so, you want to fix that first. Swapping is super expensive in terms of performance.If the system is not swapping, check out the cpu bars. Are they near 100% most of the time? Well, then your CPU is busy :)
Next, quit
htopand startsudo iotopand check the top right numbers (total and actual disk write). How high are they?If they about match the maximum reasonable throughput of your disk which is being read from / written too (and usually write operations are the bottleneck as disk write speeds are almost always lower then read speeds), then your IO is busy :)
Note I am referring to the actual reasonable throughput on your disk (what you were to approximately get if you copied a set of files manually when the server was otherwise idle), NOT the "label" performance, which is often impossible to achieve to start with.
Thus, doing a few simple check, you worked out if your system is memory, cpu or disk bound. If you want more detailed numbers or analysis, checkout some of the other tools, but this gives you a basic idea, which is often sufficient to help you scale etc.