Home / server / Questions / 878233
In Process
eng3
Asked: 2017-10-13 13:36:02 +0800 CST

Two identical systems with very different performance

  • 772

I have two Dell R730 systems which have an identical hardware configuration purchased at the same time. Both are running RHEL6.9 where were imaged from the same image. It was imaged in January. I update the packages from the repository once a month so in general everything on the system should be "nearly" identical. (ie. any software or setting I change on one system gets changed on the other but since it is a manual process, there could be something missed)

I have noticed the performance on one system is 2.5X slower than the other. The jobs I am testing are single threaded CPU intensive. Reading some data files but very low disk io utilization according to iostat. Top shows the process is constantly pegged at 100% but the system has 88 threads and the load average is only approx 1. Very little memory utilization. No network utilization. (All files that it uses are local) One is a complex python script, another is a proprietary software program, both are running much slower on one system versus the other.

/proc/cpuinfo is identical. BIOS settings are identical. Only one user on the system. The faster system is connected to the internet, the slower one is on a standalone network.

In my investigations I've only found two differences. 1. The faster system is running BIOS version 2.25 the slower system is running BIOS version 2.43 2. The slower system has auditd running. However there is zero activity in the audit log during the process.

I realize this is difficult to debug but I am running out of ideas of what to look for. Are there some builtin software tools I can use to give more insight on what might be going on?

linux

2 Answers

  1. My recommendations today with EL6 systems on enterprise hardware are the following:

    • Set your Dell servers to "OS Control" mode for power, versus a "High Performance" or "Dynamic" mode. This will allow your single-threaded processes to actually leverage Turbo Boost a bit better and give the OS CPU governor the right control.
    • Is there any reason you can't bring the firmware to the same revision?
    • For EL6, you should set the tuned-adm profile to enterprise-storage or latency-performance.
    • If your slower system isn't internet connected, check DNS and your /etc/hosts file definition to make sure that you're not being slowed down by any resolution issues.
    • Examine and compare your /etc/sysctl.conf settings across systems.
    • You can run sosreport to try to get a summary of both systems' configs.

    Of course, you could also profile the processes... top, perf top, pidstat, strace.

    Or look at the servers in realtime with Netdata and correlate all of the system metrics to see where the bottleneck(s) exist.

    I also do the following in /etc/profile.d/tzfix.sh for good reason:

    # Set TZ variable to reduce stat("/etc/localtime" activity
# See: https://blog.packagecloud.io/eng/2017/02/21/set-environment-variable-save-thousands-of-system-calls/
#
export TZ=:/etc/localtime

    Just some ideas to start.

    • 2

  2. This is probably related to power management. Try putting both servers in high performance mode (power management disabled) and redo your performance tests.

    • 0