knweiss's questions

Many of you probably have completed or are contemplating Green-IT projects with the goal to power off idle or unneeded systems when demand for computer resources is low:

How you did handle this situation in your system monitoring? I'm especially interested in solutions for Nagios.

One idea is to schedule downtime in Nagios for the poweroff hosts. However, the drawback of this solution is that the hosts would still be listed in the 'Problems' view of the Nagios web interfaces. Is there a better solution without this "pollution" (i.e. were the 'Problems' view only shows real problems that require maintenance from a system administrator).

A clean solution would be a new 'Green-IT poweroff' host state. But AFAIK this does not exist, does it? Do you have any other recommendations or solutions? What's the best way to monitor a dynamic IT environment?

Given a large Linux HPC cluster with hundreds/thousands of nodes. What are your best practices to get the best possible LINPACK benchmark (HPL) result to submit for the Top500 supercomputer list?

To give you an idea what kind of answers I would appreciate here are some sub-questions (with links):

• How to you tune the parameters (N, NB, P, Q, memory-alignment, etc) for the HPL.dat file (without spending too much time trying each possible permutation - esp with large problem sizes N)?
• Are there any Top500 submission rules to be aware of? What is allowed, what isn't?
• Which MPI product, which version? Does it make a difference?
• Any special host order in your MPI machine file?
• Do you use CPU pinning?
• How to you configure your interconnect? Which interconnect?
• Which BLAS package do you use for which CPU model? (Intel MKL, AMD ACML, GotoBLAS2, etc.)
• How do you prepare for the big run (on all nodes)? Start with small runs on a subset of nodes and then scale up? Is it really necessary to run LINPACK with a big run on all of the nodes (or is extrapolation allowed)?
• How do you optimize for the latest Intel/AMD CPUs? Hyperthreading? NUMA?
• Is it worth it to recompile the software stack or do you use precompiled binaries? Which settings? Which compiler optimizations, which compiler? (What about profile-based compilation?)
• How to get the best result given only a limited amount of time to do the benchmark run? (You can block a huge cluster forever)
• How do you prepare the individual nodes (stopping system daemons, freeing memory, etc)?
• How do you deal with hardware faults (ruining a huge run)?
• Are there any must-read documents or websites about this topic? E.g. I would love to hear about some background stories of some of the current Top500 systems and how they did their LINPACK benchmark.

I deliberately don't want to mention concrete hardware details or discuss hardware recommendations because I don't want to limit the answers. However, feel free to mention hints e.g. for specific CPU models.