Questions[grid](server)

So far as I understood,

• mem_free can be specified to submit a job in a host that has the memory free = mem_free, whereas
• h_vmem is the hard limit of the memory up to which the job can consume and if the job reaches the h_vmem, the job crashes? I think we can set the h_vmem of a host near to the total physical memory, so that the job won't start using swap and slow the server down.

Then what is h_rss? It seems to have the same definition as of h_vmem.

Or am I misinterpreting h_vmem? Is h_vmem used to reserve the extra memory that it might need than the minimum memory it's needed (mem_free)? But do not crash if it exceeds the memory, so the job can exceed h_vmem?

If my second interpretation of h_vmem is correct, then I guess, for a job to be submitted in a host, the job has to satisfy both mem_free and h_vmem (given h_vmem is not INFINITY).

And if my first interpretation of h_vmem is correct, then I guess, for a job to be submitted in a host, the job can satisfy mem_free alone and no need to satisfy h_vmem, as it only reserves the space available and if there is no space available, it doesn't matter?

I'm writing an application that could potentially be run on any cluster resource scheduler (SGE, LSF or SLURM to name a few of them), using very basic functionalities.

I'm wondering if a framework/abstraction layer does exist for interacting with such tools in a product-agnostic manner ?

We would like to have an SGE-based compute cluster with a queue that gives access to all nodes for the computational staff, and a second cluster queue that gives access to, say, half the nodes for occasional (but heavy) use by other staff.

We want to limit the resources of the second queue so that computational staff can keep doing some work, even when there is occasional (but heavy) use by non-comp. staff.

Is there a way to set up two (or more) SGE queues of one collection of nodes, such that one queue can contain all nodes, and a second queue contains a subset of the same nodes, and both queues operate simultaneously?

What specific SGE configuration parameters would I research to set up something like this?

I'm tried to build a grid-cluster based on CentOS. All the machines will have a somewhat similar structure (some with more processors than others) and I will just need to push jobs to a queue and have then run on the available nodes. One job per CPU and the rest stays in line waiting.

John T has been exceptionally helpful pointing me on Gnu Queue that seams to be a hit on what I pretend (the jobs will be essentially bath scripts). I'm still studying the issue before accepting his answer but am asking the comunity for some feedback as Gnu Queue site seams to indicate the project is dead for several years now.

I've also taken a look into Sun Grid Engine and it also seam like a candidate for the job, unfortunatelly Oracle is now killing the project and Univa is yet to release their port.

I just want to start with the right foot so my question would be, have you had any practical experience with this sort of clustering (grid computing). What would your recomendations be.

Thank you in advance.

Is it dangerous to have several parallel jobs create the same directory using mkdir -p? (This is under Linux.)

In my case, I send many jobs to a SUN grid to process them in parallel, and some of these jobs start by creating a certain directory foo. So, the execution of the different mkdir commands might happen at exactly the same time ...