Whenever I submit a job array, I get one job ID in this form: 463880-1.hostname with 463880 being the job ID and 1 the array ID, one job for each "array position".
Today I'm using a different machine, and it looks like a different PBS version, because an job array, size 10, resumes to this single job: 3310[].hostname.
I like to know list both versions, to check if I need to change something in my scripts.
I am using torque 4.0.1 on openSUSE 12.1 in a cluster environment. When I qsub a job (simple as "echo hello"), it remains in 'Q' state, and never gets scheduled. I can force the job to run with qrun, and it is executed on the first node without error.
I tried to find the solutions for the past few days but failed. I read the manual, the logs, even the source code, but still can not locate the problem. Of course I googled a lot, tried various solutions, however no one worked.
Here is some info that maybe helpful:
05/13/2012 18:55:08;0002; pbs_sched;Svr;Log;Log opened
05/13/2012 18:55:08;0002; pbs_sched;Svr;TokenAct;Account file /var/spool/torque/sched_priv/accounting/20120513 opened
05/13/2012 18:55:08;0002; pbs_sched;Svr;main;pbs_sched startup pid 32604
05/13/2012 19:33:08;0002;PBS_Server;Svr;PBS_Server;Torque Server Version = 4.0.1, loglevel = 0
05/13/2012 19:33:56;0100;PBS_Server;Job;16.head;enqueuing into batch, state 1 hop 1
05/13/2012 19:33:56;0008;PBS_Server;Job;16.head;Job Queued at request of pubuser@head, owner = pubuser@head, job name = STDIN, queue = batch
Job Id: 16.head
Job_Name = STDIN
Job_Owner = pubuser@head
job_state = Q
queue = batch
server = head
Checkpoint = u
ctime = Sun May 13 19:33:56 2012
Error_Path = head:/fserver/home/pubuser/STDIN.e16
Hold_Types = n
Join_Path = n
Keep_Files = n
Mail_Points = a
mtime = Sun May 13 19:33:56 2012
Output_Path = head:/fserver/home/pubuser/STDIN.o16
Priority = 0
qtime = Sun May 13 19:33:56 2012
Rerunable = True
Resource_List.walltime = 01:00:00
substate = 10
Variable_List = PBS_O_QUEUE=batch,PBS_O_HOME=/,
PBS_O_WORKDIR=/fserver/home/pubuser,PBS_O_HOST=head,PBS_O_SERVER=head,
PBS_O_WORKDIR=/fserver/home/pubuser
euser = pubuser
egroup = users
queue_rank = 4
queue_type = E
etime = Sun May 13 19:33:56 2012
fault_tolerant = False
job_radix = 0
submit_host = head
init_work_dir = /fserver/home/pubuser
sun1
state = free
np = 2
ntype = cluster
status = rectime=1336910403,varattr=,jobs=,state=free,netload=44492032184,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1697420kb,totmem=1802616kb,idletime=241085,nusers=0,nsessions=0,uname=Linux sun1 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun2
state = free
np = 2
ntype = cluster
status = rectime=1336910408,varattr=,jobs=,state=free,netload=39762812881,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1701012kb,totmem=1802616kb,idletime=239982,nusers=0,nsessions=0,uname=Linux sun2 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun3
state = free
np = 2
ntype = cluster
status = rectime=1336910400,varattr=,jobs=,state=free,netload=45984311925,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1699772kb,totmem=1802616kb,idletime=212303,nusers=0,nsessions=0,uname=Linux sun3 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun4
state = free
np = 2
ntype = cluster
status = rectime=1336910407,varattr=,jobs=,state=free,netload=37538584401,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1805480kb,totmem=1908308kb,idletime=211197,nusers=0,nsessions=0,uname=Linux sun4 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun5
state = free
np = 2
ntype = cluster
status = rectime=1336910411,varattr=,jobs=,state=free,netload=173547166,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1803816kb,totmem=1908308kb,idletime=211199,nusers=0,nsessions=0,uname=Linux sun5 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun6
state = free
np = 2
ntype = cluster
status = rectime=1336910411,varattr=,jobs=,state=free,netload=24641446,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1805704kb,totmem=1908308kb,idletime=212999,nusers=0,nsessions=0,uname=Linux sun6 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun7
state = free
np = 2
ntype = cluster
status = rectime=1336910412,varattr=,jobs=,state=free,netload=1548383055,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1805432kb,totmem=1908308kb,idletime=215630,nusers=0,nsessions=0,uname=Linux sun7 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun8
state = free
np = 2
ntype = cluster
status = rectime=1336910400,varattr=,jobs=,state=free,netload=128755968,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1803448kb,totmem=1908308kb,idletime=211866,nusers=0,nsessions=0,uname=Linux sun8 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
sun9
state = free
np = 2
ntype = cluster
status = rectime=1336910374,varattr=,jobs=,state=free,netload=1371896399,gres=,loadave=0.00,ncpus=2,physmem=888188kb,availmem=1805664kb,totmem=1908308kb,idletime=211161,nusers=0,nsessions=0,uname=Linux sun9 3.1.0-1.2-desktop #1 SMP PREEMPT Thu Nov 3 14:45:45 UTC 2011 (187dde0) x86_64,opsys=linux
mom_service_port = 15002
mom_manager_port = 15003
gpus = 0
#
# Create queues and set their attributes.
#
#
# Create and define queue batch
#
create queue batch
set queue batch queue_type = Execution
set queue batch resources_default.walltime = 01:00:00
set queue batch enabled = True
set queue batch started = True
#
# Set server attributes.
#
set server scheduling = True
set server acl_hosts = head
set server managers = pubuser@head
set server managers += root@head
set server operators = pubuser@head
set server operators += root@head
set server default_queue = batch
set server log_events = 511
set server mail_from = adm
set server scheduler_iteration = 600
set server node_check_rate = 150
set server tcp_timeout = 300
set server job_stat_rate = 45
set server poll_jobs = True
set server mom_job_sync = True
set server keep_completed = 0
set server submit_hosts = head
set server next_job_number = 17
set server moab_array_compatible = True
Host: sun1/sun1 Version: 4.0.1 PID: 5362
Server[0]: head (192.168.0.1:15001)
Last Msg From Server: 1584 seconds (DeleteJob)
Last Msg To Server: 7 seconds
HomeDirectory: /var/spool/torque/mom_priv
stdout/stderr spool directory: '/var/spool/torque/spool/' (4457492 blocks available)
MOM active: 229485 seconds
Check Poll Time: 45 seconds
Server Update Interval: 45 seconds
LogLevel: 0 (use SIGUSR1/SIGUSR2 to adjust)
Communication Model: TCP
MemLocked: TRUE (mlock)
TCP Timeout: 0 seconds
Trusted Client List: 127.0.0.1:0,192.168.0.1:0,192.168.0.101:0,192.168.0.101:15003,192.168.0.102:15003,192.168.0.103:15003,192.168.0.104:15003,192.168.0.105:15003,192.168.0.106:15003,192.168.0.107:15003,192.168.0.108:15003,192.168.0.109:15003: 0
Copy Command: /usr/bin/scp -rpB
NOTE: no local jobs detected
diagnostics complete
The problem is that TCP Timeout is 0 seconds, which does not seem to be normal. During the diagnostics, the following log was found in mom_logs
05/13/2012 20:30:10;0001; pbs_mom;Svr;pbs_mom;LOG_ERROR::Resource temporarily unavailable (11) in tcp_read_proto_version, no protocol version number End of File (errno 2)
I googled it, but found nothing.
I hope someone can solve this problem. Thank you!
I share a server with HAL. The server has 32 GB of memory.
I rarely use more than 1 GB of memory, and when I do, it is for a few minutes at a time, and I don't mind sending such jobs to the back of the line.
HAL read/writes large files (e.g. using gunzip). This can take up to 100% of the memory CPU, intermittently, for hours. This is usually done overnight, but when running, will make even simple commands such as cd take 30s, opening emacs can take minutes.
I would like to be able to reserve 1 GB for use by processes that use << 1GB (like a text editor). I would also like to stay out of HAL's way, and see no reason that this should be an issue.
HAL says that a queueing system (like PBS) can not be used to put a low priority on read/write, e.g. to leave 1 GB of memory always available when large jobs are running. In his words:
the script used to gunzip snags all the processors it can because the data is large... queueing would not solve this... during transfer of files from (that server) to (this server), an inflation step does lots of read/write
Why couldn't queuing solve this problem? What could?
I have access to a cluster that uses 'torque' (I think) and we use PBS scripts to submit jobs. I need to run more then 200 instances of an app that I've developed in java. The app acts as a peer forming a P2P network, which means that those instances communicate with each other through sockets.
I was able to do my tests with 100 instances running on a single node on the cluster, but when running 200 instances on a single node it doesn't work, and I can't ask for more resources (mem, cores, etc.)
My question is: should I do this the way I'm doing it? With a serial script in which I start all my instances one by one sending them to the background and then wait for them?
Could this be accomplished with a parallel script in which I could ask for 2 nodes and instantiate 100 instances of my app in each node? In this case, I have some other questions: How can I do it? and is there any guarantee that both jobs run at the same time? All 200 instances must be running at the same time.
This is part of the script that I'm currently using...
#PBS -l nodes=1:ppn=4
#PBS -l pmem=6GB
#PBS -l walltime=00:20:00
IP=`/sbin/ifconfig eth0 | grep 'inet ' | awk '{print $2}' | sed 's/addr://'`
PORT_PEER=3000
java -jar $JAR $JAR_PARAMS -ip=$IP -port=$PORT_PEER & # first peer, others connect to this one..
for i in {1..99}
do
PORT_PEER=`expr $PORT_PEER + 2`;
java -jar $JAR $JAR_PARAMS -ip=$IP -port=$PORT_PEER -bootstrap=$IP:3000 &
sleep 1s
done
wait # wait here until all instances terminates
Jobs I add to the queue stays there in "Queued" state without attempts to be executed (unless I manually qrun them)
/var/spool/torque/server_logs say just
04/11/2011 12:43:27;0100;PBS_Server;Job;16.localhost;enqueuing into batch, state 1 hop 1
04/11/2011 12:43:27;0008;PBS_Server;Job;16.localhost;Job Queued at request of test@localhost, owner = test@localhost, job name = Qqq, queue = batch
The job requires just 1 CPU on 1 node.
# qmgr -c "list queue batch"
Queue batch
queue_type = Execution
total_jobs = 0
state_count = Transit:0 Queued:0 Held:0 Waiting:0 Running:0 Exiting:0
max_running = 3
acl_host_enable = True
acl_hosts = localhost
resources_min.ncpus = 1
resources_min.nodect = 1
resources_default.ncpus = 1
resources_default.nodes = 1
resources_default.walltime = 00:00:10
mtime = Mon Apr 11 12:07:10 2011
resources_assigned.ncpus = 0
resources_assigned.nodect = 0
kill_delay = 3
enabled = True
started = True
I can't set resources_assigned to nonzero because of Cannot set attribute, read only or insufficient permission resources_assigned.ncpus.
When I qrun some task, this goes to mom's log:
04/11/2011 21:27:48;0001; pbs_mom;Svr;pbs_mom;LOG_DEBUG::mom_checkpoint_job_has_checkpoint, FALSE
04/11/2011 21:27:48;0001; pbs_mom;Job;TMomFinalizeJob3;job 18.localhost started, pid = 28592
04/11/2011 21:27:48;0080; pbs_mom;Job;18.localhost;scan_for_terminated: job 18.localhost task 1 terminated, sid=28592
04/11/2011 21:27:48;0008; pbs_mom;Job;18.localhost;job was terminated
04/11/2011 21:27:48;0080; pbs_mom;Svr;preobit_reply;top of preobit_reply
04/11/2011 21:27:48;0080; pbs_mom;Svr;preobit_reply;DIS_reply_read/decode_DIS_replySvr worked, top of while loop
04/11/2011 21:27:48;0080; pbs_mom;Svr;preobit_reply;in while loop, no error from job stat
04/11/2011 21:27:48;0080; pbs_mom;Job;18.localhost;obit sent to server
Scheduler log (/var/spool/torque/sched_logs/20110705):
07/05/2011 21:44:53;0002; pbs_sched;Svr;Log;Log opened
07/05/2011 21:44:53;0002; pbs_sched;Svr;TokenAct;Account file /var/spool/torque/sched_priv/accounting/20110705 opened
07/05/2011 21:44:53;0002; pbs_sched;Svr;main;/usr/sbin/pbs_sched startup pid 16234
qstat -f:
Job Id: 26.localhost
Job_Name = qwe
Job_Owner = test@localhost
job_state = Q
queue = batch
server = localhost
Checkpoint = u
ctime = Tue Jul 5 21:43:31 2011
Error_Path = localhost:/home/test/jscfi/default/0.738784810485275/qwe.e26
Hold_Types = n
Join_Path = n
Keep_Files = n
Mail_Points = a
mtime = Tue Jul 5 21:43:31 2011
Output_Path = localhost:/home/test/jscfi/default/0.738784810485275/qwe.o26
Priority = 0
qtime = Tue Jul 5 21:43:31 2011
Rerunable = True
Resource_List.ncpus = 1
Resource_List.neednodes = 1:ppn=1
Resource_List.nodect = 1
Resource_List.nodes = 1:ppn=1
Resource_List.walltime = 00:01:00
substate = 10
Variable_List = PBS_O_HOME=/home/test,PBS_O_LANG=en_US.UTF-8,
PBS_O_LOGNAME=test,
PBS_O_PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games,
PBS_O_MAIL=/var/mail/test,PBS_O_SHELL=/bin/sh,PBS_SERVER=127.0.0.1,
PBS_O_WORKDIR=/home/test/jscfi/default/0.738784810485275,
PBS_O_QUEUE=batch,PBS_O_HOST=localhost
euser = test
egroup = test
queue_rank = 1
queue_type = E
etime = Tue Jul 5 21:43:31 2011
submit_args = run.pbs
Walltime.Remaining = 6
fault_tolerant = False
How to make it execute jobs automatically, without manual qrun?