Home / ubuntu / Questions / 763730
In Process
Hamm
Asked: 2016-04-28 04:39:06 +0800 CST

How to run a job in several machines that have Ubuntu as operating system

  • 772

I need to run a job in several virtual machines exists in a specific cloud and I have the IP address for all of them and I configured the hostname for all of them. How to run a command in a network of virtual machine ? note that the script of this job divides the job into 4 steps to run in parallel since each step needs 2 thread that run in parallel on one machine, now I need to use more than one machine to run the same script in which to let a threads to perform the parallelism using all theses machines by using this command:

./script.py '100 digits number' slaves.hostnames=hostname1,hostname2,hostname3 --slaves 4 --client-threads 2

The above number is number that I need to factor using all these machines to decrease the overall execution time.

software-installation

1 Answers

  1. There are several tools for having a kind of "parallel ssh". As you use Python, pssh should be fine. Install it with pip:

    $ sudo pip install pssh

    Create a pssh hosts file like this:

    192.168.0.10:22
192.168.0.11:22

    and then invoke:

    $ pssh -h pssh-hosts -l root -A echo "Hi!"

    You will asked for your password, but you can use certificates to avoid manual entering of passwords.

    See more:

    http://www.tecmint.com/execute-commands-on-multiple-linux-servers-using-pssh/

    One important note: If you want to solve a problem in parallel, you may need MPI (Message Passing Interface). I suggest MPICH. MPI provides a standard means of communication to solve a problem collaboratively with multiple machines in parallel. Although, writing fast applications to find prime numbers/factors is nothing new.

    If you have a multicore machine, use multithreading. But if you are going to use multiple computers, you will need MPI.

    • 1