Resource Manager: Torque

From Debian Clusters
Jump to: navigation, search

This is the second part of a four-part tutorial on installing and configuring a queuing system and scheduler. The full tutorial includes:

There is also a troubleshooting page:

Contents

About Torque

From the Cluster Resources page on Torque,

"TORQUE is an open source resource manager providing control over batch jobs and distributed compute nodes. It is a community effort based on the original *PBS project..."

Because torque branched off from PBS, it still retains a lot of the old commands and names. PBS stands for portable batch system, and from here, I'll still call it torque, but commands may have "pbs" in them rather than "torque".

Installing Torque

Before you get and install torque, you'll want to make sure you have all the compilers installed that are necessary. If you don't, it will give you errors about which ones you're missing.

To get the most recent version of torque, visit http://www.clusterresources.com/downloads/torque/ and find the most recent version of it. At the time if this writing, that happens to be torque-2.2.1.tar.gz. Copy of the link location of the file. From /usr/local/src, issue the following command for the most current file:

wget http://www.clusterresources.com/downloads/torque/torque-2.2.1.tar.gz

Next, untar the file with

tar xvf torque-2.2.1.tar.gz

Move into the directory that that just created with cd torque-2.2.1, or whatever your directory is. We're ready to run ./configure (as part of the Source Installation Paradigm, which you might want to check out if this seems unfamiliar to you). We'll add a number of arguments to the compiler in order to let torque know we want a server, and how to set up the server. To see all of the possible arguments, type ./configure -help. What we'll use is this:

./configure --with-default-server=<your server name> --with-server-home=/var/spool/pbs --with-rcp=scp
  • --with-default-server specifies the head node, which will run the server torque process. Be sure to replace your server name with your actual head node's hostname!
  • --with-server-home sets the directory where torque will run from. /var/spool/pbs is by no means standard, but it's the paradigm I'll be using. Others use a directory like /home/torque. I don't like confusing my processes with users.
  • --with-rcp=scp sets the default file-copying mechanism. Technically, scp (for secure copy) is the default, but if you don't specify it and scp isn't found, it'll move onto trying to find the next one, which we don't want.

If the ./configure finishes successfully, you're ready to move onto the next step. If not, address the issues before running the command again. When it does finish successfully, it will end with a line like config.status: executing depfiles commands, but no message about being finished. Next, run

make

A lot of what looks like gibberish will scroll by, and it may take somewhere around five minutes. Again, it will finish without a confirmation message. The last part of the script finished on mine with

make[3]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[2]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[1]: Leaving directory `/usr/local/src/torque-2.2.1/doc'
make[1]: Entering directory `/usr/local/src/torque-2.2.1'
make[1]: Nothing to be done for `all-am'.
make[1]: Leaving directory `/usr/local/src/torque-2.2.1'

Finally, you're ready to run

make install

You won't get a confirmation message for this, either, and it'll finish similarly to the way the last one finished. To make sure it was installed correctly, try using which to locate one of the binaries, like this:

gyrfalcon:~# which pbs_server
/usr/local/sbin/pbs_server

If it can't find it, double check that the binary was installed with ls and grep:

gyrfalcon:~# ls /usr/local/sbin | grep pbs       
pbs_demux
pbs_iff
pbs_mom
pbs_sched
pbs_server

If it's there in /usr/local/sbin, but which doesn't find it, you'll need to edit /etc/login.defs. Locate the line for ENV_SUPATH and add /usr/local/bin and /usr/local/sbin to it. The line for ENV_PATH should be right below it; add /usr/local/bin to it.

Configuring Torque

To start the torque server running on the head node and create a new database of jobs, issue

pbs_server -t create

Now, if you run ps aux | grep pbs, you'll see the server running. However, if you run a command to list the queues and their statuses,

qstat -a

you'll see nothing because no queues have been set up. To begin configuring queues for torque, we need qmgr, an interface to the batch system. You can run

qmgr

to start it up in an interactive mode, or enter the commands one at a time on the command line:

qmgr -c "set server scheduling=true"
qmgr -c "create queue batch queue_type=execution"
qmgr -c "set queue batch started=true"
qmgr -c "set queue batch enabled=true"
qmgr -c "set queue batch resources_default.nodes=1"
qmgr -c "set queue batch resources_default.walltime=3600"
qmgr -c "set server default_queue=batch"

Additionally, you can run commands to set the administrators' e-mail:

qmgr -c "set server operators = root@localhost"
qmgr -c "set server operators += kwanous@localhost"

Sanity Check

At this point, running qstat -q to view available queues should give you something like this:

gyrfalcon:~# qstat -q

server: gyrfalcon

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
batch              --      --       --      --    0   0 --   E R
                                               ----- -----
                                                   0     0

Excellent, we have a queue called "batch" and it's empty. You can also view your qmgr settings with

qmgr -c "print server"

Time to try submitting a job to the queue. First, switch over to a different user account (don't run this as root) with su - <username>. Then, try to submit a job that just sleeps for thirty seconds and does nothing:

echo "sleep 30" | qsub

The purpose of this is to see whether the job shows in the queue when you run qstat after submitting it. Below is a script of my testing it.

kwanous@gyrfalcon:~$ echo "sleep 30" | qsub
0.gyrfalcon
kwanous@gyrfalcon:~$ qstat
Job id                    Name             User            Time Use S Queue
------------------------- ---------------- --------------- -------- - -----
0.gyrfalcon               STDIN            kwanous                0 Q batch          

Excellent, the job shows up! Unfortunately, though, it won't run... the state is "Q" (I assume for "queued"), and it needs to be scheduled. That's what we'll install Maui for later.

Introducing Torque to the Worker Nodes

Now we need to tell the pbs_server which worker nodes are available and will be running pbs_mom, a client that allows the the server to give them jobs to run. We do this by creating the file /var/spool/pbs/server_priv/nodes. With your favorite text editor, add each worker node hostname on a line by itself. If they have more than one processor, add np=X next to the line. Mine looks like this:

eagle np=4
goshawk np=4
harrier np=4
kestrel np=4
kite np=4
osprey np=4
owl np=4
peregrine np=4

Which that, configuration on the head node for torque is done.

Installing Torque on the Worker Nodes

Now we need to install a smaller version of torque, called pbs_mom, on all of the worker nodes. Move back into the directory we untarred earlier, /usr/local/src/torque*. There's a handy way to create the packages for the torque clients. Run

make packages

and they'll be created for you. This time you'll get a confirmation message:

Done.

The package files are self-extracting packages that can be copied
and executed on your production machines.  Use --help for options.

You'll see some new files in the directory now if you run an ls. The one we're interested in is torque-package-mom-linux-*.sh where the * is your architecture. We need to copy that file to all the the worker nodes. You can either copy it over to a shared NFS mount, or see my Cluster Time-saving Tricks on how to copy a file to all the nodes using the rsync command. I'm copying it over to my NFS mount with

cp torque-package-mom-linux-i686.sh /shared/usr/local/src/

Once it's on each worker node, they each need to run the script with

torque-package-mom-linux-i686.sh --install

You have a couple of options for doing this on each node. You can ssh over and run it manually, or you can check out my Cluster Time-saving Tricks page to learn to how to write a quick script to run the command over ssh without having to log into each node. If you're going with the second route, the command to use is

for x in `cat machines`; do ssh $x /<full path to package>/torque-package-mom-linux-i686.sh --install; done

Before we can start up pbs_mom on each of the nodes, they need to know who the server is. You can do this by creating a file /var/spool/pbs/server_name that contains the hostname of the head node on each worker node, or you can copy the file to all of the nodes at once with a short script (assuming you've created a file at ~/machines with the hostnames of the worker nodes as outlined in the Cluster Time-saving Tricks page):

for x in `cat ~/machines`; do rsync -plarv /var/spool/pbs/server_name $x:/var/spool/pbs/; done

Next, if you're using a NFS-mounted file system, you need to create a file on each of the worker nodes at /var/spool/pbs/mom_priv/config with the contents

$usecp <full hostname of head node>:<home directory path on head node> <home directory path on worker node>

The path is the same for me on my head node or worker node, and my file looks like this:

$usecp gyrfalcon.raptor.loc:/shared/home /shared/home

Again, this file can be created on each of the worker nodes, or you can create it and copy it over to each of the nodes. If you're using the latter technique, assuming you've created a machines file with all the host names, and you've created a config file, the command to run from the head node is

for x in `cat ~/machines`; do rsync -plarv config $x:/var/spool/pbs/mom_priv/; done

After you've done that, pbs_mom is ready to be started on each of the worker nodes. Again, you can ssh in to each node and run pbs_mom, or the script equivalent is

for x in `cat ~/machines`; do ssh $x pbs_mom; done

Everyone Placing Nice on Torque

Finally, it's time to make sure the server monitors the pbs_moms that are running. Terminate the current queues with

qterm

and then start up the pbs server process again

pbs_server

Then, to see all the available worker nodes in the queue, run

pbsnodes -a

(I don't know why this command doesn't have an underscore.) Each of the nodes should check in with a little report like my node peregrine's below.

peregrine
     state = free
     np = 4
     ntype = cluster
     status = opsys=linux,uname=Linux peregrine 2.6.21-2-686 #1 SMP Wed Jul 11 0
     3:53:02 UTC 2007 i686,sessions=? 0,nsessions=? ,nusers=0,idletime=1910856,totme
     m=3004480kb,availmem=2953608kb,physmem=1028496kb,ncpus=8,loadave=0.00,netload=18
     0898837,state=free,jobs=,varattr=,rectime=1200191204

Ready to continue? Move on to installing Maui, the scheduler.

References

Personal tools
Namespaces

Variants
Actions
About
services
node images
clustering
web monitoring
Tools