Cloning Worker Nodes
From Debian Clusters
If you've been walking through the tutorials with me, you're at the point where you have the head node and one worker node set up. Rather than having to install everything again on all the other worker nodes, we can turn all the others into copies of the one that's already set up. All the other worker nodes will receive an exact copy (a clone) of the entire hard drive of the worker node that's already set up. The data to be cloned across the different nodes is called the image. This is called cloning or imaging.
There are many different ways to clone the worker nodes, but they all follow the same basic principle. We have one machine's hard drive completely set up as a worker node image, and then the image on this hard drive will be copied onto all of the hard drives. Cloning a image prevents having to install the same software and configure the same files multiple times, possibly allowing inconsistencies and errors to slip in. Instead of doing something x times for x number of worker nodes, it can be configured once and tweaked until it's perfect, and then all the other nodes become exactly the same.
Because of that, however, the original worker node needs some tweaking before the image is ready to be compatible for all of the nodes.
Preparing the Original Worker Node Image
The Basics
First, the Debian/Linux basics: for one, make sure you're running an up-to-date copy of Debian. Do an apt-get update and apt-get upgrade (and even apt-get dist-upgrade if you need to) before you clone the other machines. It's a lot easier to do it once now than having to do it on each machine separately after cloning.
The machine providing the image should also have all of its services properly set up - it should be mounting the NFS share, receiving an IP address from the DHCP server, resolving DNS properly, and be set up to authenticate correctly using LDAP or another paradigm. It's much easier to fix this problem on one machine before cloning than having to fix it on x number of machines afterward. Make sure the paths, profiles, histories, and anything else for Bash is set up.
You'll also want to have the compilers installed on the worker node. If you're going to have a dynamic domain name based on DNS, that should also be set up. Make sure the /etc/hostname and /etc/hosts are set to something that you want all the machines to initially have, (I use "amnesia"), for prior to the first time they start up and set it with DNS. In other words, I'm about to clone kestrel, so I need to take out the lines referring to kestrel put in /etc/hosts and /etc/hostname by the custom host name script. When the new machines boot up for the first time, they'll have these files properly set up for them.
SSH Keys
If you want the root account to be able to SSH into each of the machines from each of the machines without needing to supply a password, now is also the time to set this up. Do this by generating a key on the machine that will be donating the image. As root, enter
ssh-keygen
and hit enter when prompted for the file (keep the default), and also hit enter (no password) twice when prompted for the password. It should look like the following.
waxwing:~# ssh-keygen Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: ****************************************************
Next, move into the directory where the key was saved: /root/.ssh. We'll create a file called authorized_keys2. This file will contain the list of SSH fingerprints allowed to SSH into this machine without being prompted for a password. Since this image will be cloned, all of the machines will have the same SSH fingerprint (id_rsa and id_rsa.pub), so putting this key into authorized_keys2 will allow the root account to SSH amongst all the machines without a password. Copy the fingerprint from id_rsa.pub to authorized_keys with
cat id_rsa.pub >> authorized_keys2
If authorized_keys2 already exists, it will append this entry at the bottom. (Make sure the file only contains entries you want applicable for all of the machines that will be imaged.) If it doesn't exist, the file will be created.
To include the head node as one of the machines that root can SSH amongst freely, the contents of the /root/.ssh/ directory need to be sync'd over. If you'd like to include the head node, run the following command from the worker node:
rsync -plarv /root/.ssh/ root@<your head node>:/root/.ssh/
SSH StrictHostKeyChecking
You'll also want to disable StrictHostKeyChecking for SSH. The default for this setting is ask, meaning that when a user (or a program acting on behalf of a user) tries to SSH from one node to another node that it hasn't encountered before, the user will be prompted as to whether they would like to accept the identification the node gives and continue SSHing in. Unfortunately, this can be a show stopper for MPICH, a widely used parallel programming implementation, and scheduler and queue software if there's no user to enter yes when prompted.
To prevent this from being a problem, open up /etc/ssh/ssh_config and find the line that looks like this:
# StrictHostKeyChecking ask
Take out the hash (#) to uncomment this line, and change the value from ask to no.
Clearing out Current Udev MAC Information
One last important tip: everything will be exactly copied from the worker node that will be sending out its image. This includes the file that contains the MAC addresses of the network interfaces (NICs) and binds them to their names. This file will automatically be added to when the machine boots, but if it has existing entries, new ones will be appended, and you'll end up with eth2 and eth3 instead of eth0 and eth1. To fix this, edit /etc/udev/rules.d/z25_persistent-net.rules and take out any lines below these comments:
# This file was automatically generated by the /lib/udev/write_net_rules # program, probably run by the persistent-net-generator.rules rules file. # # You can modify it, as long as you keep each rule on a single line.
(For more tips on udev, see Udev: Renaming Network Interfaces.)
Cloning the Worker Nodes
I'll show three different paradigms for cloning the worker nodes, udpcast, data dumping, and using rsync.
Udpcast involves setting each of the nodes to PXE boot (network boot), and then the machine donating its image sends it out across the network to the other machines. Each of the machines needs to be configured to PXE boot, and the DHCP server needs to be configured to enable PXE booting. Udpcast also has to be tweaked for your particular set up. This is a more permanent solution and allows you to re-image the machines whenever you need to.
Data dumping, or dd, requires the hard drives to be taken out of the cases and placed in another machine, a Linux machine or one running a bootable Linux distribution. From the console, the data is then copied from the original hard drive to each of the other hard drives. This may tend to take longer, but the setup time required before you're actually imaging is less. Of course, if you want to re-image the drives again at a later date, you need to remove all of them from the worker nodes again.
Rsyncing also requires to the hard drives to be taken out and put into another machine. It's similar to data dumping, but requires slightly more command line setup and takes slightly less time to copy over. Again, if you want to re-image the drives again at a later date, you need to remove all of them from the worker nodes again.
The tutorial will split into three sections from here, so pick your paradigm:
