Re: [K12OSN] Mosix and K12ltsp HOWTO
James Jensen (jmsjnsn@linuxfreemail.com)
Sat, 16 Mar 2002 23:14:36 -0600
Thanks for this How-To! I now see some things that I am missing on my
LTSP+MOSIX test kit. I hope to give it a go on Monday and with a little
luck see some success...
I notice that client processes do not migrate in your configuration, which I
am thinking is not going to be a problem in achieving the goal of taking
some of the workstation load off of the server because all the client PCs
are really doing is displaying Xwindows anyway, right?
It would appear to me that an intense application with many threads that
user A is running on the server might get moved to user B's client PC if
it's just sitting there idle. I guess the question remains, what will
happen to user A if user B does a hard shutdown while user A's application
is still running. Will that be the end of the processes that it was running
for user A? The question of failover is still an issue for those that want
to use the LT clients/MOSIX nodes as workstations. If you've using your
nodes as workstations in this manner what has been your experience?
Again, (and I think I speak for all of us interested in LTSP+MOSIX) thank
you very much Richard for providing this guide. Can't wait to try it!
James Jensen
----- Original Message -----
From: "Richard Camp"
To: "James Jensen"
Cc: ; "Anthon Walters"
Sent: Saturday, March 16, 2002 9:34 PM
Subject: Re: [K12OSN] Mosix and K12ltsp HOWTO
> Here it is. Keep in mind that I am not using any of the MOSIX-ltsp
> packages. If you have already tried or have mosix installed clean up
> your system first. These instructions assume that the reader is doing a
> fresh install.
>
> Good Luck
> Richard Camp
>
> ---------------------------------
> Mosix Cluster with Diskless Nodes
> ---------------------------------
>
> 1. Overview
>
> Building a Linux cluster is a time consumming and difficult process.
> There many ways of setting up a cluster. Each methode has its pluses
> and minuses.
>
> The objective of this howto is to guide the reader on setting up a Mosix
> cluster with diskless nodes. The setup is based on K12ltsp Project.
> This should provide an easily scalable system.
>
> 1.1 About K12ltsp
>
> K12ltsp was chosen for the cluster. Its a solid distribution for the
> beginner as well as the advanced user. It simplifies the cluster by
> installing LTSP during the server setup.
>
> 1.2 About Linux Terminal Server
>
> Please see www.ltsp.org
>
> 1.3 About Mosix
>
> Mosix is a patch to the linux kernel which allows a cluster of linux
> machines to act as one large computer. From a programming standpoint
> this allows the programmer to write software as if it is running on an
> SMP machine. Just fork and forget.
>
> An example of what you can is as follows. Lets say you are rendering a
> 3D animation. The renderer we'll be using is povray. A script can be
> used to do the following.
>
> 1. Check to see how many nodes there are.
> 2. See how many cpus are in each node.
> 3. Calculate the total number of CPUs.
> 4. fork off a povray process for each CPU.
> 5. Wait for a process to end and fork off a new one if necessary.
> 6. take the individual files and make an avi file
> 7. encode the avi file to your favorite compression standard.
> 8. all done.
>
> 1.4 About Etherboot
>
> [ to be written ]
>
> 2. Requirements
>
> 2.1 Software Requirements
>
> The software you'll need is the following:
> - K12ltsp.iso 2.0.1
> - Mosix 1.57
> - MPI (optional)
> - PVM (optional)
> - Linux kernel 2.4.17 (from www.kernel.org)
>
> 2.2 Hardware Requirements
>
> The following hardware guidelines should be followed. The hardware
> listed below are minimum requirements. The kernel setup later will
> require at least a pentium pro.
>
> Server
>
> There is a lot of I/O tasks it will handle. A dual processor system is
> recommended. This is the computer you should spend some money on.
>
> - Pentium (pro, II, III, 4) class CPU (dual CPUs is recommended)
> - or celeron cpu
> - minimum 128M RAM (256M is recommended)
> - hard drive of at least 4Gig (SCSI perfered)
> - cdrom and floppy
> - video card - what you need depends on if you'll be using the server
> locallly or remotely.
> - 2 network cards, one must be 100base-t
> - sound card (nice)
>
> Nodes
>
> - Some type of intel CPU. at least a pentium pro class
> - 64Meg RAM (128Meg recommended)
> - floppy drive
> - 100base-t network card
> - video card (needed during troubleshooting)
> - keyboard mouse monitor (to use node as xterminal)
>
> Other
> - Network switch 100mbit
> - cabling
>
> I do not recommend using 100base-t hub. A switch provides full duplex
> operation. You need as much bandwidth to the server you can get. A
> heavily loaded cluster is going to chew up the bandwidth.
>
> 3. Hardware Installation and configuration
>
> 3.1 Server
>
> Assemble and configure your server hardware. Be sure you can
> successfully boot the linux CD. At this point you can go to the section
> on installing the software on the server. While the server is
> installing software you can build and configure the nodes.
>
> 3.2 Nodes
>
> Assemble and configure your nodes. Be sure each node can boot from a
> dos floppy.
>
> [ I haven't worked with PXE yet ]
>
> 3.3 Network
>
> If you are doing custom cabling do that now.
> (installing linux can take some time :)
>
> 3.4 The Final hardware setup
>
> Now that you have all these computers, where are you going to put them?
> The best setup for your hardware is storage racks. Did I mention that
> logging into a node is a fringe benni for my use. Most of you are
> setting the equipment up in a lab.
>
> 4. Software Installation and Configuration
>
> This section will cover the installation of the software on the server
> and the nodes. The items that will take the most time are installing
> linux, updating the packages, compiling the 2.4.17 kernel, compiling the
> 2.4.17 kernel with mosix. Hopefully your are reading this section while
> building the nodes.
>
> 4.1 Server
>
> The server is where most of the software installs will occure.
>
> 3.1.1 Installing K12ltps
>
> Boot the CD. K12ltsp.org provides good instructions to guide you. If
> it does not automatically boot check your bios settings for boot
> devices. Agree to stuff that comes up.
>
> I'm assuming that your hardware and software configuration are the
> defaults as recommended by K12ltsp.org
>
> Finish the rest of the installation steps.
>
> 3.1.2 Booting for the first time
>
> Boot you newly installed linux system. Be sure everything is working
> correctly. Set everything up the way you like it. Also make sure you
> can connect to the internet. This is required for package updating.
>
> At this point check for the latest updates. Update all the installed
> packages except the kernel.
>
> WARNING: KERNEL UPDATING FROM THE UPDATE MANAGER DOESN'T WORK. I DON'T
> CARE WHAT ANYBODY TELLS YOU. BESIDES WE'RE GOING TO MAKE OUR OWN KERNEL
> ANYWAY!.
>
> Reboot your system to be sure everything went ok. You never know when
> an installed package is going to currupt something.
>
> 3.1.3 k12ltsp system checkout
>
> Be sure that your nodes boot. Do not continue with the MOSIX install
> until your setup works.
>
> 4. MOSIX setup
>
> 4.1 Getting stuff together
>
> Download the following files:
>
> mosix 1.5.7 from www.mosix.org
> kernel 2.4.17 from www.kernel.org
> initrd_kit from www.ltsp.org
>
> 4.2 Install the software
>
> Unpack the packages into the /usr/src/ directory. From the K12ltsp cd
> install the kernel sources rpm. This will give you the default RedHat
> kernel config file.
>
> I like to unpack things in a temp directory. so.
>
> > su
> > cd /usr/src
> > mkdir tmp
>
> Copy the files you downloaded to /usr/src/tmp.
>
> > cd /usr/src/tmp
> > tar -xzf linux_kernel-2.4.17.tar.gz
> > tar -xzf MOSIX-1.5.7.tar.gz
> > tar -xzf ltsp_initrd_kit-3.0.1-i386.tgz
>
> If everything looks good than lets move unpacked stuff to /usr/src.
>
> > mv MOSIX-1.5.7 /usr/src/
> > mv ltsp_initrd_kit /usr/src/
> > mv linux /usr/src/linux-2.4.17
>
> Now we need to install a few more packages.
>
> Insert the k12ltsp cd 2
>
> > rpm -i /mnt/cdrom/RedHat/RPMS/kernel-sources-2.4.9-31.i386.rpm
> > rpm -i /mnt/cdrom/RedHat/RPMS/kernel-doc-2.4.9-31.i386.rpm
>
> 4.3 Bug fixes and cleanup
>
> The following items need to edited or fixed.
>
> type:
>
> > chmod goa+x /usr/src/MOSIX-1.5.7/inst/add_kernel_to_grub
>
> The above script was not set to be executable
>
> > mkdir /usr/local/man
>
> This man directory doesn't exist
>
> 4.4 Installing mosix on the server
>
> This is where the fun part begins :)
>
> Fist we want to create a place to store our kernel configs.
>
> > cd /usr/src
> > mkdir kernel-configs
>
> Lets get a kernel config file for a starting point.
>
> > cd /usr/src/linux-2.4.9-31/configs
> > cp kernel-2.4.9-i686-smp.config
> /usr/src/kernel-configs/kernel-2.4.17-smp.config
>
> Copy our config file into the kernel directory
>
> > cd /usr/src/
> > cp kernel-configs/kernel-2.4.17-smp.config linux-2.4.17/.config
>
> Lets get the MOSIX install going.
>
> > cd /usr/src/MOSIX-1.5.7
> > ./install.mosix
>
> Accept all the defaults. When the kernel configurator comes up be sure
> to enable MOSIX, mfs, and dfsa. If you have compiled kernels before,
> get rid of the device support you don't need. Once you are done save
> the config file and exit. Now let the installer do its thing.
>
> Lets setup the mosix.map file. Use your favorite editor and type in the
> following:
>
> #
> # MOSIX map file
> #
> 1 192.168.0.254 1
> 2 192.168.0.1 253
> <>
>
> I like the server to be node 1 and the clients to be nodes 2 through
> 253. This is a bit overkill on the number of nodes but I wanted to keep
> it consistant with the distrobution setup.
>
> when mosix finishes do the following
>
> > cp /usr/src/linux-2.4.17/System.map /boot/System-2.4.17-mosix.map
> > mkinitrd /boot/initrd-2.4.17-mosix.img 2.4.17
>
> Mosix is bad and clobbered the grub.conf file. So lets fix it.
>
> > cp /etc/grub.conf /boot/grub/grub.conf
> > rm -f /etc/grub.conf
> > ln -sf /boot/grub/grub.conf /etc/grub.conf
>
> Mosix didn't add the initrd entry so we have to.
>
> > pico /boot/grub/grub.conf # or your favorite editor
>
> Add the following line to the mosix configuration
>
> initrd /initrd-2.4.17-mosix.img
>
> reboot the system.
>
> Boot to your new mosix kernel. Test your server and make sure nothing
> got broken. Before continuing make sure your clients still boot.
>
>
> 4.5 Seting up mosix for the clients
>
> First lets clean up the kernel directory. We're also remembering to save
> our config file :)
>
> > cp /usr/src/linux-2.4.17/.config
> /usr/src/kernel-configs/mosix-2.4.17.config
> > cd /usr/src/linux-2.4.17
> > make mrproper
>
> Lets get our default ltsp config file
>
> > cp /usr/src/ltsp_initrd_kit/config.2.4.9-ltsp-5 .config
>
> Now on to compiling the kernel
>
> > make xconfig
>
> Enable the mosix stuff and save and exit.
>
> Now we need to add extra version info to the kernel makefile
>
> > pico Makefile
>
> Change the EXTRAVERSION line to read:
>
> EXTRAVERSION = ltsp
>
> Save and exit. Remember to remove the extra version info when you are
> done compiling ltsp kernels
>
> Now we compile the kernel
>
> > make dep
> > make bzImage
> > make modules
> > make modulae_install
>
> Lets save a copy of our config file.
>
> > cp .config /usr/src/kernel-configs/mosix-ltsp-2.4.17.config
>
> Now we need to setup the kernel for ltsp. LTSP provides a script for this.
>
> > cd /usr/src/ltsp_initrd_kit
> > pico buildk
>
> Edit this file. Go to the end of the file. Comment out the last
> prepare_kernel line. Edit the first one to read the following:
>
> prepare_kernel /usr/src/linux-2.4.17 2.4.17ltsp
>
> Save the file and type the following.
>
> > ./buildk
> > cp /lib/modules/2.4.17ltsp /opt/ltsp/i386/lib/modules/
>
> PXE NOTE: I have not worked with PXE yet. Hence I've not setup a PXE
> kernel yet.
>
> Our kernel has now been installed. We need to edit our dhcpd.conf file.
>
> > pico /etc/dhcpd.conf
>
> Add the following line above the trick from Peter comment.
>
> option host-name = concat( "ws" , binary-to-ascii( 10, 8, "", substring(
> reverse( 1, leased-address), 0, 1)));
>
> Mosix needs the hostname set on each client. DHCPD does not pass the
> hostname when you set up everything you're supposed to. Now edit the
> filename parameter to point to the new kernel.
>
> filename "/lts/vmlinux-2.4.17ltsp";
>
> Now save and quit. Lets restart dhcpd
>
> > service dhcpd restart
>
> Mosix isn't completely setup at this point but we should be sure our new
> kernel boots. At this point boot a client and make sure everything is
> working ok. If something goes wrong than you'll prob have to fiddle with
> the kernel config options and build a new kernel.
>
> Everything worked! GREAT! The hard part is over. Now we just edit and
> copy a few files :)
>
> Fist lets copy the user programs into the ltsp directory tree. Type:
>
> > cp /sbin/setpe /opt/ltsp/i386/sbin/
> > cp /sbin/tune /opt/ltsp/i386/sbin/
> > cp /bin/mosrun /opt/ltsp/i386/bin/
> > cp /usr/bin/mon /opt/ltsp/i386/usr/bin/
> > cp /usr/bin/mosctl /opt/ltsp/i386/usr/bin/
> > cp /usr/bin/migrate /opt/ltsp/i386/usr/bin/
> > cp /bin/touch /opt/ltsp/i386/bin/
>
> Copy our mosix.map file to ltsp. Remember to edit both files if you
> make changes.
>
> > cp /etc/mosix.map /opt/ltsp/i386/etc/
>
> Copy the hosts file. The one ltsp generates won't work with mosix.
>
> > rm /opt/ltsp/i386/etc/hosts
> > cp /etc/hosts /opt/ltsp/i386/etc/
>
> Now for the mosix startup script
>
> > cp /etc/rc.d/init.d/mosix /opt/ltsp/i386/etc/rc.mosix
>
> We need a mfs mount point. so:
>
> > mkdir /opt/ltsp/i386/mfs
>
> Now to edit some files.
>
> > pico /opt/ltsp/i386/etc/fstab
>
> Add the following line
>
> none /mfs mfs dfsa=1 0 0
>
> Save and exit.
>
> > pico /opt/ltsp/i386/stc/rc.local
>
> At the end of the file add the following lines
>
> # mosix startup section
> # we don't want any terminal processes to migrate
> echo 1 > /proc/mosix/admin/stay
> # start mosix
> /etc/rc.mosix start
> # mount mfs filesystem. doesn't work when done earlier
> mount /mfs
> # end mosix startup
>
> Save and exit
>
> We are done! OK now boot a couple of clients. Type:
>
> > mon
>
> Look for your nodes to show up in the monitor.
>
> Enjoy your new cluster.
>
> 5. Testing and Checkout
>
> 5.1 Using seti@home to test the cluster
>
> I use seti@home for cluster testing. Its very cpu intensive. But at
> times it does I/O which requires it to be migrated back to the server.
_______________________________________________
K12OSN mailing list
K12OSN@redhat.com
https://listman.redhat.com/mailman/listinfo/k12osn
For more info see