1.1. openMosix HOWTO

In the beginning there was Mosix, then came openMosix, in my opinion a more interesting project. Not only from a technical point of view but also due to the more correct license. I made the decision to focus this HOWTO on openMosix rather than on Mosix, mainly based on the fact that openMosix has a bigger userbase. (Moshe Bar states that about 97% of the old Mosix community has switched over to openMosix.) (20020705) Given the above, lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest release of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 My intention is to focus on the openMosix HOWTO, however not neglecting the Mosix users. More info on http://howto.ipng.be/Mosix-HOWTO/

1.2. Introduction

This document gives a brief description of openMosix, a software package that turns a network of GNU/Linux computers into a computer cluster. Along the way, some background to parallel processing is given, as well as a brief introduction to programs that make special use of openMosix's capabilities. The HOWTO expands on the documentation as it provides more background information and discusses the quirks of various distributions.

Since the creation of this HOWTO some people of the Mosix team created openMosix (more info later), initially both openMosix and Mosix were discussed in this HOWTO. Although lots of information might be valuable to both users of Mosix and openMosix. I decided to split the HOWTO. The latest relase of the Mosix HOWTO, containing info about both Mosix and OpenMosix will be 0.20 and can be found on http://howto.ipng.be/Mosix-HOWTO/Mosix-HOWTO/

Kris Buytaert got involved in this piece of work when Scot Stevenson was looking for somebody to take over the Job: this was during February 2002. While initially we discussed both Mosix and openMosix, this version of the HOWTO now mainly focuses on openMosix. Please note that the document often still mentions Mosix where it should read openMosix.

You will notice that some of the headings are not as serious as they should be. Scot had planned to write the HOWTO in a slightly lighter style, as the world (and even the part of the world with a burping penguin as a mascot) is full of technical literature that is deadly. Therefore some parts still have these comments.

1.3. Disclaimer

Use the information in this document at your own risk. I disavow potential liability for the contents of this document. Use of these concepts, examples, and/or other content of this document is entirely at your own risk.

All copyrights are owned by their respective owners, unless specified otherwise. Use of a term in this document should not be regarded affecting the validity of any trademark or service mark. openMosix is Copyright (c) by Moshe Bar. Mosix is Copyright (c) by Amnon Barak. Linux is a Registered Trademark of Linus Torvalds. openMosix is licensed under version 2 of the GNU General Public License as published by the Free Software Foundation.

Naming of particular products or brands should not be seen as endorsements.

You are strongly recommended to take a backup of your system before major installation and backups at regular intervals.

1.4. Distribution policy

Copyright (c) 2002 by Kris Buytaert and Scot W. Stevenson. This document may be distributed under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the appendix entitled "GNU Free Documentation License".

1.5. New versions of this document

Official New versions of this document can be found on the web pages of the Linux Documentation Project Drafts and Beta versions will be available on howto.ipng.be in the appropriate sub folder. Changes to this document will usually be discussed on the openMosix Mailing Lists. See the openMosix for details.

1.6. Feedback

Currently this HOWTO is being maintained by Kris Buytaert. Please do send remarks and updates for the howto to him.

If you have a technical question about openMosix/Mosix itself, please post them on the more appropriate mailing list.

2.1. A very, very brief introduction to clustering

Most of the time, your computer is bored. Start a program like xload or top that monitors your system use, and you will probably find that your processor load is not even hitting the 1.0 mark. If you have two or more computers, chances are that at any given time, at least one of them is doing nothing. Unfortunately, when you really do need CPU power - during a C++ compile, or encoding Ogg Vorbis music files - you need a lot of it at once. The idea behind clustering is to spread these loads among all available computers, using the resources that are free on other machines.

The basic unit of a cluster is a single computer, also called a "node". Clusters can grow in size - they "scale" - by adding more machines. A cluster as a whole will be more powerful the faster the individual computers and the faster their connection speeds are. In addition, the operating system of the cluster must make the best use of the available hardware in response to changing conditions. This becomes more of a challenge if the cluster is composed of different hardware types (a "heterogeneous" cluster), if the configuration of the cluster changes unpredictably (machines joining and leaving the cluster), and the loads cannot be predicted ahead of time.

2.2. The story so far

Announcing MO6 for BSD/OS 3.0
Oren Laadan (orenl@cs.huji.ac.il)
Tue, 9 Sep 1997 19:50:12 +0300 (IDT)

Hi:

We are pleased to announce the availability of MO6 Version 3.0
Release 1.04 (beta-4) - compatible with BSD/OS 3.0, patch level
K300-001 through M300-029.

MO6 is a 6 processor version of the MOSIX multicomputer enhancements
of BSD/OS for a PC Cluster. If you have 2 to 6 PC's connected by a
LAN, you can experience truly multi-computing environment by using
the MO6 enhancements.

The MO6 Distribution
--------------------
MO6 is available either in "source" or "binary" distribution. It is
installed as a patch to BSD/OS, using an interactive installation
script.

MO6 is available at http://www.cnds.jhu.edu/mirrors/mosix/
or at our site: http://www.cs.huji.ac.il/mosix/

Main highlights of the current release:
--------------------------------------
- Memory ushering (depletion prevention) by process migration.
- Improved installation procedure.
- Enhanced migration control.
- Improved administration tools.
- More user utilities.
- More documentation and new man pages.
- Dynamic configurations.

Please send feedback and comments to mosix@cs.huji.ac.il.
-------------------

2.3. openMosix in action: An example

openMosix clusters can take various forms. To demonstrate this, let's assume you are a student and share a dorm room with a rich computer science guy, with whom you have linked computers to form an openMosix cluster. Let's also assume you are currently converting music files from your CDs to Ogg Vorbis for your private use, which is legal in your country. Your roommate is working on a project in C++ that he says will bring World Peace. However, at just this moment he is in the bathroom doing unspeakable things, and his computer is idle.

So when you start a program like bladeenc to convert Bach's .... from .wav to .ogg format, the openMosix routines on your machine compare the load on both nodes and decide that things will go faster if that process is sent from your Pentium-233 to his Athlon XP. This happens automatically: you just type or click your commands as you would if you were on a standalone machine. All you notice is that when you start two more coding runs, things go a lot faster, and the response time doesn't go down.

Now while you're still typing ...., your roommate comes back, mumbling something about red chili peppers in cafeteria food. He resumes his tests, using a program called 'pmake', a version of 'make' optimized for parallel execution. Whatever he's doing, it uses up so much CPU time that openMosix even starts to send subprocesses to your machine to balance the load.

This setup is called *single-pool*: all computers are used as a single cluster. The advantage/disadvantage of this is that your computer is part of the pool: your stuff will run on other computers, but their stuff will run on yours too.

2.4. Components

2.5. openMosix Test Drive

In support of openMosix, Major Chai Mee Joon is giving OM users a free trial account to his online openMosix cluster service, which users can use to test and experiment openMosix with.

The availability of this online openMosix cluster service will help both new users overcome the initial openMosix configuration issues, and also provides higher computing power to openMosix users who are developing or porting their applications.

Please send an email to <om@majorlinux.com> for a trial account.

2.6. Pros of openMosix

2.7. Cons of openMosix

3.1. Hardware requirements

Installing a basic cluster requires at least 2 network connected machines, either using a cross-cable between the two network cards or using a switch or hub (a switch is much better than a hub though and only costs a few bucks more). Of course the faster your network-cards the easier you will get better performance for your cluster.

These days Fast Ethernet (100 Mbps) is standard; putting multiple ports in a machine isn't that difficult, but make sure to connect them through other physical networks in order to gain the speed you want. Gigabit Ethernet is getting cheaper every day now but I suggest that you don't rush to the shop spending your money before you have actually tested your setup with multiple 100Mbit cards and noticed that you really do need the extra network capacity. Next to putting a Gigabit card you might also want to try bonding different 100Mbit cards together. An even cheaper alternative can be found in Firewire, as discussed in this paper

3.2. Hardware Setup Guidelines

Setting up a big cluster requires some thinking to be done: where are you going to put the machines? Not under a table somewhere or in the middle of your office I hope! It's ok if you just want to do some small tests, but if you are planning to deploy a N node cluster you will have to make sure that the environment that will hold these machines is capable of doing so.

I'm talking about preparing one or more 19" racks to host the machines, configuring the appropriate network topology, either straight, single connected or even a 1 to 1 cross connected network between all your nodes. You will also need to make sure that there is enough power to support such a range of machines, that your air-conditioning system supports the load and that in case of power-failure your UPS can cleanly shut down all the required systems. You might want to invest in a KVM (Keyboard, Video, Mouse) Switch in order to facility access to the machines' consoles.

But even if you don't have the number of nodes that justifies such an investment, make sure that you can always easily access the different nodes, you never know when you have to replace a CPU fan or an hard-disk of a machine in trouble. If that means that you have to unload a stack of machines to reach the bottom one, hence shutting down your cluster, you are in trouble.

3.3. Software requirements

The systems we plan to use will need a basic Linux installation of your choice: Red Hat, SuSe, Debian, Gentoo or any another distribution: it doesn't really matter which one. What does matter is that the kernel is at least on 2.4 level, and that your network-cards are configured correctly; next to that you'll need a healthy space of swap.

3.4. Planning your Cluster

When it comes to configuring openMosix Clusters with a pool of servers and a set of (personal) workstations, you have different options that will have their advantages and disadvantages.

• In a Single-pool configuration all the servers and workstations are used as a single cluster: each machine is a part of the cluster and can migrate processes to each other existing node. This of course makes your workstation a part of the pool.

• In an environment that is called a Server-pool, servers are a part of the cluster while workstations aren't part of it, they don't even have openMosix kernel. If you want to run applications on the cluster you will need to specifically log on to these servers. However your workstation will also stay clean and no remote processes will migrate to it.

• A third alternative is called an Adaptive-pool configuration: here servers are shared while workstations join or leave the cluster. Imagine your workstation being used during daytime by yourself but, as soon as you log out in the evening, a script tells the workstation to join the cluster and start crunching numbers. This way your machine is being used while you don't need it. If you need the resources of the machine again just run the openmosix stop script and your processes will stay away from the cluster and vice-versa.

  Practically this means that you will change the role of your machine by using mosctl.

3.5. Classrooms

Although it might seem a good idea to convert your classroom into an openMosix cluster at night, you'll have to consider training your end users not to pull the power switch of those machines when they want to use them again. More recent machines support automatic shutdowns when hitting the power button, but with older machines you might loose some data when this actually happens.

4.1. Installing openMosix

This Chapter deals with openMosix 2.4

This chapter deals with installing openMosix on different distributions. It won't be an exhaustive list of all the possible combinations. However throughout the chapter you should find enough information on installing openMosix in your environment.

Techniques for installing multiple machines with openMosix will be discussed in one of the next chapters.

4.2. Before getting openMosix

First of all, you must understand that openMosix is made up of a kernel patch and some user-space tools. The kernel patch is needed to make the kernel capable of talking to other openMosix-enabled machines on the network. If you download openMosix as a binary package (such as an rpm file), you don't even need to take care about the kernel patch because the kernel has been patched and compiled with the most common default options and modules for you.

The user-space tools are needed in order to make an effective use of an openMosix-enabled kernel. They are needed to start/stop the migration daemon, the openMosix File System, to migrate jobs to certain nodes and other tasks which are usually accomplished with the help our good old friend: the command line interface. About binary packages: the same as in the kernel patch goes for the user-space tools: if you install an rpm you don't need to care about compiling them or configuring anything; just let them install and run. That's all. Really :)

Once you get to the download page (which we'll talk about in a second), you'll need to get two distinct parts: the kernel and the user-space tools. You can either download two binary packages or get the kernel patch plus the user-space tools' sources. The kernel patch is usually named after this scheme: openMosix-x.y.z-w where x.y.z is the version of the vanilla Linux Kernel against which the patch should be applied and w is the patch revision for that particular kernel release. For the precompiled kernel binaries, please refer to the README-openMosix-kernel.txt file you'll find in the download page. This file also contains updated info about manually compiling a kernel.

About the user-space tools: you'll find those in a package named openmosix-tools. We use the terms user-space tools, userspace-tools and openmosix-tools interchangeably. Updated info about precompiled binaries and manually compiling the tools are also provided in the README-openmosix-tools.txt file. Please note that since version 0.3 of the openmosix-tools, the openmosix.map file is deprecated and the use of the autodiscovery daemon is highly encouraged since it tends to make your life easier.

4.3. Getting openMosix

You can download the latest versions of openMosix from http://sourceforge.net/project/showfiles.php?group_id=46729. You can either choose the binary (even in rpm) compiled for UP or SMP or download the source code. You will need both the kernel patch or binaries and the userland tools. Alternatively you can get the CVS version:

cvs -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix login
cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co linux-openmosix
cvs -z3 -d:pserver:anonymous@cvs.openmosix.sourceforge.net:/cvsroot/openmosix co userspace-tools

4.4. openMosix General Instructions

mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20
cd /usr/src/linux-2.4.20
zcat openMosix-2.4.20-2.gz | patch -Np1

mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20
cd /usr/src/linux-2.4.20
gunzip openMosix-2.4.20-2.gz
cat openMosix-2.4.20-2 | patch -Np1

mv /root/openMosix-2.4.20-2.gz /usr/src/linux-2.4.20
cd /usr/src/linux-2.4.20
gunzip openMosix-2.4.20-2.gz
patch -Np1 < openMosix-2.4.20-2

...
CONFIG_MOSIX=y
# CONFIG_MOSIX_TOPOLOGY is not set
CONFIG_MOSIX_UDB=y
# CONFIG_MOSIX_DEBUG is not set
# CONFIG_MOSIX_CHEAT_MIGSELF is not set
CONFIG_MOSIX_WEEEEEEEEE=y
CONFIG_MOSIX_DIAG=y
CONFIG_MOSIX_SECUREPORTS=y
CONFIG_MOSIX_DISCLOSURE=3
CONFIG_QKERNEL_EXT=y
CONFIG_MOSIX_DFSA=y
CONFIG_MOSIX_FS=y
CONFIG_MOSIX_PIPE_EXCEPTIONS=y
CONFIG_QOS_JID=y
...

make config | menuconfig | xconfig

make dep bzImage modules modules_install

openMosix-Node_ID               IP-Address(or hostname)          Range-size

1       node1   1
2       node2   1
3       node3   1
4       node4   1

1       192.168.1.1     1
2       192.168.1.2     1
3       192.168.1.3     1
4       192.168.1.4     1

1       192.168.1.1     4

1       192.168.1.1     1
2       192.168.1.2     1
3       192.168.1.3     1
4       192.168.1.4     1
4       192.168.10.10   ALIAS

setpe -w -f /etc/openmosix.map

/etc/init.d/openmosix stop
/etc/init.d/openmosix start
/etc/init.d/openmosix restart

mfs_mnt         /mfs            mfs     dfsa=1          0 0

mfs_mnt          /mfs           mfs     dfsa=0          0 0

[device_name]           [mount_point]   mfs     defaults        0 0

on node2 :      ln -s /mfs/1/work /work
on node3 :      ln -s /mfs/1/work /work
on node3 :      ln -s /mfs/1/work /work
...

ln -s /mfs/1/mfs/1/usr         

ln -s /mfs/1/mfs/3/usr

4.5. Red Hat and openMosix

If you are running a RedHat 7.2, 7.3 or 8.0 version, this is probably the easiest *Mosix install you have ever done. Choose the appropriate openMosix RPMs from sourceforge. They have precompiled kernels (as I write this 2.4.20) that work seamlessly: I have tested them on several machines including Laptops with PCMCIA cards and Servers with SCSI disks. If you are a grub user, the kernel rpm even modifies your grub.conf. So all you have to do is install 2 RPMs:

rpm -Uvh openmosix-kernel-2.4.20-openmosix2.i686.rpm openmosix-tools-0.2.4-1.i386.rpm

[root@oscar0 root]# more /etc/openmosix.map 
# openMosix CONFIGURATION
# ===================
#
# Each line should contain 3 fields, mapping IP addresses to openMosix node-numbers:
# 1) first openMosix node-number in range.
# 2) IP address of the above node (or node-name from /etc/hosts).
# 3) number of nodes in this range.
#
# Example: 10 machines with IP 192.168.1.50 - 192.168.1.59
# 1	   192.168.1.50	    10
#
# openMosix-#  IP  number-of-nodes
# ============================
1 192.168.10.220 1
2 192.168.10.78  1
3 192.168.10.84  1

Most RedHat installations have one extra thing to fix. You often get the following error:

[root@inspon root]# /etc/init.d/openmosix start 
Initializing openMosix...
setpe: the supplied table is well-formatted,
but my IP address (127.0.0.1) is not there!

127.0.0.1	omosix1.localhost.org localhost 

192.168.10.78   omosix1.localhost.org
127.0.0.1       localhost 

[root@inspon root]# /etc/init.d/openmosix start 
Initializing openMosix...
[root@inspon root]# /etc/init.d/openmosix status
This is openMosix node #2
Network protocol: 2 (AF_INET)
openMosix range     1-1     begins at 192.168.10.220
openMosix range     2-2     begins at inspon.localhost.be
openMosix range     3-3     begins at 192.168.10.84
Total configured: 3

If you would like to use more bleeding edge patches, you can always opt for the src rpm and run rpmbuild --rebuild on it. This will install the source for you and create an initial config file. From there you can go further applying patches to openMosix

A tutorial on how to build your own openMosix RPM's can be found in the Appendixes.

As new RedHat versions come out, they might be supported out of the box so, feel free to drop the author a note and help him keeping this information updated.

4.6. Suse and openMosix

Although the RPMs are being built on a RedHat based environment, you can use most of them on other RPM based systems.

Suse however has /sbin/mk_initrd as a link to /sbin/mkinitrd, which makes rpms before release 20-2 fail. Newer version should have a fix for this.

4.7. Debian and openMosix

Installing openMosix ``the Debian way'' can be easily done as described below.

The first step consists in downloading the packages from the net. I had to use a 2.4.19 kernel since the openMosix patches package is not yet available for 2.4.20 at the moment I write this. Since we are using a Debian setup we needed: http://packages.debian.org/unstable/net/openmosix.html, http://packages.debian.org/unstable/net/kernel-patch-openmosix.html, http://packages.debian.org/unstable/misc/kernel-package.html, http://packages.debian.org/unstable/devel/kernel-source-2.4.19.html. You can also apt-get install them ;).

The next part is making the kernel openMosix capable.

Basically, the procedure to follow is:

cd /usr/src
apt-get install kernel-source-2.4.19 kernel-package \
        openmosix kernel-patch-openmosix
tar vxjf kernel-source-2.4.19.tar.bz2
ln -s /usr/src/kernel-source-2.4.19 /usr/src/linux
cd /usr/src/linux
../kernel-patches/i386/apply/openmosix
make menuconfig
make-kpkg kernel_image modules_image
cd ..
dpkg -i kernel-image-*-openmosix-*.deb

After rebooting with this kernel and a configured /etc/openmosix.map, you should then have a cluster of openMosix machines that talk to each-other and that do migration of processes.

You can test that by running the following small script:

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}'

We also setup openMosixView on the Debian machine:

apt-get install openmosixview

openMosixView gives you a nice interface that shows the load of different machines and gives you the possibility to migrate processes manually.

A detailed discussion of openMosixView can be found elsewhere in this document.

4.8. openMosix and Gentoo

First Install Gentoo Linux

Then, install openMosix: type "emerge sys-apps/openmosix-user", which will install an openMosix kernel source tree in /usr/src/linux along with the openMosix userland tools.

Michael Imhof, aka tantive, keeps Gentoo current for the latest openMosix version.

Daniel Robbins, the President/CEO of Gentoo Technologies, Inc. and the creator of Gentoo Linux, wrote the artitles we use as our Introduction to openMosix Clusters.

4.9. Other distributions

Based on the explanations above you should be able to install openMosix on most other Linux platforms.

5.1. Easy Configuration

The auto-discovery daemon (omdiscd) provides a way to automatically configure an openMosix cluster hence eliminating the need of a /etc/mosix.map or similar manual configurations. Auto-discovery uses multicast packages to notify other nodes that it is an openMosix node. This way adding an extra node to your mosix cluster means that you just have to start the omdiscd on your machine and it will join the cluster.

However there are some small requirements, Like with any openMosix cluster , you need to have networking configured correctly. mainly the routing. Without a default route, you must specify an interface to omdiscd with the -i option. Otherwise omdiscd will exit with an error like.

Aug 31 20:41:49 localhost omdiscd[1290]: Unable to determine address of 
default interface.  This may happen because there is no default route 
configured.  Without a default route, an interface must be: Network is 
unreachable 
Aug 31 20:41:49 localhost omdiscd[1290]: Unable to initialize network.  
Exiting. 

[root@localhost log]# route -n 
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        0.0.0.0         255.0.0.0       U     0      0        0 eth0
127.0.0.0       0.0.0.0         255.0.0.0       U     0      0        0 lo
0.0.0.0         10.0.0.99       0.0.0.0         UG    0      0        0 eth0

omdiscd

Sep  2 10:00:49 oscar0 kernel: openMosix configuration changed: This is openMosix #2780 (of 6 configured)
Sep  2 10:00:49 oscar0 kernel: openMosix #2780 is at IP address 192.168.10.220
Sep  2 10:00:49 oscar0 kernel: openMosix #2638 is at IP address 192.168.10.78
Sep  2 10:00:49 oscar0 kernel: openMosix #2646 is at IP address 192.168.10.86
Sep  2 10:00:49 oscar0 kernel: openMosix #2627 is at IP address 192.168.10.67
Sep  2 10:00:49 oscar0 kernel: openMosix #2634 is at IP address 192.168.10.74

omdiscd has some other options that you can use. You can either run omdiscd as a daemon (default) or in the foreground where output goes to the screen (standard output) omdiscd -n . An interface can be specified with the -i option.

Now lets still have a short look at the other tool , it's showmap. This tool will show you the newly auto generated openMosix map.

[root@oscar0 root]# showmap
My Node-Id: 0x0adc

Base Node-Id Address          Count
------------ ---------------- -----
0x0adc       192.168.10.220   1
0x0a4e       192.168.10.78    1
0x0a56       192.168.10.86    1
0x0a43       192.168.10.67    1
0x0a4a       192.168.10.74    1

Auto-discovery has some other features not listed here such as a routing mechanism for clusters with more than one network. More detailed information can be found in the README and DESIGN files in the user-land tools source tree.

More recent versions of the openMosix rc scripts will first verify wether an /etc/openmosix.map file or similar exists before trying to use autoconfiguration.

5.2. Compiling auto-discovering

If you are compiling autodiscovery from source you will need to make a small modification to openmosix.c. One of the first lines will be

#define ALPHA

% make clean
% make

5.3. Troubleshooting autodiscovery

Sometimes however autodiscovery does not function as you would like, for example a node might not see multicast traffic from other nodes. This has occurred with some PCMCIA ethernet drivers. One solution is to place the interface in promiscuous and or multicast mode as detailed below:

Aug 31 20:45:58 localhost kernel: openMosix configuration changed: This is openMosix #98 (of 1 configured)
Aug 31 20:45:58 localhost kernel: openMosix #98 is at IP address 10.0.0.98Aug 31 20:45:58 localhost omdiscd[1627]: Notified kernel to activate 
openMosix  Aug 31 20:45:58 localhost kernel: Received an unauthorized information request from 10.0.0.99

ifconfig ethx promisc
or
ifconfig ethx multicast 

tcpdump -i eth0 ether multicast 

On some Layer 3 switches you other configs might be required. An openMosix user found out that on his Switch Summit48Si (Extreme Networks) he had to run

disable ipmcforwarding (to deactivate the routing of multicast paquets)
disable igmp snooping

Aug 31 22:14:43 inspon omdiscd[1422]: Simulated notification to activate openMosix
[root@inspon root]# showmap
My Node-Id: 0x0063

Base Node-Id Address          Count
------------ ---------------- -----
0x0063       10.0.0.99        1
[root@inspon root]# /etc/init.d/openmosix status
OpenMosix is currently disabled
[root@inspon root]# 

#define ALPHA

I have also noticed that autodiscovery does not work with FireWire based network cards.

6.1. What Plump/OS is

PlumpOS is a CD-based GNU/Linux/openMosix mini-distribution designed to allow users to quickly, or temporarily, add nodes to an openMosix cluster; as I write this in march 2003 the version (release 6.9 RC1) is a 16.7M ISO download.

This chapter is a quick hack up by Peter Willis of a very similar chapter contributed by Jean-David Marrow (who is the author of Clump/OS and inspirer of PlumpOS - props to Jean-David for his fine work on the departed Clump/OS).

6.2. How does it work?

First of all, set up your machine's BIOS in order to make it boot from CD. Check your motherboard's manual to see how to set up this. Upon booting up you will receive a prompt to boot PlumpOS using a specific kernel; several ones have been provided for you but, you also have the option of providing your own kernel + modules. This will be explained later in the ``Getting Started'' section. The boot menu should tell you all the kernels available for you to use; simply type the name of one (and optionally some kernel arguments) and press return/enter.

At boot-time, PlumpOS will auto-probe for network cards and, if any gets detected, it'll try and configure them via DHCP. If successful, it will fire up omdiscd on each interface a DHCP lease is received on. Currently this has only been tested in PCs with 1 network card so YMMV.

It works for myself, but may not work for you; if you experience difficulties, please email me with as much information about your system as possible -- after you have investigated the problem. (Check the sourceforge homepage for PlumpOS's mailing list, sign up and post your question with as much detailed information as possible. The openMosix-general and openMosix-devel lists are NOT for PlumpOS problems, they are for openMosix-specific problems, and if you can't tell the difference just send it to the PlumpOS list or my contact email).

6.3. Requirements

As the purpose of PlumpOS is to add nodes to a cluster, it is assumed that you already have a running openMosix cluster -- or perhaps only a single openMosix node -- from which you will be initiating jobs. All machines in the cluster must conform to the following requirements:

• PlumpOS Machine(s) 586+ CPU

• bootable CD-ROM drive

• Network Interface Card

• Between 32M and 128M of RAM. Because of some oddities in the openMosix kernels at the time of this writing, simply passing the size of the ramdisk is not working so one has to actually use a ramdisk pre-made for a specific size. There should be several ramdisks avaliable at PlumpOS mirrors which can be placed in the disks' root directory on the actual cdrom ISO (or rootdisk/disks/ if you're inside the PlumpOS package directory).

• Master Machine(s) GNU/Linux/openMosix kernel (same version as all the PlumpOS kernel you are booting on the Child/Slave PlumpOS Machine(s))

• Network Environment Running DHCP server (if you don't, or won't, run DHCP, you can still manually configure your system: simply go to each PlumpOS machine and enter in your desired configuration information with the supplied networking tools. Using DHCP is highly recommended however, and will greatly simplify your life in the long run.

The following network modules are present in most if not all of the supplied kernels' modules tarball (in /kernels/KERNELNAME/modules.tgz), although not all support auto-probing; if you don't see support for your card in this list, then PlumpOS will not work for you.

3c501.o, 3c503.o, 3c505.o, 3c507.o, 3c509.o, 3c515.o, 3c59x.o, 8139cp.o, 8139too.o, 82596.o, ac3200.o, acenic.o, aironet4500_card.o, aironet4500_core.o, aironet4500_proc.o, arlan-proc.o, arlan.o, at1700.o, bsd_comp.o, cs89x0.o, de4x5.o, depca.o, dgrs.o, dl2k.o, dmfe.o, dummy.o, e100/e100.o, e1000/e1000.o, e2100.o, eepro.o, eepro100.o, eexpress.o, epic100.o, eth16i.o, ewrk3.o, fealnx.o, hamachi.o, hp-plus.o, hp.o, hp100.o, lance.o, lp486e.o, mii.o, natsemi.o, ne.o, ne2k-pci.o, ni5010.o, ni52.o, ni65.o, ns83820.o, pcmcia, pcnet32.o, ppp_async.o, ppp_deflate.o, ppp_generic.o, ppp_synctty.o, pppoe.o, pppox.o, sis900.o, sk98lin/sk98lin.o, slhc.o, smc-ultra.o, smc9194.o, starfire.o, strip.o, sundance.o, sungem.o, sunhme.o, tc35815.o, tg3.o, tlan.o, tokenring/{3c359.o abyss.o ibmtr.o lanstreamer.o olympic.o smctr.o tms380tr.o tmsisa.o tmspci.o}, tulip/tulip.o, via-rhine.o, wavelan.o, wd.o, winbond-840.o, wireless/{airo.o airo_cs.o hermes.o orinoco.o orinoco_cs.o orinoco_pci.o orinoco_plx.o}, yellowfin.o

Please also note that PlumpOS may not work on a laptop: it definitely doesn't support PCMCIA cards (yet), and will probably not configure openMosix properly if your machine contains multiple connected Ethernet adapters. This is a temporary limitation of the configuration scripts, and should be resolved in future releases.

6.4. Getting Started

You can download the latest PlumpOS package under the terms of the GPL, without warranty of any kind, from any PlumpOS mirror. Upon downloading, unpack the archive

$ tar -xvzf plumpos-6.9-rc1.tar.gz

First let's familiarize with the directory structure here. There should be 3 directories:

• rootdisk, which contains the layout of the iso to be created;

• scripts, which contains small programs used to help in the creation of the iso;

• final, which will contain the ISO generated by the whole process.

If you intend on using your own kernel with PlumpOS, there are several steps you must follow in order for it to boot properly. First you must make a new directory in the rootdisk/kernels/ directory that will be in the old and often blamed DoS 8.3 character format and only contain letters and numbers (and a single '.'). In that directory you should put a file named bzImage which is your kernel and a file named modules.tgz which is a gzipped tarball of your kernel's modules (with the relative path of lib/modules/ so it can be extracted from the root dir). Optionally you can also provide the System.map and config file for your kernel. When this is done you may simply run the install program and it will detect and install your kernel(s) for use with the generated ISO.

Now, say you want to include a 3rd party add-on package. This is done very simply: create a gzipped tarball containing all the files you want to include in your package (relative to "/") and put these packages into the rootdisk/packages/ directory. Then edit the file rootdisk/packages/list and add the path relative to / on the ramdisk where the package can be located (in other words for a package named "openssl.tar.gz", add the line "/cdrom/packages/openssl.tar.gz" to the file rootdisk/packages/list). Optionally you can use a line such as "cdrom:openssl.tar.gz" which will automatically search the cdrom's packages directory for the package "openssl.tar.gz". In future releases this will be useful for things like nfs and boot floppies, so for now don't worry about it ;)

7.1. Cluster Installations

This chapter does not deal with installing openMosix as such, it does however deal with installing multiple machines with openMosix. Automated or semi automated mass installs.

7.2. DSH, Distributed Shell

At the time of this writing (May 2003) DSH's most current release is available from http://www.netfort.gr.jp/~dancer/software/downloads/ More info on the package can be found on http://www.netfort.gr.jp/~dancer/software/dsh.html The latest version available for download is 0.23.6 You will need both libdshconfig-0.20.8.tar.gz and dsh-0.23.5.tar.gz Start with installing libdshconfig

./configure
make
make install 

Say we have a small cluster with a couple of nodes. To make life easier we want type each command once but have it executed on each node. You then have to create a file in $HOME/.dsh/group/clusterwname that lists the ip's of your cluster. eg.

[root@inspon root]# cat .dsh/group/mosix 
192.168.10.220
192.168.10.84

[root@inspon root]# dsh -r ssh -g mosix ls
192.168.10.84: anaconda-ks.cfg
192.168.10.84: id_rsa.pub
192.168.10.84: install.log
192.168.10.84: install.log.syslog
192.168.10.84: openmosix-kernel-2.4.17-openmosix1.i686.rpm
192.168.10.84: openmosix-tools-0.2.0-1.i386.rpm
192.168.10.220: anaconda-ks.cfg
192.168.10.220: id_dsa.pub
192.168.10.220: id_rsa.pub
192.168.10.220: openmosix-kernel-2.4.17-openmosix1.i686.rpm
192.168.10.220: openmosix-tools-0.2.0-1.i386.rpm
192.168.10.220: oscar-1.2.1rh72
192.168.10.220: oscar-1.2.1rh72.tar.gz

[root@inspon root]# dsh -r ssh -g mosix "uname -a"
192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 
Wed May 29 14:32:28 CEST 2002 i686 unknown
192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 
2002 i686 unknown

[root@inspon root]# dsh -r ssh -g mosix -c -- uname -a 
192.168.10.220: Linux oscar0 2.4.17-openmosix1 #1 Wed May 29 14:32:28 CEST 
2002 i686 unknown
192.168.10.84: Linux omosix2.office.be.stone-it.com 2.4.17-openmosix1 #1 
Wed May 29 14:32:28 CEST 2002 i686 unknown

8.1. Basic Administration

openMosix provides the advantage of process migration to HPC-applications. The administrator can configure and tune the openMosix-cluster by using the openMosix-user-space-tools or the /proc/hpc interface which will be now described in detail.

Up till openMosix version 2.4.16 the /proc interface was named /proc/mosix ! Until openMosix version 2.4.17 it was named /proc/hpc.

8.2. Configuration

The values in the flat files in the /proc/hpc/admin directory presenting the current configuration of the cluster. Also the administrator can write its own values into these files to change the configuration during runtime, e.g.

...

8.3. the userspace-tools

These following tools are providing easy administration to openMosix clusters.

migrate -send a migrate request to a process
                syntax: 
                        migrate [PID] [openMosix_ID]

mon             -is a ncurses-based terminal monitor
                 several informations about the current status are displayed in bar-charts

mosctl          -is the openMosix main configuration utility
                syntax:
                        mosctl  [stay|nostay]
                                [lstay|nolstay]
                                [block|noblock]
                                [quiet|noquiet]
                                [nomfs|mfs]
                                [expel|bring]
                                [gettune|getyard|getdecay]

                        mosctl  whois   [openMosix_ID|IP-address|hostname]

                        mosctl  [getload|getspeed|status|isup|getmem|getfree|getutil]   [openMosix_ID]

                        mosctl  setyard [Processor-Type|openMosix_ID||this]

                        mosctl  setspeed        interger-value

                        mosctl  setdecay interval       [slow fast]

mosrun          -run a special configured command on a chosen node
                syntax:
                        mosrun  [-h|openMosix_ID| list_of_openMosix_IDs] command [arguments]

The mosrun command can be executed with several more commandline options. To ease this up there are several preconfigured run-scripts for executing jobs with a special (openMosix) configuration.

setpe           -manual node configuration utility
                syntax:
                        setpe   -w -f   [hpc_map]
                        setpe   -r [-f  [hpc_map]]
                        setpe   -off

-w reads the openMosix configuration from a file (typically /etc/hpc.map)
-r writes the current openMosix configuration to a file (typically /etc/hpc.map)
-off turns the current openMosix configuration off

tune            openMosix calibration and optimizations utility.
                (for further informations review the tune-man page)

Additional to the /proc interface and the commandline-openMosix utilities (which are using the /proc interface) there is a patched "ps" and "top" available (they are called "mps" and "mtop") which displays also the openMosix-node ID on a column. This is useful for finding out where a specific process is currently being computed.

This actually summarised the command line tools, but have a look at openMosixview which is a GUI for the most common administration tasks, and which ill be discussed in a future chapter.

8.4. Cluster Mask

(by Moshe Bar)

Several people have asked for a feature in openMosix which allows to specifiy to which nodes a given process and it's children can migrate and to which nodes it cannot.

Simone Ettore has just committed a new patch to the CVS which allows you to do just that.

Here is how it works:

• /proc/[pid]/migfilter enable/disable the capability of filter migration.

• /proc/[pid]/mignodes is a bit-list of nodes. The bit position of a node is calculated as 2^(PE-1). PE is node number.

• /proc/[pid]/migpolicy is the policy of the filtering: 0=DENY: the process can migrate in all nodes except when the relative bit on mignodes is 1 1=ALLOW: the process can migrate in all nodes where the relative bit on mignodes is 1

We are shortly going to release also a simple user-land tool to set the node mask, but I would like you guys to give it a try asap before we release it as openMosix 2.4.20-3.

9.1. Introduction

Some of the parts below are still from the old Mosix Howto, as time passes these parts will get replaced by relevant openMosix parts, however some things are still the same , but your mileage may vary.

9.2. Creating a "Master" node

Although openMosix architcture does not require a master node as such, you might want to have a head node from where you launch processes, this might be a multihomed node from where users log in to your cluster. You want to configure your machine to make processes migrate away

You have to trick the node in thinking it is the slowest node around and it'd better migrate all it's processes to the faster nodes.

You will have to make it "slow" with :

mosctl setspeed [n]

mosctl getspeed

9.3. Optimizing Mosix

Editorial Comment: To be checked with openMosix versions

Login a normal terminal as root. Type

       setpe -r 

        setpe -w -f /etc/mosix.map 

       cat /proc/$$/lock

        echo 0 > /proc/$$/lock

        init 1

        /etc/init.d/network start
        /etc/init.d/mosix start
        echo 1 > /proc/mosix/admin/quiet

        /etc/init.d/pcmcia start

9.4. Channel Bonding Made Easy

Contributed by Evan Hisey

Channel bonding is actually horrible easy. This may explain the lack of documentation on this subject A bonded network appears as a normal network to the applications. All machines on a subnet must be bonded the same way. Bonded and non-bonded machine really don't talk well to each other.

Channel bonding needs at least two physical sub-nets but can have more(Currently I have a tri-bonded cluster). To enable bonding you need to either compile in to the kernel or as a module (bonding.o) the Channel Bonding kernel code, as of 2.4.x is it a standard option of the kernel. The NIC's are setup as normal with except that you only us 'ifconfig' to initialize the first card of the bond. 'ifenslave' is used to initialize the remaining cards in the bonded connection. 'ifenslave' can be locate in the linux/Documentation/network/ directory. It will need to be compiled as it is a .c file. The basic format for use is

ifenslave <master> <slave1> <slave2> ...

9.5. openMosix and FireWire

openMosix does gain performance by using another type of network device, as described within the paper about openMosix and FireWire

10.1. Introduction

openMosixview is the next version and a complete rewrite of Mosixview. It is a cluster-management GUI for openMosix-cluster and everybody is invited to download and use it (at your own risk and responsibility). The openMosixview-suite contains 5 useful applications for monitoring and administrating openMosix-cluster.

All parts are accessible from the main application window. The most common openMosix-commands are executable by a few mouse-clicks. An advanced execution dialog helps to start applications on the cluster. "Priority-sliders" for each node simplifying the manual and automatic load-balancing. openMosixview is now adapted to the openMosix-auto-discovery and gets all configuration-values from the openMosix /proc-interface.

10.2. openMosixview vs Mosixview

openMosixview is fully designed for openMosix cluster only. The Mosixview-website (and all mirrors) will stay as they are but all further developing will continue with openMosixview located at the new domain www.openmosixview.com

If you have: questions, features wanted, problems during installation, comments, exchange of experiences etc. feel free to mail me, Matt Rechenburg or subscribe to the openMosix/Mosixview-mailing-list and mail to the openMosix/Mosixview-mailing-list

changes: (to Mosixview 1.1) openMosixview is a complete rewrite "from the scratch" of Mosixview! It has the same functionalities but there are fundamental changes in ALL parts of the openMosixview source-code. It is tested with a constantly changing cluster topography (required for the openMosix auto-discovery) All "buggy" parts are removed or rewritten and it (should ;) run much more stable now.

10.3. Installation

Requirements

Documentation about openMosixview There is a full HTML-documentation about openMosixview included in every package. You find the startpage of the documentation in your openMosixview installation directory: openmosixview/openmosixview/docs/en/index.html

The RPM-packages have their installation directories in: /usr/local/openmosixview

rpm -i openmosixview-1.4.rpm 

rpm -e openmosixview 

gunzip openmosixview-1.4.tar.gz 
tar -xvf openmosixview-1.4.tar 

./setup [your_qt_2.3.x_installation_directory] 

export QTDIR=/usr/lib/qt-2.3.0 (for bash) 
or 
setenv QTDIR /usr/lib/qt-2.3.0 (for csh) 

ln -s /usr/local/qt-2.3.0 /usr/lib/qt 

export QTDIR=/usr/lib/qt (for bash) 
or 
setenv QTDIR /usr/lib/qt (for csh) 

./configure 
make 

cp openmosixview/openmosixview /usr/bin 
cp openmosixviewproc/openmosixviewprocs/mosixviewprocs /usr/bin 
cp openmosixcollector/openmosixcollector/openmosixcollector /usr/bin 
cp openmosixanalyzer/openmosixanalyzer/openmosixanalyzer /usr/bin 
cp openmosixhistory/openmosixhistory/openmosixhistory /usr/bin 

cp openmosixcollector/openmosixcollector.init /etc/init.d/openmosixcollector 
or 
cp openmosixcollector/openmosixcollector.init /etc/rc.d/init.d/openmosixcollector 

rcp openmosixprocs/openmosixprocs your_node:/usr/bin/openmosixprocs 

openmosixview 

10.4. using openMosixview

automigration on/off 
quiet yes/no 
bring/lstay yes/no 
exspel yes/no 
openMosix start/stop 

10.5. openMosixprocs

10.6. openMosixcollector

The openMosixcollector is a daemon which should/could be started on one cluster-member. It logs the openMosix-load of each node to the directory /tmp/openmosixcollector/* These history log-files analyzed by the openMosixanalyzer (as described later) gives an nonstop overview of the load, memory and processes in your cluster. There is one main log-file called /tmp/openmosixcollector/cluster Additional to this there are additional files in this directory to which the data is written.

At startup the openMosixcollector writes its PID (process id) to /var/run/openMosixcollector.pid

The openMosixcollector-daemon restarts every 12 hours and saves the current history to /tmp/openmosixcollector[date]/* These backups are done automatically but you can also trigger this manual.

There is an option to write a checkpoint to the history. These checkpoints are graphically marked as a blue vertical line if you analyze the history log-files with the openMosixanalyzer. For example you can set a checkpoint when you start a job on your cluster and another one at the end..

Here is the explanation of the possible commandline-arguments:

openmosixcollector -d      //starts the collector as a daemon 
openmosixcollector -k      //stops the collector 
openmosixcollector -n      //writes a checkpoint to the history 
openmosixcollector -r      //saves the current history and starts a new one 
openmosixcollector         //print out a short help 

You can start this daemon with its init-script in /etc/init.d or /etc/rc.d/init.d. You just have to create a symbolic link to one of the runlevels for automatic startup.

How to analyze the created logfiles is described in the openMosixanalyzer-section.

10.7. openMosixanalyzer

/proc/hpc/nodes/[openMosix-ID]/mem. 
/proc/hpc/nodes/[openMosix-ID]/rmem. 
/proc/hpc/nodes/[openMosix-ID]/tmem. 

10.8. openMosixmigmon

10.9. openmosixview FAQ

10.10. openMosixview + ssh:

(this HowTo is for SSH2) You can read the reasons why you should use SSH instead of RSH everyday on the newspaper when another script-kiddy hacked into an insecure system/network. So SSH is a good decision at all.

freedom x security = constant    (from a security newsgroup) 

At first a running secure-shell daemon on the remote site is required. If it is not already installed install it! (rpm -i [sshd_rpm_packeage_from_your_linux_distribution_cd]) If it is not already running start it with:

/etc/init.d/ssh start 

ssh-keygen 

/root/.ssh/identity    //your private key 
and 
/root/.ssh/identity.pub     //your public key 

If you ssh to this remote host now you will be prompted for the passphrase of your public-key. Giving the right passphrase should give you a login.

What is the advantage right now??? The passphrase is normally a lot longer than a password!

The advantage you can get using the ssh-agent. It manages the passphrase during ssh login.

ssh-agent 

echo $SSH_AUTH_SOCK 
and 
echo $SSH_AGENT_PID 

SSH_AUTH_SOCK=/tmp/ssh-XXYqbMRe/agent.1065 
export SSH_AUTH_SOCK 
SSH_AGENT_PID=1066 
export SSH_AGENT_PID 

setenv SSH_AUTH_SOCK /tmp/ssh-XXYqbMRe/agent.1065 
setenv SSH_AGENT_PID 1066 

You just have to add your public-key to the ssh-agent with the ssh-add command.

ssh-add 

You could (should) add the ssh-agent and ssh-add commands in your login-profile e.g.

eval `ssh-agent` 
 ssh-add 

openMosixview There is a menu-entry which toggles using rsh/ssh with openMosixview. Just enable this and you can use openMosixview even in insecure network-environments. You should also save this configuration (the possibility for saveing the current config in openMosixview was added in the 0.7 version) because it gets initial data from the slave using rsh or ssh (just like you configured).

If you choose a service wich is not installed properly openMosixview will not work! (e.g. if you cannot rsh to a slave without being prompted for a password you cannot use openMosixview with RSH; if you cannot ssh to a slave without being prompted for a password you cannot use openMosixview with SSH)

11.1. Introduction

There are a couple of different applications available to monitor and admin openMosix, we give a short overview of them in this chapter, we won't really go in detail.

11.2. openMosixView

openMosixview is the most used and the best known applet for openMosix administration, you can read more about it in the openMosix adminstration chapter.

11.3. openMosixapplet

The openMosixApplet lets you watch the realtime load of your openMosix cluster. It consists of a local daemon which listens for connections by applets. The applet uses chart2D to provide a good-lookin' feeling.

11.4. wmonload

wmomload is a simple, but handy and small dockapp for overviewing the load of cluster nodes in a small openMosix-based cluster.

11.5. openMosixWebView

openMosixWebView - Produces web charts for monitoring an openMosix cluster. openMosixWebView is a PHP script for monitoring an openMosix cluster via the WEB. It uses openMosixview's openMosixCollector logs. Download now the last release openmosixwebview-0.2.12.tar.gz (16 Feb 2003)

See openMosixWebView screenshots and running :-) Released under the GNU General Public License (GPL). See README and FAQ files.

12.1. Introduction

Although most of the issues in this chapter could be a part of the FAQ. There where the FAQ will give a short "how to solve" answer, we have taken a closer look at them and explained why they are problems and how to solve them.

12.2. My processes won't migrate

Help process XYZ doesn't migrate. Moshe Bar explains below why some processes migrate and why some don't. But before that you can always look in /proc/$pid/, there often is a cantmove file which will tell you why a certain process can't migrate.

Processes can also be locked. You can check if a process is locked with:

cat /proc/$PID/lock

Now listen to what Moshe himself has to say about this topic.

Often people have the same kernel but on a different distribution, say a mixed environment of RedHat and Debian ,rc scripts from different distros tend to start openmosix differently. Some implementations completely modify /etc/inittab to start all daemons (and their children) with

mosrun -h

Ok, this simple program should always migrate if launched more times than number of local CPUs. So for a 2-way SMP system, starting this program 3 times will start migration if the other nodes in the cluster have at least the same speed like the local ones:

int main() {
    unsigned int i;
    while (1) {
        i++;
    }
    return 0;
}

This sample program with content like this will never migrate:

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>

...

key_t key; /* key to be passed to shmget() */
int shmflg; /* shmflg to be passed to shmget() */
int shmid; /* return value from shmget() */
int size; /* size to be passed to shmget() */

...

key = ...
size = ...
shmflg) = ...

if ((shmid = shmget (key, size, shmflg)) == -1) {
   perror("shmget: shmget failed"); exit(1); } else {
   (void) fprintf(stderr, "shmget: shmget returned %d\n", shmid);
   exit(0);
  }
...

Program using pipes like this do migrate nicely:

int pdes[2];

pipe(pdes);
if ( fork() == 0 )
  { /* child */
                                 close(pdes[1]); /* not required */
                                 read( pdes[0]); /* read from parent */
                                 .....
                 }
else
                 { close(pdes[0]); /* not required */
                                 write( pdes[1]); /* write to child */
                                 .....
                 }

//
// Very simple program demonstrating the use of threads.
//
// Command-line argument is P (number of threads).
//
// Each thread writes "hello" message to standard output, with
//   no attempt to synchronize.  Output will likely be garbled.
//
#include <iostream>
#include <cstdlib>              // has exit(), etc.
#include <unistd.h>             // has usleep()
#include <pthread.h>            // has pthread_ routines

// declaration for function to be executed by each thread
void * printHello(void * threadArg);

// ---- Main program -------------------------------------------------

int main(int argc, char* argv[]) {

  if (argc < 2) {
    cerr << "Usage:  " << argv[0] << " numThreads\n";
    exit(EXIT_FAILURE);
  }
  int P = atoi(argv[1]);

  // Set up IDs for threads (need a separate variable for each
  //   since they're shared among threads).
  int * threadIDs = new int[P];
  for (int i = 0; i < P; ++i)
    threadIDs[i] = i;

  // Start P new threads, each with different ID.
  pthread_t * threads = new pthread_t[P];
  for (int i = 0; i < P; ++i)
    pthread_create(&threads[i], NULL, printHello,
                   (void *) &threadIDs[i]);

  // Wait for all threads to complete.
  for (int i = 0; i < P; ++i)
    pthread_join(threads[i], NULL);

  // Clean up and exit.
  delete [] threadIDs;
  delete [] threads;
  cout << "That's all, folks!\n";
  return EXIT_SUCCESS;
}

// ---- Code to be executed by each thread ---------------------------

// pre:  *threadArg is an integer "thread ID".
// post:  "hello" message printed to standard output.
//        return value is null pointer.
void * printHello(void * threadArg) {
  int * myID = (int *) threadArg;
  cout << "hello, world, ";
  // pointless pause, included to make the effects of
  //   synchronization (or lack thereof) more obvious
  usleep(10);
  cout << "from thread " << *myID << endl;
  pthread_exit((void* ) NULL);
}

(all above code is by Moshe as Moshe Bar or by Moshe as CTO of Qlusters, Inc.)

Please also refer to the man pages of openMosix , they also give an adequate explanation why processes don't migrate.

If for some reason your processes stay locked while they shouldn't. You can try to allow locked processes to migrate by simply putting

# tell shells to allow subprocs to migrate to other nodes
echo 0 > /proc/self/lock

12.3. I don't see all my nodes

First of all , are you using the same kernel version on each machine ? The 'same-kernel' refers to the version. You can build different kernel images of the same source version to meet the hardware/software needs of a given node. However you wil need toe make sure that when you install openMosix on your cluster, all your machines should have the openmosix-x.x.x-y kernel installed, in contrast to having one machine running openmosix-x.x.z-x, another running openmosix-x.x.x-y, another running openmosix x.x.x-z, and so on and so forth

When you run mosmon, press t to see the total of machines running. Does it warn you that mosix is not running?

If yes, then make sure your machine's ip is included in /etc/mosix.map (don't use 127.0.0.1 - if your machine's ip is such, then you probably have problems with your dhcp server/nameserver). If it does not tell you that mosix is not running, see what machines show up. Do you see only your machine?

If yes, then your machine is most likely running a firewall and is not letting openmosix through.

If not, then the problem is most likely with the machine that doesn't show up. Also: Do you have two nic cards on a node? then you have to edit the /etc/hosts file to have a line that has the following format

non-cluster_ip  cluster-hostname.cluster-domain cluster-hostname

1 cnode1 	1
2 cnode2     	1 

127.0.0.1 localhost
123.456.7.89 usual.name.domain nickname
192.168.0.1 cnode1
192.168.0.2 cnode2

setpe -p1 -f /etc/openmosix.map 

You might also need to set up a routing table, which is a whole different subject.

Maybe you used different kernel-parameters on each machine? Especially if you use the 'Support clusters with a complex network topology' option you should take care that you use the same value for the also appearing option 'Maximum network-topology complexity support' on each machine.

12.4. I often get errors: No such process

I often get the error

bash: child setpgid (4061 to 4061): No such process

The above line meas that the shell you were using has acutallly migrated to another node ? This printout from bash is caused by a bug in old version of openmosix, but a fix has been commited. (Muli Ben-Yehuda mulix@actcom.co.il)

12.5. DFSA ? MFS ?

As of 2.4.26-om1 oMFS has been removed from openMosix

People often get confused about what exactly MFS and DFSA are. As discussed before in the howto MFS is the feature of openMosix that enables you access to remote filesystems as if those filesystems were locally mounted. They are mostly mounted on /mfs . A common misunderstanding is that you need MFS in order to have openMosix working, this is not true, however it can make things easier.

With DFSA enabled, system calls will be executed on the remote node withouth migrating the process back to it's home node. This behaviour (direct filesystem access) causes processes migratiing to the data and not the other way around (which is common). If DFSA is not enabled MfS is "just" a non-caching network-filesystem.

Very generally speaking, if you don't have DFSA turned on, each and every I/O will go to the home node for execution. With DFSA turned on, if the file happens to be residing on the node where the process finds itself then the I/O will happen locally.

A very common error is that people mix kernels with DFSA enabled and disabled. So one has to have a way to find out wether DFSA is actually enabled. This information can be obtained by typing

cat /proc/hpc/admin/version

12.6. Python Troubles

Some people have reported problems with Python, closer research showed that these problems were not with openMosix but rather with glibc issues, however it seemed that issues manifested themselves especially in openMosix.

One user solved the problem by removig /lib/i686/lib* on his machine and let the applications link dynamically against /lib/libpthread (and other) However bugfixes in newer glibc versions combined with more recent openMosix version seem to have solved these problems.

13.1. Locked Processes

If for some reason you find your processes are always locked in your home node and you can't find the reason, you can put the following lines into your ~/.profile as a stop-gap measure to automatically enable migration:

        if [ -x /proc/$$/lock ]; then
           echo 0 > /proc/$$/lock
        fi

13.2. Choosing your processes

You will probably want to test your setup before deciding which programs you want to enable migration for. For example, if you are running KDE2 on a slow machine and have a significantly faster machine has part of your Mosix cluster, you might find resource-hungry programs like kmail are migrated out. This is not a bad thing as such, however, it can lead to a brief moment when your writing is not displayed on the screen immediately.

13.3. Java and openMosix

Green Threads JVM's, allow for migration because each Java thread is a separate process. Threads other than Java green thread JVM's cannot be migrated by Linux, so openMosix cannot migrate programs that use them.

If you have the source so your Java application you might be able to compile the application native. In this case you might be able to migrate your applications to another node.

Gian Paolo Ghilardi wrote a paper titled Consideration on OpenMosix it deals amongst other topics with Java an dopenMosix. http://www.democritos.it/events/openMosix/papers/crema.pdf

13.4. openMosix and Hyperthreading

Basically openMosix performance increases with the current Linux scheduler when Hypethreading is disabled. You can do this by either entering 'noht' as a boot option or disabling HT in the bios.

For those who are still wondering what hypetrheading is : Intel explains it

13.5. openMosix and Firewalls

People often have questions regarding openMosix and firewalls. Amit helped me out on this matter:

from hpc/comm.c:

#define MIG_DAEMON_PORT         0x3412
#define INFO_DAEMON_PORT        0x3415

the mig_daemon port is a tcp port, the info_daemon port is udp. Hence tcp/4660 and udp/5428, Matt also mentions tcp/723 somewhere.

14.1. A small Test Script

The fastest way to test your openMosix cluster is by creating a small script with the following content.

awk 'BEGIN {for(i=0;i<10000;i++)for(j=0;j<10000;j++);}' &

for i in `ls /etc/` ; do ./test_mosix ; done 

14.2. Perl Proggie by Charles Nadeau

Perl program to test an openMosix Cluster.

Here is a is quick program I wrote to test an openMosix cluster. This is taken from a posting I made to the openMosix-devel mailing list on March 6th, 2002: "Charles wrote this little program (in Perl) to stress test his home cluster (3 P200MMX and a P166). It is a program simulating different sets of stocks in a portfolio for a given period of time. The code is well documented and it should be easy to add/remove stocks and change the average monthly yield and standard deviation for each stock. Since the problem of portfolio optimization cannot be solved analytically, it simulate a lot of portfolios and report the results at the end. Please note that this program does not take stock correlation into account. It is not finished yet but it's a good start. I plan to add more code at the end of the program to improve the reporting format of the data (generating SVG graph on the fly). But the simulation part works very well. In order to take advantage of the parallelism offered by openMosix, it uses the Perl module Parallel::?ForkManager (from CPAN) to span threads that openMosix can then assign to all the machines of the cluster (it also require another module for the statistical calculations, don't forget to install both, I provide the URLs in the comments of the code). Take a look at it and tell me what you think. Cheers!"

#! /usr/bin/perl -w

# this mill unlock this process and all tis childs
sub unlock { 
open (OUTFILE,">/proc/self/lock") || 
die "Could not unlock myself!\n"; 
print OUTFILE "0"; 
} 

unlock;

# this will count the nodes
sub cpucount {
 $CLUSTERDIR="/proc/hpc/nodes/";
 $howmany=0;
 opendir($nodes, $CLUSTERDIR);
 while(readdir($nodes)) {
  $howmany++;
 }

 $howmany--;
 $howmany--;
 closedir ($nodes);
 return $howmany;
}

my $processes=cpucount;
$processes=$processes*3;

print("starting $processes processes\n");

#Portfolio.pl, version 0.1
#Perl program that simulate a portfolios for various stock composition for a given period of time
#We run various scenarios to find the mix of assets that give the best performance/risk ratio
#This method is base on the book "The intelligent asset allocator" by William Bernstein
#Can be used to test an OpenMosix cluster
#This program is licensed under GPL
#Author: Charles-E. Nadeau Ph.D., (c) 2002
#E-mail address: charlesnadeau AT hotmail DOT com

use Parallel::ForkManager; #We use a module to parallelize the calculation

#Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2589

use Statistics::Descriptive::Discrete; #A module providing statistical values

#Available at http://theoryx5.uwinnipeg.ca/mod_perl/cpan-search?new=Search;filetype=%20distribution%20name%20or%20description;join=and;arrange=file;download=auto;stem=no;case=clike;site=ftp.funet.fi;age=;distinfo=2988

srand; #We initialize the random number generator

#Initializing constant
$NumberOfSimulation=$processes;  #Number of simulation to run
$NumberOfMonth=100000; #Number of month for wich to run the simulation
$NumberOfStock=6; #Number of different stocks in the simulation

#Portfolio to simulate
#TODO: Read the stock details from a file
$Stock[0][0]="BRKB"; #Stock ticker
$Stock[0][1]=0.01469184; #Stock average monthly return
$Stock[0][2]=0.071724934; #Stock average monthly standard deviation
$Stock[1][0]="TEST "; #Stock ticker
$Stock[1][1]=-0.01519; #Stock average monthly return
$Stock[1][2]=0.063773903; #Stock average monthly standard deviation
$Stock[2][0]="SPDR"; #Stock ticker
$Stock[2][1]=0.008922718; #Stock average monthly return
$Stock[2][2]=0.041688404; #Stock average monthly standard deviation
$Stock[3][0]="BRKB"; #Stock ticker
$Stock[3][1]=0.01469184; #Stock average monthly return
$Stock[3][2]=0.071724934; #Stock average monthly standard deviation
$Stock[4][0]="TEST "; #Stock ticker
$Stock[4][1]=-0.01519; #Stock average monthly return
$Stock[4][2]=0.063773903; #Stock average monthly standard deviation
$Stock[5][0]="SPDR"; #Stock ticker
$Stock[5][1]=0.008922718; #Stock average monthly return
$Stock[5][2]=0.041688404; #Stock average monthly standard deviation




my $pm = new Parallel::ForkManager($NumberOfSimulation); #Specify the number of threads to span

$pm->run_on_start(
  sub { my ($pid,$ident)=@_;
  print "started, pid: $pid\n";
  }
);



#We initialize the array that will contain the results
@Results=();
for $i (0..$NumberOfSimulation-1){
	for $j (0..$NumberOfStock+3){
		$Results[$i][$j]=0.0; #Equal to 0.0 to start
	}
}

for $i (0..$NumberOfSimulation-1){ #Loop on the number of simulation to run
	$Results[$i][0]=$i; #The first column of each line is the number of the simulation
	$pm->start and next; #Start the thread

		$TotalRatio=1; #The sum of the proprtion of each stock

		#Here we calculate the portion of each investment in the portfolio for a given simulation
		for $j (0..$NumberOfStock-2){ #We loop on all stock until the second to last one
			#TODO: Replace rand by something from Math::TrulyRandom
			$Ratio[$j]=rand($TotalRatio);
			$Results[$i][$j+1]=$Ratio[$j]; #We store the ratio associated to this stock
			$TotalRatio=$TotalRatio-$Ratio[$j];
		}
		$Ratio[$NumberOfStock-1]=$TotalRatio; #In order to have a total of the ratios equal to one, we set the last ratio to be the remainder
		$Results[$i][$NumberOfStock]=$Ratio[$NumberOfStock-1]; #We store the ratio associated to this stock. Special case for the last stock

		$InvestmentValue=1; #Initially the investment value is 1 time the initial capital amount.
		my $stats=new Statistics::Descriptive::Discrete; #We initialize the module used to calculate the means and standard deviations

		for $j (1..$NumberOfMonth){ #Loop on the number of months

			$MonthlyGrowth[$j]=0.0; #By how much did we grow this month. Initially, no growth yet.

			#We loop on each stock to find its monthly contribution to the yield 
			for $k (0..$NumberOfStock-1){

			$MonthlyGrowth[$j]=$MonthlyGrowth[$j]+($Ratio[$k]*((gaussian_rand()*$Stock[$k][2])+$Stock[$k][1])); #We had the growth for this stock to the stock already calculated for the preceding stocks
			}

			$stats->add_data($MonthlyGrowth[$j]); #Add the yield for this month so we can later on have the mean and standard deviation for this simulation
			$InvestmentValue=$InvestmentValue*(1+$MonthlyGrowth[$j]); #Value of the Investment after this month
		}
		$Results[$i][$NumberOfStock+1]=$stats->mean(); #Calculate the average monthly growth
		$Results[$i][$NumberOfStock+2]=$stats->standard_deviation(); #Calculate the standard deviation of the monthly growth

	$pm->finish; #Finish the thread
}
$pm->wait_all_children; #We wait until all threads are finished

#Printing the results
print "Simulation ";
for $j (0..$NumberOfStock-1){
	print "Ratio$Stock[$j][0] ";
}
print "  Mean StdDev YieldRatio\n";
for $i (0..$NumberOfSimulation-1){
	printf "%10d ", $Results[$i][0];
	for $j (1..$NumberOfStock){
		printf "   %6.2f ",$Results[$i][$j];
	}

	if($Results[$i][$NumberOfStock+2]!=0) {
		printf "%5.4f %5.4f    %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], ($Results[$i][$NumberOfStock+1]/$Results[$i][$NumberOfStock+2]);

	} else {
		printf "%5.4f %5.4f    %5.4f\n", $Results[$i][$NumberOfStock+1], $Results[$i][$NumberOfStock+2], 0;

	}
}



#Subroutines

#Subroutine to generate two numbers normally distributed
#From "The Perl Cookbook", recipe 2.10
sub gaussian_rand {
	my ($u1, $u2);
	my $w;
	my ($g1, $g2);

	do {
		$u1=2*rand()-1;
		$u2=2*rand()-1;
		$w=$u1*$u1+$u2*$u2;
	} while ($w>=1 || $w==0); #There was an error in the recipe, I corrected it here

	$w=sqrt(-2*log($w)/$w);
	$g2=$u1*$w;
	$g1=$u2*$w;
	return wantarray ? ($g1,$g2) : $g1;

}

14.3. the openMosix stress-test

by Matt Rechenburg

[root@dhcp51 omtest]# ./start_openMosix_test.sh


starting the openMosix stress test now!

the results will be saved in : /tmp/openMosix-stress-test-report-03/16/2003-11:27:02.txt

oMFS is not mounted at /mfs! oMFS-test will be disabled.
Please mount oMFS before running this openMosix-test
You will find instructions how to configure and use oMFS at:
http://howto.ipng.be/openMosix-HOWTO/x222.htm#AEN243


(return to continue, ctrl-c for cancel)

15.1. Introduction

This Section is a Work in Progress

Lots of people try to use openMosix as a kind of compile farm, ofthen they come back very disappointed. This chapter of the howto will try to explain in which cases your compilations will benefit from openMosix and how to improve your successlevel.

First of all you have to remember 1 thing. openMosix will not migrate all processes you start on your cluster, only the ones that will benefit from migration to another node. For compiling this means that a process has to last long enough. Kernel compiles typically consist of numerous short compiles, each of them not being long enough to acutally migrate.

16.1. Introduction

This Section is a Work in Progress

Computer Graphics have always been applications that required a lot of CPU power, this hasn't changed. Within this chapter I wil demonstrate with some practical examples how Computer Graphics can benefit from openMosix.

16.2. Povray

The Persistence of Vision Raytracer is a high-quality, totally free tool for creating stunning three-dimensional graphics.

Ray-tracing is a rendering technique that calculates an image of a scene by simulating the way rays of light travel in the real world. However it does its job backwards. In the real world, rays of light are emitted from a light source and illuminate objects. The light reflects off of the objects or passes through transparent objects. This reflected light hits our eyes or perhaps a camera lens. Because the vast majority of rays never hit an observer, it would take forever to trace a scene.

These kind of applications can be easily made parrallel by using pvmpovray. Pvmpovray expects to working on a Beowulf style cluster and spread it's load to other nodes using pvm. The openMosix way of doing this is the same, however we just do this on 1 machine and have openMosix do the load spreading work fo you !

A GREAT Howto on PVM Povray will show you how to setup PVMPovray. Below is a small summary.

$ cd pvmpov3_1g_2
$ tar xfz ../povuni_s.tgz
$ tar xfz ../povuni_d.tgz                                                      
$ ./inst-pvm
Trying to apply the patch.
Searching for rejected files
$

If you are on a RH 8.0 box id moved libpng and zlib to .notused .. This in order to prevent version issues .. with other libpng and zlib versions.

export PATH=$PATH:/usr/share/pvm3/lib
export PVMROOt=/usr/share/pvm3

And a last thing that is not known by a novice pvm user is that pvm does use its own paths, and you have to put pvmpov either in that path or launch it with the complete pathname.

[root@dhcp71 povray31]# /usr/local/bin/pvmpov -L 
/usr/src/povray/pvmpov3_1g_2/povray31/include/ +Iskyvase.pov  
+Oskyvase.tga +NT16 +NW64 +NH64 +v +w1024 +h768
Persistence of Vision(tm) Ray Tracer Version 3.1g.Linux.gcc
  This is an unofficial version compiled by:
  Jakob Flierl  - PVMPOV Version 3.1g.2
  The POV-Ray Team(tm) is not responsible for supporting this version.
Copyright 1999 POV-Ray Team(tm)
Never found section  in file 
/usr/src/povray/pvmpov3_1g_2/povray31/include/.
Initializing PVMPOV
  Spawning /usr/local/bin/pvmpov with 16 PVM tasks on 1 hosts...
  ...16 PVM tasks successfully spawned.
  Waiting up to 120s for first slave to start...
  Slave 0 successfully started.
Parsing Options
  Input file: skyvase.pov (compatible to version 3.1)
  Remove bounds........On  Split unions........Off
  Library paths: /usr/local/lib/povray31 /usr/local/lib/povray31/include
Output Options
  Image resolution 1024 by 768 (rows 1 to 768, columns 1 to 1024).
  Output file: skyvase.tga, 24 bpp PNG
  Graphic display.....Off
  Mosaic preview......Off
  CPU usage histogram.Off
  Continued trace.....Off  Allow interruption...On  Pause when 
done.....Off
  Verbose messages.....On
Tracing Options
  Quality:  9
  Bounding boxes.......On  Bounding threshold: 25
  Light Buffer.........On  Vista Buffer.........On
  Antialiasing........Off
  Radiosity...........Off
Animation Options
  Clock value....   0.000  (Animation off)
PVM Options
  Block Width....      64  Block Height...      64
  PVM Tasks......      16
  PVM Nice.......       5
  PVM Arch.......
  PVM Slave...... /usr/local/bin/pvmpov
  PVM WorkingDir. /usr/src/povray/pvmpov3_1g_2/povray31
Redirecting Options
  All Streams to console..........On
  Debug Stream to console.........On
  Fatal Stream to console.........On
  Render Stream to console........On
  Statistics Stream to console....On
  Warning Stream to console.......On

Starting frame 0...
 Slave 1 at dhcp71.office.be.stone-it.com successfully started.
 Slave 2 at dhcp71.office.be.stone-it.com successfully started.
 Slave 3 at dhcp71.office.be.stone-it.com successfully started.
 Slave 4 at dhcp71.office.be.stone-it.com successfully started.
 Slave 5 at dhcp71.office.be.stone-it.com successfully started.
 Slave 6 at dhcp71.office.be.stone-it.com successfully started.
 Slave 7 at dhcp71.office.be.stone-it.com successfully started.
 Slave 8 at dhcp71.office.be.stone-it.com successfully started.
 Slave 9 at dhcp71.office.be.stone-it.com successfully started.
 Slave 10 at dhcp71.office.be.stone-it.com successfully started.
 Slave 11 at dhcp71.office.be.stone-it.com successfully started.
 Slave 12 at dhcp71.office.be.stone-it.com successfully started.
 Slave 13 at dhcp71.office.be.stone-it.com successfully started.
 Slave 14 at dhcp71.office.be.stone-it.com successfully started.
 Slave 15 at dhcp71.office.be.stone-it.com successfully started.
  0:00:53 86.46 of blocks complete.Not using dhcp71.office.be.stone-it.com 
for reassignment (77%)
  0:00:53 86.98 of blocks complete.Not using dhcp71.office.be.stone-it.com 
for reassignment (67%)
Not using dhcp71.office.be.stone-it.com for reassignment (86%)
Not using dhcp71.office.be.stone-it.com for reassignment (85%)
  0:00:55 89.06 of blocks complete.   640 of  768 lines finished (in frame 
0).Not using dhcp71.office.be.stone-it.com for reassignment (65%)
  0:00:56 91.67 of blocks complete.Not using dhcp71.office.be.stone-it.com 
for reassignment (72%)
  0:00:56 92.71 of blocks complete.Not using dhcp71.office.be.stone-it.com 
for reassignment (80%)
  0:00:57 93.75 of blocks complete.
Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:57 94.79 of blocks complete.
Slave at dhcp71.office.be.stone-it.com has exited.

Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:58 95.83 of blocks complete.
Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:58 96.35 of blocks complete.   672 of  768 lines finished (in frame 
0).Not using dhcp71.office.be.stone-it.com for reassignment (77%)

Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:58 97.14 of blocks complete.   688 of  768 lines finished (in frame 
0).
Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:59 97.92 of blocks complete.
Slave at dhcp71.office.be.stone-it.com has exited.
  0:00:60 98.44 of blocks complete.   704 of  768 lines finished (in frame 
0).
Slave at dhcp71.office.be.stone-it.com has exited.
  0:01:03 100.00 of blocks complete.   768 of  768 lines finished (in 
frame 0).
Finishing frame 0...rtw. 768


Waiting for remaining slave stats.


PVM Task Distribution Statistics:
           host name  [ done ] [ late ]           host name  [ done ] [ 
late ]
dhcp71.office.be.stone-it.com  [ 5.21%] [ 
0.00%]dhcp71.office.be.stone-it.com  [ 7.81%] [ 0.07%]
dhcp71.office.be.stone-it.com  [ 8.85%] [ 
1.17%]dhcp71.office.be.stone-it.com  [ 4.69%] [ 0.00%]
dhcp71.office.be.stone-it.com  [ 8.85%] [ 
0.98%]dhcp71.office.be.stone-it.com  [ 4.17%] [ 0.00%]
dhcp71.office.be.stone-it.com  [ 5.21%] [ 
0.00%]dhcp71.office.be.stone-it.com  [ 8.33%] [ 0.52%]
dhcp71.office.be.stone-it.com  [ 5.21%] [ 
0.00%]dhcp71.office.be.stone-it.com  [ 5.73%] [ 0.72%]
dhcp71.office.be.stone-it.com  [ 7.29%] [ 
2.73%]dhcp71.office.be.stone-it.com  [ 4.17%] [ 0.00%]
dhcp71.office.be.stone-it.com  [ 5.21%] [ 
0.00%]dhcp71.office.be.stone-it.com  [ 6.77%] [ 0.13%]
dhcp71.office.be.stone-it.com  [ 4.69%] [ 
0.00%]dhcp71.office.be.stone-it.com  [ 7.81%] [ 0.00%]


POV-Ray statistics for finished frames:
skyvase.pov Statistics (Partial Image Rendered), Resolution 1024 x 768
----------------------------------------------------------------------------
Pixels:          303104   Samples:          303104   Smpls/Pxl: 1.00
Rays:           1192710   Saved:                 0   Max Level: 0/5
----------------------------------------------------------------------------
Ray->Shape Intersection          Tests       Succeeded  Percentage
----------------------------------------------------------------------------
Cone/Cylinder                  1842227          900504     48.88
CSG Intersection               2742731          323346     11.79
CSG Union                      1801008          521692     28.97
Plane                         20223278        11233348     55.55
Quadric                        1801008         1196533     66.44
Sphere                         1801008          461786     25.64
Bounding Object                1842227          900504     48.88
----------------------------------------------------------------------------
Calls to Noise:            1201944   Calls to DNoise:        2108954
----------------------------------------------------------------------------
Shadow Ray Tests:          2856188   Succeeded:                85620
Reflected Rays:             889606
----------------------------------------------------------------------------
Smallest Alloc:                  9 bytes   Largest:            20508
Peak memory used:          5643343 bytes
----------------------------------------------------------------------------
Time For Trace:    0 hours  1 minutes   7.0 seconds (67 seconds)
    Total Time:    0 hours  1 minutes   7.0 seconds (67 seconds)

I had good results with 2 to 3 times the the number of cpu's I had available

17.1. Introduction

17.2. Blast

One of the more frequent used application in this field is Blast, Blast has a patch available that makes it work smoother with openMosix, but that's not the only alternative.

First of all there are some known problems with this patch, and other versions of blast , blast tends to segfault sometimes, this mostly happens with the preformatted databases you download from the internet. If you run formatdb on a raw database these errors tend to go away.

Next to the openMosix blast patch a lot of people run MPIBlast Given the fact that openMosix tends to speed up MPI, adding openMosix to this config might even give you more power for your money, however we will have to do some extra research to be able to confirm this.

18.1. Introduction

this part has been written by Amit Shah

There's not much documentation available right now for the kernel. I hope to write some in the coming weeks. Anyways, here's how the sources are laid out:

The openMosix code resides largely in hpc/ and include/hpc. There are lots of patches to the core kernel files everywhere, right from the arch/i386 directories to mm/, fs/, etc. You need to read up the code which interests you and think that would matter for the present situation (that shouldn't be a problem, since you've done kernel coding).

here's what you should expect in each of the source files:

• hpc/badops.c: one file to handle all the bad operations: mostly return err codes

• hpc/balance.c: The load balancer code (load + mem usage + n/w usage)

• hpc/comm.c: The intra-cluster communication setup

• hpc/config.c: The config code for openMosix: after you run the startup script

• hpc/decay.c: decay (age) the stats and info collected from other nodes

• hpc/deputy.c: Code executed on the deputy: service remote syscalls (ie. after the process has migrated), signals, etc.

• hpc/dfsa.c: Direct File System Access code: the distributed file system abstraction layer

• hpc/div.c: the algorithms to do floating point divisions

• hpc/export.c: export symbols needed in other files

• hpc/freemem.c: to keep track of free, avl. memory and to free it if need be. hugely taken from the Linux mm/ code.

• hpc/hpcadmin.c: tune openMosix admin values (through /proc/hpc)

• hpc/hpcproc.c: The /proc/hpc code is handled here

• hpc/info.c: The info daemon: sends and receives (multicast) load+mem usage stats throughout the cluster

• hpc/init.c: Initialization code: initializes the daemons, etc.

• hpc/kernel.c: most of the "core" code: all the important algorithms, decisions, etc. made here.

• hpc/load.c: calculation of local load, etc.

• hpc/mig.c: Code that handles the migration. Code in this file is invoked on any migration: deputy->remote, remote->deputy; remote->remote

• hpc/prequest.c: handles the process's requests: signals, more memory, etc.

• hpc/remote.c: Code executed when the process is on the remote: syscalls handling on remote, passing control to deputy, etc.

• hpc/rinode.c: fs/ related stuff: used mostly for DFSA

• hpc/service.c: setting up daemons, getting memory, etc.

• hpc/syscalls.c: handles all the remote syscalls here

• hpc/ucache.c: handles the ucache: mostly mm/, fs/ stuff.

19.1. General

19.2. Getting, building, installing and running openMosix

19.3. Kernel Questions

19.4. File Systems

19.5. Programming openMosix

19.6. Resources

19.7. Miscellaneous

<non-cluster ip> <cluster-hostname>.<cluster-domain> <cluster-hostname>

127.0.0.1 hostname.domain.com localhost

192.168.10.250 hostname.domain.com 
127.0.0.1 localhost

20.1. Frequently Asked Questions