1.1. What is this all about?

Recent linux kernels offer the possibility to boot a linux box entirely from network, by loading its kernel and root filesystem from a server. In that case, the client may use several ways to get the first instructions it has to execute when booting: home made eproms, special network cards implementing the RARP, BOOTP or DHCP protocols, cdroms, or bootloaders loaded from a boot floppy or a local hard drive.

1.2. Thanks

Logilab sponsored this HOWTO. Check their website for new versions of this document. I also thank the etherboot, netboot, plume and linux terminal server project developers and webmasters, who made it really possible to boot a Linux worstation over a network.

Very special thanks go to Ken Yap, member of the etherboot project, whose comments greatly helped to improve the quality of this document.

I also thank Jerome Warnier, main developer of the plume project, Pierre Mondié, Kyle Bateman, Peter T. Breuer, Charles Howes, and Thomas Marteau for their comments and contributions.

1.3. Diskless booting advocacy

1.4. Requirements

1.5. Acknowledgements and related documentation

This documentation has been written for experimented system administrators, who are already aware of linux fundamentals, like the use of grep, sed, and awk, basic shell programming, the init process and the boot scripts, kernel compilation, and NFS server configuration. Experience of kernel arguments passing should also help. Information on these subjects can be found respectively in the grep, sed, awk, and bash man/info pages, in the Bootdisk-HOWTO, the From-PowerUp-To-Bash-Prompt-HOWTO, the Kernel-HOWTO, the BootPrompt-HOWTO, the bootparam man page, the rdev man page, the NFS-HOWTO, and the exports manual page.

There are many sources of information on network booting, but, and this is why I wrote this HOWTO, none describes all the existing ways of booting over a network, and much of them are specific to a way of operating. The most useful to me has been the documentation provided by the linux terminal server project, although I did not use the packages they recommend, and I have chosen to describe here how to proceed without these packages, because they configure things so that every application program is executed remotely on a server. Useful information can also be found on the etherboot project's homepage.

At last, you can also find useful but succinct information in the kernel's source tree, in /usr/src/linux/Documentation, assuming your kernel source tree resides in /usr/src/linux.

1.6. Feedback

I will highly appreciate any feedback about this document. Please feel free to mail me at <bbp@via.ecp.fr> if you have any comment, correction, or suggestion. You may also use <contact@logilab.fr>.

1.7. Copyright Information

This document is copyrighted (c) 2001 and is distributed under the terms of the GNU Free Documentation License. You should have received a copy along with it. If not, it is available from http://www.fsf.org/licenses/fdl.html.

2.1. Obtaining IP parameters

One could wonder how a station may boot over an IP network if it doesn't even know its own IP address. In fact, three protocols enable the client to obtain this information and some additional configuration parameters:

• RARP: this is the simplest of these protocols. However I guess it does not enable the server to specify how the client should download the kernel, so we won't use it (In fact, there is a convention that uses the IP address of the workstation as filename, e.g. a client getting the address 192.168.42.12 by RARP might ask for /tftpboot/192.168.42.12 by TFTP, as the linux kernel does. The filename might also be the hex form of the IP address, this is implementation dependant, and is not mandatory.).

• BOOTP: this protocol allows a server to provide the client (identified by its hardware MAC address) with much information, in particular its IP address, subnet mask, broadcast address, network address, gateway address, host name, and kernel loading path. This is the one we will use.

• DHCP: this is an extension of BOOTP.

2.2. Loading the kernel

When the client has got its IP parameters, if the kernel is not on a local support (like a floppy, a cdrom, or a hard drive), the client will start to download it via TFTP. Its location is given by the BOOTP/DHCP server. A server (not necessarily the BOOTP/DHCP server) will also have to run a TFTP daemon for non local kernels. The kernel one obtains after compilation can not be used "as is" for BOOTP/DHCP operation, its binary image has to be modified with the mknbi utility (and then modified again with the imggen utility if you use LanWorks EPROMs). The mknbi utility should also be used to modify kernels that will be written in a ROM.

2.3. Mounting the root filesystem

After the kernel has started, it will try to mount its root filesystem. The location of this filesystem is also obtained through BOOTP/DHCP, and it is mounted via NFS. It means a client may use BOOTP twice for booting: the first time to get its kernel, and the second time to learn the location of the root filesystem (which may be on a third server).

Another solution is to use a ramdisk as root filesystem. In this case, the ramdisk image is obtained with the kernel via TFTP.

2.4. Terminating the boot process

When the root filesystem is mounted, you can start breathing: you can at least use your swiss army knife with its sh, sed, and awk blades. In fact, you will have to customize the initialization scripts of the client's filesystem: for instance, you will have to remove all hard drive, floppy or cdrom related stuff from /etc/fstab (when your stations are not equipped with these devices), you may also have to inhibit swap partitions activation (note there is a way to swap over NFS or network block devices). You also will have to automagically generate all network configuration files at boot time if several clients use the same remote root filesystem.

3.1. When the root filesystem is on a ramdisk

It is possible to use a ramdisk for the root filesystem. In this case, the command used to modify the kernel's binary image is slightly different. If you choose to do so, you have to enable support for initial ramdisk (initrd), and you probably don't need NFS support, or you probably can compile it as a module.

Its time to give an overview of what happens when you use initrd. The full documentation for this is in your kernel source tree, in the Documentation/initrd.txt file. I have to warn you I did never try this :).

When initrd is enabled, the boot loader first loads the kernel and the inital ramdisk into memory. Then, the ramdisk is mounted read-write as root filesystem. The kernel looks for a /linuxrc file (a binary executable or a script beginning with #!). When /linuxrc terminates, the traditionnal root filesystem is mounted as /, and the usual boot sequence is performed. So, if you want to run your box entirely from ramdisk, you just have to create a link from /linuxrc to /sbin/init, or to write there a shell script to perform any action you like, and then shutdown the computer.

After the kernel has been compiled, you have to build a root filesystem for your installation. This is explained in the "Clients setup, creation of the root filesystem" section. I will assume here that this is already done and that the root filesystem for your clients temporarily resides in /tmp/rootfs. You now have to create a ramdisk image. A simple way to do so is the following:

• Make sure the computer you are working on has support for ramdisks and has such a device (/dev/ram0).

• Create an empty filesystem with the appropriate size on this ramdisk:

  # mke2fs -m0 /dev/ram0 300

• Mount it somewhere:

  # mount -t ext2 /dev/ram0 /mnt

• Copy what you need for your new root filesystem, and create your future /linuxrc if you did not create it in /tmp/rootfs/linuxrc:

  # cp -a /tmp/rootfs/* /mnt

• Unmount the ramdisk:

  # umount /mnt

• Save the ramdisk image to some file and free it:

  # dd if=/dev/ram0 of=initrd bs=1024 count=300 # freeramdisk /dev/ram0

Then, you have to modify the kernel image, as was told above, with the mknbi-linux utility. Its invocation will slightly differ from the above, though (I will assume your just compiled zImage resides in /tftpboot/zImage and your initial ramdisk image resides in /tmp/initrd):

        # cd /tftpboot
        # chmod 0555 zImage
        # chown root:root zImage
	# rdev zImage /dev/ram0
        # mknbi-linux zImage --output=kernel --rootdir=/dev/ram0 /tmp/initrd
      

4.1. NFS daemon

Just export the directory in which the client's root filesystem will reside (see the exports manpage for more information about this topic). The simplest is to export it no_root_squash and rw, but a perfect setup would export most of the root filesystem root_squash and ro, and have separate lines in the /etc/exports for directories which really require no_root_squash and/or rw. Just start with everything rw and no_root_squash, the fine tuning will be done later.

Of course, you don't need any NFS server at all if you plan to run your clients entirely from ramdisk.

4.2. BOOTP daemon

I assume you have installed the bootpd package. The default configuration file is /etc/bootptab, and its syntax is detailed in the bootptab manpage. Let's create it.

First, open as root your favourite text editor. It is vim. Yes, it is. If it is not, it has to become. Now, enter the following lines (they are the default attributes). All the attributes you give here and do not override in a machine's specific attributes list will be given to clients):

        .default\
                :sm=your subnet mask\
                :ds=the IP address of your DNS server\
                :ht=ethernet\
                :dn=your domain name\
                :gw=the IP address of your gateway\
                :sa=the IP address of the TFTP server\
                :bf=path to find the kernel image\
                :rp=path of the root filesystem\
                :hn
      

Then, add an entry per client in your network. An entry should look like this:

        dns of the client\
                :ha=MAC address of the client\
                :ip=IP address of the client
      

Here is a sample /etc/bootptab file:

              .default\
                      :sm=255.255.0.0\
                      :ds=192.168.0.2\
                      :ht=ethernet\
                      :dn=frtest.org\
                      :gw=192.168.0.1\
                      :sa=192.168.0.2\
                      :bf=/tftpboot/kernel\
                      :rp=/nfsroot\
                      :hn

              foo\
                      :ha=001122334455\
                      :ip=192.168.2.12

              bar\
                      :ha=00FFEEDDCCBB\
                      :ip=192.168.12.42\
                      :ds=192.168.2.42
      

Then, run the bootpd daemon with the bootpd -s command (it is also a good idea to add it to your startup scripts), or add the following line to your /etc/inetd.conf:

        bootps dgram udp wait root /usr/sbin/tcpd bootpd -i -t 120
      

If you want to test the BOOTP server, add an entry to your /etc/bootptab and use the bootptest program.

4.3. TFTP

Setting up the TFTP daemon is not the hard part: just install the tftpd package if you have one, and add the following line to your /etc/inetd.conf (again, I assume /tftpboot is the directory where the kernel image resides):

        tftp dgram udp wait root /usr/sbin/tcpd in.tftpd /tftpboot
      

You should be aware of the limitations implied by running the TFTP daemon from the inetd. Most inetd's will shutdown a service if it is spawned to frequently. So if you have many clients, you should look for another inetd like xinetd, or run a standalone TFTP daemon.

Now you have properly setup all daemons, you can restart the inetd and take a coffee. Don't forget to tell everyone the server setup is over, so you think you're a hero before you start building the root filesystem for the clients.

5.1. Creating the first files and directories

First, cd to your future station's root directory. You can safely create the future /home directory with the mkdir command, or by copying it from anywhere you want (you can use cp -a to do a recursive copy preserving owners, groups, symlinks, and permissions). Same thing for the future /mnt, /root, /tmp (don't forget to chmod 0 it, this is only a mount point for the actual /tmp we will use, because each workstation needs to have its own /tmp). Then, copy some existing /bin, /sbin, /boot, and /usr into this future root directory (use cp -a). You can create the /proc directory with mkdir, and chmod 0 it. Note some applications need write access to their user's home directory.

The /lib directory can be safely copied from somewhere else, but you will have to put the proper modules in it. To do so, use the following commands (assuming you have compiled the kernel for your clients on the server in /usr/src/linux, and the root filesystem will reside in /nfsroot):

        # cd /usr/src/linux
        # make modules_install INSTALL_MOD_PATH=/nfsroot
      

Don't forget to put the System.map file in /nfsroot/boot. A first problem we will have to fix is that, depending on your configuration, your system may try to run fsck on the root filesystem at boot time. It shouldn't if there is no hard drive in the box. Most distributions will also skip this fsck if they find a fastboot file in the root directory. So, issue the following commands if you do not plan to mount any hard drive:

        # cd /nfsroot
        # touch fastboot
        # chmod 0 fastboot
      

      # cd /nfsroot/sbin
      # ln -s ../bin/true fsck.nfs
    

The /dev directory can also be safely copied from another place into /nfsroot. But permissions and symlinks have to be preserved, so use cp -a. Another solution is to use kernel 2.2.x devfs feature, which will reduce memory consumption and improve performance, but the drawback of this method is that all symlinks created in /dev will be lost. The point to remember is that each workstation needs to have its own /dev, so you will have to copy it on a ramdisk if you plan to use several clients and not to use devfs.

5.2. The /var and /etc directories

We will use ramdisks for these directories, because each client needs to have its own one. But we still need them at the beginning to create their standard structure. Note you are not required to do so if you use a single client. So copy these directories (cp -a) from another place into /nfsroot. Then you can make some cleanup in /var: you can remove everything in /nfsroot/var/log and /nfsroot/var/run. You also probably can remove everything in /nfsroot/var/spool/mail, if you plan to export it via NFS. You also will have to remove the files containing host specific information in /nfsroot/etc to build them on the fly during the boot process.

The startup scripts will have to be customized in order to mount some parts of the filesystem: the /dev directory, if you don't use devfs, the /tmp, the /var, and the /etc directories. Here is some code which will achieve this:

        # this part only if you don't use devfs
        mke2fs -q -i 1024 /dev/ram0 16384
        mount -n -t ext2 -o rw,suid,dev,exec, \
            async,nocheck /dev/ram0 /dev
        # this part for everyone
        mke2fs -q -i 1024 /dev/ram1 16384
        mount -n -t ext2 -o rw,suid,dev,exec, \
            async,nocheck /dev/ram1 /tmp
        chmod 1777 /tmp
        cp -a /etc /tmp
        mke2fs -q -i 1024 /dev/ram2 16384
        mount -n -t ext2 -o rw,suid,dev,exec, \
            async,nocheck /dev/ram2 /etc
        find /tmp/etc -maxdepth 1 -exec cp -a '{}' /etc ';'
        mount -f -t ext2 -o rw,suid,dev,exec, \
            async,nocheck,remount /dev/ram2 /etc
        mount -f -o remount /
        cp -a /var /tmp
        mke2fs -q -i 1024 /dev/ram3 16384
        mount -t ext2 -o rw,suid,dev,exec, \
            async,nocheck /dev/ram3 /var
        find /tmp/var -maxdepth 1 -exec cp -a '{}' /var ';'
      

If you plan to use more than a single client, you will also have to change files dynamically at boot time in /etc: the files which contain the IP and hostname of the client. These files depend on your distribution, but you will easily find them with a few greps. Just remove client-specific information from them, and add code into your startup files to generate this information again at boot time but only once the new /etc has been mounted on the ramdisk! A way to obtain your IP address and hostname at bootup is the following (if you have the bootpc package installed on the workstations'filesystem):

        IPADDR="$(bootpc | awk '/IPADDR/ \
                                {
                                  match($0,"[A-Za-z]+")
                                  s=substr($0,RSTART+RLENGTH)
                                  match(s,"[0-9.]+")
                                  print substr(s,RSTART,RLENGTH)
                                }
                               ')"

        HOST="$(bootpc | awk '/HOSTNAME/ \
                              {
                                match($0,"[A-Za-z]+")
                                s=substr($0,RSTART+RLENGTH)
                                match(s,"[A-Za-z0-9-]+")
                                print substr(s,RSTART,RLENGTH)
                              }')"

        DOMAIN="$(bootpc | awk '/DOMAIN/ \
                                {
                                  match($0,"[A-Za-z]+")
                                  s=substr($0,RSTART+RLENGTH)
                                  match(s,"[A-Za-z0-9-.]+")
                                  print substr(s,RSTART,RLENGTH)
                                }')"
      

Then, the hostname should be set with the hostname $HOSTNAME command. When this is done, it is time to generate on the fly the configuration files which contain the IP address or the hostname of the client.

5.3. Last details

Now, it's time to do the fine tuning of the client. As /var will be mounted on a ramdisk (unless you have a single client), you will have to send the logs to a logs server if you want to keep them. One way to do that is to delete the /nfsroot/etc/syslog.conf file and replacing it by the following file (see man syslog.conf for details):

        *.*     /dev/tty12
        *.*     @dns or IP of the logs server
      

If you use logrotate and you have done the preceding operation, you should replace the logrotate configuration file (/etc/logrotate.conf on most boxes) by an empty file:

        # rm -f /etc/logrotate.conf
        # touch /etc/logrotate.conf
      

In the /nfsroot/etc/fstab file, remove anything related to the hard drive, floppy disk reader, or cdrom if you don't have such devices on your workstations. Add an entry for the /var/spool/mail directory, which should be exported by the server through NFS or any other network filesystem. You probably also want to put an entry for the /home directory in this file.

You can also comment the lines running newaliases, activating swap, and running depmod -a and remove the /nfsroot/etc/mtab file. Comment out the line(s) removing /fastboot, /fsckoptions, and /forcefsck in your startup scripts. Also remove or comment any line in the startup scripts that would try to write on the root filesystem except for really necessary writes, which should all be redirected to some ramdisk location if you use several clients.

5.4. Trial...

Time has come for a small trial. MAKE A BACKUP OF YOUR NEWLY CREATED /nfsroot. tar -cvvIf should do fine. Take a minute to verify we didn't forget anything. Try to boot a client.

5.5. And Error!

Look carefully at the client's screen during the boot process. Oh, I didn't tell you to connect a screen... Run, forest! Run an get one. You will probably see some error messages. Fix the problems, and make frequent backups of your /nfsroot. One day, the client will boot properly. This day, you will have to fix errors occurring during shutdown;=P.

6.1. BOOTP or DHCP capable NICs

This is the most simple case: some network cards provide a supplement to the BIOS, containing a BOOTP or DHCP client, so just setup them for BOOTP or DHCP operation in the BIOS, and you're done.

6.2. Kernel on a local floppy or hard drive

These cases are also quite simple: the kernel is loaded from a local drive, and all the kernel has to do is to obtain its network parameters from BOOTP, and mount its root filesystem over NFS; this should not cause any problem. By the way, a local hard drive is a good place to leave a /var, /tmp, and a /dev...

If you have a local hard drive, all you have to do is to use lilo or your favourite boot loader as usual. If you use a floppy, you can use a bootloader or simply write the kernel on the floppy: a kernel is directly bootable.This enables you to use a command like the following:

      # dd if=zImage of=/dev/fd0 bs=8192
    

However, Alan Cox told in a linux-kernel thread that this feature of the linux kernel will be removed sooner or later, you thus will have to use a bootloader even on floppies some day. I know this still works with 2.4.11 kernels, but support seems to have been removed in the 2.4.13 version. See the sixth chapter of the boot-disk-HOWTO for this topic.

6.3. Bootloader without kernel on a local floppy or hard drive

Certain bootloaders are network aware, you may thus use them to download the kernel image from the network. Some of them are listed below:

• netboot, a bootloader dedicated to network boot.

• GRUB, the GNU project's GRand Unified Bootloader, which is a very general purpose bootloader.

6.4. Creating ROMs for the clients

Many network cards include a slot in which one can insert an EPROM with additional BIOS code. This enables one to add, for instance, BOOTP capabilities to the BI0S. To do so, you will first have to find how to enable the EPROM socket. You may need a jumper or a special software to do so. Some cards like the 3Com 905B have slots for EEPROMs which enable one to change the software in the EEPROM in place. In appendix, you'll find the information about EPROM and various types of memory chips.

For a list of EPROM burner manufacturers visit the Yahoo site and go to economy->company->Hardware->Peripherals->Device programmers or check out the old Diskless-HOWTO List of EPROM burner manufacturers section.

If you choose to create your own ROMS, you will have to load a BOOTP or DHCP capable software in the ROM, and then, you will be in the case of BOOTP or DHCP capable NICs described above.

You will also need to find the proper EPROM size and speed for your NIC. Some methods to do so are provided in appendix, because the NICs manufacturers often do not provide this information.

6.5. Local CDROM

This section was originally written by Hans de Goede <j.w.r.degoede@et.tudelft.nl> for the Diskless-root-NFS-HOWTO. I modified it slightly in order to reflect some differences between this document and the Diskless-root-NFS-HOWTO.

Much of the above also goes for booting from cdrom. Why would one want to boot a machine from cdrom? Booting from cdrom is interesting everywhere one wants to run a very specific application, like a kiosk, a library database program or an internet cafe, and one doesn't have a network or a server to use a root over nfs setup.

            # cdrecord -v speed=<desired writing speed> dev=<path to your writers generic scsi device> boot.iso
          

8.1. Transparently handling workstations'specific files

The previous sections discussed a simple way to handle workstations'specific files and directories like /var. Most of them are simply build on the fly and put on ramdisks, you may however prefer to deal with this problem on the NFS server. The clusternfs project provides a network filesystem server that can serve different files based on several criteria including the client's IP address or host name. The basic idea is that if the client whose IP address is 10.2.12.42 requests a file named, for instance, myfile, the server will look for a file named myfile$$IP=10.2.12.42$$ and serve this file instead of myfile if it is available.

8.2. Reducing diskless workstations'memory usage

One simple way to reduce memory consumption is to put several dynamically created directories on the same ramdisk. For instance, let's say the first ramdisk will contain the /tmp directory. Then, one may move the /var/tmp directory on that ramdisk with the following commands issued on the server:

        # mkdir /nfsroot/tmp/var
        # chmod 0 /nfsroot/tmp/var
        # ln -s /tmp/var /nfsroot/var/tmp
      

Another good way to reduce memory consumption if you don't have local hard drives and do not swap over a network block device is to disable the Swapping to block devices option during kernel compilation.

8.3. Swapping over NFS

If your stations do not have enough memory and do not have local drives, you may want to swap over NFS. You have to be warned the cod eto do so is still under development and this method is generally quite slow. The full documentation for this can be found at http://www.instmath.rwth-aachen.de/~heine/nfs-swap/.

The first thing to do if you want to apply this solution is to patch your kernel (you need a kernel version 2.2 or above). First download the patch at the above url, and cd to /usr/src/linux. I assume the patch is in /usr/src/patch. Then issue the following command:

        # cat ../patch | patch -p1 -l -s
      

Then export a directory read-write and no_root_squash from the NFS server. Setup the clients so that they will mount it somewhere (say on /mnt/swap). It should be mounted with a rsize and wsize smaller than the page size used by the kernel (ie. 4 kilobytes on Intel architectures), otherwise your machine may run out of memory due to memory fragmentation; see the nfs manual page for details about rsize and wsize. Now, to create a 20 MB swap file, issue the following commands (which should be placed in the clients'initialization scripts):

        # dd if=/dev/zero of=/mnt/swap/swapfile bs=1k count=20480
	# mkswap /mnt/swap/swapfile
	# swapon /mnt/swap/swapfile
      

Let's say a word about the drawbacks of NFS swapping: the first drawback is that it is generally slow, except you have specially fast network cards. Then, this possibility has not been very well tested yet. At last, this is not secure at all: anyone on the network is able to read the swapped data.

8.4. Swapping over network block devices

Although I have never tried it personally, I got report that the trick described below works, at least with recent kernels.

The general principle for swapping over network block devices is the same than to swap over NFS. The good point is you won't have to patch the kernel. But most of the same drawbacks also apply to the NBD method.

To create a 20 MB swap file, you will have to first create it on the server, export it to the client, and do an mkswap on the file. Note that the mkswap must be done on the server, because mkswap uses system calls which are not handled by NBD. Moreover, this command must be issued after the server starts exporting the file, because the data on the file may be destroyed when the server starts exporting it. If we assume the server's name is NBDserver, the client's name is NBDclient, and the TCP port used for the export is 1024, the commands to issue on the server are the following:

        # dd if=/dev/zero of=/swap/swapfile bs=1k count=20480
	# nbd-server NBDclient 1024 /swap/swapfile
	# mkswap /swap/swapfile
      

        # swapon /dev/nd0
      

Again, this was just to show the general principle. The files'names should also be dependant on the workstations'names or IPs.

Another solution to swap over a network block device is to create an ext2 filesystem on the NBD, then create a regular file on this filesystem, and at last, use mkswap and swapon to start swapping on this file. This second method is closer to the swap over NFS method than the first solution.

8.5. Getting rid of error messages about /etc/mtab or unmounted directories on shutdown

The following commands, issued on the server may solve the problem:

        # ln -s /proc/mounts /nfsroot/etc/mtab
        # touch /nfsroot/proc/mounts
      

8.6. Installing new packages on workstations

A simple way to do so is to use, on the server, a chroot and then execute your favourite installation commands normally. To chroot to the appropriate place, use the following command:

        # chroot /nfsroot
      

Debian users will be particularly interested in the --root option of dpkg, which simply tells dpkg where the root of the target system is.