Network Block Devices: Using Hardware Over a Network

Most Linux administrators today are familiar with protocols such as the Network File System (NFS) and the Server Message Block/Common Internet File System (SMB/CIFS, the protocol used by Samba). These protocols enable you to share filesystems across a network — the Linux computer sitting on your desk can access files stored on another Linux system, a Windows computer, or whatnot across the office (or on the other side of the planet!).

While NFS, SMB/CIFS, and similar protocols are handy, they aren’t always ideally suited to what you need to do. One obscure Linux tool can sometimes help on this score: Network block devices (NBDs). This tool is a way to enable one computer to provide another computer with direct low-level access to block device hardware. (Block devices exchange data in fixed-size multi-byte blocks, whereas character devices communicate in byte-size chunks. Disks, including hard disks, floppy disks, and CD-ROM/DVD drives, are the most common block devices.)

Why Use NBDs?

Why would you want to use an NBD rather than a more common file-sharing protocol? Several scenarios come to mind:

• If client can provide better tools for low-level maintenance of disks (newer versions of fsck, for instance) than the server, providing NBD access may make sense.
• The client may have need of expanded network disk space, but a traditional network filesystem may not be adequate. For instance, you might be testing new filesystem drivers or need features of a particular filesystem and those features might not be properly handled by NFS or SMB/CIFS.
• The server might not support the filesystem or data structures on the device you wish to export. Note that NBD server software is available for Windows and other non-Linux OSes, so in principle a Windows system could host a ReiserFS or XFS partition for a Linux client, or you could export CD-ROMs burned with Rock Ridge extensions from a Windows server.
• Under some circumstances, using NBDs can provide superior performance compared to using traditional network filesystems. This can be true if diskless clients need to boot simultaneously, for instance.

You may be able to find workarounds or superior alternatives to NBDs in many of these cases. For instance, you could upgrade your server’s low-level disk utilities or expand the disk space on the client. Sometimes, though, NBDs may make more sense. You’ll need to be the judge of which is true in your case.

You should be aware that NBDs have their limitations, too. The most important of these is that providing read/write access to NBDs to more than one client is extremely dangerous. Ordinarily, you’ll either limit the NBD access to a single client or use NBDs to provide read-only access to data (as is possible for a diskless workstation’s root filesystem).

NBDs can also encounter deadlock conditions, particularly if you try to access the NBD from the server. If you need to update the NBD’s partition from the server, you should do so directly — and when no client is using it, particularly if read/write client access is desirable.

Obtaining NBD Software

NBD software for Linux includes three components:

• NBD user-space server software
• NBD user-space client software
• NBD kernel-space client software

The user-space software can be obtained from http://sourceforge.net/projects/nbd. The nbdpackage includes both client and server software. Some distributions provide NBD packages; for instance, Ubuntu calls them nbd-client and nbd-server. It’s usually easiest to install a ready-made package, but if you can’t find one for your distribution, the installation process is conventional:

1. Download the source tarball, unpack it in a convenient location, and cd into the resulting directory.
2. Type ./configure
3. Type make
4. As root, type make install

The result will be the installation of the nbd-server and nbd-client programs, along with associated man pages. As of version 2.99, the NBD source package includes a subdirectory called winnbd, which includes a Windows NBD server. I didn’t test it for this column, though.

Your NBD client requires kernel support to use NBDs, but the kernel support is not needed on the server. To add this support, you must compile it into your kernel (or as a module). The module is called nbd, so you can look for it on your system before you recompile your kernel. (On my systems, it’s stored in /lib/modules/version/kernel/drivers/block, where versionis the kernel version number.)

Completely describing the process of recompiling your kernel is beyond the scope of this column. Assuming you know how to do it, you can find the NBD option under Device Drivers -> Block Devices, as Network block device support. Activate this option (either compiling it directly into the kernel or building it as a module) and recompile your kernel. (In some cases, recompiling just your kernel modules will do.) You’ll also need to install your kernel modules, and probably the kernel. If you changed the kernel proper, you may need to modify your LILO or GRUB configuration and reboot.

Preparing an NBD Server

The first step in preparing an NBD server is to set aside a block device for use by the server. Typically, this will be a hard disk partition, although it could be a logical volume manager (LVM) volume, redundant array of independent disks (RAID) device, CD-ROM/DVD-ROM drive, or some other block device. You may also create a regular file and export it as a block device. This approach is similar to using a loopback device locally, but you give a remote system access to the file as if it were a block device. This can be handy if you want to provide clients with access to a variety of image files (CD-ROM images, say).

Depending on how you want to use the NBD, you might want or need to create a filesystem on the device and populate it with files beforehand. This would be necessary, for instance, if you intend to use the NBD as a read-only root filesystem for diskless network clients. In other cases, you might leave the preparation of the storage space to the client. This would be necessary if the server doesn’t support the filesystem or other low-level format the client uses, as for instance when using a Windows NBD server.

Whether or not you prepare the NBD storage space using the server computer, you should be sure to never access the NBD space using the server when the clients do so! In the case of read/write client access, having two clients write to a filesystem simultaneously is almost certain to produce data corruption. In the case of read-only access, unexpected writes might confuse the client. You’ll need to be careful about how you prepare and use your disk space on the server side to avoid such problems.

With the disk space prepared, you can run the NBD server:

# nbd-server 2000 /dev/sdb1

This command exports /dev/sdb1 using port 2000. Note that the device filename must use an absolute path, not a relative path. The version of NBD I tested issues a warning message about being unable to open its config file. This message seems to be harmless, so you can ignore it. In principle, you can run nbd-server as a non-root user; however, the user must have read (and perhaps write) access to whatever file or device you export. Note that both the port number and block device are required parameters to nbd-server. You can read the nbd-server man page to learn about its options; some highlights include:

• You can precede the port number with an IP address, as in 192.168.17.2:2000 to listen to port 2000 on IP address 192.168.17.2. This is most useful if your server has multiple network cards and you only want to export the device using one interface.
• -r exports the device in read-only mode, which is a highly recommended precaution if clients should not be able to write to the device.
• -c creates a copy-on-write export, meaning that client write operations are performed on a temporary file, rather than on the original file. The temporary file is discarded when the client disconnects.
• -a timeout specifies a timeout period (in seconds), after which the NBD server terminates the connection.
• -l host_list_file specifies a file that includes the IP addresses of hosts that may connect to the NBD server. The default value is nbd_server.allow. If the file is missing, any host may connect. This tool is obviously useful for controlling access to your NBD server.
• -C config_file tells nbd-server where to find its configuration file (described shortly).

The -c (copy-on-write) option deserves a few more comments. Because most modern Linux distributions use udev to create a dynamic /dev directory tree on a virtual filesystem,-c can produce write errors after a while. To avoid this problem, try creating symbolic links on a conventional filesystem to point to the real device file and then export the symbolic links rather than the device files to which they point. The -c option also has the drawback of reducing performance.

On the plus side, it’s a way around the danger associated with providing write access to multiple clients; each client gets a unique diff file on the server, so each client can safely write to the “shared” file without damaging other clients’ files.

As an example of some of nbd-server‘s options in action, consider the following extended command:

# nbd-server 2000 /dev/sdb1 -r -l /etc/nbd.allow

This command provides read-only access to /dev/sdb1 to those clients whose IP addresses appear in /etc/nbd.allow.

Preparing a Server Configuration File

Instead of specifying options on the command line, you can create an NBD server configuration file. This file consists of named sections, each section name enclosed in square brackets ([]). Within each section, lines contain parameter/value pairs, with parameters and values separated by equal signs (=). The [generic] section is required and sets global options. Each subsequent named section sets options for particular devices. Listing One presents an example, which demonstrates two NBD exports, one for/dev/sdb1 and one for /dev/fd0.

Listing One: Sample NBD Configuration File

[generic]
  listenaddr = 0.0.0.0
  authfile = /etc/nbd-server/allow

[diskfile]
  exportname = /dev/sdb1
  copyonwrite = true
  port = 2000

[floppy]
  exportname = /dev/fd0
  port = 2001

The easiest way to create a configuration file is to specify the options you want and use the -o section_name option to nbd-server, where section_name is the name you want to give to the section. You can then cut-and-paste the output into your configuration file. Be sure each section specifies a unique port number! You may also want to peruse theREADME file that comes with the NBD source package; it provides examples of a few additional options that the configuration file supports.

With configuration file created, you can pass it to nbd-server via its -C option. The server will then share all the listed block devices, each on its specified port and with its specified options, with just one call to nbd-server.

Preparing an NBD Client

With an NBD-enabled kernel running, you can use the nbd-client program on the client computer to map a remote NBD to a local device file:

# nbd-client nbdserver 2000 /dev/nbd0

In this example, nbdserver is the hostname or IP address of the server computer, 2000is the port number associated with the device you want to use, and /dev/nbd0 is the local device filename you want to link to the remote server. A few caveats and tricks require attention:

• You may need to explicitly load the nbd module before running nbd-client.
• Once the nbd module is loaded, some systems (including Fedora and Ubuntu systems I’ve checked) automatically create /dev/nbd# devices, where # is a number from 0 up. Upon creation, these device files are not assigned to any server, but you should specify one of these devices in your nbd-client command line.
• The nbd-client program fails silently if it can’t find an NBD server on the specified computer and port. If it does find an NBD server, it displays a couple of lines of text summarizing the size of the device it finds. If something goes wrong during this process, it may hang with the word Negotiation on the screen. This looks like a prompt, but it isn’t. If you see this, chances are there’s something wrong with your NBD server configuration.
• The nbd-client program must be run as root.

Although the basic nbd-client command specified earlier works fine in many cases, the program supports some additional options, including:

• -swap tells the system that the device should be used as swap space. This helps prevent deadlocks.
• timeout=seconds specifies the timeout period for NBD operations.

Using NBDs

With your NBD client pointing to a network-accessible NBD server, you can begin using NBD. You can treat your /dev/nbd# device just as you would any local block device. Typically, you’ll mount it as a filesystem (perhaps first creating a filesystem on it) withmount:

# mount /dev/nbd0 /mnt/nbd

This command mounts the device at /mnt/nbd. If the device was exported for read/write operations, you’ll be able to write to it, with the caveat that two clients should not connect to the same read/write device simultaneously unless you use the -c option on the server!

Of course, you can also perform other operations on an NBD, such as use mkfs to create a filesystem, check a filesystem with fsck, and so on. Many of these actions require that you have read/write access to the device, though.

When you’re done using an NBD that you’ve mounted, you can unmount it using umount, just as you would a local filesystem. You can then type nbd-client -d /dev/nbd0(substituting the correct device filename) to terminate the client/server NBD network link. Although this last step may not always be strictly necessary, it can help you avoid confusion over what devices are active, and it causes the server to delete diff files it created if you used the -c option on the server.

One of the problems with NBDs is that, if a network connection goes down or a server crashes, clients will have a hard time recovering. Data may be lost, much as in a local disk crash. To guard against such problems, I recommend using NBDs for brief periods, if possible; if you don’t need access to the NBD now, unmount it. Try to avoid using NBDs over anything but local networks. NBD traffic is unencrypted, so passing it over wireless networks or the Internet can be risky from a security point of view.

Overall, NBDs can fill some specialized needs in a Linux network. If NFS or SMB/CIFS just doesn’t seem to fill your needs, give NBDs a try.