One of the joys of working on a server from a remote location is dealing with the issue caused by a broken /etc/fstab file. Even the best admins make mistakes and mistakes in that file can lead to a server that won't boot.

Which is fine; if you have an IP-based KVM where you can get console access without actually being at the facility. But not so great when a screwed up fstab file requires you to go physically visit the location.

So how do we verify that our fstab file makes sense prior to a reboot? The answer lies in the mount command. There are (3) useful options that can be passed to the mount command which will help us check the fstab file prior to a reboot.

-f - FAKE IT (Causes everything to be done except for the actual system call). This tells mount to do everything, but don't actually change anything.

-a - Mount all filesystems mentioned in fstab.

-v - Be verbose about it.

Here's an example where everything is fine and dandy.

# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
mount: /dev/md4 already mounted on /var/log
nothing was mounted

Same example, except that I screwed up the name of /dev/md4 in the fstab file:

# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
/dev/md4x on /var/log type ext3 (rw,noatime)
nothing was mounted

Now, there's probably a better way to do this, but this serves as at least a moderate check against shooting yourself in the foot.

Labels: 2009, CentOS5, Linux

There are currently (3) basic methods for getting a remote control GUI on a Linux server (like we do with Remote Desktop for Windows servers):

1) X-Windows over TCP/IP

All GUI operations on Unix/Linux are handled by the X-Windows sub-system. Window interfaces like KDE, Gnome, and others are merely layered on top of the X sub-system. One of the useful things about X is that any window can be forwarded over TCP/IP to any other X server. So you could run an application on the linux server, but display the output window on your PC (as long as you run a local X server program).

The downside of all this is that accessing remote servers requires the use of SSH port forwarding, and a bit of arcane magic. It's nowhere near as clean of a solution as RDP (Terminal Services). But it can be ultra-secure (by using SSH keys) and it works fairly well across the WAN.

2) VNC

VNC is a screen-scraper solution for GUI desktops, very similar to the old pcAnywhere and e/pop solutions that we used to use.

The downsides of VNC are:
- security is non-existent in the base spec
- different VNC server use different encryptions
- authentication tends to be done via plain text passwords
- rather slow across the WAN

3) NX/FreeNX

A company called NoMachines came out with a different solution called "NX". NX is a protocol that is very similar to RDP and the client works rather similar to Remote Desktop. You used to have to pay for the product, but over the years, they've opened up the source code. So now there are (3) different server implementations (NX, FreeNX, and another) and you can download the NX client from NoMachines for free.

The big advantage here is that security is better and performance is better over slow WAN links.

Labels: 2008, FreeNX, Linux, NX, VNC, X11

With age comes wisdom? After working with SoftwareRAID and Linux servers for a while, I've changed my preferred disk system design and layout.

RAID

Under the old system, I was running a (2) disk RAID1 (mirror) with a hot-spare disk setup and ready for action. But if you're going to have a hot-spare dedicated to the RAID1 array, why not use it as an active array member? That way, if a disk fails, you still have two good disks. Unfortunately, when a RAID element fails, the load from the rebuild process can often kill the one of the remaining disks in the array.

Is it a likely scenario? Probably not. But Linux's Software RAID handles a triple-active RAID1 mirror without any slowdown, so there's not much reason *not* to implement it that way. Plus it's a useful trick to know for situations where you really *do* need to be that paranoid.

(I'm not sure whether any hardware RAID cards provide for a triple-active mirroring RAID1 configuration.)

Partitions

I've also simplified how many partitions I like to have on the disk. My current disk layouts typically look like:

/dev/sdX1 - /dev/md0 - 250MB - /boot
/dev/sdX2 - /dev/md1 - 12GB - / (primary root)
/dev/sdX3 - /dev/md2 - 12GB - / (backup root)
/dev/sdX5 - /dev/md3 - 32GB - /var/log
/dev/sdX6 - /dev/md4 - 2GB - swap
/dev/sdX7 - /dev/md5 - 64GB - /backup/system
/dev/sdX8 - /dev/md6 - (remainder) - LVM area

During normal operations, we boot and run /dev/md1 as our / (root) partition. The /dev/md2 partition is kept offline and is never mounted. Periodically, after validating that the server is in good health, we will copy the contents of /dev/md1 to /dev/md2, make adjustments to /etc/fstab and the hostname. This requires some server downtime (long enough to setup the 2nd root partition).

In the case where the primary OS is hosed, we can boot from the backup OS partition and get back up and running quickly. That gives us the luxury to continue operations until we can schedule downtime to fix the primary OS partition.

Notice that I've broken /var/log out to its own partition. I do this so that an overflowing set of logs won't take the server box down. Plus, by putting the log files in their own physical partition, it's easy to use a boot CD or USB key to gain access to the logs in case of severe issues.

The other physical partition that I consider necessary is /backup/system. This partition is used to hold images of the boot and root partitions, along with information about the partition layout and images of the MBRs. Basically, it's used to store disaster recovery backups. You should not have this partition mounted during normal operations. Taking the contents of this partition offsite is also a good idea. A basic text file of how the backups were created along with information for how to restore these backups is recommended.

Summary

This setup tries to walk the fine line between keeping it simple, but having enough flexibility to deal with a large set of potential failures. Anything from a two-disk failure, to the primary OS being hosed, to both OS partitions having problems all the way up to boot records or the /boot partition being killed.

Labels: 2007, DisasterRecovery, Linux, SoftwareRAID

A few more tools that I've found useful for console-only Linux server administration. All of these have Gentoo ebuilds so they can simply be emerge'd into your system.

atop - Similar to the built-in "top" command in Linux/Unix, except that it gives additional details about what is going on with the system. Of particular use is that it will show you individual disk utilization statistics so that you can locate spindles that are in heavy use. (Which is much needed when tuning a database server.)

tree - Allows you to output a hierachial display of all files and directories on your system in a "tree" view. Which is useful for documenting the directory layout of your system.

Labels: 2005, Linux

The whole concept behind bare-metal restore is that it's the shortest point from pulling a replacement set of server hardware out of a box to having a working system to replace the one that's down. Ideally, without having to go through re-installing the operating system (which, in the case of Gentoo, can take a few hours). It also (generally) only works when you have exactly the same hardware in the replacement box as the box that's kaput. So if you're using any exotic hardware (e.g. a fancy RAID card), you'd best buy (3). That way when the first one dies, you can put in one of the (2) spare cards and still get at your data.

Some backup software makes the process as simple as dropping a boot CD in the drive and inserting the most recent bare-metal backup (sometimes called a "gold" tape?) tape. It can also be a multi-step process, where the first step restores the operating system to a known state and the second step handles restoring the data and applications.

Since I'm not ready to tackle writing a guide, I'll settle for a few links:

Linux Complete Backup and Recovery HOWTO (by Charles Curley) - A very good starting point.

Linux Journal: Bare Metal Recovery, Revisited (by Charles Curley, Aug 2002) - A good sidebar discussion of the HOWTO document that he wrote.

About.com: Linux backup - A collection of links to backup software.

Unix SysAdm Resources: Backup & Archival Software for Unix (Stokely Consulting) - Listing of backup software for unix.

Linux Backups Mini-FAQ (Karsten M. Self) - Good article on simply using "tar".

Labels: 2004, DisasterRecovery, Linux