Setting up our test iSCSI SAN box this week. The original plans were to run this on top of Gentoo (which is very powerful and flexible) but after 3 years, I'm not very pleased with Gentoo as a server OS. Which is a whole different topic. So we've migrated over to using CentOS5, which is derived from Red Hat Enterprise Linux 5, a distro that is more suited for corporate use.

There's not much to talk about in terms of the base system. It's a pretty vanilla 64bit CentOS5 install (from DVD) running on top of a dual-CPU dual-core pair of Socket F Opterons. The primary packages that I've installed so far are "Yum Extender" (from stock repositories) and "rdiff-backup" (downloaded as an RPM). The OS runs on top of a 3-disk RAID1 (mirror, all drives active) Software RAID for safety.

I use a semi-customized partition layout on the (3) operating system disks. I have:

a) /boot
b) / (root, the primary OS install area)
c) swap
d) a backup root partition (which is basically a clone of the primary, except for a small change in /etc/fstab) designed for quick recovery from a situation that would hose the primary root partition
e) /var/log (broken out to its own area)
f) /backup/system (a place to store system backups)
g) LVM area (no allocated areas yet)

I mention all that because the first step before installing iscsitarget is to make sure I can recover if things go awry. Since installing iscsitarget involves mucking with the running kernel, I want a good backup of /boot along with making sure GRUB offers me options to boot an older kernel. I'll also freshen my root backup partition.

Step 1 - Backing up /, /boot, and the existing kernel

Simplicity is often best when dealing with the base OS. My methods are crude, but designed to get me back up and running without needing much in the way of software. The primary requirement is a bootable USB pen drive or bootable LiveCD (such as RIPLinuX) with the necessary tools. You could also use the CentOS5 boot DVD.

I'll run with the CentOS install DVD since that's what I have sitting in the optical drive at the moment. When CentOS boots up, enter "linux rescue" at the boot prompt. Note, if you have multiple NICs installed, it's probably better to not start networking (because the CentOS rescue mode takes forever to initialize unconnected NICs).

Select "Skip" when asked about mounting the existing install at /mnt/sysimage. We'll be doing things our own way instead.

Start up Software RAID on the key partitions (/boot, /, the backup /, and the backup partition). The following commands will (usually) startup your existing RAID devices automatically.

# mdadm --examine --scan >> /etc/mdadm.conf
# mdadm --assemble --scan

In my case "md0" is /boot, "md2" is my base CentOS install, "md3" is the backup root partition, and "md5" is where I can store image files. So let's double-check that.

# mkdir /mnt/root ; mount /dev/md3 /mnt/root
# mkdir /mnt/backuproot ; mount /dev/md3 /mnt/backuproot

If we then examine the output of "df -h" or by using "ls" on the mounted volumes we can verify that we know which is which. Let's mount our backup area and create image files. I prefer to kick off the "dd" commands in the background so that I can monitor progress and keep multiple CPUs busy.

# mkdir /mnt/backup ; mount /dev/md5 /mnt/backup
# cd /mnt/backup ; mkdir images ; cd images
# dd if=/dev/md0 | gzip > dd-md0-boot-20070525.img.gz &
# dd if=/dev/md2 | gzip > dd-md2-root-20070525.img.gz &
# dd if=/dev/md3 | gzip > dd-md3-bkproot-20070525.img.gz &

We should also backup the master boot records on each of the hard drives in the unit.

# for i in a b c; do dd if=/dev/sd$i count=1 bs=512 of=dd-sd$i-mbr-20070525.img; done

Unfortunately, the CentOS5 DVD doesn't include tools like "G4L" (Ghost for Linux) or I'd make a second set of backup files using that. I may boot my RIPLinuX CD and see what tools are there. (Because you can never have too many backups.)

Now I can dump the contents of "md2" (the original root) to "md3" (our backup root).

# dd if=/dev/md2 of=/dev/md3

Now for some cleanup stuff...

# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/fstab

We'll need to change any references of "md2" to "md3". Basically flip them around so that "md3" is the official root when /etc/fstab gets processed. I also like to change the prompt and system name to remind myself that I'm using the emergency system. Again, our primary goal is to be able to get a box back up and operational in the case where the primary root partition is hosed. Get it up quickly, then schedule some downtime to deal with it properly.

Now would also be a good time to tune the ext3 file system on your partitions.

The last thing we need to do is edit GRUB's configuration so that we can select our backup root OS from the selection menu.

# mkdir /mnt/boot
# mount /dev/md0 /mnt/boot
# vi /mnt/boot/grub/grub.conf

Things that we'll want to do here (you could also accomplish this by booting the server in normal mode and editing grub.conf there using a more comfortable text editor):

a) Change the timeout=5 value to timeout=15 (or 30 or 60). By default, CentOS doesn't give you very long to pick an alternate boot. I find 5 seconds to be too short of a window, especially on a unit where the storage controller takes a minute or two to scan and setup the drives.

b) Copy the latest "title" section and change "root=/dev/md2" to "root=/dev/md3". I always make the "EMERGENCY" boot option the 2nd one in the list.

# mkdir /mnt/backuproot
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/sysconfig/network

I like to change the hostname to have "-emergency" tacked onto the end. Which should make it fairly obvious that we are booting up in emergency mode using the backup root partition. I also edit root's .bash_profile to set PS1.

Okay, that was a lot of setup work just to prepare for implementing iSCSITarget (or any other kernel rebuild), but it's always worth it.

Final notes:

- When I test booted the emergency root partition, things didn't work as planned. So while my concept is sound, I may have screwed something up. I think it's an error with /etc/fstab in the emergency partition, so I'll troubleshoot that later.

- It's also possible that you'll need to do a GRUB install on all (3) of the primary mirror disks.

Step 2 - Downloading and compiling the iSCSITarget software

So far, I've found (2) links to be useful here. One is Moving on.... and the other is iSCSI Enterprise Target を CentOS5 にインストールする (japanese). While the 2nd link is in Japanese, it shows the commands in english.

Head over to the The iSCSI Enterprise Target page and download the latest tarball containing the source code. The current version is 0.4.15. If you're using Firefox in CentOS's Gnome shell, it will probably prompt you to open the file with the archive manager. I created a subfolder under /root/iscsitarget-0.4.15 and extracted the contents there.

You will also need to go into Applications -> Add/Remove Software and add the development tools and libraries to your system. (Mostly you just need gcc.)

As noted on jackshck's page, you will also need to install the following packages:

openssl-devel (I installed the x86_64 version)
kernel-devel (again, I'm using the x86_64 version)

Open up a terminal window and go to where you extracted the iscsitarget tarball (I put mine in /root/iscsitarget-0.4.15).

# ls -l /usr/src/kernels
(make note of the kernel folder)
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/ install

Now we can start up the ietd daemon:

# /etc/init.d/iscsi-target start

And add it to our default runlevel (this is similar to the rc-update command in Gentoo Linux):

# chkconfig iscsi-target on

Step 3 - Creating a target

This is where we get into the nitty-gritty and where I need to take a break and do some research. The /etc/ietd.conf file already exists at this point, but only contains a commented out sample configuration.

Notes:

Dec 21 2007 - The comment about iSCSITarget software for Microsoft Windows really isn't on-topic. But I'll go ahead and list the link to it, but not as an HTML link. Pricing for the real version is currently $395 (Server) or $995 (Professional). And personally, there's no way that I'd recommend running a SAN on top of Microsoft Windows (even Server 2003, which is a nice product).

Labels: Backups, CentOS5, iSCSITarget, SoftwareRAID

Today's trick is moving a CentOS5 system from an old set of disks over to a new set of disks. Along the way, I'll create an image of the system to allow me to restore it later on.

The CentOS5 system is a fresh install running RAID-1 across (3) disks using Linux Software RAID (mdadm). There are (4) primary partitions (boot, root, swap, LVM) with no data on the LVM partition.

(Why 3 active disks? The normal setup for this server was RAID-1 across 2 disks with a hot-spare. Rather then have a risky window of time where one disk has failed and the hot-spare is synchronizing with the remaining good disk, I prefer to have all 3 disks running. That way, when a disk dies, we still have 2 disks in action. The mdadm / Software RAID doesn't seem to care and it doesn't seem to affect performance at all.)

Because this is RAID-1, capturing the disk layout and migrating over to the new disks will be very easy. It's also a very fresh install, so I'm just going to grab the disk contents using "dd" (most of the partition's sectors are still zeroed out from the original install). Once I've backed up the (3) partitions on the first drive, I'm going to pull the (3) drives and replace them with the new ones.

I'll get the machine up and running with the first replacement drive, then configure the blank 2nd and 3rd drives and add them to the RAID set. That is, if mdadm doesn't beat me to the punch and start the sync on the 2nd/3rd disks automatically.

If things go bad, I can always drop the original disks back in the unit and power it back up. I plan on keeping them around for a few days, just in case. I'll have to recreate the LVM volumes, but there aren't any yet (just a PV and a VG).

One advantage of pulling the old drives out completely and rebuilding using fresh drives - I'll end up with a tested disaster recovery process.

Now for the nitty gritty. I'm using a USB pocket drive formatted with ext3 for the rescue work. Make sure that you plug this in before booting the rescue CD.

1. Login to the system and power it down.
   
2. Boot the CentOS5 install DVD
   
3. At the "boot:" enter "linux rescue"
   
4. Work your way through the startup dialogs
   
5. When prompted whether to mount your linux install, choose "Skip"
   

This should give you a command shell with useful tools. So let's poke around and check on our system.

1. Looking at "cat /proc/mdstat" shows that while the mdadm software is running, it has not assembled any RAID arrays.
   
2. The "fdisk -l" command shows us that the (3) existing disks are named sda, sdb, sdc. Each has (4) partitions (boot, root, swap, LVM).
   
3. My USB drive showed up as "/dev/sdd" so I'll create a "/backup" folder and mount it using "mkdir /backup ; mount /dev/sdd1 /backup ; df -h"
   

Naturally, we should create a sub-folder under /backup for each machine and possibly create another folder underneath it using today's date. We should grab information about the current disk layout and store it in a text file (fdisk.txt).

1. # cd /backup ; mkdir machinename ; cd machinename
   
2. # mkdir todaysdate ; cd todaysdate
   
3. # fdisk -l > fdisk.txt
   

Now to grab the boot loader and image the two critical partitions (boot and root). We'll grab the boot loader off of all (3) drives because it's so small (and it may not be properly synchronized).

1. dd if=/dev/sda bs=512 count=1 of=machinename-date-sda.mbr
   
2. dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdb.mbr
   
3. dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdc.mbr
   
4. dd if=/dev/sda1 | gzip > machinename-date-ddcopy-sda1.img.gz
   
5. dd if=/dev/sda2 | gzip > machinename-date-ddcopy-sda2.img.gz
   

Total disk space for my system was around 1.75GB worth of compressed files (8GB root, 250MB boot). You could also use bzip2 if you need more compression. Unfortunately, the CentOS5 DVD does not include the "split" command, which could cause issues if you're trying to write to a filesystem that can't handle files over 2GB in size.

Now you should shut the box back down, burn those files to DVD-R, install the new (blank) disks, and boot from the install DVD again. Again, mount the drive that holds the rescue image files to a suitable path.

1. dd of=/dev/sda bs=512 count=1 if=machinename-date-sda.mbr
   
2. fdisk /dev/sda (fix the last partition)
   
3. dd if=/dev/sda bs=512 count=1 of=/dev/sdb
   
4. dd if=/dev/sda bs=512 count=1 of=/dev/sdc
   

That will restore the MBR and partition table from the old drive to the new one. If your new drive has a different size, then the last partition will be incorrectly sized for the disk. Fire up "fdisk" and delete / recreate the last partition on the disk.

Restore the two partition images:

1. # gzip -dc machinename-date-ddcopy-sda1.img.gz | dd of=/dev/sda1
   
2. # gzip -dc machinename-date-ddcopy-sda2.img.gz | dd of=/dev/sda2
   

At this point, we should be able to boot the system on the primary drive and have Software RAID come up in degraded mode for the arrays. Things that will need to be done once the unit boots:

1. Tell mdadm about the 2nd (and 3rd) disks and tell it to bring those partitions into the arrays and synchronize them.
   
2. Create a new swap area
   
3. Recreate the LVM physical volume (PV) and volume group (VG)
   
4. Restore any data from the LVM area (we had none in this example)
   

Getting the Software RAID back up and happy is the trickiest of the steps.

1. Login as root, open up a terminal window
   
2. # cat /proc/mdstat
   
3. # fdisk -l
   
4. Notice that the swap area on our system is sda3, sdb3, sdc3 and will need to be loaded as /dev/md1.
   
5. # mdadm --create /dev/md1 -v --level=raid1 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc3
   
6. # mkswap /dev/md1 ; swapon /dev/md1
   
7. Now we're ready to recreate the LVM area
   
8. # mknod /dev/md3 b 9 3
   
9. # mdadm --create /dev/md3 -v --level=raid1 --raid-devices=3 /dev/sda4 /dev/sdb4 /dev/sdc4
   
10. # pvcreate /dev/md3 ; vgcreate vg /dev/md3
    
11. Finally, we should add the 2nd and 3rd drive to md0 and md2.
    
12. mdadm --add /dev/md0 /dev/sdc1
    
13. mdadm --add /dev/md0 /dev/sdb1
    

Note: If your triple mirror RAID array puts the additional disks in as spares, make sure that you have (a) grown the number of raid devices to 3 for the RAID1 set and (b) make sure that there are no other arrays synchronizing as the same time. It's also best to add the elements one at a time, rather then adding both at the same time. I'm not sure if it's a bug in mdadm or just the way it works, but it took me two tries to get my triple mirror back up with all disks marked as "active" instead of (2) active and (1) hot-spare.

Labels: Backups, CentOS5, DisasterRecovery, SoftwareRAID

This was a slightly odd one that did not show up on Google at all. We had a bunch of backup jobs on our main Windows 2003 file server and had recently promoted the Win2003 server to a domain controller using DCPROMO.EXE. In the process of doing that, some tasks refused to run and we also had to delete and re-add a lot of tasks.

In the Scheduled Tasks window, we saw a result code of "0x57" in the Last Result column. In the schedule log (Scheduled Tasks, Advanced, View Log):

"Backup-FileServer-DailyAppend-Week2.job" (ntbackup.exe)
Finished 7/28/2006 8:30:48 PM
Result: The task completed with an exit code of (57).

We checked a few things and finally took a very close look at the "Run" field in the task. Turns out that we were missing a double-quote in the middle of the NTBACKUP command line.

Labels: Backups, Win2003

This is how I backup my PST files whenever I login to my laptop. Because I always have MSOutlook open, I find it difficult to get a good backup of the PST files.

1) You will want a copy of Info-Zip's ZIP and UNZIP executables.

2) Create a folder on your hard drive to hold these executables. I recommend creating a folder called "C:\bin" and adding that folder to your system's PATH statement.

Right-click on My Computer, Properties, Advanced, Environment Variables... Under System Variables, highlight "PATH" and click "Edit"... place C:\BIN at the start. It should look similar to "C:\bin;%SystemRoot%\system32;%SystemRoot%...". Make sure you leave the existing directories in the list and just add the "C:\bin;" at the start of the listing.

Place the zip.exe and unzip.exe files in the C:\BIN folder.

3) Locate your PST files. Create the following as a text file in the same directory. Name it as "backupemail.cmd".

@echo off
rem Backup my Outlook PST files

IF NOT EXIST "X:\EMailBackup\*" GOTO quit
IF NOT EXIST "C:\Data\EMail\*.pst" GOTO quit

C:
CD "\Data\EMail"
ZIP -u1 X:\EMailBackup\MyPSTs.zip *.pst

X:
CD \EMailBackup

IF EXIST MyPSTs-3.zip DEL MyPSTs-3.zip
IF EXIST MyPSTs-2.zip REN MyPSTs-2.zip MyPSTs-3.zip
IF EXIST MyPSTs-1.zip REN MyPSTs-1.zip MyPSTs-2.zip
IF EXIST MyPSTs.zip REN MyPSTs.zip MyPSTs-1.zip

:quit

Notes:

a) My PST files are stored in C:\Data\EMail. Anywhere that you see "C:\Data\EMail" you should replace with the folder name where your PST files are stored.

b) I backup my PST files to X:\EMailBackup. You will need to replace this path with the location where you keep your ZIP file backups.

c) This script keeps the past 3 backups by renaming them out of the way.

d) You must have at least one file in the X:\EMailBackup folder in order for the backup to run the first time. After that, the script will work properly.

e) If you find that ZIP is too slow, you may wish to change the "-u9" to "-u1" in order to get faster compression (but less compression).

f) I'm pretty sure the above script is foolproof and correct, but as always you should have a good backup before you attempt things like this.

4) Create a shortcut link to "backupemail.cmd" and place it in your Programs -> Startup folder. That way it will run as soon as you login.

Labels: Backups