Friday, November 27, 2009
FSVS: Install on CentOS 5.4
(Also see my older post on this: FSVS - Install on CentOS 5. Or the original post where I explained the power of FSVS for sysadmins.)
I'm going to start with the assumption that this is a base CentOS 5.4 install without *any* package groups selected during the initial install. In my case, this is a DomU that I'm setting up under Xen to serve as testing server for web development. The only thing I've done so far is setting the root password and configuring it to use a static IP address.
The basic steps will be:
Setting up RPMForge
In order to get the latest Subversion packages for your system, you'll have to add RPMForge as a source repository. The CentOS base repository only has Subversion 1.4.2 and the latest is currently 1.6.6. I recommend doing this in conjunction with the yum-priorities package.
After installing the yum-priorities package, you should edit the CentOS-Base.repo file found under /etc/yum.repos.d/. For the base repositories, I recommend setting them to priority values of 1 through 3. For example:
I generally give the [base], [updates], [addons], [extras] repositories a priority of "1", with [centosplus] and [contrib] getting a priority of "3".
In addition, you'll need to add or edit the "exclude=" line in the [base] repository section to exclude Subversion from being sourced from that repository. This will allow the Yum package manager to look in other repositories to find Subversion.
Now we can install the RPMForge repository (see Using RPMForge).
Now you should edit the rpmforge.repo file and insert a priority= line. I recommend a value of 10 or 25.
You can now verify that you'll pull in the latest Subversion package by the following command:
Install the packages needed for FSVS
Download and compile FSVS
As always, you shouldn't compile code as the root user.
Creating the repository on the SVN server
This is how we setup users on our SVN server. Machine accounts are prefixed as "sys-" in front of the machine name. The SVN repository name matches the name of the machine. In general, only the machine account should have write access to the repository, although you may wish to add other users to the group so that they can gain read-only access.
Back on the source machine (our test machine), we'll need to create an SSH key that can be used on our SVN server. You may wish to use a slightly larger RSA key (3200 bits or 4096 bits) if you're working on an extra sensitive server. But a key size of 2048 bits should be secure for another decade for this purpose.
Copy this key into the clipboard or send it to the SVN server or the SVN server administrator. Then we'll need to create a ~/.ssh/config file to tell the user what account name, port and key file to use when talking to the SVN server.
Back on the SVN server, you'll need to finish configuration of the user that will add files to the SVN repository.
Now you'll want to prepend the following in front of the key line in the authorized_keys file.
That ensures (mostly) that the key can only be used to run the svnserve command and that it can't be used to access a command shell on the SVN server.
Test the configuration back on the original server by issuing the "svn info" command. Alternately, you can try to ssh to the SVN repository server. Errors will usually either be logged in /var/log/secure on the source server or in the same log file on the SVN repository server.
Here's an example of a successful connection:
This shows that they key is running the "svnserve" command automatically.
Connect the system to the SVN repository
The very first command that you'll need to issue for FSVS is the "urls" (or "initialize") command. This tells FSVS what repository will be used to store the files.
You may see the following error, which means you need to create the /var/spool/fsvs folder, then reissue the fsvs urls command.
The following error means that you forgot to create the /etc/fsvs/ folder.
Configure ignore patterns and doing the base check-in
When constructing ignore patterns, generally work on adding a few directories at a time to the SVN repository. Everyone has different directories that they won't want to version, so you'll need to tailor the following to match your configuration. However, I generally recommend starting with the following:
Then you'll either want to ignore (or encrypt) the SSH key files.
You can check what FSVS is going to version by using the "fsvs status pathname" command (such as "fsvs status /etc"). Once you are happy with the selection in a particular path, you can do the following command:
Repeat this for the various top level trees until you have checked everything in. Then you should do one last check-in at the root level that catches anything you might have missed.
I'm going to start with the assumption that this is a base CentOS 5.4 install without *any* package groups selected during the initial install. In my case, this is a DomU that I'm setting up under Xen to serve as testing server for web development. The only thing I've done so far is setting the root password and configuring it to use a static IP address.
The basic steps will be:
- Setup the RPMForge repository
- Install the packages needed for FSVS
- Download and compile FSVS
- Configure ignore patterns
- Do the base check-ins
Setting up RPMForge
In order to get the latest Subversion packages for your system, you'll have to add RPMForge as a source repository. The CentOS base repository only has Subversion 1.4.2 and the latest is currently 1.6.6. I recommend doing this in conjunction with the yum-priorities package.
# yum install yum-prioritiesAfter installing the yum-priorities package, you should edit the CentOS-Base.repo file found under /etc/yum.repos.d/. For the base repositories, I recommend setting them to priority values of 1 through 3. For example:
[base]
name=CentOS-$releasever - Base
mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5
priority=1
exclude=subversion-*I generally give the [base], [updates], [addons], [extras] repositories a priority of "1", with [centosplus] and [contrib] getting a priority of "3".
In addition, you'll need to add or edit the "exclude=" line in the [base] repository section to exclude Subversion from being sourced from that repository. This will allow the Yum package manager to look in other repositories to find Subversion.
Now we can install the RPMForge repository (see Using RPMForge).
# cd /root/
# wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
# rpm -Uhv rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
# cd /etc/yum.repos.d/Now you should edit the rpmforge.repo file and insert a priority= line. I recommend a value of 10 or 25.
You can now verify that you'll pull in the latest Subversion package by the following command:
# yum info subversion
Available Packages
Name : subversion
Arch : x86_64
Version : 1.6.6
Release : 0.1.el5.rf
Size : 6.8 M
Repo : rpmforgeInstall the packages needed for FSVS
# yum install subversion subversion-devel ctags apr apr-devel gcc gdbm gdbm-devel pcre pcre-devel apr-util-develDownload and compile FSVS
As always, you shouldn't compile code as the root user.
# su username
$ mkdir ~/fsvs
$ cd ~/fsvs
$ wget http://download.fsvs-software.org/fsvs-1.2.1.tar.bz2
$ tar xjf fsvs-1.2.1.tar.bz2
$ cd fsvs-1.2.1
$ ./configure
$ make
$ exit
# cp /home/username/fsvs/fsvs-1.2.1/src/fsvs /usr/local/bin/
# chmod 755 /usr/local/bin/fsvsCreating the repository on the SVN server
This is how we setup users on our SVN server. Machine accounts are prefixed as "sys-" in front of the machine name. The SVN repository name matches the name of the machine. In general, only the machine account should have write access to the repository, although you may wish to add other users to the group so that they can gain read-only access.
# useradd -m sys-www-test
# passwd sys-www-test
# svnadmin create /var/svn/sys-www-test
# chmod -R 740 sys-www-test
# chmod -R g+s sys-www-test/db
# chown -R sys-www-test:sys-www-test sys-www-testBack on the source machine (our test machine), we'll need to create an SSH key that can be used on our SVN server. You may wish to use a slightly larger RSA key (3200 bits or 4096 bits) if you're working on an extra sensitive server. But a key size of 2048 bits should be secure for another decade for this purpose.
# cd /root/
# mkdir .ssh
# chmod .ssh 700
# cd .ssh
# /usr/bin/ssh-keygen -N '' -C 'svn key for root@hostname' -t rsa -b 2048 -f root@hostname
# cat root@hostname.pubCopy this key into the clipboard or send it to the SVN server or the SVN server administrator. Then we'll need to create a ~/.ssh/config file to tell the user what account name, port and key file to use when talking to the SVN server.
# vi /root/.ssh/config
Host svn.tgharold.com
Port 22
User sys-www-test
IdentityFile /root/.ssh/root@hostnameBack on the SVN server, you'll need to finish configuration of the user that will add files to the SVN repository.
# su username
$ cd ~/
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ cat >> authorized_keys
(paste in the SSH key from the other server)
$ chmod 600 authorized_keysNow you'll want to prepend the following in front of the key line in the authorized_keys file.
command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty,no-port-forwarding,no-X11-forwardingThat ensures (mostly) that the key can only be used to run the svnserve command and that it can't be used to access a command shell on the SVN server.
Test the configuration back on the original server by issuing the "svn info" command. Alternately, you can try to ssh to the SVN repository server. Errors will usually either be logged in /var/log/secure on the source server or in the same log file on the SVN repository server.
Here's an example of a successful connection:
# ssh svn.tgharold.com
( success ( 2 2 ( ) ( edit-pipeline svndiff1 absent-entries commit-revprops depth log-revprops partial-replay ) ) )This shows that they key is running the "svnserve" command automatically.
Connect the system to the SVN repository
The very first command that you'll need to issue for FSVS is the "urls" (or "initialize") command. This tells FSVS what repository will be used to store the files.
# cd /
# mkdir /var/spool/fsvs
# mkdir /etc/fsvs/
# fsvs urls svn+ssh://svn.tgharold.com/sys-www-test/You may see the following error, which means you need to create the /var/spool/fsvs folder, then reissue the fsvs urls command.
stat() of waa-path "/var/spool/fsvs/" failed. Does your local WAA storage area exist?The following error means that you forgot to create the /etc/fsvs/ folder.
Cannot write to the FSVS_CONF path "/etc/fsvs/".Configure ignore patterns and doing the base check-in
When constructing ignore patterns, generally work on adding a few directories at a time to the SVN repository. Everyone has different directories that they won't want to version, so you'll need to tailor the following to match your configuration. However, I generally recommend starting with the following:
# cd /
# fsvs ignore group:ignore,./dev
# fsvs ignore group:ignore,./etc/fsvs/
# fsvs ignore group:ignore,./etc/gconf/
# fsvs ignore group:ignore,./etc/gdm/
# fsvs ignore group:ignore,./home/
# fsvs ignore group:ignore,./lost+found
# fsvs ignore group:ignore,./media/
# fsvs ignore group:ignore,./mnt/
# fsvs ignore group:ignore,./proc
# fsvs ignore group:ignore,./root/.gconf
# fsvs ignore group:ignore,./root/.nautilus
# fsvs ignore group:ignore,./selinux/
# fsvs ignore group:ignore,./srv
# fsvs ignore group:ignore,./sys
# fsvs ignore group:ignore,./tmp/
# fsvs ignore group:ignore,./usr/tmp/
# fsvs ignore group:ignore,./var/gdm/
# fsvs ignore group:ignore,./var/lib/mlocate/
# fsvs ignore group:ignore,./var/lock/
# fsvs ignore group:ignore,./var/log/
# fsvs ignore group:ignore,./var/mail/
# fsvs ignore group:ignore,./var/run/
# fsvs ignore group:ignore,./var/spool/
# fsvs ignore group:ignore,./var/tmp/
Then you'll either want to ignore (or encrypt) the SSH key files.
# cd /
# fsvs ignore group:ignore,./root/.ssh
# fsvs ignore group:ignore,./etc/ssh/shadow*
# fsvs ignore group:ignore,./etc/ssh/ssh_host_key
# fsvs ignore group:ignore,./etc/ssh/ssh_host_dsa_key
# fsvs ignore group:ignore,./etc/ssh/ssh_host_rsa_keyYou can check what FSVS is going to version by using the "fsvs status pathname" command (such as "fsvs status /etc"). Once you are happy with the selection in a particular path, you can do the following command:
# fsvs ci -m "base check-in" /etcRepeat this for the various top level trees until you have checked everything in. Then you should do one last check-in at the root level that catches anything you might have missed.
Monday, November 09, 2009
CentOS 5, ClamAV 0.95 and /etc/sysconfig/clamav
Trying to configure the new ClamAV 0.95 as a milter for our Postfix install this week. So I've been doing some digging into the configuration files. Here's what I've found so far.
In order to get the newer ClamAV for Red Hat Enterprise Linux 5 (RHEL5) and CentOS 5, I had to use the RPMForge repository in order to get the 0.95 version.
The old clamav-milter package is outdated and should not be installed (use the newer clamav 0.95 or later package).
The /etc/rc.d/init.d/clamav script is still from 2008 and is very old. It references /etc/sysconfig/clamav which has an outdated setting called "CLAMAV_MILTER=yes". In ClamAV 0.95+, the milter was rewritten and now uses a configuration file (/etc/clamav-milter.conf) instead of command-line arguments. The init.d script that manages the clamd daemon is still for the older milter. It works fine for starting and stopping clamd, but you should not use the "CLAMAV_MILTER=yes" setting in the sysconfig file.
If you were using the /etc/sysconfig/clamav file to turn on the milter in RHEL5, then you will probably see the following error when you upgrade to ClamAV 0.95 or later:
You'll need to convert your old command line options into configuration file options.
In order to get the newer ClamAV for Red Hat Enterprise Linux 5 (RHEL5) and CentOS 5, I had to use the RPMForge repository in order to get the 0.95 version.
The old clamav-milter package is outdated and should not be installed (use the newer clamav 0.95 or later package).
The /etc/rc.d/init.d/clamav script is still from 2008 and is very old. It references /etc/sysconfig/clamav which has an outdated setting called "CLAMAV_MILTER=yes". In ClamAV 0.95+, the milter was rewritten and now uses a configuration file (/etc/clamav-milter.conf) instead of command-line arguments. The init.d script that manages the clamd daemon is still for the older milter. It works fine for starting and stopping clamd, but you should not use the "CLAMAV_MILTER=yes" setting in the sysconfig file.
If you were using the /etc/sysconfig/clamav file to turn on the milter in RHEL5, then you will probably see the following error when you upgrade to ClamAV 0.95 or later:
Starting clamav-milter: clamav-milter: unrecognized option `--max-children=10'
ERROR: Unknown option passed
ERROR: Can't parse command line optionsYou'll need to convert your old command line options into configuration file options.
Wednesday, July 22, 2009
3ware SATA RAID - Reboot and Select proper Boot device
Reboot and Select proper Boot device
or Insert Boot Media in selected Boot device and press a keyIn the process of setting up my 15-disk array, I kept encountering the above error message while attempting to boot into CentOS 5 off of the 3ware 9650SE array. If I turned off the offending drive (or removed it), the error went away and CentOS would boot properly.
This particular error message seems to be generated by an MS-DOS / Windows boot loader on the hard drive. Some motherboards seem to prefer this particular MBR, rather then loading GRUB/LILO as desired. So, when scanning the drives, they tend to fixate on whatever boot loader they find first.
If you hook the machine to a regular SATA port (or use an external USB device), you can overwrite the first few gigabytes of the drive with zeroes. At which point, adding the drive to the array will no longer cause the problem. Attempting to remedy the issue while the drive is connected to the 3ware controller will not work as the 3ware controller seems to hide the first few cylinders of the drive.
(It might work in JBOD mode, but you can't setup a JBOD disk on the fly using the tw_cli command on a 9650SE controller. So an external USB enclosure / tray / drive toaster is the best route.)
Updates:
- After playing with a little more, it seems to be a BIOS issue where the BIOS gets confused if the 3ware controller presents more then 12 units to the BIOS.
- The 3ware controller does not "hide the first few cylinders of the drive". Instead, it seems to store it's metadata for the drive at the end. Aside from the problem of losing data in the last few cylinders, you can take a drive configured as a "single-unit" and hook it up to a regular SATA controller with no issues.
3ware 9650SE SATA RAID and CentOS 5 Linux
A few months ago, I picked up a 3ware 9650SE 16-port controller for use in my primary office server which runs CentOS 5. So far, it's been an up and down ride.
Problem #1 - The boot process could not find the array disks.
I have a triple-mirror set that I use for my primary operating system drive. On the old system, they were hooked up directly to the SATA ports on the motherboard.
(Triple-mirror means that I created the elements in mdadm, a.k.a. Software RAID, where all 3 drives are active mirror copies of each other. This offers a slight speed-up for reads, allows you to survive 2 drive failures, and puts what would otherwise be an idle hot-swap disk to use.)
On the new system, I decided to attach them to the 3ware card and configure them as JBOD. However, the kernel initrd file (2.6.18-92.1.22.el5) that I was using for the old motherboard did not have drivers installed for the 3ware card.
So I had to create a custom initrd file by using gzip and cpio to unpack the contents into a directory. Then I copied the 3ware kernel driver binary from the 2.6.18-92.1.22.el5 folder, added it to the list of modules to be loaded. Finally, I repacked the initrd file, added a grub.conf entry to point at it, and booted cleanly.
Unfortunately, this was all done using the CentOS 5.3 install CD, so I was unable to log the session or keep careful track of what I did.
Problem #2 - JBOD is not really supported on the 9650SE
In the BIOS interface, you *can* setup disks as JBOD and tell the controller to export JBOD disks (see controller options). Setting a drive as JBOD does not currently overwrite/erase data on the drive, so it is a fairly safe operation and a good way to hook up a drive with existing data.
Note that if you have "export JBOD" set in the controller options, you can simply hook the drives up, rescan (using the BIOS or 3ware command-line utility called "tw_cli"), and the drives should show up as /dev/sd? in your list.
The major downside of JBOD mode is that write caching is always disabled by default. This means that your drives are going to have a much higher utilization percentage (as seen by "atop") then if you had enabled write caching.
Now, you can enable write caching for JBOD drives, but the unit has to be told to do that after every reboot. The command (assuming that your controller is "c0" and the unit is "u12") is:
# tw_cli /c0/u12 set cache=on
A final note. If you're going to use write caching, you should spring for the BBU (Battery Backup Unit).
Problem #3 - Controlling the RAID
Download and install the "tw_cli" tarball from 3ware. Since you have to have the 3ware driver installed to talk to the card, and the 9650SE prefers "Single Disk" over "JBOD", you're probably going to want to use 3ware RAID instead of Software RAID.
The problem with "Single Disk" mode is that it overwrites the first few sectors on the disk with 3ware control information. So all of the disks in "Single Disk" mode are going to slightly smaller then a JBOD disk. Be aware that putting a disk into a 3ware array will cause the loss of anything at the start of the disk (such as the partition table) and the number of cylinders will be slightly smaller.
Of course, due to the strange geometry of a disk touched by the 3ware controller, you'll probably have to move the disk to another 3ware controller in order to read the data in the future. Well, maybe, if you're using Software RAID1 mirroring on top of 3ware Single Disks, then the data is highly likely to be in an easy to read format, other then the odd starting point for the partitions.
Anyway, some key commands when using the tw_cli application:
# tw_cli show
- Displays the list of controllers installed. Make note of the "c#" nomenclature as you will use those "c#" labels in later commands to refer to a specific controller.
# tw_cli /c0 show
- Displays units/ports for the *first* controller installed.
# tw_cli /c0 rescan
- Use this after inserting/removing a disk using a hot-swap enclosure.
# tw_cli /c0/ux show all
- Displays configuration information for whichever unit # you provided. Replace the "x" with the unit # that you want to look at (such as /c0/u3 or /c0/u12).
Problem #4 - Performance (a.k.a. I/O wait hell)
Unfortunately, the 3ware Linux kernel driver in Red Hat / CentOS 2.6.18-92.1.22.el5, is not very good. The symptoms are as follows:
1) Create multiple "single disk" units.
2) Make heavy writes to one or more of the units. Such as using "dd" to overwrite the unit with zeros.
3) Attempt to access data on the other units.
What you will find is that:
- Performance of the system starts to feel extremely sluggish for any operations that touch drives on the 3ware controller.
- Looking at "atop", you will see that the other drives are now reporting seek times of 100-200ms instead of 1-10ms. Their utilization numbers will be up around 90-100%, even though the number of reads/writes are only in the 2-3 digit range.
- Turning write caching on/off doesn't make a difference.
From my web searches, it seems like this may be a problem specific to kernel versions prior to 2.6.26. Unfortunately, the stock kernel in Red Hat / CentOS is based off of 2.6.18 and I haven't found out yet whether Red Hat / CentOS have backported the fix.
Updates:
- Even the 2.6.18-128 RHEL/CentOS kernel displays sluggishness any time that we access units (a RAID 6, 8 drive unit that is the only thing on the array). We have zero performance problems with drive attached to a different SATA controller running Software RAID.
- I can't recommend using the 9650SE controller with RHEL/CentOS currently. Performance is absolutely horrid under load.
Problem #1 - The boot process could not find the array disks.
I have a triple-mirror set that I use for my primary operating system drive. On the old system, they were hooked up directly to the SATA ports on the motherboard.
(Triple-mirror means that I created the elements in mdadm, a.k.a. Software RAID, where all 3 drives are active mirror copies of each other. This offers a slight speed-up for reads, allows you to survive 2 drive failures, and puts what would otherwise be an idle hot-swap disk to use.)
On the new system, I decided to attach them to the 3ware card and configure them as JBOD. However, the kernel initrd file (2.6.18-92.1.22.el5) that I was using for the old motherboard did not have drivers installed for the 3ware card.
So I had to create a custom initrd file by using gzip and cpio to unpack the contents into a directory. Then I copied the 3ware kernel driver binary from the 2.6.18-92.1.22.el5 folder, added it to the list of modules to be loaded. Finally, I repacked the initrd file, added a grub.conf entry to point at it, and booted cleanly.
Unfortunately, this was all done using the CentOS 5.3 install CD, so I was unable to log the session or keep careful track of what I did.
Problem #2 - JBOD is not really supported on the 9650SE
In the BIOS interface, you *can* setup disks as JBOD and tell the controller to export JBOD disks (see controller options). Setting a drive as JBOD does not currently overwrite/erase data on the drive, so it is a fairly safe operation and a good way to hook up a drive with existing data.
Note that if you have "export JBOD" set in the controller options, you can simply hook the drives up, rescan (using the BIOS or 3ware command-line utility called "tw_cli"), and the drives should show up as /dev/sd? in your list.
The major downside of JBOD mode is that write caching is always disabled by default. This means that your drives are going to have a much higher utilization percentage (as seen by "atop") then if you had enabled write caching.
Now, you can enable write caching for JBOD drives, but the unit has to be told to do that after every reboot. The command (assuming that your controller is "c0" and the unit is "u12") is:
# tw_cli /c0/u12 set cache=on
A final note. If you're going to use write caching, you should spring for the BBU (Battery Backup Unit).
Problem #3 - Controlling the RAID
Download and install the "tw_cli" tarball from 3ware. Since you have to have the 3ware driver installed to talk to the card, and the 9650SE prefers "Single Disk" over "JBOD", you're probably going to want to use 3ware RAID instead of Software RAID.
The problem with "Single Disk" mode is that it overwrites the first few sectors on the disk with 3ware control information. So all of the disks in "Single Disk" mode are going to slightly smaller then a JBOD disk. Be aware that putting a disk into a 3ware array will cause the loss of anything at the start of the disk (such as the partition table) and the number of cylinders will be slightly smaller.
Of course, due to the strange geometry of a disk touched by the 3ware controller, you'll probably have to move the disk to another 3ware controller in order to read the data in the future. Well, maybe, if you're using Software RAID1 mirroring on top of 3ware Single Disks, then the data is highly likely to be in an easy to read format, other then the odd starting point for the partitions.
Anyway, some key commands when using the tw_cli application:
# tw_cli show
- Displays the list of controllers installed. Make note of the "c#" nomenclature as you will use those "c#" labels in later commands to refer to a specific controller.
# tw_cli /c0 show
- Displays units/ports for the *first* controller installed.
# tw_cli /c0 rescan
- Use this after inserting/removing a disk using a hot-swap enclosure.
# tw_cli /c0/ux show all
- Displays configuration information for whichever unit # you provided. Replace the "x" with the unit # that you want to look at (such as /c0/u3 or /c0/u12).
Problem #4 - Performance (a.k.a. I/O wait hell)
Unfortunately, the 3ware Linux kernel driver in Red Hat / CentOS 2.6.18-92.1.22.el5, is not very good. The symptoms are as follows:
1) Create multiple "single disk" units.
2) Make heavy writes to one or more of the units. Such as using "dd" to overwrite the unit with zeros.
3) Attempt to access data on the other units.
What you will find is that:
- Performance of the system starts to feel extremely sluggish for any operations that touch drives on the 3ware controller.
- Looking at "atop", you will see that the other drives are now reporting seek times of 100-200ms instead of 1-10ms. Their utilization numbers will be up around 90-100%, even though the number of reads/writes are only in the 2-3 digit range.
- Turning write caching on/off doesn't make a difference.
From my web searches, it seems like this may be a problem specific to kernel versions prior to 2.6.26. Unfortunately, the stock kernel in Red Hat / CentOS is based off of 2.6.18 and I haven't found out yet whether Red Hat / CentOS have backported the fix.
Updates:
- Even the 2.6.18-128 RHEL/CentOS kernel displays sluggishness any time that we access units (a RAID 6, 8 drive unit that is the only thing on the array). We have zero performance problems with drive attached to a different SATA controller running Software RAID.
- I can't recommend using the 9650SE controller with RHEL/CentOS currently. Performance is absolutely horrid under load.
Labels: 2009, 3ware, CentOS5, SoftwareRAID
Sunday, March 08, 2009
Linux: Checking the fstab prior to a reboot
One of the joys of working on a server from a remote location is dealing with the issue caused by a broken /etc/fstab file. Even the best admins make mistakes and mistakes in that file can lead to a server that won't boot.
Which is fine; if you have an IP-based KVM where you can get console access without actually being at the facility. But not so great when a screwed up fstab file requires you to go physically visit the location.
So how do we verify that our fstab file makes sense prior to a reboot? The answer lies in the mount command. There are (3) useful options that can be passed to the mount command which will help us check the fstab file prior to a reboot.
-f - FAKE IT (Causes everything to be done except for the actual system call). This tells mount to do everything, but don't actually change anything.
-a - Mount all filesystems mentioned in fstab.
-v - Be verbose about it.
Here's an example where everything is fine and dandy.
Same example, except that I screwed up the name of /dev/md4 in the fstab file:
Now, there's probably a better way to do this, but this serves as at least a moderate check against shooting yourself in the foot.
Which is fine; if you have an IP-based KVM where you can get console access without actually being at the facility. But not so great when a screwed up fstab file requires you to go physically visit the location.
So how do we verify that our fstab file makes sense prior to a reboot? The answer lies in the mount command. There are (3) useful options that can be passed to the mount command which will help us check the fstab file prior to a reboot.
-f - FAKE IT (Causes everything to be done except for the actual system call). This tells mount to do everything, but don't actually change anything.
-a - Mount all filesystems mentioned in fstab.
-v - Be verbose about it.
Here's an example where everything is fine and dandy.
# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
mount: /dev/md4 already mounted on /var/log
nothing was mountedSame example, except that I screwed up the name of /dev/md4 in the fstab file:
# mount -fav
mount: /dev/md0 already mounted on /boot
mount: devpts already mounted on /dev/pts
mount: tmpfs already mounted on /dev/shm
mount: proc already mounted on /proc
mount: sysfs already mounted on /sys
/dev/md4x on /var/log type ext3 (rw,noatime)
nothing was mountedNow, there's probably a better way to do this, but this serves as at least a moderate check against shooting yourself in the foot.
Wednesday, January 14, 2009
Yum Error - Metadata file does not match checksum
Ran into this issue today when the pgsql folks updated their repository. All of our CentOS 5 machines are behind a transparent HTTP proxy cache server (squid).
It doesn't really matter what the package name is, the primary issue is the "[Errno -1] Metadata file does not match checksum" error message.
Solution:
1) Edit /etc/yum.conf and add the following line
http_caching=packages
2) Run "yum cleanup metadata"
3) Retry the yum install
References:
FedoraForum.org > Fedora Support > Installation Help > yum "Metadata file does not match checksum" problem
filelists.sqlite.bz2 100% |=========================| 157 kB 00:00
http://yum.pgsqlrpms.org/8.3/redhat/rhel-5-x86_64/repodata/filelists.sqlite.bz2: [Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/filelists.sqlite.bz2 from pgdg83: [Errno 256] No more mirrors to try.
It doesn't really matter what the package name is, the primary issue is the "[Errno -1] Metadata file does not match checksum" error message.
Solution:
1) Edit /etc/yum.conf and add the following line
http_caching=packages
2) Run "yum cleanup metadata"
3) Retry the yum install
References:
FedoraForum.org > Fedora Support > Installation Help > yum "Metadata file does not match checksum" problem
Tuesday, January 06, 2009
Setting up FreeNX/NX on CentOS 5
Quick guide to setting up FreeNX/NX. This the approximate minimums on a fresh CentOS 5.1 box. We're limiting things to using public-key authentication from the outside and we already have a second ssh daemon running (listening on localhost, allowing password authentication).
Note: If you have ATRPMs configured as a repository, make sure that you exclude nx* and freenx*. (Add/edit the exclude= line in the ATRPMs .repo file.)
# yum install nx freenx
# cp /etc/nxserver/node.conf.sample /etc/nxserver/node.conf
# vi /etc/nxserver/node.conf
Change the following lines in the node.conf file:
ENABLE_SSH_AUTHENTICATION="1"
-- remove the '#' at the start of the line
ENABLE_SU_AUTHENTICATION="1"
-- remove the '#' at the start of the line
-- change the zero to a one
ENABLE_FORCE_ENCRYPTION="1"
-- remove the '#' at the start of the line
-- change the zero to a one
Change the server's public/private key pair:
# mv /etc/nxserver/client.id_dsa.key /etc/nxserver/client.id_dsa.key.OLD
# mv /etc/nxserver/server.id_dsa.pub.key /etc/nxserver/server.id_dsa.pub.key.OLD
# ssh-keygen -t dsa -N '' -f /etc/nxserver/client.id_dsa.key
# mv /etc/nxserver/client.id_dsa.key.pub /etc/nxserver/server.id_dsa.pub.key
# cat /etc/nxserver/client.id_dsa.key
You'll need to give the DSA Private Key information to people who should be allowed to use FreeNX/NX to access the server.
You'll also need to put the new public key into the authorized_keys2 file:
# cat /etc/nxserver/server.id_dsa.pub.key >> /var/lib/nxserver/home/.ssh/authorized_keys2
# vi /var/lib/nxserver/home/.ssh/authorized_keys2
Comment out the old key, put the following at the start of the good key line.
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="/usr/bin/nxserver"
Restart the FreeNX/NX service:
# service freenx-server restart
You should now be able to connect (assuming that you specify the proper SSH port and paste the private key into the configuration).
Note: If you have ATRPMs configured as a repository, make sure that you exclude nx* and freenx*. (Add/edit the exclude= line in the ATRPMs .repo file.)
# yum install nx freenx
# cp /etc/nxserver/node.conf.sample /etc/nxserver/node.conf
# vi /etc/nxserver/node.conf
Change the following lines in the node.conf file:
ENABLE_SSH_AUTHENTICATION="1"
-- remove the '#' at the start of the line
ENABLE_SU_AUTHENTICATION="1"
-- remove the '#' at the start of the line
-- change the zero to a one
ENABLE_FORCE_ENCRYPTION="1"
-- remove the '#' at the start of the line
-- change the zero to a one
Change the server's public/private key pair:
# mv /etc/nxserver/client.id_dsa.key /etc/nxserver/client.id_dsa.key.OLD
# mv /etc/nxserver/server.id_dsa.pub.key /etc/nxserver/server.id_dsa.pub.key.OLD
# ssh-keygen -t dsa -N '' -f /etc/nxserver/client.id_dsa.key
# mv /etc/nxserver/client.id_dsa.key.pub /etc/nxserver/server.id_dsa.pub.key
# cat /etc/nxserver/client.id_dsa.key
You'll need to give the DSA Private Key information to people who should be allowed to use FreeNX/NX to access the server.
You'll also need to put the new public key into the authorized_keys2 file:
# cat /etc/nxserver/server.id_dsa.pub.key >> /var/lib/nxserver/home/.ssh/authorized_keys2
# vi /var/lib/nxserver/home/.ssh/authorized_keys2
Comment out the old key, put the following at the start of the good key line.
no-port-forwarding,no-X11-forwarding,no-agent-forwarding,command="/usr/bin/nxserver"
Restart the FreeNX/NX service:
# service freenx-server restart
You should now be able to connect (assuming that you specify the proper SSH port and paste the private key into the configuration).
Friday, January 02, 2009
Samba3: Upgrading to v3.2 on CentOS 5
CentOS 5 currently only has Samba 3.0.28 in their BASE repository. The DAG/RPMForge projects don't have updated Samba3 RPMs either (although I do see an OpenPkg RPM). So the question that I've been dealing with for the past few weeks is "where do I get newer Samba RPMs"?
Ideally, I would get these RPMs from a repository, so that I could be notified via "yum check-update" for when there are security / feature updates. While I don't mind the occasional source package in .tar.gz or .tar.bz2 format, they rapidly become a maintenance nightmare. Especially for security-sensitive packages like Samba which tend to be attack targets.
What I've found that looks promising is:
http://ftp.sernet.de/pub/samba/recent/centos/5/
Which has a .repo file and looks like it might be usable as a repository for yum. (See "Get the latest Samba from Sernet" for confirmation of this.)
# cd /etc/yum.repos.d/
# wget http://ftp.sernet.de/pub/samba/recent/centos/5/sernet-samba.repo
Now, the major change is that the RedHat/CentOS packages are named "samba.x86_64" while the sernet.de packages are named "samba3.x86_64". Also, the sernet.de folks don't sign their packages, so you will need to add "gpgcheck=0" to the end of the .repo file.
(At least, I don't think they do...)
Note: As always, before doing a major upgrade like this, make backups. At a minimum, make sure you have good backups of your Samba configuration files. We use FSVS with a SVN backend for all of our configuration files, which makes an excellent change tracking tool for Linux servers.
# yum remove samba.x86_64
# yum install samba3.x86_64
# service smb start
With luck, you should now be up and running with v3.2 of Samba. You can verify this by looking at the latest log file in the /var/log/samba/ directory.
Ideally, I would get these RPMs from a repository, so that I could be notified via "yum check-update" for when there are security / feature updates. While I don't mind the occasional source package in .tar.gz or .tar.bz2 format, they rapidly become a maintenance nightmare. Especially for security-sensitive packages like Samba which tend to be attack targets.
What I've found that looks promising is:
http://ftp.sernet.de/pub/samba/recent/centos/5/
Which has a .repo file and looks like it might be usable as a repository for yum. (See "Get the latest Samba from Sernet" for confirmation of this.)
# cd /etc/yum.repos.d/
# wget http://ftp.sernet.de/pub/samba/recent/centos/5/sernet-samba.repo
Now, the major change is that the RedHat/CentOS packages are named "samba.x86_64" while the sernet.de packages are named "samba3.x86_64". Also, the sernet.de folks don't sign their packages, so you will need to add "gpgcheck=0" to the end of the .repo file.
(At least, I don't think they do...)
Note: As always, before doing a major upgrade like this, make backups. At a minimum, make sure you have good backups of your Samba configuration files. We use FSVS with a SVN backend for all of our configuration files, which makes an excellent change tracking tool for Linux servers.
# yum remove samba.x86_64
# yum install samba3.x86_64
# service smb start
With luck, you should now be up and running with v3.2 of Samba. You can verify this by looking at the latest log file in the /var/log/samba/ directory.
Wednesday, August 15, 2007
Installing Angband on CentOS 5
Installation of Angband 3.0.9 on RedHat or CentOS5.
1) Grab the latest source release from http://rephial.org/release
# cd /root
# wget http://rephial.org/downloads/3.0/angband-3.0.9-src.tar.gz
# tar xzf angband-3.0.9-src.tar.gz
2) Compile the source code (the following is for running angband from the location where you unpacked the source, see Compiling for other options)
# cd angband-3.0.9
# ./configure
# make
# make install
3a) Errors: Make can't find "ncurses.h" (see also Compiling)
# make
CC main-gcu.c
main-gcu.c:63:22: error: ncurses.h: No such file or directory
Which indicates that you need to install the ncurses library. You can fix that by installing the "ncurses-devel" and re-running "./configure".
# yum install ncurses-devel
# ./configure
4) If you do a system install (making Angband available for all users on the system), make sure you add the users to the "games" group. Otherwise, when your users attempt to run Angband, they will get error messages about not being able to write to various files in the /usr/local/games/lib/angband folders.
# ./configure --with-setgid=games --with-libpath=/usr/local/games/lib/angband --bindir=/usr/local/games
# make
# make install
1) Grab the latest source release from http://rephial.org/release
# cd /root
# wget http://rephial.org/downloads/3.0/angband-3.0.9-src.tar.gz
# tar xzf angband-3.0.9-src.tar.gz
2) Compile the source code (the following is for running angband from the location where you unpacked the source, see Compiling for other options)
# cd angband-3.0.9
# ./configure
# make
# make install
3a) Errors: Make can't find "ncurses.h" (see also Compiling)
# make
CC main-gcu.c
main-gcu.c:63:22: error: ncurses.h: No such file or directory
Which indicates that you need to install the ncurses library. You can fix that by installing the "ncurses-devel" and re-running "./configure".
# yum install ncurses-devel
# ./configure
4) If you do a system install (making Angband available for all users on the system), make sure you add the users to the "games" group. Otherwise, when your users attempt to run Angband, they will get error messages about not being able to write to various files in the /usr/local/games/lib/angband folders.
# ./configure --with-setgid=games --with-libpath=/usr/local/games/lib/angband --bindir=/usr/local/games
# make
# make install
Thursday, July 05, 2007
SELinux is preventing named from write access
It seems like the SELinux profile in CentOS5 may not be correct by default. In my /var/log/messages file, I have thousands of entries per month consisting of:
Jul 4 05:01:04 fw1-hosho setroubleshoot: SELinux is preventing /usr/sbin/named (named_t) "write" access to named (named_conf_t). For complete SELinux messages. run sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
Here's the output of the sealert command:
The most helpful web page that I've found so far is the thread "Permissions Issue starting Bind 9.3.1". The gist seems to be that RedHat (and CentOS) are using a chroot bind installation in conjunction with an SELinux policy that expects the bind configuration files to be in a non-chroot setup. But there aren't very clear instructions there on fixing it.
Jul 4 05:01:04 fw1-hosho setroubleshoot: SELinux is preventing /usr/sbin/named (named_t) "write" access to named (named_conf_t). For complete SELinux messages. run sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
Here's the output of the sealert command:
# sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
Summary
SELinux is preventing /usr/sbin/named (named_t) "write" access to named
(named_conf_t).
Detailed Description
SELinux denied access requested by /usr/sbin/named. It is not expected that
this access is required by /usr/sbin/named and this access may signal an
intrusion attempt. It is also possible that the specific version or
configuration of the application is causing it to require additional access.
Please file a http://bugzilla.redhat.com/bugzilla/enter_bug.cgi against this
package.
Allowing Access
Sometimes labeling problems can cause SELinux denials. You could try to
restore the default system file context for named, restorecon -v named.
There is currently no automatic way to allow this access. Instead, you can
generate a local policy module to allow this access - see
http://fedora.redhat.com/docs/selinux-faq-fc5/#id2961385 - or you can
disable SELinux protection entirely for the application. Disabling SELinux
protection is not recommended. Please file a
http://bugzilla.redhat.com/bugzilla/enter_bug.cgi against this package.
Changing the "named_disable_trans" boolean to true will disable SELinux
protection this application: "setsebool -P named_disable_trans=1."
The following command will allow this access:
setsebool -P named_disable_trans=1
Additional Information
Source Context system_u:system_r:named_t
Target Context root:object_r:named_conf_t
Target Objects named [ dir ]
Affected RPM Packages bind-9.3.3-8.el5 [application]
Policy RPM selinux-policy-2.4.6-30.el5
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Enforcing
Plugin Name plugins.disable_trans
Host Name fw1-shimo.hq.example.org.
Platform Linux fw1-shimo.hq.example.org.
2.6.18-8.1.6.el5 #1 SMP Thu Jun 14 17:29:04 EDT
2007 x86_64 x86_64
Alert Count 70481
Line Numbers
Raw Audit Messages
avc: denied { write } for comm="named" dev=md1 egid=25 euid=25
exe="/usr/sbin/named" exit=-13 fsgid=25 fsuid=25 gid=25 items=0 name="named"
pid=2628 scontext=system_u:system_r:named_t:s0 sgid=25
subj=system_u:system_r:named_t:s0 suid=25 tclass=dir
tcontext=root:object_r:named_conf_t:s0 tty=(none) uid=25
The most helpful web page that I've found so far is the thread "Permissions Issue starting Bind 9.3.1". The gist seems to be that RedHat (and CentOS) are using a chroot bind installation in conjunction with an SELinux policy that expects the bind configuration files to be in a non-chroot setup. But there aren't very clear instructions there on fixing it.
Sunday, July 01, 2007
CentOS5: Moving /var/log to a separate volume
One thing I like to do is put /var/log on its own volume. That keeps the root volume from overflowing and also gets the log files out of the way. However, in CentOS5 (and probably RHEL5), SELinux is probably going to complain unless we tell it to "fixup" the new filesystem.
AFAIK, that's the extent of what's needed. Looking at the directory listings using "ls -lZ" seems to show the correct SELinux flags on the files between the two different directories.
- Create the filesystem (I use ext3, so # /sbin/mke2fs -j /dev/mdX)
- Mount it at a temporary location: # mkdir /mnt/log ; mount /dev/mdX /mnt/log
- Copy the contents: # cp -a /var/log/* /mnt/log/
- It may be necessary to "fixup" the new volume: # cd /mnt/log ; /sbin/restorecon -R *
- Edit the /etc/fstab file to mount the new volume at /var/log
- Reboot
AFAIK, that's the extent of what's needed. Looking at the directory listings using "ls -lZ" seems to show the correct SELinux flags on the files between the two different directories.
Labels: 2007, CentOS5, SELinux, ServerAdministration
Friday, June 22, 2007
FSVS ignore patterns (v1.1.5)
Here's a list of the current ignore patterns that I use on my CentOS5 box.
# fsvs ignore dump
./backup
./dev
./home
./lost+found
./media
./mnt
./proc/**
./root/.mozilla/firefox/**/Cache/**
./root/.thumbnails/**
./selinux
./sys
./tmp
./var/cache/**
./var/lock/**
./var/log/**
./var/named/chroot/proc
./var/run/**
./var/spool/**
./var/tmp/**
There are a few commands that I use to keep my sanity:
# fsvs dump ignore | sort > /root/fsvs-ignore.txt
# sort /root/fsvs-ignore.txt | fsvs ignore load
I find that keeping my ignore files in a .txt file under /root makes it easier to work with them. I'm able to edit the text file, load the ignore patterns into FSVS and see whether it does what it should. If it's wrong, I re-edit the text file and load them back into FSVS.
...
After mucking with a new box for a week, here's the set of ignore filters that I'm using on another CentOS5 box. On this particular box, I'm only versioning configuration data (/etc, /var/named).
[root@fw1-shimo /]# fsvs ignore dump | sort
./backup/
./bin/
./dev/
./home/
./lib/
./lib64/
./lost+found
./media/
./mnt/
./proc/
./root/
./sbin/
./selinux/
./srv/
./sys/
./tmp/
./usr/bin/
./usr/include/
./usr/kerberos/
./usr/lib/
./usr/lib64
./usr/libexec/
./usr/local/bin/
./usr/local/include/
./usr/local/lib/
./usr/local/libexec/
./usr/local/share/
./usr/local/src/
./usr/sbin/
./usr/share/
./usr/share/applications/
./usr/share/backgrounds/
./usr/share/dict/
./usr/share/doc/
./usr/share/i18n/
./usr/share/info/
./usr/share/locale/
./usr/share/man/
./usr/share/pixmaps/
./usr/share/X11/
./usr/share/zoneinfo/
./usr/src/
./usr/tmp/
./usr/X11R6/
./var/cache/
./var/lib/
./var/lock/
./var/log/
./var/named/chroot/dev/
./var/named/chroot/proc/
./var/named/chroot/var/run/
./var/run/
./var/spool/
./var/svn/
./var/tmp/
./var/www/
[root@fw1-shimo /]#
# fsvs ignore dump
./backup
./dev
./home
./lost+found
./media
./mnt
./proc/**
./root/.mozilla/firefox/**/Cache/**
./root/.thumbnails/**
./selinux
./sys
./tmp
./var/cache/**
./var/lock/**
./var/log/**
./var/named/chroot/proc
./var/run/**
./var/spool/**
./var/tmp/**
There are a few commands that I use to keep my sanity:
# fsvs dump ignore | sort > /root/fsvs-ignore.txt
# sort /root/fsvs-ignore.txt | fsvs ignore load
I find that keeping my ignore files in a .txt file under /root makes it easier to work with them. I'm able to edit the text file, load the ignore patterns into FSVS and see whether it does what it should. If it's wrong, I re-edit the text file and load them back into FSVS.
...
After mucking with a new box for a week, here's the set of ignore filters that I'm using on another CentOS5 box. On this particular box, I'm only versioning configuration data (/etc, /var/named).
[root@fw1-shimo /]# fsvs ignore dump | sort
./backup/
./bin/
./dev/
./home/
./lib/
./lib64/
./lost+found
./media/
./mnt/
./proc/
./root/
./sbin/
./selinux/
./srv/
./sys/
./tmp/
./usr/bin/
./usr/include/
./usr/kerberos/
./usr/lib/
./usr/lib64
./usr/libexec/
./usr/local/bin/
./usr/local/include/
./usr/local/lib/
./usr/local/libexec/
./usr/local/share/
./usr/local/src/
./usr/sbin/
./usr/share/
./usr/share/applications/
./usr/share/backgrounds/
./usr/share/dict/
./usr/share/doc/
./usr/share/i18n/
./usr/share/info/
./usr/share/locale/
./usr/share/man/
./usr/share/pixmaps/
./usr/share/X11/
./usr/share/zoneinfo/
./usr/src/
./usr/tmp/
./usr/X11R6/
./var/cache/
./var/lib/
./var/lock/
./var/log/
./var/named/chroot/dev/
./var/named/chroot/proc/
./var/named/chroot/var/run/
./var/run/
./var/spool/
./var/svn/
./var/tmp/
./var/www/
[root@fw1-shimo /]#
Wednesday, June 20, 2007
Remote GUI administration of CentOS5 using Windows
Over the years, I've become very spoiled by Windows Terminal Services that we use to administer our Windows 2000 and Windows 2003 servers. It's fast, it's slick, it allows copy-paste and with a bit of command line fu you can connect to the physical display (instead of one of the two virtual sessions). It also uses built-in Windows authentication and offers encryption.
So, now that I'm rolling out CentOS 5 servers - I need something similar that allows me to look at the graphical UI on the box from elsewhere. From what I can tell, my options are:
KVM that supports TCP/IP
Probably one of the holy grails of remote administration. It allows you to see everything from the BIOS setup screen onward without needing to be physically at the machine. The downside is cost. So while I will eventually be hooking one of these up, it's not in the budget for this quarter.
VNC over SSH
I have a love/hate relationship with VNC. On the Windows clients, we use UltraVNC with built-in Windows authentication and the AES encryption plug-in.
But if you want to wrap VNC with SSH, you have to configure port forwarding all the time in PuTTY. Which turns connecting to a remote server into a multi-step process. With Windows' RDP, I just say "connect to IP address X" and I'm done (and I can connect in as anyone that I want). For PuTTY+VNC, I have to jump through a lot more hoops.
There's also the (possible) issue that VNC is nowhere as efficient over the network as RDP. Once you use Terminal Services' RDP, you'll be spoiled and never want to use older technologies. It (almost) never glitches, it's lightning fast and responsive, and it's just pure remote GUI goodness (except for being a MS-only protocol).
X11 over SSH
This is where I'm heading at the moment. It uses SSH for authentication, so we can lock things down that way (forcing the use of public keys).
Now, a word of caution. A misconfigured SSH or X11 server is a security breach waiting to happen. Pay close attention to chapter 9 in SSH, The Secure Shell, The Definitive Guide by Barrett, Silverman & Byrnes (published by O'Reilly).
Installing Xming on Windows
In order to do X11 on Microsoft Windows, you need to install "X Server" software on the Windows box. While there are pay options out there, I'd suggest starting with Xming which is free (GPLv2). You'll want to download and install both Xming and Xming-fonts.
Configuration of sshd and X11
In order for the local X Server (Xming - running on your Windows system) to talk to the remote Linux server, you'll need to verify some settings on the Linux server. First up is configuration of the sshd daemon (typically /etc/ssh/sshd_config for OpenSSH). Look for the following 2 lines and make sure they are configured correctly:
X11Forwarding yes
#X11UseLocalhost yes
By default, OpenSSH ships with X11Forwarding set to "no" but the default for X11UseLocalhost is "yes". So you should only have to add the "X11Forwarding yes" line.
Create a PuTTY session
I'll make the assumption that you're going to use a PuTTY public-key pair. If you need to install a generated PuTTY key (maybe you want to use a separate PuTTY key for X11 forwarding), then here are the directions for OpenSSH.
(login as yourself or as root and then "su" to your username)
# cd ~/.ssh
# cat > machinename@svn.pub
(paste in PuTTY key)
# ssh-keygen -i -f machinename@svn.pub >> authorized_keys
(Ctrl-D to exit)
If all goes well, you should see a line like:
/usr/bin/xauth: creating new authority file /home/thomas/.Xauthority
Which tells us that SSH is ready to do some X forwarding.
Fire up Xming
If you haven't already ran Xming you should run XLaunch and just roll through the defaults. Now, in the PuTTY window that is sitting at a command prompt, try:
# xeyes
And you should see the xeyes application open up on your Windows system. If you want to continue to start up other X applications, put an ampersand (&) at the end of the line.
More advanced stuff
If all goes well, you should see the Gnome desktop!
Final thoughts (for the moment)
Now, it's still not as slick as Terminal Services. But it seems to work just fine and gives me a GUI desktop. I still plan on doing most of my administration from the command line, but this provides a nice GUI for those who follow in my footsteps.
So, now that I'm rolling out CentOS 5 servers - I need something similar that allows me to look at the graphical UI on the box from elsewhere. From what I can tell, my options are:
KVM that supports TCP/IP
Probably one of the holy grails of remote administration. It allows you to see everything from the BIOS setup screen onward without needing to be physically at the machine. The downside is cost. So while I will eventually be hooking one of these up, it's not in the budget for this quarter.
VNC over SSH
I have a love/hate relationship with VNC. On the Windows clients, we use UltraVNC with built-in Windows authentication and the AES encryption plug-in.
But if you want to wrap VNC with SSH, you have to configure port forwarding all the time in PuTTY. Which turns connecting to a remote server into a multi-step process. With Windows' RDP, I just say "connect to IP address X" and I'm done (and I can connect in as anyone that I want). For PuTTY+VNC, I have to jump through a lot more hoops.
There's also the (possible) issue that VNC is nowhere as efficient over the network as RDP. Once you use Terminal Services' RDP, you'll be spoiled and never want to use older technologies. It (almost) never glitches, it's lightning fast and responsive, and it's just pure remote GUI goodness (except for being a MS-only protocol).
X11 over SSH
This is where I'm heading at the moment. It uses SSH for authentication, so we can lock things down that way (forcing the use of public keys).
Now, a word of caution. A misconfigured SSH or X11 server is a security breach waiting to happen. Pay close attention to chapter 9 in SSH, The Secure Shell, The Definitive Guide by Barrett, Silverman & Byrnes (published by O'Reilly).
Installing Xming on Windows
In order to do X11 on Microsoft Windows, you need to install "X Server" software on the Windows box. While there are pay options out there, I'd suggest starting with Xming which is free (GPLv2). You'll want to download and install both Xming and Xming-fonts.
Configuration of sshd and X11
In order for the local X Server (Xming - running on your Windows system) to talk to the remote Linux server, you'll need to verify some settings on the Linux server. First up is configuration of the sshd daemon (typically /etc/ssh/sshd_config for OpenSSH). Look for the following 2 lines and make sure they are configured correctly:
X11Forwarding yes
#X11UseLocalhost yes
By default, OpenSSH ships with X11Forwarding set to "no" but the default for X11UseLocalhost is "yes". So you should only have to add the "X11Forwarding yes" line.
Create a PuTTY session
I'll make the assumption that you're going to use a PuTTY public-key pair. If you need to install a generated PuTTY key (maybe you want to use a separate PuTTY key for X11 forwarding), then here are the directions for OpenSSH.
(login as yourself or as root and then "su" to your username)
# cd ~/.ssh
# cat > machinename@svn.pub
(paste in PuTTY key)
# ssh-keygen -i -f machinename@svn.pub >> authorized_keys
(Ctrl-D to exit)
- Right-click on the Pageant icon in the system tray and choose "New Session".
- Enter the hostname (i.e. 192.168.1.1)
- Go to the Connection -> SSH -> X11 tab
- Turn ON "X11 forwarding"
- Display location should be: localhost:0
- Go back to the Session tab
- Enter a name in the Saved Sessions text box (i.e. "MyHost-X11") and click on "Save"
- Click the "Open" button to connect to the server
If all goes well, you should see a line like:
/usr/bin/xauth: creating new authority file /home/thomas/.Xauthority
Which tells us that SSH is ready to do some X forwarding.
Fire up Xming
If you haven't already ran Xming you should run XLaunch and just roll through the defaults. Now, in the PuTTY window that is sitting at a command prompt, try:
# xeyes
And you should see the xeyes application open up on your Windows system. If you want to continue to start up other X applications, put an ampersand (&) at the end of the line.
More advanced stuff
- Fire up XLaunch
- Select "One window" and click "Next"
- Select "Start a program" and click "Next"
- The start program should be either "gnome-session" or "startkde"
- Select Run Remote using PuTTY (plink.exe) and turn on the compression option.
- Enter the IP address or hostname in "Connect to computer" of the Linux box that you are connecting to
- Enter your username in the "Login as user"
- Click the "Next" button
- In the "Additional parameters", enter "-screen 0 1024 768" which will set screen zero to be 1024x768
- If you run your SSH server on a non-standard port, enter "-P port" in the PuTTY extra options field (run "plink" at a Windows command prompt to see the possible options)
- Save your configuration file and click "Finish"
If all goes well, you should see the Gnome desktop!
Final thoughts (for the moment)
Now, it's still not as slick as Terminal Services. But it seems to work just fine and gives me a GUI desktop. I still plan on doing most of my administration from the command line, but this provides a nice GUI for those who follow in my footsteps.
Thursday, May 31, 2007
FSVS for sysadmins
Notes: This entry was based on v1.1.4. The 1.1.5 and later versions of FSVS also place a few files in /etc/fsvs.
Original post follows
Okay, I'm heading back to trying FSVS again for doing system configuration management. The FSVS website has some documentation, but for the full documentation you'll want to download the source tarball and look in the "doc/" folder.
Resource links (and other useful information):
FSVS (fsvs.tigris.org) - the home page for FSVS. See also the Purpose of FSVS and Backup pages.
Subversion (software) - The SubVersion explanatory page over at Wikipedia.
SSH tricks - One of the most important documents to read if you want to setup secure SSH access to your SVN server. It specifies how to lock things down so that "/usr/bin/svnserve" is the only thing they can do with a particular public key.
Setting up Subversion in Linux
Creating Subversion Repositories
Now, here's what I do know about FSVS:
- It uses SubVersion for the backend storage. Which means that if you already have a SVN server up and running, you can use it for the FSVS storage. This also means that you could pull configuration files down onto your laptop with SVN to take a gander at the revision history of a particular file.
- FSVS doesn't pollute your directories with ".svn" folders. Instead, it keeps a central storage database elsewhere (by default this goes in /var/spool/fsvs, but you can move it). This WAA (Working copy Administrative Area) directory only contains file lists and hashes. It does not contain "pristine copies" of any files, so it will use up a lot less space then ".svn" folders.
- FSVS will keep track of file metadata (such as timestamps, chmod flags, etc. - see the FSVS website for particulars). I'm not sure whether this includes information needed by SELinux.
- You can use FSVS to push changes to machines. Not something that I'm interested in (yet), but I might use it down the road.
- And most (all?) tricks you can use in SVN repositories apply to FSVS. Such as cloning a machine from another's configuration using "svn cp" or comparing files between two machines. Or creating a branch for a new configuration.
- FSVS allows you to make "empty" commits to the repository. If nothing has changed in the system and you do a fsvs ci -m "commit message" then FSVS will create a new revision in the repository, but with no actual changes. That may come in handy in certain circumstances.
The method to my madness
My preference for system administration is to have a separate user account and SSH key for each machine that I manage. This allows me to use a no-password SSH key on the machine so that I can do svn/fsvs commands easily (or script svn/fsvs commands). Because the SSH server is locked down, and the keys are locked down with the "command=" syntax of SSH - I'm not terribly worried about keys that don't have passwords. Since SVN doesn't allow you to permanently delete files from a repository, there's a limited amount of damage that an attacker could do if they swipe the private key.
(Lastly, because the SSH private key is stored inside of /root/, it means they've cracked the server security already. We're just trying to limit the damage and keep them from being able to erase things on the central SVN server.)
Naturally, after talking about SSH keys, I'm only using the "svn+ssh" method of accessing the central repositories.
SSH Security considerations
On the SVN server, you should edit /etc/ssh/sshd_config and verify that the following are enforced in the SSH daemon configuration:
AllowTcpForwarding no
X11Forwarding no
PermitTunnel no
That eliminates most abuses that are possible, even if someone edits their ~/.ssh/authorized_key file on the SVN server.
Creating a user account on the SSH server
Make sure you have a naming scheme in place. For us, regular developers and administrators get normal looking usernames (i.e. "thomas" or "tgh" or "haroldt"). For machine accounts, we prefix the server name with "sys-" to create the username (i.e. "sys-fw1", "sys-mail1", "sys-gracie"). Which should make it easy to see if a machine has been added to groups that it shouldn't be in. Any groups that are used to control access to SVN directories are prefixed with "svn-repositoryname" such as "svn-sys-fw" (which owns the "sys-fw" repository).
A) On the client machine:
Login as root (or "su" to root).
# cd /root/
(skip the next 2 commands if the .ssh subfolder already exists)
# mkdir .ssh
# chmod .ssh 700
# cd .ssh
# /usr/bin/ssh-keygen -N '' -C 'svn key for root@hostname' -t rsa -b 2048 -f root@hostname
# cp /root/.ssh/root@svn /root/.ssh/id_rsa
# cat root@hostname.pub
(copy this into the clipboard or send it to the SVN server or the SVN server administrator)
B) On the SVN server
Note: You should use some sort of random password creator (or the output of /dev/random or /dev/urandom) to create a long password that can be copied and pasted into the password prompt. Since we're using SSH keys, the account doesn't need a password that anyone knows.
(I know there's a better way to do make an account with an unguessable password, yet still allow SSH access via pub keys, but I can't find it at the moment.)
# useradd -m username
(i.e. "useradd -m sys-fw1-pri")
# passwd username
(paste in a super-long randomized password, such as a few bytes from /dev/urandom shoved through md5sum)
# cd /home/username
# su username
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ cat > root@hostname.pub
(paste in the public key file from the client system)
$ cat root@hostname.pub >> authorized_keys
$ chmod 600 *
Now to lock the key down, edit the ~/.ssh/authorized_keys file and put the following on the front of the key line that will be used by the client machine:
command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty
This forces the connection to run the "svnserve" command in tunnel mode. So this SSH key cannot be used to login or run any other commands on the server. It also changes the SVN root path to /var/svn. You will want to also add "no-port-forwarding" and "no-X11-forwarding" if you have not disabled those in your /etc/ssh/sshd_config file.
We now have a user account that can be used with SVN. Go ahead and [Ctrl-D] to escape the "su username" session and get back into the root account.
Setting up the repository (on the SVN server)
Reasons to use a common repository for all similar machines:
- Ability to do a svn diff between two different machines
- Ability to clone machines ("svn cp")
- Backup scripts are less complex (fewer repositories)
- You can "svn diff" between machines
Reasons to use individual repositories
- Easy to secure using chmod/chown
- Easy to get report sizes on how much space a machine is using on the repository server
- Easy to dump/load an individual machine's repository
- Easy to take a machine offline and remove the repository to save space
- A machine's SSH key can only be used to look at their repository (unless you configure per-directory authentication in SVN)
Note: I'm using "username", "hostname" and "machinename" fairly interchangeably on this page. (Someday I'll go back and clean it up.)
On the SVN server:
# cd /var/svn
(your repositories may be stored elsewhere)
# svnadmin create /var/svn/sys-machinename
# chmod -R 770 sys-machinename
# chmod -R g+s sys-machinename/db
# chown -R sys-machinename:sys-machinename sys-machinename
Notes:
- A chmod of "770" allows read/write access to everyone in the same group.
- A chmod of "700" only allows read/write to the owner account.
- The third octet should probably always be "0" to prevent the repository from being world-readable.
Verifying the connection (on the client)
You will need to customize the following URL to point at your SVN repository location.
# svn info svn+ssh://sys-machinename@svn.intra.example.com/sys-machinename/
That should prompt you to accept the server's public key, then display a response fron the SVN server. If things don't work, then you've got connectivity (firewall), account (wrong name? wrong key?) or permissions (chmod or chown goofs?).
Installing FSVS (on the client)
Notes:
In order for the install to succeed, you must have installed the "subversion", "subversion-devel" "apr", "apr-devel", "gcc" and "ctags" packages. Two others that you need are "gdbm" and "pcre" (and the associated developer packages). There may be other dependencies that will also be installed that are required by those packages. The following command worked for me on CentOS5:
# yum install subversion subversion-devel ctags apr apr-devel gcc gdbm gdbm-devel pcre pcre-devel
Head on over to the official project page at freshmeat.net and download the tarball. This is currently "fsvs-1.14.tar.gz". Extract the tarball to a folder somewhere (i.e. /root/fsvs-1.14.tar.gz) and use a terminal session to go to that folder.
# cat README
(look for the section that talks about the install)
# cd src
# make
(you will receive a message that the Makefile has now been updated and that you need to run make again)
# make
(you should see a large stream of gcc output with the following at the end)
-rwxr-xr-x 1 root root 201510 May 31 09:54 fsvs
(you must see the above line to know that you got a good compile)
# cp fsvs /usr/local/bin
At this point, FSVS *should* be installed correctly.
Getting started with FSVS
By default, FSVS will want to use /var/spool/fsvs unless you define the "WAA" variable and point it somewhere else. So according to the README you will need to create that folder and then initialize it (which lets FSVS create the administrative files it needs).
Note #1: I'm not sure how large /var/spool/fsvs will get. You may eventually want to break it off into a separate LVM volume of its own. Since it is mostly file lists and hashes, it shouldn't get too large, but you may want to put it on an ext3 volume with more inodes then normal.
Note #2: You should read both the README and the output of "fsvs help urls".
# mkdir -p /var/spool/fsvs
# chmod 700 /var/spool/fsvs
# cd /
# fsvs urls svn+ssh://username@machine/path/to/repos
(The "fsvs urls" won't display any confirmation text.)
The above will connect the root folder to the repository path. You could also do multiple URLs and only link sub-folders (such as /etc, /usr, /home) up against the SVN repository.
Setting up ignore filters
After telling FSVS that we want to use "/" as our working copy, we'll want to also tell it to ignore various directories and files. While this tends to be somewhat similar across Linux distributions, you should also plan on modifying this list to match your distribution.
Make sure you read "fsvs-*/doc/IGNORING" and the output of "# fsvs help ignore".
# fsvs ignore ./backup
# fsvs ignore ./dev
# fsvs ignore ./mnt
# fsvs ignore './proc/*'
# fsvs ignore ./sys
# fsvs ignore ./tmp
# fsvs ignore ./var/tmp
# fsvs ignore ./var/spool
In addition you may wish to initially ignore all of the binary file directories (such as ./lib, ./lib64, ./sbin, ./usr, ./var) and focus solely on /etc, /home, /boot and /root. That will give you a much slimmer listing when you "fsvs status" from the root directory.
You can use the "fsvs ignore dump" and "fsvs ignore load" commands to backup your listing, edit it, then load it back into FSVS. Note that you must be in the base directory of your working copy, otherwise "fsvs ignore dump" will return an empty listing.
# cd /
# fsvs ignore dump > ~/fsvs-ignore.txt
# vi ~/fsvs-ignore.txt
# sort ~/fsvs-ignore.txt | fsvs ignore load
My initial listing on CentOS5 is:
Putting /etc under version control
Assuming that you setup FSVS where "/" (root) is the base of the working copy, we can now add the contents of /etc to SVN.
# cd /
# fsvs commit -m "Base check-in of /etc" /etc
# fsvs commit -m "Base check-in of /boot" /boot
If you want to deep-commit a single folder (such as /usr/local/sbin) without doing the intervening folders:
# cd /
# fsvs commit -m "Base check-in" /usr/local/sbin
Working with FSVS
Create a test file in /etc
# cat > /etc/testfile.txt
foo
# fsvs commit -m "Checking in a test file" /etc/testfile.txt
Now delete the test file
That just scratches the surface, but covers the majority of day-to-day use. Unlike SVN, FSVS knows (assumes) that when a file is missing that it should implicity do a "delete" operation in the repository to make the repository match the file system.
Other useful commands to know are "fsvs unversion" and "fsvs diff".
Original post follows
Okay, I'm heading back to trying FSVS again for doing system configuration management. The FSVS website has some documentation, but for the full documentation you'll want to download the source tarball and look in the "doc/" folder.
Resource links (and other useful information):
FSVS (fsvs.tigris.org) - the home page for FSVS. See also the Purpose of FSVS and Backup pages.
Subversion (software) - The SubVersion explanatory page over at Wikipedia.
SSH tricks - One of the most important documents to read if you want to setup secure SSH access to your SVN server. It specifies how to lock things down so that "/usr/bin/svnserve" is the only thing they can do with a particular public key.
Setting up Subversion in Linux
Creating Subversion Repositories
Now, here's what I do know about FSVS:
- It uses SubVersion for the backend storage. Which means that if you already have a SVN server up and running, you can use it for the FSVS storage. This also means that you could pull configuration files down onto your laptop with SVN to take a gander at the revision history of a particular file.
- FSVS doesn't pollute your directories with ".svn" folders. Instead, it keeps a central storage database elsewhere (by default this goes in /var/spool/fsvs, but you can move it). This WAA (Working copy Administrative Area) directory only contains file lists and hashes. It does not contain "pristine copies" of any files, so it will use up a lot less space then ".svn" folders.
- FSVS will keep track of file metadata (such as timestamps, chmod flags, etc. - see the FSVS website for particulars). I'm not sure whether this includes information needed by SELinux.
- You can use FSVS to push changes to machines. Not something that I'm interested in (yet), but I might use it down the road.
- And most (all?) tricks you can use in SVN repositories apply to FSVS. Such as cloning a machine from another's configuration using "svn cp" or comparing files between two machines. Or creating a branch for a new configuration.
- FSVS allows you to make "empty" commits to the repository. If nothing has changed in the system and you do a fsvs ci -m "commit message" then FSVS will create a new revision in the repository, but with no actual changes. That may come in handy in certain circumstances.
The method to my madness
My preference for system administration is to have a separate user account and SSH key for each machine that I manage. This allows me to use a no-password SSH key on the machine so that I can do svn/fsvs commands easily (or script svn/fsvs commands). Because the SSH server is locked down, and the keys are locked down with the "command=" syntax of SSH - I'm not terribly worried about keys that don't have passwords. Since SVN doesn't allow you to permanently delete files from a repository, there's a limited amount of damage that an attacker could do if they swipe the private key.
(Lastly, because the SSH private key is stored inside of /root/, it means they've cracked the server security already. We're just trying to limit the damage and keep them from being able to erase things on the central SVN server.)
Naturally, after talking about SSH keys, I'm only using the "svn+ssh" method of accessing the central repositories.
SSH Security considerations
On the SVN server, you should edit /etc/ssh/sshd_config and verify that the following are enforced in the SSH daemon configuration:
AllowTcpForwarding no
X11Forwarding no
PermitTunnel no
That eliminates most abuses that are possible, even if someone edits their ~/.ssh/authorized_key file on the SVN server.
Creating a user account on the SSH server
Make sure you have a naming scheme in place. For us, regular developers and administrators get normal looking usernames (i.e. "thomas" or "tgh" or "haroldt"). For machine accounts, we prefix the server name with "sys-" to create the username (i.e. "sys-fw1", "sys-mail1", "sys-gracie"). Which should make it easy to see if a machine has been added to groups that it shouldn't be in. Any groups that are used to control access to SVN directories are prefixed with "svn-repositoryname" such as "svn-sys-fw" (which owns the "sys-fw" repository).
A) On the client machine:
Login as root (or "su" to root).
# cd /root/
(skip the next 2 commands if the .ssh subfolder already exists)
# mkdir .ssh
# chmod .ssh 700
# cd .ssh
# /usr/bin/ssh-keygen -N '' -C 'svn key for root@hostname' -t rsa -b 2048 -f root@hostname
# cp /root/.ssh/root@svn /root/.ssh/id_rsa
# cat root@hostname.pub
(copy this into the clipboard or send it to the SVN server or the SVN server administrator)
B) On the SVN server
Note: You should use some sort of random password creator (or the output of /dev/random or /dev/urandom) to create a long password that can be copied and pasted into the password prompt. Since we're using SSH keys, the account doesn't need a password that anyone knows.
(I know there's a better way to do make an account with an unguessable password, yet still allow SSH access via pub keys, but I can't find it at the moment.)
# useradd -m username
(i.e. "useradd -m sys-fw1-pri")
# passwd username
(paste in a super-long randomized password, such as a few bytes from /dev/urandom shoved through md5sum)
# cd /home/username
# su username
$ mkdir .ssh
$ chmod 700 .ssh
$ cd .ssh
$ cat > root@hostname.pub
(paste in the public key file from the client system)
$ cat root@hostname.pub >> authorized_keys
$ chmod 600 *
Now to lock the key down, edit the ~/.ssh/authorized_keys file and put the following on the front of the key line that will be used by the client machine:
command="/usr/bin/svnserve -t -r /var/svn",no-agent-forwarding,no-pty
This forces the connection to run the "svnserve" command in tunnel mode. So this SSH key cannot be used to login or run any other commands on the server. It also changes the SVN root path to /var/svn. You will want to also add "no-port-forwarding" and "no-X11-forwarding" if you have not disabled those in your /etc/ssh/sshd_config file.
We now have a user account that can be used with SVN. Go ahead and [Ctrl-D] to escape the "su username" session and get back into the root account.
Setting up the repository (on the SVN server)
Reasons to use a common repository for all similar machines:
- Ability to do a svn diff between two different machines
- Ability to clone machines ("svn cp")
- Backup scripts are less complex (fewer repositories)
- You can "svn diff" between machines
Reasons to use individual repositories
- Easy to secure using chmod/chown
- Easy to get report sizes on how much space a machine is using on the repository server
- Easy to dump/load an individual machine's repository
- Easy to take a machine offline and remove the repository to save space
- A machine's SSH key can only be used to look at their repository (unless you configure per-directory authentication in SVN)
Note: I'm using "username", "hostname" and "machinename" fairly interchangeably on this page. (Someday I'll go back and clean it up.)
On the SVN server:
# cd /var/svn
(your repositories may be stored elsewhere)
# svnadmin create /var/svn/sys-machinename
# chmod -R 770 sys-machinename
# chmod -R g+s sys-machinename/db
# chown -R sys-machinename:sys-machinename sys-machinename
Notes:
- A chmod of "770" allows read/write access to everyone in the same group.
- A chmod of "700" only allows read/write to the owner account.
- The third octet should probably always be "0" to prevent the repository from being world-readable.
Verifying the connection (on the client)
You will need to customize the following URL to point at your SVN repository location.
# svn info svn+ssh://sys-machinename@svn.intra.example.com/sys-machinename/
That should prompt you to accept the server's public key, then display a response fron the SVN server. If things don't work, then you've got connectivity (firewall), account (wrong name? wrong key?) or permissions (chmod or chown goofs?).
Installing FSVS (on the client)
Notes:
In order for the install to succeed, you must have installed the "subversion", "subversion-devel" "apr", "apr-devel", "gcc" and "ctags" packages. Two others that you need are "gdbm" and "pcre" (and the associated developer packages). There may be other dependencies that will also be installed that are required by those packages. The following command worked for me on CentOS5:
# yum install subversion subversion-devel ctags apr apr-devel gcc gdbm gdbm-devel pcre pcre-devel
Head on over to the official project page at freshmeat.net and download the tarball. This is currently "fsvs-1.14.tar.gz". Extract the tarball to a folder somewhere (i.e. /root/fsvs-1.14.tar.gz) and use a terminal session to go to that folder.
# cat README
(look for the section that talks about the install)
# cd src
# make
(you will receive a message that the Makefile has now been updated and that you need to run make again)
# make
(you should see a large stream of gcc output with the following at the end)
-rwxr-xr-x 1 root root 201510 May 31 09:54 fsvs
(you must see the above line to know that you got a good compile)
# cp fsvs /usr/local/bin
At this point, FSVS *should* be installed correctly.
Getting started with FSVS
By default, FSVS will want to use /var/spool/fsvs unless you define the "WAA" variable and point it somewhere else. So according to the README you will need to create that folder and then initialize it (which lets FSVS create the administrative files it needs).
Note #1: I'm not sure how large /var/spool/fsvs will get. You may eventually want to break it off into a separate LVM volume of its own. Since it is mostly file lists and hashes, it shouldn't get too large, but you may want to put it on an ext3 volume with more inodes then normal.
Note #2: You should read both the README and the output of "fsvs help urls".
# mkdir -p /var/spool/fsvs
# chmod 700 /var/spool/fsvs
# cd /
# fsvs urls svn+ssh://username@machine/path/to/repos
(The "fsvs urls" won't display any confirmation text.)
The above will connect the root folder to the repository path. You could also do multiple URLs and only link sub-folders (such as /etc, /usr, /home) up against the SVN repository.
Setting up ignore filters
After telling FSVS that we want to use "/" as our working copy, we'll want to also tell it to ignore various directories and files. While this tends to be somewhat similar across Linux distributions, you should also plan on modifying this list to match your distribution.
Make sure you read "fsvs-*/doc/IGNORING" and the output of "# fsvs help ignore".
# fsvs ignore ./backup
# fsvs ignore ./dev
# fsvs ignore ./mnt
# fsvs ignore './proc/*'
# fsvs ignore ./sys
# fsvs ignore ./tmp
# fsvs ignore ./var/tmp
# fsvs ignore ./var/spool
In addition you may wish to initially ignore all of the binary file directories (such as ./lib, ./lib64, ./sbin, ./usr, ./var) and focus solely on /etc, /home, /boot and /root. That will give you a much slimmer listing when you "fsvs status" from the root directory.
You can use the "fsvs ignore dump" and "fsvs ignore load" commands to backup your listing, edit it, then load it back into FSVS. Note that you must be in the base directory of your working copy, otherwise "fsvs ignore dump" will return an empty listing.
# cd /
# fsvs ignore dump > ~/fsvs-ignore.txt
# vi ~/fsvs-ignore.txt
# sort ~/fsvs-ignore.txt | fsvs ignore load
My initial listing on CentOS5 is:
./backup
./dev
./lost+found
./media
./mnt
./proc/*
./selinux
./sys
./tmp
./var/named/chroot/proc
./var/spool
./var/tmpPutting /etc under version control
Assuming that you setup FSVS where "/" (root) is the base of the working copy, we can now add the contents of /etc to SVN.
# cd /
# fsvs commit -m "Base check-in of /etc" /etc
# fsvs commit -m "Base check-in of /boot" /boot
If you want to deep-commit a single folder (such as /usr/local/sbin) without doing the intervening folders:
# cd /
# fsvs commit -m "Base check-in" /usr/local/sbin
Working with FSVS
Create a test file in /etc
# cat > /etc/testfile.txt
foo
# fsvs commit -m "Checking in a test file" /etc/testfile.txt
Now delete the test file
# rm testfile.txt
# fsvs status .
D... 4 ./etc/testfile
.mC. dir ./etc
# fsvs commit -m "Removed test file" .
Committing to svn+ssh://sys-fw1-pri@svn.example.com/sys-fw1-pri
.mC. dir ./etc
D... 4 ./etc/testfile
committed revision 7 on 2007-06-04T00:17:13.808327Z as sys-fw1-priThat just scratches the surface, but covers the majority of day-to-day use. Unlike SVN, FSVS knows (assumes) that when a file is missing that it should implicity do a "delete" operation in the repository to make the repository match the file system.
Other useful commands to know are "fsvs unversion" and "fsvs diff".
Labels: 2007, CentOS5, FSVS, SubVersion, SystemAdministration
Tuesday, May 29, 2007
Remote install of CentOS5 using VNC
I haven't tried this yet, but there are times when it could come in handy.
Centos 5 vnc remote installation
Post details: Upgrading to CentOS4, over a remote vnc connection
I figured it was possible, I just hadn't looked.
Centos 5 vnc remote installation
Post details: Upgrading to CentOS4, over a remote vnc connection
I figured it was possible, I just hadn't looked.
Sunday, May 27, 2007
Squid, SELinux and using a separate volume for the cache_dir
This was a slightly tricky one. I'm running CentOS5 with SELinux and I was trying to setup Squid to put its cache_dir on a LVM volume (to keep it from using up space on the root partition).
# /etc/init.d/squid stop
# cd /var/spool
# lvcreate -L64G -nvar-spool-squid vg
# mke2fs -j /dev/vg/var-spool-squid
# mkdir /mnt/squid ; mount /dev/vg/var-spool-squid squid
# cp -a /var/spool/squid/* /mnt/squid/
# cd /var/spool/squid
# rm -rf *
# cd /var/spool
# mount /dev/vg/var-spools-squid squid
# /etc/init.d/squid start
Starting squid: /etc/init.d/squid: line 53: 9440 Aborted $SQUID $SQUID_OPTS >>/var/log/squid/squid.out 2>&1
[FAILED]
# tail /var/log/messages
May 27 21:50:48 fw1-hosho setroubleshoot: SELinux is preventing /usr/sbin/named (named_t) "write" access to named (named_conf_t). For complete SELinux messages. run sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
May 27 22:39:26 fw1-hosho squid: cache_dir /var/spool/squid: (13) Permission denied
...
So, the problem is that SELinux had not yet been told to look at the newly created volume (a LVM volume mounted on /var/spool/squid). Fixing this was rather simple once you know about the restorecon command.
# cd /var/spool/squid
# /usr/sbin/squid -z
# /sbin/restorecon -R *
# /etc/init.d/squid start
# /etc/init.d/squid stop
# cd /var/spool
# lvcreate -L64G -nvar-spool-squid vg
# mke2fs -j /dev/vg/var-spool-squid
# mkdir /mnt/squid ; mount /dev/vg/var-spool-squid squid
# cp -a /var/spool/squid/* /mnt/squid/
# cd /var/spool/squid
# rm -rf *
# cd /var/spool
# mount /dev/vg/var-spools-squid squid
# /etc/init.d/squid start
Starting squid: /etc/init.d/squid: line 53: 9440 Aborted $SQUID $SQUID_OPTS >>/var/log/squid/squid.out 2>&1
[FAILED]
# tail /var/log/messages
May 27 21:50:48 fw1-hosho setroubleshoot: SELinux is preventing /usr/sbin/named (named_t) "write" access to named (named_conf_t). For complete SELinux messages. run sealert -l 663ea169-d194-4c49-a5bb-a6a4bb707990
May 27 22:39:26 fw1-hosho squid: cache_dir /var/spool/squid: (13) Permission denied
# /usr/bin/sealert -l 626e75b4-32aa-4a61-88f7-f36a68fecd35
Summary
SELinux is preventing access to files with the label, file_t.
Detailed Description
SELinux permission checks on files labeled file_t are being denied. file_t
is the context the SELinux kernel gives to files that do not have a label.
This indicates a serious labeling problem. No files on an SELinux box should
ever be labeled file_t. If you have just added a new disk drive to the
system you can relabel it using the restorecon command. Otherwise you
should relabel the entire files system.
Allowing Access
You can execute the following command as root to relabel your computer
system: "touch /.autorelabel; reboot"
Additional Information
Source Context user_u:system_r:squid_t
Target Context user_u:object_r:file_t
Target Objects /var/spool/squid/00 [ dir ]
Affected RPM Packages squid-2.6.STABLE6-4.el5 [application]
Policy RPM selinux-policy-2.4.6-30.el5
Selinux Enabled True
Policy Type targeted
MLS Enabled True
Enforcing Mode Enforcing
Plugin Name plugins.file
Host Name fw1-hosho.intra.example.com.
Platform Linux fw1-hosho.intra.example.com. 2.6.18-8.1.4.el5
#1 SMP Thu May 17 03:16:52 EDT 2007 x86_64 x86_64
Alert Count 10
Line Numbers
Raw Audit Messages
avc: denied { getattr } for comm="squid" dev=dm-0 egid=23 euid=23
exe="/usr/sbin/squid" exit=-13 fsgid=23 fsuid=23 gid=23 items=0 name="00"
path="/var/spool/squid/00" pid=9584 scontext=user_u:system_r:squid_t:s0 sgid=23
subj=user_u:system_r:squid_t:s0 suid=0 tclass=dir
tcontext=user_u:object_r:file_t:s0 tty=(none) uid=23...
So, the problem is that SELinux had not yet been told to look at the newly created volume (a LVM volume mounted on /var/spool/squid). Fixing this was rather simple once you know about the restorecon command.
# cd /var/spool/squid
# /usr/sbin/squid -z
# /sbin/restorecon -R *
# /etc/init.d/squid start
Friday, May 25, 2007
iSCSITarget on CentOS5
Setting up our test iSCSI SAN box this week. The original plans were to run this on top of Gentoo (which is very powerful and flexible) but after 3 years, I'm not very pleased with Gentoo as a server OS. Which is a whole different topic. So we've migrated over to using CentOS5, which is derived from Red Hat Enterprise Linux 5, a distro that is more suited for corporate use.
There's not much to talk about in terms of the base system. It's a pretty vanilla 64bit CentOS5 install (from DVD) running on top of a dual-CPU dual-core pair of Socket F Opterons. The primary packages that I've installed so far are "Yum Extender" (from stock repositories) and "rdiff-backup" (downloaded as an RPM). The OS runs on top of a 3-disk RAID1 (mirror, all drives active) Software RAID for safety.
I use a semi-customized partition layout on the (3) operating system disks. I have:
a) /boot
b) / (root, the primary OS install area)
c) swap
d) a backup root partition (which is basically a clone of the primary, except for a small change in /etc/fstab) designed for quick recovery from a situation that would hose the primary root partition
e) /var/log (broken out to its own area)
f) /backup/system (a place to store system backups)
g) LVM area (no allocated areas yet)
I mention all that because the first step before installing iscsitarget is to make sure I can recover if things go awry. Since installing iscsitarget involves mucking with the running kernel, I want a good backup of /boot along with making sure GRUB offers me options to boot an older kernel. I'll also freshen my root backup partition.
Step 1 - Backing up /, /boot, and the existing kernel
Simplicity is often best when dealing with the base OS. My methods are crude, but designed to get me back up and running without needing much in the way of software. The primary requirement is a bootable USB pen drive or bootable LiveCD (such as RIPLinuX) with the necessary tools. You could also use the CentOS5 boot DVD.
I'll run with the CentOS install DVD since that's what I have sitting in the optical drive at the moment. When CentOS boots up, enter "linux rescue" at the boot prompt. Note, if you have multiple NICs installed, it's probably better to not start networking (because the CentOS rescue mode takes forever to initialize unconnected NICs).
Select "Skip" when asked about mounting the existing install at /mnt/sysimage. We'll be doing things our own way instead.
Start up Software RAID on the key partitions (/boot, /, the backup /, and the backup partition). The following commands will (usually) startup your existing RAID devices automatically.
# mdadm --examine --scan >> /etc/mdadm.conf
# mdadm --assemble --scan
In my case "md0" is /boot, "md2" is my base CentOS install, "md3" is the backup root partition, and "md5" is where I can store image files. So let's double-check that.
# mkdir /mnt/root ; mount /dev/md3 /mnt/root
# mkdir /mnt/backuproot ; mount /dev/md3 /mnt/backuproot
If we then examine the output of "df -h" or by using "ls" on the mounted volumes we can verify that we know which is which. Let's mount our backup area and create image files. I prefer to kick off the "dd" commands in the background so that I can monitor progress and keep multiple CPUs busy.
# mkdir /mnt/backup ; mount /dev/md5 /mnt/backup
# cd /mnt/backup ; mkdir images ; cd images
# dd if=/dev/md0 | gzip > dd-md0-boot-20070525.img.gz &
# dd if=/dev/md2 | gzip > dd-md2-root-20070525.img.gz &
# dd if=/dev/md3 | gzip > dd-md3-bkproot-20070525.img.gz &
We should also backup the master boot records on each of the hard drives in the unit.
# for i in a b c; do dd if=/dev/sd$i count=1 bs=512 of=dd-sd$i-mbr-20070525.img; done
Unfortunately, the CentOS5 DVD doesn't include tools like "G4L" (Ghost for Linux) or I'd make a second set of backup files using that. I may boot my RIPLinuX CD and see what tools are there. (Because you can never have too many backups.)
Now I can dump the contents of "md2" (the original root) to "md3" (our backup root).
# dd if=/dev/md2 of=/dev/md3
Now for some cleanup stuff...
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/fstab
We'll need to change any references of "md2" to "md3". Basically flip them around so that "md3" is the official root when /etc/fstab gets processed. I also like to change the prompt and system name to remind myself that I'm using the emergency system. Again, our primary goal is to be able to get a box back up and operational in the case where the primary root partition is hosed. Get it up quickly, then schedule some downtime to deal with it properly.
Now would also be a good time to tune the ext3 file system on your partitions.
The last thing we need to do is edit GRUB's configuration so that we can select our backup root OS from the selection menu.
# mkdir /mnt/boot
# mount /dev/md0 /mnt/boot
# vi /mnt/boot/grub/grub.conf
Things that we'll want to do here (you could also accomplish this by booting the server in normal mode and editing grub.conf there using a more comfortable text editor):
a) Change the timeout=5 value to timeout=15 (or 30 or 60). By default, CentOS doesn't give you very long to pick an alternate boot. I find 5 seconds to be too short of a window, especially on a unit where the storage controller takes a minute or two to scan and setup the drives.
b) Copy the latest "title" section and change "root=/dev/md2" to "root=/dev/md3". I always make the "EMERGENCY" boot option the 2nd one in the list.
# mkdir /mnt/backuproot
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/sysconfig/network
I like to change the hostname to have "-emergency" tacked onto the end. Which should make it fairly obvious that we are booting up in emergency mode using the backup root partition. I also edit root's .bash_profile to set PS1.
Okay, that was a lot of setup work just to prepare for implementing iSCSITarget (or any other kernel rebuild), but it's always worth it.
Final notes:
- When I test booted the emergency root partition, things didn't work as planned. So while my concept is sound, I may have screwed something up. I think it's an error with /etc/fstab in the emergency partition, so I'll troubleshoot that later.
- It's also possible that you'll need to do a GRUB install on all (3) of the primary mirror disks.
Step 2 - Downloading and compiling the iSCSITarget software
So far, I've found (2) links to be useful here. One is Moving on.... and the other is iSCSI Enterprise Target を CentOS5 にインストールする (japanese). While the 2nd link is in Japanese, it shows the commands in english.
Head over to the The iSCSI Enterprise Target page and download the latest tarball containing the source code. The current version is 0.4.15. If you're using Firefox in CentOS's Gnome shell, it will probably prompt you to open the file with the archive manager. I created a subfolder under /root/iscsitarget-0.4.15 and extracted the contents there.
You will also need to go into Applications -> Add/Remove Software and add the development tools and libraries to your system. (Mostly you just need gcc.)
As noted on jackshck's page, you will also need to install the following packages:
openssl-devel (I installed the x86_64 version)
kernel-devel (again, I'm using the x86_64 version)
Open up a terminal window and go to where you extracted the iscsitarget tarball (I put mine in /root/iscsitarget-0.4.15).
# ls -l /usr/src/kernels
(make note of the kernel folder)
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/ install
Now we can start up the ietd daemon:
# /etc/init.d/iscsi-target start
And add it to our default runlevel (this is similar to the rc-update command in Gentoo Linux):
# chkconfig iscsi-target on
Step 3 - Creating a target
This is where we get into the nitty-gritty and where I need to take a break and do some research. The /etc/ietd.conf file already exists at this point, but only contains a commented out sample configuration.
Notes:
Dec 21 2007 - The comment about iSCSITarget software for Microsoft Windows really isn't on-topic. But I'll go ahead and list the link to it, but not as an HTML link. Pricing for the real version is currently $395 (Server) or $995 (Professional). And personally, there's no way that I'd recommend running a SAN on top of Microsoft Windows (even Server 2003, which is a nice product).
There's not much to talk about in terms of the base system. It's a pretty vanilla 64bit CentOS5 install (from DVD) running on top of a dual-CPU dual-core pair of Socket F Opterons. The primary packages that I've installed so far are "Yum Extender" (from stock repositories) and "rdiff-backup" (downloaded as an RPM). The OS runs on top of a 3-disk RAID1 (mirror, all drives active) Software RAID for safety.
I use a semi-customized partition layout on the (3) operating system disks. I have:
a) /boot
b) / (root, the primary OS install area)
c) swap
d) a backup root partition (which is basically a clone of the primary, except for a small change in /etc/fstab) designed for quick recovery from a situation that would hose the primary root partition
e) /var/log (broken out to its own area)
f) /backup/system (a place to store system backups)
g) LVM area (no allocated areas yet)
I mention all that because the first step before installing iscsitarget is to make sure I can recover if things go awry. Since installing iscsitarget involves mucking with the running kernel, I want a good backup of /boot along with making sure GRUB offers me options to boot an older kernel. I'll also freshen my root backup partition.
Step 1 - Backing up /, /boot, and the existing kernel
Simplicity is often best when dealing with the base OS. My methods are crude, but designed to get me back up and running without needing much in the way of software. The primary requirement is a bootable USB pen drive or bootable LiveCD (such as RIPLinuX) with the necessary tools. You could also use the CentOS5 boot DVD.
I'll run with the CentOS install DVD since that's what I have sitting in the optical drive at the moment. When CentOS boots up, enter "linux rescue" at the boot prompt. Note, if you have multiple NICs installed, it's probably better to not start networking (because the CentOS rescue mode takes forever to initialize unconnected NICs).
Select "Skip" when asked about mounting the existing install at /mnt/sysimage. We'll be doing things our own way instead.
Start up Software RAID on the key partitions (/boot, /, the backup /, and the backup partition). The following commands will (usually) startup your existing RAID devices automatically.
# mdadm --examine --scan >> /etc/mdadm.conf
# mdadm --assemble --scan
In my case "md0" is /boot, "md2" is my base CentOS install, "md3" is the backup root partition, and "md5" is where I can store image files. So let's double-check that.
# mkdir /mnt/root ; mount /dev/md3 /mnt/root
# mkdir /mnt/backuproot ; mount /dev/md3 /mnt/backuproot
If we then examine the output of "df -h" or by using "ls" on the mounted volumes we can verify that we know which is which. Let's mount our backup area and create image files. I prefer to kick off the "dd" commands in the background so that I can monitor progress and keep multiple CPUs busy.
# mkdir /mnt/backup ; mount /dev/md5 /mnt/backup
# cd /mnt/backup ; mkdir images ; cd images
# dd if=/dev/md0 | gzip > dd-md0-boot-20070525.img.gz &
# dd if=/dev/md2 | gzip > dd-md2-root-20070525.img.gz &
# dd if=/dev/md3 | gzip > dd-md3-bkproot-20070525.img.gz &
We should also backup the master boot records on each of the hard drives in the unit.
# for i in a b c; do dd if=/dev/sd$i count=1 bs=512 of=dd-sd$i-mbr-20070525.img; done
Unfortunately, the CentOS5 DVD doesn't include tools like "G4L" (Ghost for Linux) or I'd make a second set of backup files using that. I may boot my RIPLinuX CD and see what tools are there. (Because you can never have too many backups.)
Now I can dump the contents of "md2" (the original root) to "md3" (our backup root).
# dd if=/dev/md2 of=/dev/md3
Now for some cleanup stuff...
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/fstab
We'll need to change any references of "md2" to "md3". Basically flip them around so that "md3" is the official root when /etc/fstab gets processed. I also like to change the prompt and system name to remind myself that I'm using the emergency system. Again, our primary goal is to be able to get a box back up and operational in the case where the primary root partition is hosed. Get it up quickly, then schedule some downtime to deal with it properly.
Now would also be a good time to tune the ext3 file system on your partitions.
The last thing we need to do is edit GRUB's configuration so that we can select our backup root OS from the selection menu.
# mkdir /mnt/boot
# mount /dev/md0 /mnt/boot
# vi /mnt/boot/grub/grub.conf
Things that we'll want to do here (you could also accomplish this by booting the server in normal mode and editing grub.conf there using a more comfortable text editor):
a) Change the timeout=5 value to timeout=15 (or 30 or 60). By default, CentOS doesn't give you very long to pick an alternate boot. I find 5 seconds to be too short of a window, especially on a unit where the storage controller takes a minute or two to scan and setup the drives.
b) Copy the latest "title" section and change "root=/dev/md2" to "root=/dev/md3". I always make the "EMERGENCY" boot option the 2nd one in the list.
# mkdir /mnt/backuproot
# mount /dev/md3 /mnt/backuproot
# vi /mnt/backuproot/etc/sysconfig/network
I like to change the hostname to have "-emergency" tacked onto the end. Which should make it fairly obvious that we are booting up in emergency mode using the backup root partition. I also edit root's .bash_profile to set PS1.
Okay, that was a lot of setup work just to prepare for implementing iSCSITarget (or any other kernel rebuild), but it's always worth it.
Final notes:
- When I test booted the emergency root partition, things didn't work as planned. So while my concept is sound, I may have screwed something up. I think it's an error with /etc/fstab in the emergency partition, so I'll troubleshoot that later.
- It's also possible that you'll need to do a GRUB install on all (3) of the primary mirror disks.
Step 2 - Downloading and compiling the iSCSITarget software
So far, I've found (2) links to be useful here. One is Moving on.... and the other is iSCSI Enterprise Target を CentOS5 にインストールする (japanese). While the 2nd link is in Japanese, it shows the commands in english.
Head over to the The iSCSI Enterprise Target page and download the latest tarball containing the source code. The current version is 0.4.15. If you're using Firefox in CentOS's Gnome shell, it will probably prompt you to open the file with the archive manager. I created a subfolder under /root/iscsitarget-0.4.15 and extracted the contents there.
You will also need to go into Applications -> Add/Remove Software and add the development tools and libraries to your system. (Mostly you just need gcc.)
As noted on jackshck's page, you will also need to install the following packages:
openssl-devel (I installed the x86_64 version)
kernel-devel (again, I'm using the x86_64 version)
Open up a terminal window and go to where you extracted the iscsitarget tarball (I put mine in /root/iscsitarget-0.4.15).
# ls -l /usr/src/kernels
(make note of the kernel folder)
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/
# make KSRC=/usr/src/kernels/2.6.18-8.1.4.el5-x86_64/ install
Now we can start up the ietd daemon:
# /etc/init.d/iscsi-target start
And add it to our default runlevel (this is similar to the rc-update command in Gentoo Linux):
# chkconfig iscsi-target on
Step 3 - Creating a target
This is where we get into the nitty-gritty and where I need to take a break and do some research. The /etc/ietd.conf file already exists at this point, but only contains a commented out sample configuration.
Notes:
Dec 21 2007 - The comment about iSCSITarget software for Microsoft Windows really isn't on-topic. But I'll go ahead and list the link to it, but not as an HTML link. Pricing for the real version is currently $395 (Server) or $995 (Professional). And personally, there's no way that I'd recommend running a SAN on top of Microsoft Windows (even Server 2003, which is a nice product).
Labels: 2007, Backups, CentOS5, iSCSITarget, SoftwareRAID
Monday, May 07, 2007
Brute force disaster recovery for CentOS5
Today's trick is moving a CentOS5 system from an old set of disks over to a new set of disks. Along the way, I'll create an image of the system to allow me to restore it later on.
The CentOS5 system is a fresh install running RAID-1 across (3) disks using Linux Software RAID (mdadm). There are (4) primary partitions (boot, root, swap, LVM) with no data on the LVM partition.
(Why 3 active disks? The normal setup for this server was RAID-1 across 2 disks with a hot-spare. Rather then have a risky window of time where one disk has failed and the hot-spare is synchronizing with the remaining good disk, I prefer to have all 3 disks running. That way, when a disk dies, we still have 2 disks in action. The mdadm / Software RAID doesn't seem to care and it doesn't seem to affect performance at all.)
Because this is RAID-1, capturing the disk layout and migrating over to the new disks will be very easy. It's also a very fresh install, so I'm just going to grab the disk contents using "dd" (most of the partition's sectors are still zeroed out from the original install). Once I've backed up the (3) partitions on the first drive, I'm going to pull the (3) drives and replace them with the new ones.
I'll get the machine up and running with the first replacement drive, then configure the blank 2nd and 3rd drives and add them to the RAID set. That is, if mdadm doesn't beat me to the punch and start the sync on the 2nd/3rd disks automatically.
If things go bad, I can always drop the original disks back in the unit and power it back up. I plan on keeping them around for a few days, just in case. I'll have to recreate the LVM volumes, but there aren't any yet (just a PV and a VG).
One advantage of pulling the old drives out completely and rebuilding using fresh drives - I'll end up with a tested disaster recovery process.
Now for the nitty gritty. I'm using a USB pocket drive formatted with ext3 for the rescue work. Make sure that you plug this in before booting the rescue CD.
This should give you a command shell with useful tools. So let's poke around and check on our system.
Naturally, we should create a sub-folder under /backup for each machine and possibly create another folder underneath it using today's date. We should grab information about the current disk layout and store it in a text file (fdisk.txt).
Now to grab the boot loader and image the two critical partitions (boot and root). We'll grab the boot loader off of all (3) drives because it's so small (and it may not be properly synchronized).
Total disk space for my system was around 1.75GB worth of compressed files (8GB root, 250MB boot). You could also use bzip2 if you need more compression. Unfortunately, the CentOS5 DVD does not include the "split" command, which could cause issues if you're trying to write to a filesystem that can't handle files over 2GB in size.
Now you should shut the box back down, burn those files to DVD-R, install the new (blank) disks, and boot from the install DVD again. Again, mount the drive that holds the rescue image files to a suitable path.
That will restore the MBR and partition table from the old drive to the new one. If your new drive has a different size, then the last partition will be incorrectly sized for the disk. Fire up "fdisk" and delete / recreate the last partition on the disk.
Restore the two partition images:
At this point, we should be able to boot the system on the primary drive and have Software RAID come up in degraded mode for the arrays. Things that will need to be done once the unit boots:
Getting the Software RAID back up and happy is the trickiest of the steps.
Note: If your triple mirror RAID array puts the additional disks in as spares, make sure that you have (a) grown the number of raid devices to 3 for the RAID1 set and (b) make sure that there are no other arrays synchronizing as the same time. It's also best to add the elements one at a time, rather then adding both at the same time. I'm not sure if it's a bug in mdadm or just the way it works, but it took me two tries to get my triple mirror back up with all disks marked as "active" instead of (2) active and (1) hot-spare.
The CentOS5 system is a fresh install running RAID-1 across (3) disks using Linux Software RAID (mdadm). There are (4) primary partitions (boot, root, swap, LVM) with no data on the LVM partition.
(Why 3 active disks? The normal setup for this server was RAID-1 across 2 disks with a hot-spare. Rather then have a risky window of time where one disk has failed and the hot-spare is synchronizing with the remaining good disk, I prefer to have all 3 disks running. That way, when a disk dies, we still have 2 disks in action. The mdadm / Software RAID doesn't seem to care and it doesn't seem to affect performance at all.)
Because this is RAID-1, capturing the disk layout and migrating over to the new disks will be very easy. It's also a very fresh install, so I'm just going to grab the disk contents using "dd" (most of the partition's sectors are still zeroed out from the original install). Once I've backed up the (3) partitions on the first drive, I'm going to pull the (3) drives and replace them with the new ones.
I'll get the machine up and running with the first replacement drive, then configure the blank 2nd and 3rd drives and add them to the RAID set. That is, if mdadm doesn't beat me to the punch and start the sync on the 2nd/3rd disks automatically.
If things go bad, I can always drop the original disks back in the unit and power it back up. I plan on keeping them around for a few days, just in case. I'll have to recreate the LVM volumes, but there aren't any yet (just a PV and a VG).
One advantage of pulling the old drives out completely and rebuilding using fresh drives - I'll end up with a tested disaster recovery process.
Now for the nitty gritty. I'm using a USB pocket drive formatted with ext3 for the rescue work. Make sure that you plug this in before booting the rescue CD.
- Login to the system and power it down.
- Boot the CentOS5 install DVD
- At the "boot:" enter "linux rescue"
- Work your way through the startup dialogs
- When prompted whether to mount your linux install, choose "Skip"
This should give you a command shell with useful tools. So let's poke around and check on our system.
- Looking at "cat /proc/mdstat" shows that while the mdadm software is running, it has not assembled any RAID arrays.
- The "fdisk -l" command shows us that the (3) existing disks are named sda, sdb, sdc. Each has (4) partitions (boot, root, swap, LVM).
- My USB drive showed up as "/dev/sdd" so I'll create a "/backup" folder and mount it using "mkdir /backup ; mount /dev/sdd1 /backup ; df -h"
Naturally, we should create a sub-folder under /backup for each machine and possibly create another folder underneath it using today's date. We should grab information about the current disk layout and store it in a text file (fdisk.txt).
- # cd /backup ; mkdir machinename ; cd machinename
- # mkdir todaysdate ; cd todaysdate
- # fdisk -l > fdisk.txt
Now to grab the boot loader and image the two critical partitions (boot and root). We'll grab the boot loader off of all (3) drives because it's so small (and it may not be properly synchronized).
- dd if=/dev/sda bs=512 count=1 of=machinename-date-sda.mbr
- dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdb.mbr
- dd if=/dev/sdb bs=512 count=1 of=machinename-date-sdc.mbr
- dd if=/dev/sda1 | gzip > machinename-date-ddcopy-sda1.img.gz
- dd if=/dev/sda2 | gzip > machinename-date-ddcopy-sda2.img.gz
Total disk space for my system was around 1.75GB worth of compressed files (8GB root, 250MB boot). You could also use bzip2 if you need more compression. Unfortunately, the CentOS5 DVD does not include the "split" command, which could cause issues if you're trying to write to a filesystem that can't handle files over 2GB in size.
Now you should shut the box back down, burn those files to DVD-R, install the new (blank) disks, and boot from the install DVD again. Again, mount the drive that holds the rescue image files to a suitable path.
- dd of=/dev/sda bs=512 count=1 if=machinename-date-sda.mbr
- fdisk /dev/sda (fix the last partition)
- dd if=/dev/sda bs=512 count=1 of=/dev/sdb
- dd if=/dev/sda bs=512 count=1 of=/dev/sdc
That will restore the MBR and partition table from the old drive to the new one. If your new drive has a different size, then the last partition will be incorrectly sized for the disk. Fire up "fdisk" and delete / recreate the last partition on the disk.
Restore the two partition images:
- # gzip -dc machinename-date-ddcopy-sda1.img.gz | dd of=/dev/sda1
- # gzip -dc machinename-date-ddcopy-sda2.img.gz | dd of=/dev/sda2
At this point, we should be able to boot the system on the primary drive and have Software RAID come up in degraded mode for the arrays. Things that will need to be done once the unit boots:
- Tell mdadm about the 2nd (and 3rd) disks and tell it to bring those partitions into the arrays and synchronize them.
- Create a new swap area
- Recreate the LVM physical volume (PV) and volume group (VG)
- Restore any data from the LVM area (we had none in this example)
Getting the Software RAID back up and happy is the trickiest of the steps.
- Login as root, open up a terminal window
- # cat /proc/mdstat
- # fdisk -l
- Notice that the swap area on our system is sda3, sdb3, sdc3 and will need to be loaded as /dev/md1.
- # mdadm --create /dev/md1 -v --level=raid1 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc3
- # mkswap /dev/md1 ; swapon /dev/md1
- Now we're ready to recreate the LVM area
- # mknod /dev/md3 b 9 3
- # mdadm --create /dev/md3 -v --level=raid1 --raid-devices=3 /dev/sda4 /dev/sdb4 /dev/sdc4
- # pvcreate /dev/md3 ; vgcreate vg /dev/md3
- Finally, we should add the 2nd and 3rd drive to md0 and md2.
- mdadm --add /dev/md0 /dev/sdc1
- mdadm --add /dev/md0 /dev/sdb1
Note: If your triple mirror RAID array puts the additional disks in as spares, make sure that you have (a) grown the number of raid devices to 3 for the RAID1 set and (b) make sure that there are no other arrays synchronizing as the same time. It's also best to add the elements one at a time, rather then adding both at the same time. I'm not sure if it's a bug in mdadm or just the way it works, but it took me two tries to get my triple mirror back up with all disks marked as "active" instead of (2) active and (1) hot-spare.
Labels: 2007, Backups, CentOS5, DisasterRecovery, SoftwareRAID
Sections:
Recent posts:
[an error occurred while processing this directive]
Monthly archives:
2003/04
2003/05
2003/06
2003/07
2003/08
2003/09
2003/10
2003/11
2003/12
2004/01
2004/02
2004/03
2004/04
2004/05
2004/06
2004/07
2004/08
2004/09
2004/10
2004/11
2004/12
2005/01
2005/02
2005/03
2005/04
2005/05
2005/06
2005/07
2005/08
2005/09
2005/10
2005/11
2005/12
2006/01
2006/02
2006/03
2006/04
2006/05
2006/06
2006/07
2006/08
2006/09
2006/10
2006/11
2006/12
2007/01
2007/02
2007/03
2007/04
2007/05
2007/06
2007/07
2007/08
2007/12
2008/01
2008/03
2008/04
2008/05
2008/06
2008/07
2008/08
2008/09
2008/10
2008/11
2008/12
2009/01
2009/02
2009/03
2009/04
2009/05
2009/06
2009/07
2009/08
2009/09
2009/10
2009/11
2009/12
2010/06
Copyright 2003-2010, Thomas G. Harold, All Rights Reserved
