Monday, July 02, 2007
LVM and SELinux
I was a bit perplexed... I had created a LV called /dev/vg/svn, had it mounted, was reading/writing data to it with no issues. But after I rebooted the CentOS5 server, I'm unable to mount the LV.
So lvdisplay knows that the LV is there, but only if I tell it to look at the VG named "vg".
...
Turns out that it's an SELinux issue. Because SELinux was blocking access to the /etc/lvm/.cache file, it was causing problems. Fixing it was as simple as:
[root@localhost /]# /usr/sbin/pvscan
PV /dev/md6 VG vg lvm2 [144.78 GB / 59.78 GB free]
Total: 1 [144.78 GB] / in use: 1 [144.78 GB] / in no VG: 0 [0 ]
[root@localhost /]# /usr/sbin/vgscan
Reading all physical volumes. This may take a while...
Found volume group "vg" using metadata type lvm2
[root@localhost /]# /usr/sbin/lvscan
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay
No volume groups found
[root@localhost /]# /usr/sbin/lvdisplay vg
--- Logical volume ---
LV Name /dev/vg/svn
VG Name vg
LV UUID taYjia-BWWs-IWG3-313k-VoC2-ghik-01mFCg
LV Write Access read/write
LV Status NOT available
LV Size 85.00 GB
Current LE 21760
Segments 1
Allocation inherit
Read ahead sectors 0
[root@localhost /]#
So lvdisplay knows that the LV is there, but only if I tell it to look at the VG named "vg".
...
Turns out that it's an SELinux issue. Because SELinux was blocking access to the /etc/lvm/.cache file, it was causing problems. Fixing it was as simple as:
# cd /etc/lvm
# /sbin/restorecon -v .cache
# /usr/sbin/lvscan
inactive '/dev/vg/svn' [85.00 GB] inherit
Saturday, June 19, 2004
Gentoo: Segmentation fault in vgscan during boot
Now for the other error that I got during the initial bootup.
My initial guess is that the software RAID is not loading up prior to the LVM stuff trying to load. Possibly, I'll have to edit the ordering in "/etc/init.d/checkfs", however since RAID is compiled into the kernel as built-in, and the LVM stuff is a module, the RAID should've already started prior to the LVM stuff.
Looking closer at the boot screen, I can see the "md:" lines correctly autodetecting the RAID arrays. So RAID support seems to be working fine. In fact, if I login to maintenance mode, and "cat /proc/mdstat", all of the RAID arrays show up correctly.
Attempt #1:
Moved things around in /etc/init.d/checkfs. No change in the end-result, except that the messages are re-ordered ("Starting up RAID devices" now appears before "Setting up the Logical Volume Manager") and the error message changes to "4422 Segmentation Fault /sbin/vgscan >/dev/nul". Probably a dry-hole in terms of finding and fixing the real problem.
Attempt #2:
Flipped back to the Gentoo LVM2 documents to see if I missed anything in setting up the LVM set to auto-mount at startup. Booted my way into maintenance mode and use "vgscan -v" to let vgscan attempt to find all of the volume groups. "vgscan" will take a while to run, at least with verbose (-v) mode you'll be able to see some status. On my setup, "vgscan" correctly located the "vgmirror" volume group.
Did a look at the "/etc/lvm" folder on the root volume using "ls -la /etc/lvm" and saw something surprising. There is a ".cache" file which is huge (mine was 10881785 in size). Doing a "cat" of the contents, I see some entries like "/dev/discs/disc2/discs/disc2.../disc2/md/254" which looks like a recursive loop of some sort.
Hint #2, run "vgdisplay -vv" and I see the error message "Too many levels of symbolic links" after each of those long entries. I also see this problem if I run "vgscan -vv". I finally changed my "/etc/lvm/lvm.conf" file to look like the following, and vgscan and vgdisplay are very quick at finding the volume group on my raid array and no longer segfault while looking at other items:
Note that this filter only allows vgscan to scan the "md3" device. This keeps vgscan from scanning other devices that don't need to be scanned on my system (and fixes the segfault issue where it goes into infinite recursion on certain devices). If you need to scan other RAID devices (/dev/md1, etc.) or other physical partitions, then you'll need to adjust the "accept" portion of the filter.
Save, shutdown the raid (raidstop -a /dev/md0 for each /dev/md* device), and reboot. My server now boots up correctly.
Links:
Re: mkfs.xfs on software raid5 (2.6.5 kernel) - MD array /dev/md2 not in clean state (alt.os.linux.gentoo) - Shows the exact error message that I'm seeing.
Re: [gentoo-user] LVM2 Date: 2004-03-31 15:40:13 PST (linux.gentoo.user) - Talks about the ordering in which software RAID and the LVM modules load.
Example of lvm.conf file - shows a more complex lvm.conf file, complete with multiple filters.
A more complete example lvm.conf file
* Using /etc/modules.autoload.d/kernel-2.6 as config:
* Loading module dm-mod... [ ok ]
* Autoloaded 1 module(s)
* Setting up the Logical Volume Manager...
/sbin/rc: line 429: 4422 Segmentation Fault /sbin/vgscan >/dev/nul [ ok ]
* Starting up RAID devices: ...
* Checking all filesystems...
/dev/md0: clean, 39/18072 files, 5573/72192 blocks
fsck.ext: No such file or directory while trying to open /dev/vgmirror/opt
/dev/vgmirror/opt:
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193
("No such file or directory..." error repeats for all of the other
logical volumes in the volume group(s) on the system)
My initial guess is that the software RAID is not loading up prior to the LVM stuff trying to load. Possibly, I'll have to edit the ordering in "/etc/init.d/checkfs", however since RAID is compiled into the kernel as built-in, and the LVM stuff is a module, the RAID should've already started prior to the LVM stuff.
Looking closer at the boot screen, I can see the "md:" lines correctly autodetecting the RAID arrays. So RAID support seems to be working fine. In fact, if I login to maintenance mode, and "cat /proc/mdstat", all of the RAID arrays show up correctly.
Attempt #1:
Moved things around in /etc/init.d/checkfs. No change in the end-result, except that the messages are re-ordered ("Starting up RAID devices" now appears before "Setting up the Logical Volume Manager") and the error message changes to "4422 Segmentation Fault /sbin/vgscan >/dev/nul". Probably a dry-hole in terms of finding and fixing the real problem.
Attempt #2:
Flipped back to the Gentoo LVM2 documents to see if I missed anything in setting up the LVM set to auto-mount at startup. Booted my way into maintenance mode and use "vgscan -v" to let vgscan attempt to find all of the volume groups. "vgscan" will take a while to run, at least with verbose (-v) mode you'll be able to see some status. On my setup, "vgscan" correctly located the "vgmirror" volume group.
Did a look at the "/etc/lvm" folder on the root volume using "ls -la /etc/lvm" and saw something surprising. There is a ".cache" file which is huge (mine was 10881785 in size). Doing a "cat" of the contents, I see some entries like "/dev/discs/disc2/discs/disc2.../disc2/md/254" which looks like a recursive loop of some sort.
Hint #2, run "vgdisplay -vv" and I see the error message "Too many levels of symbolic links" after each of those long entries. I also see this problem if I run "vgscan -vv". I finally changed my "/etc/lvm/lvm.conf" file to look like the following, and vgscan and vgdisplay are very quick at finding the volume group on my raid array and no longer segfault while looking at other items:
devices = {
scan=["/dev/md"]
filter=["a|^/dev/md/3$|","r/.*/"]
}Note that this filter only allows vgscan to scan the "md3" device. This keeps vgscan from scanning other devices that don't need to be scanned on my system (and fixes the segfault issue where it goes into infinite recursion on certain devices). If you need to scan other RAID devices (/dev/md1, etc.) or other physical partitions, then you'll need to adjust the "accept" portion of the filter.
Save, shutdown the raid (raidstop -a /dev/md0 for each /dev/md* device), and reboot. My server now boots up correctly.
Links:
Re: mkfs.xfs on software raid5 (2.6.5 kernel) - MD array /dev/md2 not in clean state (alt.os.linux.gentoo) - Shows the exact error message that I'm seeing.
Re: [gentoo-user] LVM2 Date: 2004-03-31 15:40:13 PST (linux.gentoo.user) - Talks about the ordering in which software RAID and the LVM modules load.
Example of lvm.conf file - shows a more complex lvm.conf file, complete with multiple filters.
A more complete example lvm.conf file
Labels: LVM
Tuesday, June 15, 2004
LVM and Software RAID
Links:
Linux Logical Volume Manager (LVM) on Software RAID - explains the benefits of using LVM, and also how to get LVM running on top of software RAID in RedHat 8.
Google search for how to do software RAID and LVM
The Software-RAID HOWTO - Updated June 2004, so it's not an old outdated document. Also see the official URL.
Gentoo Server Project Wiki
Notes on Building a Linux Storage Server
Gentoo Forums: How to do a gentoo install on a software RAID - Thread was started back in July 2002, but there are posts as recently as June 2004.
Summary:
The "grub" boot loader needs a real physical partition to boot from (or possibly a RAID1 setup, RAID0 definitely doesn't work). What some folks do is to create (2) identical physical partitions on the drives in the RAID array, and then periodically copy from the primary partition to the secondary partition. (See gentoo forum post or search for user "wrex". Also look for posts by "hover" or "blake121666") The copy command is as simple as "dd if=/dev/hda1 of=/dev/hdb1 bs=8192b". You need to customize that to match your configuration (e.g. changing the source/dest partitions and the block size). If I can puzzle it all out, I think I'm going to go with /boot on RAID1. See section 7.3 of the Software RAID HOWTO
The "swap" partition should be mirrored. Otherwise when the disk fails, existing applications with data in the swap partition of the failed drive will die a nasty death. See section 2.3 of the Software RAID HOWTO.
There's a special set of command that need to be done in "grub" if you want to be able to pull the primary disk and still have the system bootable. (In case the primary disk would fail.) This is all mentioned in the gentoo forums thread about software RAID, look for the post by "hover". Alternatively, you can simply use a boot floppy and fixup the grub settings after the primary drive fails.
Replacing a failed disk in a software RAID1 array requires partitioning the drive by hand prior to rebuilding the array. This is also mentioned in the gentoo forums thread about software RAID, look for the post by "hover".
Searching around on my Gentoo 2004.0 Universal CD, I don't see the "mdadm" tools anywhere, but I do see "raidtools".
Linux Logical Volume Manager (LVM) on Software RAID - explains the benefits of using LVM, and also how to get LVM running on top of software RAID in RedHat 8.
Google search for how to do software RAID and LVM
The Software-RAID HOWTO - Updated June 2004, so it's not an old outdated document. Also see the official URL.
Gentoo Server Project Wiki
Notes on Building a Linux Storage Server
Gentoo Forums: How to do a gentoo install on a software RAID - Thread was started back in July 2002, but there are posts as recently as June 2004.
Summary:
The "grub" boot loader needs a real physical partition to boot from (or possibly a RAID1 setup, RAID0 definitely doesn't work). What some folks do is to create (2) identical physical partitions on the drives in the RAID array, and then periodically copy from the primary partition to the secondary partition. (See gentoo forum post or search for user "wrex". Also look for posts by "hover" or "blake121666") The copy command is as simple as "dd if=/dev/hda1 of=/dev/hdb1 bs=8192b". You need to customize that to match your configuration (e.g. changing the source/dest partitions and the block size). If I can puzzle it all out, I think I'm going to go with /boot on RAID1. See section 7.3 of the Software RAID HOWTO
The "swap" partition should be mirrored. Otherwise when the disk fails, existing applications with data in the swap partition of the failed drive will die a nasty death. See section 2.3 of the Software RAID HOWTO.
There's a special set of command that need to be done in "grub" if you want to be able to pull the primary disk and still have the system bootable. (In case the primary disk would fail.) This is all mentioned in the gentoo forums thread about software RAID, look for the post by "hover". Alternatively, you can simply use a boot floppy and fixup the grub settings after the primary drive fails.
Replacing a failed disk in a software RAID1 array requires partitioning the drive by hand prior to rebuilding the array. This is also mentioned in the gentoo forums thread about software RAID, look for the post by "hover".
Searching around on my Gentoo 2004.0 Universal CD, I don't see the "mdadm" tools anywhere, but I do see "raidtools".
Labels: LVM, SoftwareRAID
Friday, April 30, 2004
Gentoo LVM2 stuff
vgscan - displays the list of volume groups allocated on the system (for my box, I have vgos, vguser, vgtmp and vgmedia)
lvscan - displays the list of virtual partitions inside of the volume groups (for my box, I have 6)
vgdisplay - displays a lot of data bout the volume groups, a good place to find out how much space is free within a particular volume group (vgos has 12GB free, vguser 19GB free, vgtmp 7GB free, vgmedia 92GB free).
Back when I setup the box, I never created any partitions (volumes?) inside the vgmedia volume group. So now I want to think about what sort of media I'm going to be storing, and how to seperate it. Since most audio/video files are larger, I may consider using a larger block size (if that makes sense?). I'm also going to leave a small amount of space for backup files to be written to from the primary drive. So for now, I'll only allocate 32GB out of the 92GB.
lvcreate -L32G -nmedia vgmedia
mke2fs -j -c /dev/vgmedia/media
(then use an editor to add an entry to the /etc/fstab table)
mkdir /media
mount /dev/vgmedia/media /media
Now I'm ready to configure Samba. See the gentoo documentation about Samba.
lvscan - displays the list of virtual partitions inside of the volume groups (for my box, I have 6)
vgdisplay - displays a lot of data bout the volume groups, a good place to find out how much space is free within a particular volume group (vgos has 12GB free, vguser 19GB free, vgtmp 7GB free, vgmedia 92GB free).
Back when I setup the box, I never created any partitions (volumes?) inside the vgmedia volume group. So now I want to think about what sort of media I'm going to be storing, and how to seperate it. Since most audio/video files are larger, I may consider using a larger block size (if that makes sense?). I'm also going to leave a small amount of space for backup files to be written to from the primary drive. So for now, I'll only allocate 32GB out of the 92GB.
lvcreate -L32G -nmedia vgmedia
mke2fs -j -c /dev/vgmedia/media
(then use an editor to add an entry to the /etc/fstab table)
mkdir /media
mount /dev/vgmedia/media /media
Now I'm ready to configure Samba. See the gentoo documentation about Samba.
Sections:
Recent posts:
Update #1 to Current frustrations with Thunderbird...
Current frustrations with Thunderbird
SELinux - dealing with exceptions
Linux RAID tuning and troubleshooting
ngg.js and fgg.js site infections
Annual Index:
Copyright 2003-2006, Thomas G. Harold, All Rights Reserved
