(I'm assuming that you've already created a working Xen Dom0 domain and that you already have a Xen DomU kernel with which you can boot the Gentoo guest OS. I'm skipping a lot of steps and doing what I think works based on previous Gentoo installs.)

Disk preparation: I'm installing the Gentoo guest OSs into LVM volumes managed by the Dom0 hypervisor domain. Each guest OS gets a single partition for root that is exported to the guest OS as /dev/sda1. In rare cases, I'm also providing a 2nd and 3rd LVM partition for the guest OS which are exported as /dev/sda2 and /dev/sda3.

xena-adele thomas # lvcreate -L4G vgmirror -n domu-svn1root
Logical volume "domu-svn1root" created
xena-adele thomas # mke2fs -j /dev/vgmirror/domu-svn1root
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
524288 inodes, 1048576 blocks
52428 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1073741824
32 block groups
32768 blocks per group, 32768 fragments per group
16384 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 20 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
xena-adele thomas # mkdir /mnt/gentoo
mkdir: cannot create directory `/mnt/gentoo': File exists
xena-adele thomas # mount /dev/vgmirror/domu-svn1root /mnt/gentoo

That takes care of the first 4 sections in the handbook. In my case, the vgmirror group is an LVM volume group backed by software RAID (RAID1+hotspare) that the Dom0 hypervisor manages using mdadm.

A 4GB volume is probably about the minimum useful size for a Gentoo install, even a stripped down server-only install like this one. It's safe enough as long as you have a monitoring job to watch disk space and/or are using an aggressive logrotate schedule.

I already had the portage and stage3 *.bz2 files downloaded from last month, so I don't need to download them. Otherwise, you should grab the latest stage3 tarball and the latest portage snapshot from the public servers. Both of these files should be placed into /mnt/gentoo.

Extraction of the 2 bz2 files is simple:

# cd /mnt/gentoo
# ls -l *.bz2
# tar xvjpf stage3-*.tar.bz2
# tar xvjf portage-*.tar.bz2 -C /mnt/gentoo/usr

That puts us at step 5e in the Gentoo Handbook. You can remove the *.bz2 files from /mnt/gentoo, but you'll probably want to back them up somewhere safe so you can reference them when building the next DomU.

Edit your make.conf file and configure the GENTOO_MIRROR=, SYNC=, and USE= lines.

Now you're ready to prepare for the chroot.

# cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
# mount -t proc none /mnt/gentoo/proc
# mount -o bind /dev /mnt/gentoo/dev
# chroot /mnt/gentoo /bin/bash
# env-update
>> Regenerating /etc/ld.so.cache...
# source /etc/profile
# export PS1="(chroot) $PS1"
# emerge --sync

That will go pretty quick if you have a local rsync mirror.

# ln -sf /usr/share/zoneinfo/EST5EDT /etc/localtime

You can skip configuration of the kernel and installing grub/lilo since both of those tasks are handled by the hypervisor domain. The only exception to this guideline is if you have kernel modules that need to be installed and loaded. This should be a rare occurance in a Xen guest domain.

The fstab should be pretty simple. Especially if you have the system configured with a single partition. In my case, /etc/fstab looks like:

/dev/sda1 / ext3 noatime 0 1
proc /proc proc defaults 0 0
shm /dev/shm tmpfs nodev,nosuid,noexec 0 0

Now for some final clean-up work:

(chroot) livecd linux # nano -w /etc/conf.d/hostname
(chroot) livecd linux # nano -w /etc/conf.d/net
config_eth7=( "192.168.142.100 netmask 255.255.255.0" )
routes_eth7=( "default gw 192.168.142.1" )
(chroot) livecd linux # cd /etc/init.d
(chroot) livecd init.d # ln -s net.lo net.eth0
(chroot) livecd init.d # rc-update add net.eth0 default
* net.eth0 added to runlevel default
* rc-update complete.
(chroot) livecd init.d # cat /etc/resolv.conf
(verify your DNS servers if you specified a static IP)
(chroot) livecd init.d # passwd
(set your root password to something you will remember)
New UNIX password:
Retype new UNIX password:
passwd: password updated successfully
# cd /
# emerge syslog-ng
# rc-update add syslog-ng default
(then update /etc/syslog-ng/syslog-ng.conf)
# emerge dcron
# rc-update add dcron default
# crontab /etc/crontab
# /usr/bin/ssh-keygen -t rsa -b 2048 -f /etc/ssh/ssh_host_rsa_key -N ""
(the key may take a a minute to generate)
# /usr/bin/ssh-keygen -t dsa -b 1024 -f /etc/ssh/ssh_host_dsa_key -N ""
# chmod 600 /etc/ssh/ssh_host_?sa_key
# chmod 644 /etc/ssh/ssh_host_?sa_key.pub
# rc-update add sshd default
# useradd -m -G users,wheel,audio -s /bin/bash username
(then put your user's public key file in their ~/.ssh folder)

Time to exit the chroot, unmount everything, and try to start the guest domain. Odds are high that you will have trouble completely dismounting /mnt/gentoo (even if you umount /mnt/gentoo/proc and /mnt/gentoo/dev first). So you'll likely have to restart the entire machine to get things clean.

(Which is why you'll only want to create the base system *once*. Then copy it for new setups.)

I strongly suggest running GNU "screen" in the Dom0 hypervisor domain. That will allow you to create a new screen to test out the install ([Ctrl-A][C] to create a new screen). Then you can simply start the new guest domain with:

# xm create -c mydomainconfigfile

You can then shutdown (and exit) the domain by logging in as root and typing "shutdown -h now". Once the guest domain seems to be working, fire it up without the "-c" option to get it running in the background.

Labels: Gentoo

Not sure what I'm going to do about this error on the AMD64 Asus M2N32-SLI Deluxe motherboard.

Aug 27 20:16:42 san1-azure irq 7: nobody cared (try booting with the "irqpoll" option)
Aug 27 20:16:42 san1-azure
Aug 27 20:16:42 san1-azure Call Trace: {__report_bad_irq+48}
Aug 27 20:16:42 san1-azure {note_interrupt+472} {__do_IRQ+183}
Aug 27 20:16:42 san1-azure {do_IRQ+57} {default_idle+0}
Aug 27 20:16:42 san1-azure {ret_from_intr+0} {default_idle+43}
Aug 27 20:16:42 san1-azure {cpu_idle+151} {start_secondary+1141}
Aug 27 20:16:42 san1-azure handlers:
Aug 27 20:16:42 san1-azure [] (usb_hcd_irq+0x0/0x54)
Aug 27 20:16:42 san1-azure Disabling IRQ #7

Putting "irqpoll" on the end of the kernel line in grub.conf causes the system to panic during boot (has to do with the 2nd core in the X2 chip).

Labels: Gentoo

Now there were a few minor things that I had to fix after the reboot before I could SSH back in. I hadn't pointed my swap line in /etc/fstab at the proper mdadm RAID volume, the 3c509 moved from eth7 to eth4 after the reboot, I had to use the "noapic" kernel option and I'm still getting the IRQ7 warning.

But the system is mostly in a workable state at this point. So it's time to start installing administration packages and cleaning up the install. I'll save the existing kernel as my "base" configuration in case I screw things up.

The first package that I always install is emerge screen. That provides me with multiple virtual terminals in my single SSH connection. Even better, if I disconnect accidentally, I don't lose my state and programs that were running within screen sessions will continue running. After reconnecting, I can type "screen -x" and reconnect to my old sessions.

Other key packages to install, even on a basic system like this one, are:

app-benchmarks/bonnie
app-editors/vim
app-misc/colordiff
app-portage/gentoolkit
app-text/tree
dev-util/subversion
net-analyzer/iptraf
net-analyzer/nettop
net-analyzer/nload
net-misc/ntp
sys-apps/dstat
sys-apps/eject
sys-apps/smartmontools
sys-process/atop

san1-azure ~ # emerge -pv bonnie vim colordiff gentoolkit tree subversion iptraf nettop nload ntp dstat eject smartmontools atop

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild N ] app-benchmarks/bonnie-2.0.6 6 kB
[ebuild N ] dev-util/ctags-5.5.4-r2 254 kB
[ebuild N ] app-editors/vim-core-7.0.17 -acl -bash-completion -livecd +nls 5,997 kB
[ebuild N ] app-editors/vim-7.0.17 -acl -bash-completion -cscope +gpm -minimal (-mzscheme) +nls +perl +python -ruby -vim-pager -vim-with-x 0 kB
[ebuild N ] app-vim/gentoo-syntax-20051221 -ignore-glep31 18 kB
[ebuild N ] app-misc/colordiff-1.0.5-r2 13 kB
[ebuild N ] app-portage/gentoolkit-0.2.2 84 kB
[ebuild N ] app-text/tree-1.5.0 -bash-completion 25 kB
[ebuild N ] dev-libs/apr-0.9.12 +ipv6 -urandom 1,024 kB
[ebuild N ] dev-libs/apr-util-0.9.12 +berkdb -gdbm -ldap 578 kB
[ebuild N ] net-misc/neon-0.26.1 +expat -gnutls +nls -socks5 +ssl -static +zlib 763 kB
[ebuild N ] dev-util/subversion-1.3.2-r1 -apache2 -bash-completion +berkdb -emacs -java +nls -nowebdav +perl +python -ruby +zlib 6,674 kB
[ebuild N ] net-analyzer/iptraf-2.7.0-r1 +ipv6 410 kB
[ebuild N ] net-libs/libpcap-0.9.4 +ipv6 415 kB
[ebuild N ] sys-libs/slang-1.4.9-r2 -cjk -unicode 628 kB
[ebuild N ] net-analyzer/nettop-0.2.3 22 kB
[ebuild N ] net-analyzer/nload-0.6.0 118 kB
[ebuild N ] net-misc/ntp-4.2.0.20040617-r3 -caps -debug +ipv6 -logrotate -openntpd -parse-clocks (-selinux) +ssl 2,403 kB
[ebuild N ] sys-apps/dstat-0.6.0-r1 35 kB
[ebuild N ] sys-apps/eject-2.1.0-r1 +nls 65 kB
[ebuild N ] mail-client/mailx-support-20030215 8 kB
[ebuild N ] net-libs/liblockfile-1.06-r1 31 kB
[ebuild N ] mail-client/mailx-8.1.2.20040524-r1 126 kB
[ebuild N ] sys-apps/smartmontools-5.36 -static 528 kB
[ebuild N ] sys-process/acct-6.3.5-r1 300 kB
[ebuild N ] sys-process/atop-1.15 102 kB

Total size of downloads: 20,638 kB
san1-azure ~ #

That should take all of about 0.1 seconds on this Athlon64 X2. (I joke, slightly... the box is quite snappy even with only 7200rpm SATA drives.)

Now I'm going to configure SubVersion. There have been some changes in that process that I learned through trial and error. Folders that I would recommend placing under version control are:

/boot (most files)
/etc (most files, especially configuration files)
/usr/local/sbin (local sysadmin scripts that you create)
/usr/src (the .config file, make sure you add the actual directory with "svn add -N" before adding the "linux" symbolic link)

I know there are other folders to add, but I typically add them on the fly as I start customizing the system.

Labels: Gentoo

This is a record of the kernel flags that I'm going to use for my AMD64 system. It's an Asus M2N32-SLI Deluxe (NVIDIA nForce 590 SLI MCP chipset) with an Athlon64 X2 4200+ chip along with 2GB of RAM. Hard drives are hooked up to the onboard SATA-II controller (NVIDIA nForce 590 SLI MCP chipset). Plus the motherboard has a pair of onboard gigabit ethernet NICs (Marvell 88E1116) and a Silicon Image Sil3132 SATA-II controller. Other chips on the motherboard are the nVidia C51XE, nVidia MCP55PXE, AD1988B, and TSB43AB22A.

In addition, I'll have even more hard drives hooked up to a HighPoint RocketRAID 2300 PCIe card. There's also a 3Com 3C905B PCI ethernet card installed along with a pair of Intel PRO/1000 PCIe gigabit NICs.

# emerge mdadm
# emerge lvm2
# cd /usr/src/linux
# make menuconfig

Linux Kernel v2.6.17-gentoo-r4 Configuration
Code maturity level options
General setup
Loadable module support
Processor type and features
--> Processor family (changed to "AMD-Opteron/Athlon64")
--> Preemption Model (No Forced Preemption (Server))
Power management options (ACPI, APM)
Bus options (PCI, etc.)
Executable file formats
Device drivers
--> ATA/ATAPI/MFM/RLL support
--> --> generic/default IDE chipset support (should already be ON)
--> --> --> ATI IXP chipset IDE support (turn OFF)
--> --> --> Intel PIIXn chipsets support (turn OFF)
--> --> --> IT821X IDE support (turn OFF)
--> SCSI device support
--> --> SCSI generic support (turn this ON)
--> --> SCSI low-level drivers
--> --> --> Serial ATA (SATA) support (should already be ON)
--> --> --> --> Intel PIIX/ICH SATA support (turn OFF)
--> --> --> --> Silicon Image SATA support (turn OFF)
--> --> --> --> Silicon Image 3124/3132 SATA support (turn ON as BUILT-IN)
--> --> --> --> VIA SATA support (turn OFF)
--> Multi-device support (should already be ON)
--> --> RAID support (turn it ON as BUILT-IN)
--> --> --> RAID-1 mirroring mode (turn it ON as BUILT-IN)
--> --> --> RAID-10 mirroring striping mode (turn it ON as BUILT-IN)
--> --> Device mapper support (turn ON as BUILT-IN)
--> Networking support
--> --> Ethernet (1000Mbit)
--> --> --> Intel. PRO/1000 Gigabit Ethernet support (turn ON)
--> --> --> Broadcom Tigon3 support (turn OFF)
--> Character Devices
--> --> Intel/AMD/VIA HW Random Number Generator (should be ON)
--> --> Intel 440LX/BX/GX, I8xx and E7x05 chipset support (turn it OFF)
--> Sound
--> --> Sound card support (turn OFF)
File systems
--> Network File Systems
--> --> SMB file system support (turn ON as BUILT-IN)
--> --> CIFS support (turn ON as BUILT-IN)
Profiling support
Kernel hacking
Security options
Cryptographic options
--> Cryptographic API (turn ON)
--> --> HMAC support (NEW) (turn ON as BUILT-IN)
--> --> (turn ON all other options as MODULE)
Library routines

Now we can compile and copy the kernel to the /boot partition.

# make && make modules_install
# ls -l /boot
# ls -l arch/x86_64/boot
# df
# cp arch/x86_64/boot/bzImage /boot/kernel-2.6.17-25Aug2006-2300
# cp System.map /boot/System.map-2.6.17-25Aug2006-2300
# cp .config /boot/config-2.6.17-25Aug2006-2300
# ls -l /boot

Next is Chapter 8, Configuring your System.

(chroot) livecd linux # nano -w /etc/fstab

My fstab (there are lines not shown):

/dev/md0                /boot           ext2            noauto,noatime  1 2        
/dev/md1                /               ext3            noatime         0 1        
/dev/md3                none            swap            sw              0 0        
/dev/cdroms/cdrom0      /mnt/cdrom      iso9660         noauto,ro       0 0
#/dev/fd0               /mnt/floppy     auto            noauto          0 0

/dev/vgmirror/home      /home                   ext3    noatime         0 3        
/dev/vgmirror/tmp       /tmp                    ext2    noatime         0 3 
/dev/vgmirror/vartmp    /var/tmp                ext2    noatime         0 3
/dev/vgmirror/log1      /var/log                ext3    noatime         0 3
/dev/vgmirror/portage   /usr/portage            ext3    noatime         0 3

/dev/vgmirror/svn       /var/svn                ext3    noatime         0 4        
/dev/vgmirror/backupsys /backup/system          ext3    noatime         0 4

Now for some final clean-up work:

(chroot) livecd linux # nano -w /etc/conf.d/hostname
(chroot) livecd linux # nano -w /etc/conf.d/net
config_eth7=( "192.168.142.100 netmask 255.255.255.0" )
routes_eth7=( "default gw 192.168.142.1" )
(chroot) livecd linux # cd /etc/init.d
(chroot) livecd init.d # ln -s net.lo net.eth7
(chroot) livecd init.d # rc-update add net.eth7 default
* net.eth7 added to runlevel default
* rc-update complete.
(chroot) livecd init.d # cat /etc/resolv.conf
(verify your DNS servers if you specified a static IP)
(chroot) livecd init.d # nano -w /etc/conf.d/clock
CLOCK_SYSTOHC="yes"
(chroot) livecd init.d # passwd
(set your root password to something you will remember)
(chroot) livecd init.d # passwd
New UNIX password:
Retype new UNIX password:
passwd: password updated successfully
(chroot) livecd init.d #
# emerge syslog-ng
# rc-update add syslog-ng default
# emerge dcron
# rc-update add dcron default
# crontab /etc/crontab
# /usr/bin/ssh-keygen -t dsa -b 2048 -f /etc/ssh/ssh_host_dsa_key -N ""
(the key may take a a minute to generate)
# chmod 600 /etc/ssh/ssh_host_dsa_key
# chmod 644 /etc/ssh/ssh_host_dsa_key.pub
# rc-update add sshd default

Now it's time for grub.

(chroot) livecd init.d # emerge grub
(chroot) livecd init.d # ls -l /boot
total 3468
-rw-r--r-- 1 root root 1090703 Aug 26 00:35 System.map-2.6.17-25Aug2006-2300
lrwxrwxrwx 1 root root 1 Aug 25 18:09 boot -> .
-rw-r--r-- 1 root root 28714 Aug 26 00:35 config-2.6.17-25Aug2006-2300
drwxr-xr-x 2 root root 1024 Aug 26 01:03 grub
-rw-r--r-- 1 root root 2397504 Aug 26 00:35 kernel-2.6.17-25Aug2006-2300
drwx------ 2 root root 12288 Aug 25 16:46 lost+found
(chroot) livecd init.d # nano -w /boot/grub/grub.conf
# Which listing to boot as default. 0 is the first, 1 the second etc.
default 0
timeout 30

# Aug 2006 Base Installation (software RAID, LVM2)
title=Gentoo Linux 2.6.17 (Aug 25 2006) BASE INSTALL
root (hd0,0)
kernel /kernel-2.6.17-25Aug2006-2300 root=/dev/md1

# Aug 2006 Base Installation (software RAID, LVM2) - NOAPIC
title=Gentoo Linux 2.6.17 (Aug 25 2006) BASE NOAPIC
root (hd0,0)
kernel /kernel-2.6.17-25Aug2006-2300 root=/dev/md1 noapic
(chroot) livecd init.d # grub --no-floppy
grub> find /grub/stage1
(hd0,0)
(hd1,0)
grub> root (hd0,0)
grub> setup (hd0)
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

Time to exit the chroot, unmount everything, and try a reboot.

livecd / # cat /proc/mounts
rootfs / rootfs rw 0 0
tmpfs / tmpfs rw 0 0
/dev/hda /mnt/cdrom iso9660 ro 0 0
/dev/loop/0 /mnt/livecd squashfs ro 0 0
proc /proc proc rw,nodiratime 0 0
sysfs /sys sysfs rw 0 0
udev /dev tmpfs rw,nosuid 0 0
devpts /dev/pts devpts rw 0 0
tmpfs /mnt/livecd/lib64/firmware tmpfs rw 0 0
tmpfs /mnt/livecd/usr/portage tmpfs rw 0 0
usbfs /proc/bus/usb usbfs rw 0 0
/dev/md1 /mnt/gentoo ext3 rw,data=ordered 0 0
/dev/md0 /mnt/gentoo/boot ext2 rw,nogrpid 0 0
/dev/vgmirror/tmp /mnt/gentoo/tmp ext2 rw,nogrpid 0 0
/dev/vgmirror/vartmp /mnt/gentoo/var/tmp ext2 rw,nogrpid 0 0
/dev/vgmirror/home /mnt/gentoo/home ext3 rw,data=ordered 0 0
/dev/vgmirror/portage /mnt/gentoo/usr/portage ext3 rw,data=ordered 0 0
/dev/vgmirror/log1 /mnt/gentoo/var/log ext3 rw,data=ordered 0 0
/dev/vgmirror/svn /mnt/gentoo/var/svn ext3 rw,data=ordered 0 0
/dev/vgmirror/backupsys /mnt/gentoo/backup/system ext3 rw,data=ordered 0 0
none /mnt/gentoo/proc proc rw,nodiratime 0 0
udev /mnt/gentoo/dev tmpfs rw,nosuid 0 0
livecd / # unmount /mnt/gentoo/backup/system /mnt/gentoo/var/svn /mnt/gentoo/var/log /mnt/gentoo/usr/portage
-bash: unmount: command not found
livecd / # umount /mnt/gentoo/backup/system /mnt/gentoo/var/svn /mnt/gentoo/var/log /mnt/gentoo/usr/portage
livecd / # umount /mnt/gentoo/home /mnt/gentoo/var/tmp /mnt/gentoo/tmp
livecd / # umount /mnt/gentoo/boot /mnt/gentoo/dev /mnt/gentoo/proc /mnt/gentoo
livecd / # reboot

Remove the LiveCD and cross your fingers. Success!

Labels: Gentoo

It's now time to start refering to the Gentoo Installation Handbook for AMD64. While I'm pretty sure that my install method works, it's worthwhile verifying against the handbook. Plus I'm using a minimal CD to do the installation, so things will be slightly different then normal.

(I use my own recipe for the initial configuration due to the mix of Software RAID + LVM2. It has served me well over the past few years and works well.)

This page starts with section 5 in the Gentoo handbook (Installing the Gentoo Installation Files).

livecd / # date
Fri Aug 25 21:23:52 UTC 2006
livecd / # cd /mnt/gentoo
livecd gentoo # links http://www.gentoo.org/main/en/mirrors.xml

Follow the directions on the Gentoo handbook page to download the correct stage3 tarball for your install. The steps are roughly thus:

1. Pick an HTTP mirror from the list (arrow up/down then press [Enter] on the link)
   
2. Go into the "releases/" folder
   
3. Go into the "amd64/" folder (note: Not all mirrors carry the AMD64 folder, you may need to pick another mirror)
   
4. Go into the "current/" folder
   
5. Go into the "stages/" folder
   
6. Find the stage3-amd64-NNNN.N-tar.bz2 tarball, highlight it and click [D] to download
   
7. Press [Q] to quit out of links
   

Now you should have the tarball in /mnt/gentoo:

livecd gentoo # ls -l
total 105797
drwxr-xr-x 3 root root 4096 Aug 25 21:11 backup
drwxr-xr-x 3 root root 1024 Aug 25 20:46 boot
drwxr-xr-x 3 root root 4096 Aug 25 20:59 home
drwx------ 2 root root 16384 Aug 25 20:47 lost+found
-rw-r--r-- 1 root root 108186115 Aug 25 22:05 stage3-amd64-2006.0.tar.bz2
drwxrwxrwt 3 root root 4096 Aug 25 20:59 tmp
drwxr-xr-x 3 root root 4096 Aug 25 21:00 usr
drwxr-xr-x 5 root root 4096 Aug 25 21:10 var
livecd gentoo #

Extract the tarball:

livecd gentoo # tar xvjpf stage3-*.tar.bz2

You'll follow similar steps for the portage tarball. In fact, we probably should've downloaded it at the same time to /mnt/gentoo.

livecd gentoo # tar xvjf /mnt/gentoo/portage-20060123.tar.bz2 -C /mnt/gentoo/usr

Setup your make flags. Since I have an X2 CPU, I'm using "-j3" for MAKEOPTS.

livecd gentoo # vi /mnt/gentoo/etc/make.conf
# These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-march=k8 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j3"

Now we start in on Section 6 (Installing the Gentoo Base System). Time to pick mirrors and other things.

livecd gentoo # mirrorselect -i -o >> /mnt/gentoo/etc/make.conf
livecd gentoo # mirrorselect -i -r -o >> /mnt/gentoo/etc/make.conf
livecd gentoo # cat /mnt/gentoo/etc/make.conf
# These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-march=k8 -O2 -pipe"
CHOST="x86_64-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j3"


GENTOO_MIRRORS="http://gentoo.arcticnetwork.ca/ http://www.gtlib.gatech.edu/pub/gentoo http://gentoo.chem.wisc.edu/gentoo/ http://gentoo.mirrors.pair.com/ "

SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"
livecd gentoo #

Copy the resolv.conf file and mount /proc and /dev:

livecd gentoo # cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
livecd gentoo # mount -t proc none /mnt/gentoo/proc
livecd gentoo # mount -o bind /dev /mnt/gentoo/dev

We're ready to chroot and start the build.

livecd gentoo # chroot /mnt/gentoo /bin/bash
livecd / # env-update
>>> Regenerating /etc/ld.so.cache...
livecd / # source /etc/profile
livecd / # export PS1="(chroot) $PS1"
(chroot) livecd / #

Read the next section carefully! I use an extremely limited USE flag (that turns off all multimedia and graphical support).

(chroot) livecd / # emerge --sync

(chroot) livecd / # ls -FGg /etc/make.profile
lrwxrwxrwx 1 50 Aug 25 22:10 /etc/make.profile -> ../usr/portage/profiles/default-linux/amd64/2006.0/
(chroot) livecd / # nano -w /etc/make.conf
USE="-alsa -apm -arts -bitmap-fonts -gnome -gtk -gtk2 -kde -mad -mikmod -motif -opengl -oss -qt -quicktime -sdl -truetype -truetype-fonts -type1-fonts -X -xmms -xv"
(chroot) livecd / # ls /usr/share/zoneinfo
(chroot) livecd / # ln -sf /usr/share/zoneinfo/EST5EDT /etc/localtime
(chroot) livecd / # date
(chroot) livecd / # zdump GMT
(chroot) livecd / # zdump EST5EDT

Read section 7 carefully. I'm using the default "gentoo-sources" kernel.

(chroot) livecd / # USE="-doc symlink" emerge gentoo-sources

I'll cover configuration of the kernel in the next post.

Labels: Gentoo

As I said last time, I'm setting this unit up with the following partitions on the 2-disk RAID1 set (sda and sdb):

sda1 (md0) - 128MB for /boot
sda2 (md1) - 8GB for the root partition (primary)
sda3 (md2) - 8GB for a root partition (backup operating system)
sda4 - place-holder for extended partition
sda5 (md3) - 4GB swap file partition
sda6 (md4) - 679GB in LVM2 volume group (vgmirror)

Since I already created the RAID arrays last time, this time I only need to start up the RAID sets.

# modprobe md
# modprobe raid1
# for i in 0 1 2 3 4; do mknod /dev/md$i b 9 $i; done
# mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1
# mdadm --assemble /dev/md1 /dev/sda2 /dev/sdb2
# mdadm --assemble /dev/md2 /dev/sda3 /dev/sdb3
# mdadm --assemble /dev/md3 /dev/sda5 /dev/sdb5
# mdadm --assemble /dev/md4 /dev/sda6 /dev/sdb6

The LVM2 area on /dev/md4 is already created as well:

# modprobe dm-mod
# pvscan
PV /dev/md4 VG vgmirror lvm2 [679.39 GB / 679.39 GB free]
Total: 1 [679.39 GB] / in use: 1 [679.39 GB] / in no VG: 0 [0 ]
# vgscan
Reading all physical volumes. This may take a while...
Found volume group "vgmirror" using metadata type lvm2

Create the basic mdadm configuration file. While mdadm is able to figure out most things automatically, it's useful to give it hints.

# mdadm --detail --scan >> /etc/mdadm.conf
# vi /etc/mdadm.conf

Here's my mdadm.conf file. Notice the use of UUIDs by mdadm to ensure that it always matches up the correct partitions with the mdadm device numbers. This also should ensure that the mdadm device numbers never change.

ARRAY /dev/md4 level=raid1 num-devices=2 UUID=9b76e544:c3775946:1458656b:c78ce692
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=a441836c:a3801fed:a6e616da:dd829ebc
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=84c2076e:edd3ceaf:595ae236:381044ca
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=b4da9f10:265c3868:db128369:583c900e
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=ada9bc71:2044d255:74204255:b1ba5cd1

Next we create the file systems:

livecd / # mke2fs /dev/md0
livecd / # mke2fs -j /dev/md1
livecd / # mke2fs -j /dev/md2
livecd / # mkswap /dev/md3 ; swapon /dev/md3
livecd / # mount /dev/md1 /mnt/gentoo
livecd / # mkdir /mnt/gentoo/boot ; mount /dev/md0 /mnt/gentoo/boot

For the LVM2 volumes, things are a bit more complex. The majority of the action is going to take place inside of the root volumes since we are only doing a minimal build on order to prep for a Xen Domain0 kernel. However, there are a few volumes that are worthwhile placing in LVM so that they are available to both the primary and the backup operating system. (Mostly /home and /usr/portage along with the temporary volumes.)

# lvcreate -L2G -ntmp vgmirror
# lvcreate -L2G -nvartmp vgmirror
# lvcreate -L2G -nhome vgmirror
# lvcreate -L4G -nportage vgmirror
# ls -l /dev/vgmirror
# lvscan
# mke2fs /dev/vgmirror/tmp
# mke2fs /dev/vgmirror/vartmp
# mke2fs -j /dev/vgmirror/home
# mke2fs -j /dev/vgmirror/portage
# mkdir /mnt/gentoo/tmp ; mount /dev/vgmirror/tmp /mnt/gentoo/tmp
# chmod 1777 /mnt/gentoo/tmp
# mkdir /mnt/gentoo/var
# mkdir /mnt/gentoo/var/tmp ; mount /dev/vgmirror/vartmp /mnt/gentoo/var/tmp
# chmod 1777 /mnt/gentoo/var/tmp
# mkdir /mnt/gentoo/home ; mount /dev/vgmirror/home /mnt/gentoo/home
# mkdir /mnt/gentoo/usr
# mkdir /mnt/gentoo/usr/portage ; mount /dev/vgmirror/portage /mnt/gentoo/usr/portage

Now for the supplementary volumes (logs, subversion and system backup). I'm using separate volumes for the log files of the primary vs secondary operating system. The secondary (backup) O/S only gets 1GB of space for its log files.

# lvcreate -L4G -nlog1 vgmirror
# lvcreate -L1G -nlog2 vgmirror
# lvcreate -L2G -nsvn vgmirror
# lvcreate -L16G -nbackupsys vgmirror
# mke2fs -j /dev/vgmirror/log1
# mke2fs -j /dev/vgmirror/log2
# mke2fs -j /dev/vgmirror/svn
# mke2fs -j /dev/vgmirror/backupsys
# mkdir /mnt/gentoo/var/log ; mount /dev/vgmirror/log1 /mnt/gentoo/var/log
# mkdir /mnt/gentoo/var/svn ; mount /dev/vgmirror/svn /mnt/gentoo/var/svn
# mkdir /mnt/gentoo/backup
# mkdir /mnt/gentoo/backup/system ; mount /dev/vgmirror/backupsys /mnt/gentoo/backup/system

Whew, that's a big packet of LVM partitions. But it prevents problems down the road.

At this point, everything is setup and ready for the initial install (or a chroot into an existing system for repairs).

Labels: Gentoo

Time to build the base Gentoo Linux O/S. While I plan on switching over to a Xen hypervisor kernel in a few days, I still need to get a base Gentoo system up and running. But first, let me document what sort of machine I'm building and the reasoning behind some of the decisions.

The unit is a custom-built system that will serve as a test unit for building out an iSCSI SAN (and eventually serve as part of that SAN if it tests well). There will be multiple NICs so that I can bond NICs for bandwidth and so that I can connect NICs to multiple switches in the SAN mesh (for fault-tolerance). Initially, it will have only (2) SATA drives installed, but the eventual loadout will have a total of (14) SATA drives.

Since I needed (2) SATA RAID cards and (2) Intel PRO/1000 dual-port server gigabit NICs, I needed a motherboard with multiple PCIe slots. The Asus M2N32-SLI Deluxe meets that requirement with (2) x16, (1) x4, and (1) x1 slots. Plus it has (2) PCI slots. These slots will be populated as follows:

PCIe x16: Intel PRO/1000 dual-port PCIe x4
PCIe x4: 8-port SATA RAID card PCIe x4
PCIe x1: HighPoint RocketRAID 2300 SATA-II 4-port PCIe x1
PCIe x16: Intel PRO/1000 dual-port PCIe x4
PCI: 3com 3C905B NIC
PCI: old PCI video card

So I'm using the x16 slots for something other then a video card (which works on newer BIOS revisions).

Plus, the motherboard is an AM2 with the newer AMD Pacifica virtualization technology (AMD-V). That will do a better job of running Xen then the current Opteron 940pin CPUs. This motherboard also supports ECC, which I will be installing in a few weeks. Other features include: (6) SATA-II ports, (2) gigabit Marvel NICs, (1) internal SATA-II port on a Silicon Image chip, (1) ESATA on the Silicon Image chip, (10) USB ports, (2) Firewire ports.

Not a cheap board ($200 retail) and has quite a bit of headroom. The PCIe architecture should also perform better then the older PCI motherboards, which is important in a 14-disk SAN unit.

All of this is being installed in a ThermalTake Armor tower case (MODELNUMBER?). The Armor case has (2) internal hard drive bays (actually 3, but I'm only using 2 for thermal reasons) and (11) 5.25" bays on the front. One of those 5.25" bays is occupied by the floppy drive mount location, the power and reset switches and the power and HD LEDs. That leaves me with (10) 5.25" bays.

In those (10) 5.25" bays, I'm installing a DVD-RW in the top and then filling the rest of the (9) bays with 4:3 SATA hot-plug back planes. These will hold (12) SATA-II hard drives and allow us to swap out hard drives easily (even if we don't hotplug we can still minimize downtime). The two internal drives are not as easy to replace, but they are still fairly easy to get at and replace in under 30 minutes.

My initial configuration plan is a (2) drive RAID1 using the internal bays and the motherboard SATA-II connectors. That will allow me to get the OS up and running and do some limited testing. After that I will add (4) HDs to the first hotswap bay, connect them to the HighPoint 2300 card, and run them as RAID10 for twice the performance as a 2-drive RAID1 set.

Down the road we will add the 8-port SATA RAID card and fill the other 8 bays in the front. These will be configured as a (6) drive RAID10 set (3x performance) with (2) hot spare drives. That will fill out the unit. If I'm pressed for space and capacity, I could add a 3rd drive to the internal drive bay for a hot-spare (risking more heat) and setup the last (8) drives up front as a (8) drive RAID10.

The current power-supply is a 750W ThermalTake ToughPower unit. Ideally, I'd install a redundant PSU in this case, but I'll tackle that at a future date. One thing that I do wish I had done was to have bought the "modular" 750W PSU. That unit makes the component power cables connect to plugs on the PSU making it easier to swap out a failed PSU without re-wiring all of the components. However, due to all of the room inside the Armor case, re-wiring is not that difficult or time-consuming.

I'm also not ready to spend $500-$600 on a redundant PSU until I know whether the 750W can handle (15) hard drives plus an Athlon64 X2 4200+ with multiple NICs and expansion cards. I'm pretty sure that it will. The base recommendation for an Athlon64 X2 with a simple setup is 300-350W. Figure that each additional hard drive adds 20W to the load (overstatement) and 15x20W = 300W. That puts us up around 600-650W not including the extra expansion cards. So the 750W should perform very well.

...

For the install, I plan on a partition layout like so (both sda and sdb are configured identically and then mirrored together):

Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1               1          17      136521   fd  Linux raid autodetect
/dev/sda2              18        1015     8016435   fd  Linux raid autodetect
/dev/sda3            1016        2013     8016435   fd  Linux raid autodetect
/dev/sda4            2014       91201   716402610    5  Extended
/dev/sda5            2014        2512     4008186   fd  Linux raid autodetect
/dev/sda6            2513       91201   712394361   fd  Linux raid autodetect

Partition #1 is the boot partition. I generally go with 128MB as it allows me to have a dozen or so kernels setup in grub.

Partition #2 and #3 are 8GB install partitions. I plan on installing once to the first 8GB partition, then cloning the install to the second 8GB partition. That way, worst case, if the primary install gets hosed, we can boot to the second partition which is still functional.

Partition #4 is simply the extended (logical) partition place-holder.

Partition #5 will be used for the Linux swap area. I always mirror my swap so that the machine keeps running even if one of the drives fails.

Partition #6 is for LVM2 and will be sub-divided up. I estimate that I'll use about 50-100GB for the O/S which will leave 600GB or so for use by clients.

That's a fairly conservative partition layout and should provide me with enough flexibility to recover from most situations without having to toss an install CD into the unit.

...

Other sysadmin tricks that I plan on using are:

- Using SubVersion to store the contents of /boot, /etc, and other system configuration files. That will give me a history of changes to the system. That has been working very well on my other systems.

- Using Bacula or rdiff-snapshot or rsync or dd to create snapshots of the working root volumes. In worst-case, we restore the root volumes from backups.

- Sharing the portage tree between the two root partitions. This is low-risk and will save space and time.

- Sharing the /tmp and /var/tmp LVM partitions between the two root partitions. Naturally, the swap partition is also shared between the two root partitions.

- Putting /home on its own LVM partition and sharing it between the two root partitions. Moderately risky, but since this is a server-only headless setup with no users, it's not a big deal.

...

So it's a moderately complex setup, but provides me with multiple fall-back positions and restoration options. Anything from reverting a configuration file to a newer version, to booting a known-good root partition, to restoring the system from backups.

And there's always the last resort of putting a Gentoo boot CD in the unit, starting networking and the SSH daemon, and attempting to fix the unit remotely.

Labels: Gentoo

Finally got my Asus M2N32-SLI Deluxe motherboard setup and ready for installation. Tossed the 2006.0 Gentoo AMD64 CD in and told it to use 80x25 for the old PCI video card that I have in it. Unforunately, it hangs after a bit.

So my first plan of attack is to update the Asus BIOS from 0406 to 0603 (which is 2 revisions newer). The 0603 revision was released on June 29, 2006. The Asus BIOS includes an EZ-FLASH tool in the BIOS Setup (Tools menu), all I have to do is burn the BIOS file to a CD-ROM or diskette.

Update: It was actually easier to put the BIOS update on a USB flash drive and connect it one of the the USB ports.

Update #2: I had to boot the kernel using:

boot: gentoo noapic noacpi

Which gets me past the hang at:

io scheduler noop registered
io scheduler deadline registered

I'm also using a very old PCI video card so I have to specify my video mode at the prompt (I usually pick 80x43).

Instead it now hangs at "Letting udev process events". Could be time to try the "gentoo-nofb noapic" kernel option (no frame buffer). Hmm, that got me farther, but still no luck. I'm reading that the nForce 590 chipsets aren't well supported by Linux yet. Trying with the "noapic" option resulted in a hang entering kernel mode 3.

Time for plan B... seeing if the latest Ubuntu 6.06 CD works on this system. That may give me some hints. From what I'm reading I have to use the "apic=off" option on the Ubuntu 6.06 CD. Hmm... that hung as well.

# gentoo-nofb noapic noacpi nolapic

Hmm... still hangs. Off to do some more research.

Update #3: Finally got a system that seems to be working.

Reverted the BIOS from the 0604 revision back to the 0503 revision. Now I can boot the Gentoo AMD64 2006.0 minimal CD using the options gentoo-nofb noapic. I still get the IRQ7 error that seems to be bugging other Gentoo users, but the system is stable enough to start the install.

Labels: Gentoo

So as I wait for parts to be delivered for the Test SAN unit, I'm reading up on Xen and iSCSI and debating what my plan of attack will be. My options (roughly) are:

1) Build a normal Gentoo server and load iscsitarget on top of it. This is the simplest of all options and something that I'm very comfortable with. However, if I were then then need to run other services on top of the unit, I run the risk of making the iSCSI server portion unstable.

2) Xen with various services running in DomUs. A bit more complex but offers a lot more flexibility. I can start setting up virtual servers for the sub-tasks and then migrate them off of the test unit when I have more hardware available.

What I'm not sure is whether to run iscsitarget in its own DomU or if it should run in Dom0. If I run the iscsitarget in its own DomU, then it becomes easy to restart the iscsitarget server in a worst case without taking down the entire box.

Links:

Infrastructure virtualisation with Xen advisory

Labels: Gentoo, iSCSI, iSCSITarget, Xen

Currently, I was using a VIA EPIA system as my music server, but now I'm thinking about turning it into a smart router for the home office. This will entail adding a ethernet card to the unit in addition to migrating my hard drives from a pair of 300GB drives to a pair of notebook drives (for less power). Since I'm using Software RAID, moving from one set of disks to another should be nearly seamless.

The hardware has changed a little bit since my previous attempt at building the system back in March 2005, but the BIOS settings are identical. The current hardware consists of:

(1) VIA EPIA ME6000 (EPIA M series), 600Mhz fanless CPU
(2) 300GB 5400rpm hard drives
(1) DVD-ROM
(1) Morex Venus 668 Black Case
(1) 1GB PC2100 DIMM

I'm going to replace the two 300GB 3.5" drives with less power-hungry 60GB laptop drives. The basic process is:

1. Detach the DVD-ROM (which happens to be the master drive on the 2nd cable) and connect the laptop drive. That will allow me to work with the new drives one at a time while I migrate from the old to the new.
   
2. Copy the boot sector from the old /dev/hda to the new laptop drive: dd if=/dev/hda bs=512 count=1 of=/dev/hdc
   
3. Verify that the first (3) partitions on the new drive are identically sized as the original drive.
   
4. Copy the filesystem from the old drive to the new drive: dd if=/dev/hda1 of=/dev/hdc1
   
5. Install grub on the new disk.
   
6. Shutdown
   
7. Remove the old /dev/hda 300GB drive, move the new laptop drive into place
   
8. Restart the system, verify the Software RAID. I had to tell mdadm to add the /dev/hda partitions to the arrays: mdadm /dev/md0 -a /dev/hda1
   

Note: I forgot to install grub on the new disk this last time. So I need to boot from the LiveCD, chroot into the O/S and re-install grub on the new disks.

Labels: Gentoo, VIAEPIA

Got the following error while trying to emerge samba into my Gentoo box.

Compiling torture/rpctorture.c
make: *** Waiting for unfinished jobs....
torture/rpctorture.c:27: error: `global_myname' redeclared as different kind of symbol
include/proto.h:1019: error: previous declaration of `global_myname'
torture/rpctorture.c:57: warning: `struct client_info' declared inside parameter list
torture/rpctorture.c:57: warning: its scope is only this definition or declaration, which is probably not what you want
torture/rpctorture.c: In function `rpcclient_connect':
torture/rpctorture.c:62: error: dereferencing pointer to incomplete type
torture/rpctorture.c:62: error: dereferencing pointer to incomplete type
torture/rpctorture.c:63: error: dereferencing pointer to incomplete type
torture/rpctorture.c:66: error: dereferencing pointer to incomplete type
torture/rpctorture.c:66: error: dereferencing pointer to incomplete type
torture/rpctorture.c:68: error: dereferencing pointer to incomplete type
torture/rpctorture.c:68: error: dereferencing pointer to incomplete type
torture/rpctorture.c: At top level:
torture/rpctorture.c:90: warning: `struct client_info' declared inside parameter list
torture/rpctorture.c: In function `run_enums_test':
torture/rpctorture.c:96: warning: passing arg 1 of `rpcclient_connect' from incompatible pointer type
torture/rpctorture.c:102: error: dereferencing pointer to incomplete type
torture/rpctorture.c:102: error: dereferencing pointer to incomplete type
torture/rpctorture.c: At top level:
torture/rpctorture.c:134: warning: `struct client_info' declared inside parameter list
torture/rpctorture.c: In function `run_ntlogin_test':
torture/rpctorture.c:140: warning: passing arg 1 of `rpcclient_connect' from incompatible pointer type
torture/rpctorture.c:146: error: dereferencing pointer to incomplete type
torture/rpctorture.c:146: error: dereferencing pointer to incomplete type
torture/rpctorture.c: At top level:
torture/rpctorture.c:167: warning: `struct client_info' declared inside parameter list
torture/rpctorture.c: In function `main':
torture/rpctorture.c:233: error: storage size of `cli_info' isn't known
torture/rpctorture.c:377: error: `scope' undeclared (first use in this function)
torture/rpctorture.c:377: error: (Each undeclared identifier is reported only once
torture/rpctorture.c:377: error: for each function it appears in.)
torture/rpctorture.c:535: warning: passing arg 5 of `create_procs' from incompatible pointer type
torture/rpctorture.c:539: warning: passing arg 5 of `create_procs' from incompatible pointer type
make: *** [torture/rpctorture.o] Error 1
* rpctorture didn't build
running build
running build_py
running build_ext
--------------------------- ACCESS VIOLATION SUMMARY ---------------------------
LOG FILE = "/var/log/sandbox/sandbox-net-fs_-_samba-3.0.14a-r2-21241.log"

access_wr: /etc/krb5.conf
--------------------------------------------------------------------------------
#

Here are my current USE flags:

# cat /etc/make.conf

# These settings were set by the catalyst build script that automatically built this stage
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-Os -mcpu=i686"
CHOST="i386-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j2"

GENTOO_MIRRORS="http://gentoo.osuosl.org/"
SYNC="rsync://rsync.namerica.gentoo.org/gentoo-portage"


USE="apache2 kerberos ldap postgres samba -alsa -apm -arts -bitmap-fonts -gnome -gtk -gtk2 -kde -mad -mikmod -motif -opengl -oss -qt -quicktime -sdl -truetype -truetype-fonts -type1-fonts -X -xmms -xv"

# cat /etc/make.profile/make.defaults

# Copyright 1999-2005 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo-x86/profiles/default-linux/x86/2005.1/make.defaults,v 1.4 2005/08/29 22:20:25 wolf31o2 Exp $

USE="alsa apm arts avi berkdb bitmap-fonts crypt cups eds emboss encode fortran foomaticdb gdbm gif gnome gpm gstreamer gtk gtk2 imlib ipv6 jpeg kde libg++ libwww mad mikmod motif mp3 mpeg ncurses nls ogg oggvorbis opengl oss pam pdflib perl png python qt quicktime readline sdl spell ssl tcpd truetype truetype-fonts type1-fonts vorbis X xml2 xmms xv zlib"
#

I'm still searching for a solution to this issue. I've heard it has to do with trying to use the kerberos USE flag (which is not an optional flag for me). The closest possible solution in Google is on the Gentoo forums (Problems upgrading to Samba 3.0.14a-r2!). The user, "jpnag", posts a solution.

The solution involves editing the ebuild file for Samba. This is where you will need to become a bit more knowledgeable about how portage and emerge works (see "man make.conf" for details on some of this along with "man portage").

By default, portage downloads and installs packages under the "/usr/portage/" tree (defined by "PORTDIR=" in your "/etc/make.conf" file or "/etc/make.profile/make.defaults" file). There is also an optional define, "PORTDIR_OVERLAY=", which you can use to point at a tree containing user-built ebuild files that are not updated by "emerge --sync". Essentially, the second tree will overlay the first. So if you have "package X" in both trees, only the one in the overlay tree will get compiled.

Now to create the backup copy of the broken Samba ebuild. If you have not already added "PORTDIR_OVERLAY=" to your "make.conf" file, you should also do this.

# cd /etc
etc # echo 'PORTDIR_OVERLAY="/usr/local/portage"' >> /etc/make.conf
etc # cd /usr/local
local # ls /usr/portage/net-fs/samba/
local # mkdir portage ; cd portage
portage # mkdir net-fs ; cd net-fs
net-fs # mkdir samba ; cd samba
samba # cp -a /usr/portage/net-fs/samba/* .
samba # ls -l samba-3.0.14a-r2.ebuild
samba # nano -w samba-3.0.14a-r2.ebuild

Now hit [Ctrl-W] and type "src_compile", which will take you straight to the following code block:

rc_compile() {
        ebegin "Running autoconf"
                autoconf
        eend $?

        local myconf
        local mymods
        local mylangs

        if use xml || use xml2 ;
        then   
                mymods="xml,${mymods}"
        fi

Somewhere towards the start of the funciton, add the line "addpredict /etc/krb5.conf".

src_compile() {
        ebegin "Running autoconf"
                autoconf
        eend $?

        local myconf
        local mymods
        local mylangs

        addpredict /etc/krb5.conf

        if use xml || use xml2 ;
        then
                mymods="xml,${mymods}"
        fi

Create the ebuild digest (MD5 signatures) for the patched package.

samba # ebuild /usr/local/portage/net-fs/samba/samba-3.0.14a-r2.ebuild digest
>>> Generating digest file...
<<< samba-3.0.14a.tar.gz
<<< samba-vscan-0.3.6.tar.bz2
<<< samba-3-gentoo-0.3.3.tar.bz2
>>> Generating manifest file...
<<< ChangeLog
<<< metadata.xml
<<< samba-3.0.14a-r2.ebuild
<<< samba-3.0.14a-r3.ebuild
<<< samba-3.0.20-r1.ebuild
<<< samba-3.0.20a.ebuild
<<< samba-3.0.20b.ebuild
<<< files/digest-samba-3.0.14a-r2
<<< files/README.gentoo
<<< files/digest-samba-3.0.14a-r3
<<< files/digest-samba-3.0.20-r1
<<< files/digest-samba-3.0.20a
<<< files/digest-samba-3.0.20b
>>> Computed message digests.

samba # emerge -pv samba

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild N ] net-fs/samba-3.0.14a-r2 -acl +cups -doc +kerberos +ldap -libclamav -mysql -oav +pam +postgres +python -quotas +readline (-selinux) -winbind -xml +xml2 0 kB [1]

Total size of downloads: 0 kB
Portage overlays:
[1] /usr/local/portage

samba # emerge samba

(crosses fingers)

Labels: Gentoo, Samba

Time to configure the Gentoo kernel. I'm configuring this for my VIA EPIA ME6000 motherboard. (Notice that I'm now including the LVM2 components as part of my base installation.)

# emerge mdadm
# emerge lvm2
# cd /usr/src/linux
# make menuconfig

Linux Kernel v2.6.11 Configuration
(C)ode maturity level options
(G)eneral setup
--> (C)onfigure standard kernel features for small systems (turn ON)
--> --> (O)ptimize for size (turn ON)
(L)oadable module support
(P)rocessor type and features
--> (P)rocessor family (changed to "CyrixIII/VIA-C3")
--> (S)ymetric multi-processing support (turned this one OFF)
--> M(a)chine Check Exception (turned this OFF)
(P)ower management options (ACPI, APM)
(B)us options (PCI, PCMCIA, EISA< MCA, ISA)
(E)xecutable file formats
(D)evice drivers
--> (P)arallel port support (turned OFF)
--> (A)TA/ATAPI/MFM/RLL support
--> --> (V)IA82CXXX chipset support (turned ON)
--> M(u)lti-device support (turn it ON)
--> --> (R)AID support (turn it ON as BUILT-IN)
--> --> --> (R)AID-1 mirroring mode (turn it ON as BUILT-IN)
--> --> (D)evice mapper support (set to MODULE, per section 13 of LVM2 guide)
--> N(e)tworking support
--> --> (E)thernet (10 or 100Mbit)
--> --> --> (R)ealTek RTL-8139 PCI Fast Ethernet (turn it OFF)
--> --> --> (V)IA Rhine support (turn it ON)
--> --> --> --> (U)se MMIO instead of PIO (turn it ON)

--> (C)haracter Devices
--> --> (I)ntel/AMD/VIA HW Random Number Generator (turn ON as BUILT-IN)
--> --> (I)ntel 440LX/BX/GX, I8xx and E7x05 chipset support (turn it OFF)
--> --> (V)IA chipset support (turn ON)
--> (S)ound
--> --> (S)ound card support (turn OFF)
(F)ile systems
--> N(e)twork File Systems
--> --> (S)MB file system support (turn ON as BUILT-IN)
--> --> (C)IFS support (turn ON as BUILT-IN)
(P)rofiling support
(K)ernel hacking
(S)ecurity options
(C)ryptographic options
--> (C)ryptographic API (turn ON)
--> --> HM(A)C support (NEW) (turn ON as BUILT-IN)
--> --> (turn ON all other options as MODULE)
(L)ibrary routines

Exit and save your configuration. Then build the kernel (the following command is for 2.6 kernels). Expect the compile to take about an hour.

# make && make modules_install

Labels: Gentoo

Note: As always, these are works-in-progress. They might work, or they might not and are mostly here so that I can keep track of what I did or didn't do during an install.

Gentoo 2005.0 Software RAID (part 1) - Going to try again with Gentoo 2005.0 in a software RAID with LVM configuration. There's a post on the Gentoo forums that indicates it may now be possible to get it up and running (seems to indicate there was a bug with kernels prior to 2.6.12).

The hardware that I'm using is a VIA EPIA ME6000 (full details). Basically 2x160GB drives, which will be mirrored.

Start with the usual first steps (I prefer to ssh into my box during the build, it allows me to keep a log file of all commands, errors, etc.).

1) Boot the system using the Universal CD
2) ifconfig - find out the IP address of the box
3) passwd - change the root password to something you know
4) Start the SSH daemon - /etc/init.d/sshd start

Then I SSH in using SecureCRT and start my install.

As always, I start by blowing away any existing partitions by using "fdisk" or Derik's Boot and Nuke (a bootable CD which you can use to erase hard drives). Then I'll re-create my partitions and start in on the installation.

# fdisk /dev/hda

Command: n
Command action: p
Partition number: 1
First cylinder: 1
Last cylinder: +128M
Command: a
Partition number: 1
Command: t
Hex code: fd

Command: n
Command action: p
Partition number: 2
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 2
Hex code: fd

Command: n
Command action: p
Partition number: 3
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 3
Hex code: fd

Command: n
Command action: p
First cylinder: [enter]
Last cylinder: [enter]
Command: t
Partition number: 4
Hex code: fd

Command: p

Command: w

This gives me a 128MB boot area, a 2GB swap area, a 2GB root area, with the rest of the disk set aside for my LVM partitions. Repeat the above commands to configure the 2nd disk in the same fashion. Note that I'm using a different partition type then that shown in chapter 4.c. The 'fd' partition type is what I need to use since all 4 partitions on hda/hdc are going to be put into a software RAID1 set.

(Eventually ran out of time to get this working due to other projects, now waiting on 2005.1 release.)

Labels: Gentoo

Been fighting with the Gentoo install for a few weeks now, but am currently working on another project. It will be a few weeks before I get back around to working on the Gentoo issues again.

Labels: Gentoo

So, I've got a system that isn't booting yet. The boot error is:

* Mounting proc at /proc [ ok ]
* Mounting sysfs at /sys [ !! ]
can't create lock file /etc/mtab~944: Read-only file system (use -n flag to override)

First, I have to go back to the boot CD and get up and running:

livecd root # passwd
livecd root # /etc/init.d/sshd start
livecd root # ifconfig
(at this point, I switch over to using SecureCRT on the other system)
livecd root # modprobe md
livecd root # modprobe raid1
livecd root # for i in 0 1 2 3; do mknod /dev/md$i b 9 $i; done
livecd root # mdadm --assemble /dev/md0 /dev/hda1 /dev/hdc1
livecd root # mdadm --assemble /dev/md1 /dev/hda2 /dev/hdc2
livecd root # mdadm --assemble /dev/md2 /dev/hda3 /dev/hdc3
livecd root # mdadm --assemble /dev/md3 /dev/hda4 /dev/hdc4

That gets the raid arrays up and running.

livecd root # swapon /dev/md2
livecd root # mount /dev/md1 /mnt/gentoo
livecd root # mount /dev/md0 /mnt/gentoo/boot
livecd root # modprobe dm-mod
livecd root # mkdir /etc/lvm
livecd root # echo 'devices { filter=["r/cdrom/"] }' >/etc/lvm/lvm.conf
livecd / # lvscan
livecd / # lvchange -ay vgmirror

Which gets the LVM2 up and running (and the logical volumes set to active).

# mount /dev/vgmirror/opt /mnt/gentoo/opt
# mount /dev/vgmirror/usr /mnt/gentoo/usr
# mount /dev/vgmirror/var /mnt/gentoo/var
# mount /dev/vgmirror/home /mnt/gentoo/home
# mount /dev/vgmirror/tmp /mnt/gentoo/tmp
# chmod 1777 /mnt/gentoo/tmp
# mount /dev/vgmirror/vartmp /mnt/gentoo/var/tmp
# chmod 1777 /mnt/gentoo/var/tmp
# mount -t proc none /mnt/gentoo/proc

Should be ready to chroot into the hard disk.

# chroot /mnt/gentoo /bin/bash ; env-update

Gonna redo my kernel configuration (see Gentoo 2004.3 on Gigabyte GA-6VA7+ (part 4)) because I suspect that something needed to be loaded in differently.

# cd /usr/src/linux
# make menuconfig

Going to switch the LVM2 from loading as a "module" and change it to being "built-in". Could be an error on the Gentoo LVM2 page according to a note I see in the gentoo wiki.

# make && make modules_install
# cp arch/i386/boot/bzImage /boot/kernel-2.6.11-gentoo-Apr20
# cp System.map /boot/System.map-2.6.11-gentoo-Apr20
# cp .config /boot/config-2.6.11-gentoo-Apr20
# nano -w /etc/modules.autoload.d/kernel-2.6
(remove dm-mod from being auto-loaded)
livecd linux # exit
livecd / # cd /
livecd / # cat /proc/mounts

(unmount all of your mounted partitions, including the LVM mounts)

livecd / # umount ... (insert list of mounted file systems)

livecd / # vgchange -an vgmirror
livecd / # reboot

(remove the gentoo boot CD)

Now to see if it works. No luck. Redoing the above, but going to re-do my LVM2, but changing to be 'static' per the wiki.

# echo 'sys-fs/lvm2 static' >> /etc/portage/package.use
# emerge lvm2

No joy here either. Time to go do some more searching.

Attempt #3 (#4?), editing the grub.conf file and adding "udev" to the end of the "kernel" line. Nothing complex, just tack " udev" onto the end of the kernel line using nano.

No joy again.

Attempt #4 - added an initrd line to the grub.conf file. Doubtful that this will fix the issue.

Nope.

Attempt #5 - using this thread over at the gentoo forums, I added some more information to my kernel line in the grub.conf file.

Nope.

Labels: Gentoo

Well, first glitch. Looking at my boot screen with [Shift-PgUp] / [Shift-PgDn] to find the error. I see that the Software RAID is working fine (it built md0..md3 automatically).

* Mounting proc at /proc [ ok ]
* Mounting sysfs at /sys [ !! ]
can't create lock file /etc/mtab~944: Read-only file system (use -n flag to override)
* Mounting ramfs at /dev... [ ok ]
* Configuring system to use udev... [ ok ]
* Populating /dev with device nodes...
* Using /sbin/hotplug for udev management...
* Mounting devpts at /dev/pts... [ ok ]
* Activating (possible) swap... [ ok ]
* Remounting root filesystem read-only (if necessary)... [ ok ]
* Checking root filesystem...
ext2fs_check_if_mount: No such file or directory while determining whether /dev/md2 is mounted.
fsck.ext3: No such file or directory while trying to open /dev/md2
/dev/md2: The superblock could not be read or does not describe a correct ext2 filesystem. If the device is valid and it really contains an ext2 filesystem (and not swap or ufs or something else), then the superblock is corrupt, and you might try running e2fsck -b 8193 <device>
* Filesystem couldn't be fixed :( [ !! ]

Give root password for maintenance
(or type Control-D for normal startup):

So, according to a quick google, this indicates an issue with /etc/fstab.

I'll be digging into this in a few days when I get a chance. (See Troubleshooting 1.)

Labels: Gentoo

At some point, I need to upgrade my Gentoo profile from 2004.3 to 2005.0. Here's the error message that you see on screen when you need to do this.

livecd linux # emerge (something)

!!! Your current profile is deprecated and not supported anymore.
!!! Please upgrade to the following profile if possible:
default-linux/x86/2005.0

To upgrade do the following steps:
# emerge -n '>=sys-apps/portage-2.0.51'
# cd /etc/
# rm make.profile
# ln -s ../usr/portage/profiles/default-linux/x86/2005.0 make.profile

# Gentoo has switched to 2.6 as the defaults for headers/kernels. If you wish
# to use 2.4 headers/kernels, then you should do the following to upgrade:
# emerge -n '>=sys-apps/portage-2.0.51'
# cd /etc/
# rm make.profile
# ln -s ../usr/portage/profiles/default-linux/x86/2005.0/2.4 make.profile

# More information can be found at the following URLs:
# http://www.gentoo.org/doc/en/gentoo-upgrading.xml
# http://www.gentoo.org/doc/en/migration-to-2.6.xml

Labels: Gentoo

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

(previous step)

Time to install the bootloader. I plan on using GRUB:

# emerge grub

Now to configure GRUB (refer to my old post about GRUB for a more in-depth explanation of what I'm telling GRUB to do here).

# grub --no-floppy
grub> find /grub/stage1
(hd0,0)
(hd1,0)
grub> root (hd0,0)
grub> setup (hd0)
grub> device (hd0) /dev/hdc
grub> root (hd0,0)
grub> setup (hd0)
grub> quit
#

Now, edit your config file for grub:

# nano -w /boot/grub/grub.conf

Here is what mine looks like. Yours may be different, depending on how you configured things (and remember that I'm using Software RAID).

# cat /boot/grub/grub.conf
default 0
timeout 30
title=Gentoo Linux 2.6.11 (April 20 2005)
root (hd0,0)
kernel /kernel-2.6.11-gentoo-Apr20 root=/dev/md2
#

Now I install various system tools (see the handbook):

# emerge syslog-ng
# rc-update add syslog-ng default
# emerge dcron
# rc-update add dcron default
# crontab /etc/crontab

Note: Now you need to unmount everything that you can (including LVM), possibly shutdown the RAID as well prior to reboot.

livecd gentoo # exit
livecd / # cd /
livecd / # cat /proc/mounts

(unmount all of your mounted partitions, including the LVM mounts)

livecd / # umount ... (insert list of mounted file systems)

livecd / # vgchange -an vgmirror
livecd / # reboot

Pull the CD-ROM at this point, otherwise the LiveCD will probably boot.

(next step)

Labels: Gentoo, Gigabyte-GA-6VA7+

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

(previous step)

Time to configure the kernel.

# emerge lvm2
# emerge mdadm
# cd /usr/src/linux
# make menuconfig

Linux Kernel v2.6.11 Configuration
(C)ode maturity level options
(G)eneral setup
--> (C)onfigure standard kernel features for small systems (turn ON)
--> --> (O)ptimize for size (turn ON)
(L)oadable module support
(P)rocessor type and features
--> (P)rocessor family (changed to "Pentium-III...")
--> (S)ymetric multi-processing support (turned this one OFF)
--> M(a)chine Check Exception (turned this OFF)
(P)ower management options (ACPI, APM)
(B)us options (PCI, PCMCIA, EISA< MCA, ISA)
(E)xecutable file formats
(D)evice drivers
--> (P)arallel port support (turned OFF)
--> (A)TA/ATAPI/MFM/RLL support (turned ON the PDC20262 chipset support as BUILT-IN)
--> M(u)lti-device support (turn it ON)
--> --> (R)AID support (turn it ON as BUILT-IN)
--> --> --> (R)AID-1 mirroring mode (turn it ON as BUILT-IN)
--> --> (D)evice mapper support (set to MODULE, per section 13 of LVM2 guide)
--> (C)haracter Devices
--> --> (I)ntel/AMD/VIA HW Random Number Generator (turn ON as BUILT-IN)
--> (S)ound
--> --> (S)ound card support (turn OFF)
(F)ile systems
(P)rofiling support
(K)ernel hacking
(S)ecurity options
(C)ryptographic options
(L)ibrary routines

Exit and save your configuration. Then build the kernel (the following is for 2.6 kernels). Expect the compile to take about an hour.

# make && make modules_install

Now you need to install your kernel into the boot partition. Change the "2.6.6-gentoo" portion of the filenames to whatever you want.

# cp arch/i386/boot/bzImage /boot/kernel-2.6.11-gentoo-Apr20
# cp System.map /boot/System.map-2.6.11-gentoo-Apr20
# cp .config /boot/config-2.6.11-gentoo-Apr20

Now we need to configure LVM to auto-load.

# nano -w /etc/modules.autoload.d/kernel-2.6

Here is what my autoload file looks like:

# /etc/modules.autoload.d/kernel-2.6: kernel modules to load when system boots.
# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/modules.autoload.d/kernel-2.6,v 1.1 2003/07/16 18:13:45 azarah Exp $
#
# Note that this file is for 2.6 kernels.
#
# Add the names of modules that you'd like to load when the system
# starts into this file, one per line. Comments begin with # and
# are ignored. Read man modules.autoload for additional details.

# For example:
# 3c59x

dm-mod

Now, edit the /etc/fstab file:

# /etc/fstab: static file system information.
# $Header: /home/cvsroot/gentoo-src/rc-scripts/etc/fstab,v 1.14 2003/10/13 20:03:38 azarah Exp $
#
# noatime turns off atimes for increased performance (atimes normally aren't
# needed; notail increases performance of ReiserFS (at the expense of storage
# efficiency). It's safe to drop the noatime options if you want and to
# switch between notail and tail freely.

# <fs> <mountpoint> <type> <opts> <dump/pass>

# NOTE: If your BOOT partition is ReiserFS, add the notail option to opts.
/dev/md0 /boot ext2 noauto,noatime 1 2
/dev/md1 / ext3 noatime 0 1
/dev/md2 none swap sw 0 0
/dev/cdroms/cdrom0 /mnt/cdrom auto noauto,ro,user 0 0
#/dev/fd0 /mnt/floppy auto noauto 0 0

/dev/vgmirror/opt /opt ext3 noatime 0 3
/dev/vgmirror/usr /usr ext3 noatime 0 3
/dev/vgmirror/var /var ext3 noatime 0 3
/dev/vgmirror/home /home ext3 noatime 0 3
/dev/vgmirror/tmp /tmp ext2 noatime 0 3
/dev/vgmirror/vartmp /var/tmp ext2 noatime 0 3

# NOTE: The next line is critical for boot!
none /proc proc defaults 0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
# use almost no memory if not populated with files)
# Adding the following line to /etc/fstab should take care of this:

none /dev/shm tmpfs defaults 0 0

Now, some misc stuff:

# echo yourhostname > /etc/hostname
# echo yourdnsname > /etc/dnsdomainname
# rc-update add domainname default
# nano -w /etc/conf.d/net
(either use iface_eth0="dhcp" or configure your IP and gateway)
# rc-update add net.eth0 default
# cat /etc/resolv.conf
(verify your DNS servers if you specified a static IP)
# nano -w /etc/rc.conf
(change CLOCK="UTC" to CLOCK="local")
# passwod
(set your root password to something you will remember)

# useradd -m -G users,wheel,audio -s /bin/bash john
# passwd john

(add a user called 'john' and set a password)

(next step)

Labels: Gentoo, Gigabyte-GA-6VA7+

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

(previous step)

Time to configure the timezone and setup the kernel, this is chapter 7 in the Gentoo handbook.

Timezone for me is EST5EDT, so here's how to set that up.

# ls /usr/share/zoneinfo
# ln -sf /usr/share/zoneinfo/EST5EDT /etc/localtime
# date
# zdump GMT
# zdump EST5EDT

Last year, I went with development-sources for the kernel in order to get 2.6. This is no longer necessary (and development-sources has been rolled into vanilla-sources). So I'm going to go with the default gentoo-sources.

# emerge gentoo-sources
# ls -l /usr/src

This takes a while to run (maybe an hour or two).

(next step)

Labels: Gentoo, Gigabyte-GA-6VA7+

(Continuation of part 1)

# ls -l /etc/make.profile

As far as I can tell the 2004.3 already uses the 2.6 kernel, so there's nothing to do here. I also configured my USE flags in my last post, so that's already done as well.

# cd /usr/portage
# scripts/bootstrap.sh

This will take a while to run (I estimate a few hours, maybe even overnight). Once that finishes, you move from stage2 to stage3.

# emerge --emptytree system

Which will also take a few hours.

(next step)

Labels: Gentoo, Gigabyte-GA-6VA7+

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

This is a continuation of Gentoo and Software RAID (2004.3), where I configured the disks and setup the RAID array. I'm now picking up at the point where the RAID array has been configured and we're ready to start installing file systems.

/dev/md0 - 128MB boot
/dev/md1 - 2GB root partition
/dev/md2 - 2GB swap
/dev/md3 - rest of disk (user files)

Now, some folks say copy the /etc/mdadm.conf file, but in the same breath, they indicate that mdadm does not require the use of a config file at all. Since I'm documenting my configuration here, and my array is extremely straightforward, I'm going to skip creating the mdadm.conf file and see how it goes.

Links:
Software RAID (Gentoo Tips-n-Tricks)
LinuxDevCenter article on mdadm

The LinuxDevCenter article actually explains how to create the mdadm.conf file yourself (semi-automatically). Notice the use of wild cards that lets me compactly express that I want all 4 partitions on /dev/hda and /dev/hdc to be used in arrays. You will need to edit the results to match the syntax of the mdadm.conf file.

# echo 'DEVICES /dev/hda*' >> /etc/mdadm.conf
# echo 'DEVICES /dev/hdc*' >> /etc/mdadm.conf
# mdadm --detail --scan >> /etc/mdadm.conf
# nano -w /etc/mdadm.conf

Contents of my mdadm.conf file:

EVICES /dev/hda*
DEVICES /dev/hdc*
ARRAY /dev/md3 level=raid1 num-devices=2 devices=/dev/hda4,/dev/hdc4
ARRAY /dev/md2 level=raid1 num-devices=2 devices=/dev/hda3,/dev/hdc3
ARRAY /dev/md1 level=raid1 num-devices=2 devices=/dev/hda2,/dev/hdc2
ARRAY /dev/md0 level=raid1 num-devices=2 devices=/dev/hda1,/dev/hdc1

Picking up again with Chapter 4 of the installation handbook. This is also very similar to what I did back in June 2004 with Software RAID and LVM2.

# mke2fs /dev/md0
# mke2fs -j /dev/md1
# mkswap /dev/md2
# swapon /dev/md2
# mount /dev/md1 /mnt/gentoo
# mkdir /mnt/gentoo/boot
# mount /dev/md0 /mnt/gentoo/boot

Now, we initialize the 4th raid partition for LVM2 operations. See Gentoo LVM2 Documentation for more details about this.

# modprobe dm-mod
# pvcreate /dev/md3
# echo 'devices { filter=["r/cdrom/"] }' >/etc/lvm/lvm.conf
# vgcreate vgmirror /dev/md3
# vgscan

Here is my plan for logical volumes inside the vgmirror partition (this uses up 22GB):

4GB /tmp (ext2)
4GB /var/tmp (ext2)
2GB /opt (ext3)
4GB /usr (ext3)
4GB /var (ext3)
4GB /home (ext3)

Create the logical volumes. If you see the error message "/etc/lvm/backup: fsync failed: Invalid argument", you can ignore this warning (according to Gentoo's LVM2 page).

# lvcreate -L4G -ntmp vgmirror
# lvcreate -L4G -nvartmp vgmirror
# lvcreate -L2G -nopt vgmirror
# lvcreate -L4G -nusr vgmirror
# lvcreate -L4G -nvar vgmirror
# lvcreate -L4G -nhome vgmirror
# ls -l /dev/vgmirror
# lvscan

Output of the lvscan command:

ACTIVE '/dev/vgmirror/tmp' [4.00 GB] next free (default)
ACTIVE '/dev/vgmirror/vartmp' [4.00 GB] next free (default)
ACTIVE '/dev/vgmirror/opt' [2.00 GB] next free (default)
ACTIVE '/dev/vgmirror/usr' [4.00 GB] next free (default)
ACTIVE '/dev/vgmirror/var' [4.00 GB] next free (default)
ACTIVE '/dev/vgmirror/home' [4.00 GB] next free (default)

Format the logical volumes:

# mke2fs /dev/vgmirror/tmp
# mke2fs /dev/vgmirror/vartmp
# mke2fs -j /dev/vgmirror/opt
# mke2fs -j /dev/vgmirror/usr
# mke2fs -j /dev/vgmirror/var
# mke2fs -j /dev/vgmirror/home

Make the directories to hold your mounted volumes. Mount your volumes.

# mkdir /mnt/gentoo/opt
# mkdir /mnt/gentoo/usr
# mkdir /mnt/gentoo/var
# mkdir /mnt/gentoo/home
# mount /dev/vgmirror/opt /mnt/gentoo/opt
# mount /dev/vgmirror/usr /mnt/gentoo/usr
# mount /dev/vgmirror/var /mnt/gentoo/var
# mount /dev/vgmirror/home /mnt/gentoo/home

Make the special directories to hold your temp file volumes (these require special permissions). Then mount your temp file volumes. Also mount your proc folder.

# mkdir /mnt/gentoo/tmp
# mount /dev/vgmirror/tmp /mnt/gentoo/tmp
# chmod 1777 /mnt/gentoo/tmp
# mkdir /mnt/gentoo/var/tmp
# mount /dev/vgmirror/vartmp /mnt/gentoo/var/tmp
# chmod 1777 /mnt/gentoo/var/tmp
# mkdir /mnt/gentoo/proc
# mount -t proc none /mnt/gentoo/proc

Now we move into Installation (chapter 5) in the handbook. Verify your system date and then start extracting the tarballs.

# date
# ls -l /mnt/cdrom/stages
# cd /mnt/gentoo
# tar -xvjpf /mnt/cdrom/stages/stage1-x86-2004.3.tar.bz2
# ls -l /mnt/cdrom/snapshots
# cd /mnt/gentoo
# tar -xvjf /mnt/cdrom/snapshots/portage-20041022.tar.bz2 -C /mnt/gentoo/usr
# cd /mnt/gentoo
# mkdir /mnt/gentoo/usr/portage/distfiles
# cp /mnt/cdrom/distfiles/* /mnt/gentoo/usr/portage/distfiles/

Before I configure the make.conf file, I should take a look at my system configuration.

# cat /proc/version
Linux version 2.6.9-gentoo-r1 (root@inertia) (gcc version 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6)) #1 SMP Thu Nov 25 03:43:53 UTC 2004
# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Celeron (Coppermine)
stepping : 6
cpu MHz : 568.097
cache size : 128 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 sep mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips : 1114.11

Now we're ready to edit the make.conf file and change flags:

# nano -w /mnt/gentoo/etc/make.conf

Here is my personal make.conf (use at your own risk). This is for a Celeron Coppermine CPU (Pentium III). I prefer to compile for size given the small amount of installed RAM on this system.

CFLAGS="-Os -march=pentium3 -pipe -fomit-frame-pointer"
CHOST="i686-pc-linux-gnu"
CXXFLAGS="${CFLAGS}"
MAKEOPTS="-j2"
USE="apache2 kerberos ldap -apm -gif -gnome -gtk -jpeg -kde -mad -mikmod -mpeg -oggvorbis -opengl -oss -pdflib -png -qt -quicktime -sdl -truetype -xmms -xv"

Pick up again with Installing the Gentoo Base System in the handbook. Where we pick a mirror and start the move from stage1 to stage3. I see that the mirrorselect command has changed between 2004.0 and 2004.3.

# mirrorselect -i -o >> /mnt/gentoo/etc/make.conf
# mirrorselect -i -r -o >> /mnt/gentoo/etc/make.conf

This should have dumped 2 extra lines into your make.conf file (cat /mnt/gentoo/etc/make.conf). Here is what got added to my make.conf file:

GENTOO_MIRRORS="http://gentoo.osuosl.org/ http://csociety-ftp.ecn.purdue.edu/pub/gentoo/ http://gentoo.chem.wisc.edu/gentoo/"
SYNC="rsync://rsync.us.gentoo.org/gentoo-portage"

Now we need to copy some files.

# cp -L /mnt/gentoo/etc/make.conf /mnt/gentoo/boot/make.conf-backupcopy
# cp -L /etc/resolv.conf /mnt/gentoo/etc/resolv.conf
# cp -L /etc/mdadm.conf /mnt/gentoo/etc/mdadm.conf
# cp -L /etc/mdadm.conf /mnt/gentoo/boot/mdadm.conf-backupcopy
# mkdir /mnt/gentoo/etc/lvm
# cp -L /etc/lvm/lvm.conf /mnt/gentoo/etc/lvm/lvm.conf
# cp -L /etc/lvm/lvm.conf /mnt/gentoo/boot/lvm.conf-backupcopy

Change into the new system (note that we already mounted the proc filesystem earlier).

# chroot /mnt/gentoo /bin/bash
# env-update
# source /etc/profile
# emerge --sync

This should update your portage tree to the latest version (and make take a while to run).

(See the next step)

Labels: Gentoo, Gigabyte-GA-6VA7+

Going back to the 2004.3 Gentoo Universal boot CD. Trying to get past my previous issue bd_claim issues when setting up a software RAID. This is on my Gigabyte GA-6VA7+ motherboard (notes on the Gigabyte GA-6VA7+ motherboard and other hardware).

Starting with the usual tricks:

1) Boot the system using the Universal CD
2) ifconfig - find out the IP address of the box
3) passwd - change the root password to something you know
4) Start the SSH daemon - /etc/init.d/sshd start

Since these drives were nuked since my last attempt, I have to re-configure the partitions.

# fdisk /dev/hda

Command: n
Command action: p
Partition number: 1
First cylinder: 1
Last cylinder: +128M
Command: a
Partition number: 1
Command: t
Hex code: fd

Command: n
Command action: p
Partition number: 2
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 2
Hex code: fd

Command: n
Command action: p
Partition number: 3
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 3
Hex code: fd

Command: n
Command action: p
First cylinder: [enter]
Last cylinder: [enter]
Command: t
Partition number: 4
Hex code: fd

Command: p

Command: w

This gives me a 128MB boot area, a 2GB swap area, a 2GB root area, with the rest of the disk set aside for my LVM2 partitions. Repeat the above commands to configure the 2nd disk in the same fashion (/dev/hdc for me).

Now I need to configure software RAID. This is a bit easier then last year since I don't need to muck with the /etc/raidtab file (instead, I'm going to use mdadm).

The following loads the 'md' module and creates the nodes (/dev/md*).

# modprobe md
# ls /dev/md*
ls: /dev/md*: No such file or directory
# for i in 0 1 2 3; do mknod /dev/md$i b 9 $i; done
# ls /dev/md*
/dev/md0 /dev/md1 /dev/md2 /dev/md3

Now, we create our RAID1 sets.

# modprobe raid1
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hda1 /dev/hdc1
mdadm: array /dev/md0 started.
# mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/hda2 /dev/hdc2
mdadm: array /dev/md1 started.
# cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 hdc2[1] hda2[0]
2000256 blocks [2/2] [UU]
[==>..................] resync = 13.8% (277760/2000256) finish=3.6min speed=7920K/sec
md0 : active raid1 hdc1[1] hda1[0]
125376 blocks [2/2] [UU]

unused devices: <none>

Seems to be working fine. Once each RAID set finishes initialization, I'll create the next one in the series using the following commands:

# mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/hda3 /dev/hdc3
# mdadm --create /dev/md3 --level=1 --raid-devices=2 /dev/hda4 /dev/hdc4

The last RAID set will take a while to initialize (2 hours?), so I'm going to go work on other things while it runs. I also need to go back and review the documentation to see what else I need to do when doing software RAID.

Labels: Gentoo, SoftwareRAID

I'm currently fighting with this again. Apparently, there have been some changes between the 2004.0 CD and the 2005.0 CD (mostly related to the 2.6 kernal and the DevFS vs UDev systems).

First off, some tips-n-tricks:

1) ifconfig - find out the IP address of the box
2) passwd - change the root password to something you know
3) Starting the SSH daemon - The command is: /etc/init.d/sshd start

Now I'll be able to use SecureCRT and have an easier time of copy-n-paste. (Pick 'keyboard interactive' and 'ssh2' under options when setting up the connection in SecureCRT.)

Links:

Gentoo 2004.3 Software RAID Install HOWTO
Gentoo.org - x86 tips and tricks (including software RAID)
The Linux Documentation Project: Software RAID
An older gentoo.org thread on software raid

Now... to show you what I did and where it breaks. I have (2) IDE disks, attached as /dev/hda and /dev/hdc. I plan on mirroring them, and have already created /dev/hda1 (128MB for /boot), /dev/hda2 (2GB for swap), /dev/hda3 (2GB for root), and /dev/hda4 (rest of disk for LVM). All partitions are flagged as type 'fd' (linux raid auto).

# modprobe md
# ls /dev/md*
ls: /dev/md*: No such file or directory
# for i in 0 1 2 3; do mknod /dev/md$i b 9 $i; done
# ls /dev/md*
/dev/md0 /dev/md1 /dev/md2 /dev/md3
# nano -w /etc/mdadm.conf

Contents of my mdadm.conf file:

DEVICE /dev/hda1 /dev/hda2 /dev/hda3 /dev/hda4
DEVICE /dev/hdc1 /dev/hdc2 /dev/hdc3 /dev/hdc4

ARRAY /dev/md0 devices=/dev/hda1,/dev/hdc1
ARRAY /dev/md1 devices=/dev/hda2,/dev/hdc2
ARRAY /dev/md2 devices=/dev/hda3,/dev/hdc3
ARRAY /dev/md3 devices=/dev/hda4,/dev/hdc4

Pretty generic... now to create the raid. This is where we hit our first issue:

# modprobe raid1
# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/hda1 /dev/hdc1
mdadm: /dev/hda1 appears to contain an ext2fs file system
size=72192K mtime=Wed Mar 30 16:46:07 2005
mdadm: /dev/hdc1 appears to contain an ext2fs file system
size=72192K mtime=Wed Mar 30 16:46:07 2005
Continue creating array? y
mdadm: ADD_NEW_DISK for /dev/hda1 failed: Device or resource busy
#

Here are the problems:

1) Even though I have a 128MB partition for /boot, it is still showing 64MB from a previous partitioning scheme.

2) mdadm: ADD_NEW_DISK for /dev/hda1 failed: Device or resource busy

So... take a look at my partition list:

livecd root # fdisk -l

Disk /dev/hda: 76.8 GB, 76869918720 bytes
16 heads, 63 sectors/track, 148945 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot Start End Blocks Id System
/dev/hda1 * 1 249 125464+ fd Linux raid autodetect
/dev/hda2 250 4218 2000376 fd Linux raid autodetect
/dev/hda3 4219 8187 2000376 fd Linux raid autodetect
/dev/hda4 8188 148945 70942032 fd Linux raid autodetect

Disk /dev/hdc: 76.8 GB, 76869918720 bytes
16 heads, 63 sectors/track, 148945 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

Device Boot Start End Blocks Id System
/dev/hdc1 * 1 249 125464+ fd Linux raid autodetect
/dev/hdc2 250 4218 2000376 fd Linux raid autodetect
/dev/hdc3 4219 8187 2000376 fd Linux raid autodetect
/dev/hdc4 8188 148945 70942032 fd Linux raid autodetect
livecd root #

All of which looks right and proper.

Labels: Gentoo, SoftwareRAID

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

So, I've created my /etc/raidtab file. I've done the "modprobe md" and "modprobe dm-mod" commands. I'm able to "cat /proc/mdstat" which shows that I have no personalities and no unused devices. But, when I try to use the "mkraid" command, I get the following error:

# mkraid /dev/md0
cannot determine md version: no MD device file in /dev.

The fix (according to the wiki is):

cd /dev ; MAKEDEV md

The key resource to setting up Gentoo on a RAID:

Gentoo Install on Software RAID

Now, they're not putting their swap file on a RAID1. The reason that I put my swap in a RAID1 partition is that I don't want the system to crash if a drive dies (which it will, if you have 2 seperate, non-RAID swap partitions). It's a question of whether you want a slightly higher level of stability, or if you want speed.

Other possible gotches:

1) (not confirmed) After you fdisk, you may want to reboot so make sure that the disks are no longer in use. Otherwise you may get "bd_claim" errors when trying to setup the RAID array.

(still fighting with this a few hours later... going to start a new post)

Labels: Gentoo, SoftwareRAID

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

Okay, tossed the new 2005.0 boot CD in, and I'm booting it up. Just trying a standard default boot for the moment (not trying to use the "nohotplug" option yet).

Load the RAID modules, and verify some things:

# modprobe md
# modprobe dm-mod
# ifconfig { verifies that the ethernet card is working }
# ls -l /dev/hd*

Now, use fdisk to blow away and setup partitions. (Note: You will lose all data on these disks when you perform this step.) Use the 'w' command to confirm the destruction of all partitions when you're finished.

Setup the new partitions:

# fdisk /dev/hda

Command: n
Command action: p
Partition number: 1
First cylinder: 1
Last cylinder: +128M
Command: a
Partition number: 1
Command: t
Hex code: fd

Command: n
Command action: p
Partition number: 2
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 2
Hex code: fd

Command: n
Command action: p
Partition number: 3
First cylinder: [enter]
Last cylinder: +2048M
Command: t
Partition number: 3
Hex code: fd

Command: n
Command action: p
First cylinder: [enter]
Last cylinder: [enter]
Command: t
Partition number: 4
Hex code: fd

Command: p

Command: w

This gives me a 128MB boot area, a 2GB swap area, a 2GB root area, with the rest of the disk set aside for my LVM partitions. Repeat the above commands to configure the 2nd disk in the same fashion. Note that I'm using a different partition type then that shown in chapter 4.c. The 'fd' partition type is what I need to use since all 4 partitions on hda/hdc are going to be put into a software RAID1 set.

The 3rd disk is a single primary partition with the '8E' (LVM) type. (Need to verify this, but I'm pretty sure that's correct.)

Create your "/etc/raidtab" configuration file (I used "nano -w /etc/raidtab", but other text editors will work).

# this config is for mirroring /dev/hda with /dev/hdc
# /boot (RAID1)
raiddev /dev/md0
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/hda1
raid-disk 0
device /dev/hdc1
raid-disk 1

# *swap* (RAID1)
raiddev /dev/md1
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 8
persistent-superblock 1
device /dev/hda2
raid-disk 0
device /dev/hdc2
raid-disk 1

# / (RAID1)
raiddev /dev/md2
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 32
persistent-superblock 1
device /dev/hda3
raid-disk 0
device /dev/hdc3
raid-disk 1

# LVM (RAID1)
raiddev /dev/md3
raid-level 1
nr-raid-disks 2
nr-spare-disks 0
chunk-size 16
persistent-superblock 1
device /dev/hda4
raid-disk 0
device /dev/hdc4
raid-disk 1

# end of /etc/raidtab

Create the raid set(s).

# mkraid /dev/md0
# mkraid /dev/md1
# mkraid /dev/md2
# mkraid /dev/md3

If you get the error message: "raid_disks + spare_disks != nr_disks" when attempting to create any of your RAID sets, go back and verify your "/etc/raidtab" file as well as verifying your disk partitions. The RAID sets will build in the background and you should periodically monitor their progress using "cat /proc/mdstat". Another possibility is that you have set the "chunk-size" setting to be too small or too large (e.g. "chunk-size 4" did not work for me, but "chunk size 8" worked fine).

Building the raid sets may take a while, so once again, I'll come back to this point in a few hours.

Update: mkraid is tossing errors (see my next post)

Labels: Gentoo, Gigabyte-GA-6VA7+

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

While I swap around some memory modules in the old EPIA box, I'm also going to take a swipe at using the new 2005.0 Gentoo image and build a 2nd box.

This is an old Celeron 350Mhz or 400Mhz CPU with 384MB of RAM (Gigabyte GA-6VA7+). About the same power as the VIA EPIA motherboard. I've already gone into the BIOS and disabled all optional ports (serial, parallel, etc). Drives are configured as:

Pri M: 72GB - /dev/hda
Pri S: (free)
Sec M: 72GB - /dev/hdc
Sec S: CD-ROM - /dev/hdd

I also have a 3rd hard drive (80GB, /dev/hde) hooked up to an old Promise FastTrak66 PCI RAID card. It uses the PDC20262 chip and I'm really just using it as an IDE card rather then making use of its RAID functionality.

I plan on mirroring using the two 72GB master drives and using the 80GB as a backup/scratch disk. Similar setup and goals as the EPIA box from last June's install.

Labels: Gentoo, Gigabyte-GA-6VA7+

The RAID array has finally finished synchronizing. Continuing on, referecing my June 2004 notes about setting up LVM.

Things went well up until I started extracting the portage tree. Then I started to get random errors, which means I need to stop and verify that the hardware is behaving properly.

(Time to dig out my memory test programs...)

Labels: Gentoo, VIAEPIA

Note: These directions are works-in-progress... in fact, they might not even work at all until I find out why I'm ending up with non-bootable systems (looks like a bug in the 2.6 kernel).

First up, still using the 2004.0 Gentoo Boot CD and referring to my old notes from last year. Also note that I rebuilt it in June 2004, so it may be better to look at those notes. Especially the "gentoo nohotplug" command during the boot process.

Going to use last June's installation notes for the most part, with a few notes here if I change anything.

Key commands (that don't do anything other then report status):

/sbin/ifconfig - verifies networking
ls -l /dev/hd* - shows hard drives
hdparm -i /dev/hda - display information about hda

Now to fire up fdisk and wipe any existing partitions. Then I'm going to create the same partitions I did last year. (Helps to refer to the Gentoo handbook for this step.)

Currently making my raid volumes, no changes from the June 2004 instructions. Verifying progress using "cat /proc/mdstat".

Labels: Gentoo, VIAEPIA

Note: These directions are works-in-progress... in fact, they might not even work at a