Now that it's spring, it's time for us to start building out our preliminary iSCSI SAN unit. Here's the hardware shopping list:

$0600 Super Micro 4U/TOWER RM EATX BLACK ( CSE-942I-R760B )
- triple-module redundant PSU w/ 760W
- 4U case for either rack or tower use
- (9) 5.25" bays

$0020 20-pin front panel connector to breakout cable
- Converts the 20-pin connector to something that can be attached to normal ATX motherboards
- CBL-0067 30cm
- CBL-0085 15cm

$0050 Rackmount Rail Kit: CSE-PT26

$0320 (2) Spare PSU modules - PWS-0050(M)
- Spare PSU modules for the redundant PSU
- Useful to have a spare or two on-hand

$0600 (4) CSE-M35T1 (black) - SuperMicro SATA 5:3 backplanes
- These allow you to fit a total of (15) SATA drives into the (9) 5.25" bays
- There are other SATA 5:3 backplanes that you can use
- While we're only going to install (3) of these backplanes, I recommend buying a 4th for spare parts

$0167 3848163 (1) INTEL PRO/1000 PT DUAL PORT EXPI9402PT gigabit PCIe x4
- Used for SAN traffic
- Eventually, we'll upgrade to a quad-port PCIe or a 10GigE

$0167 1494573 (1) INTEL PRO/1000 PCI-X
- The PCI card is used to talk to the LAN and internet, no SAN traffic will flow over it
- You could use an inexpensive 10/100 PCI card, but with a dual-port NIC you can bond for high-availability

$0600 12-port Promise SATA-II PCIe x8 card EX12350
- CentOS5 automatically sees any drives attached to this card (when they are configured in JBOD mode)
- We're going the SoftwareRAID route

$0305 TYAN S2927G2NR dual-Opteron Socket F Thunder n3600B (S2927)
$0600 Opteron 2214 dual-core Socket F
$0200 (2) 1GB memory modules
$0100 (2) Socket F cooling fans (Cooljag CJC689C)
- (4) 1.8GHz cores should be plenty of horsepower to do use Software RAID instead of the Promise RAID software
- 2GB is probably minimal for RAM, 4GB would be better

$1800 (15) 500GB SATA-II drives
- 500GB is a good balance between price and capacity

Totals:

$3410 base system
$1800 drives

...

The drive plan for this unit is:

(3) 500GB drives in 3-way RAID1 (mirrored) for the operating system, log files, and other support software

Either:

(10) RAID10 + (2) hot-spares
(2) 5-disk RAID6 + (2) hot-spares

The pair of RAID6 arrays would give us about 20% more capacity (net of 6 disks vs 5 disks). So the RAID10 setup results in around 2.27TB while the RAID6 setup would give 2.72TB.

With an overall cost of around $5500 for the entire unit, the price per gigabytes end up as:

$2.36/GB for (1) RAID10 array
$1.97/GB for (2) RAID6 arrays

Which is not terribly bad for a starter unit.

Labels: iSCSI, SAN

Trying to decide how to allocate disks within the SAN unit. I have (14) or (17) slots. For now, I'll assume that the 5:3 bay units work which will give me a total of 17 disks.

-------------------------------------------------
BAYS / DRIVE CONFIGURATION (2 INT, 15 BAY-COOLER)
-------------------------------------------------
INT1    M/B      RAID1(A)    RAID1(A)
INT2    ''       ''          ''
BCA1    ''       RAID1(B1)   RAID1(B1)
BCA2    ''       RAID1(B2)   RAID1(B2)
BCA3    ''       RAID1(B3)   RAID1(B3)
BCA4    HP2300   RAID1(B1)   RAID1(B1)
BCA5    ''       RAID1(B2)   RAID1(B2)
BCB1    ''       RAID1(B3)   RAID1(B3)
BCB2    ''       HOT SPARE   HOT SPARE
BCB3    HP2320   RAID1 (C1)  RAID6(C)
BCB4    ''       ''          ''
BCB5    ''       RAID1 (C2)  ''
BCC1    ''       ''          ''
BCC2    ''       RAID1 (C3)  ''
BCC3    ''       ''          ''
BCC4    ''       RAID1 (C4)  ''
BCC5    ''       ''          ''

INTx - Internal drive bay at back of case
BCAx - 5:3 bay cooler
BCBx - 5:3 bay cooler
BCCx - 5:3 bay cooler

M/B - Indicates that I'm using the 5 SATA ports on the motherboard
HP2300 - HighPoint RocketRAID 2300 PCIe x1 SATA 4-port
HP2320 - HighPoint RocketRAID 2320 PCIe x4 SATA 8-port

A) In the first configuration I have:

700GB RAID1
2100GB RAID0 over (3) RAID1 sets
2800GB RAID0 over (4) RAID1 sets
====
5600GB total (11200GB gross capacity)

B) The second configuration sets up a 8-disk RAID6 array

700GB RAID1
2100GB RAID0 over (3) RAID1 sets
4200GB RAID6 over 8 disks
====
7000GB total (11200GB gross capacity)

The RAID0 over (3) RAID1 sets (a.k.a. RAID 10) should give me roughly 3x the performance of a regular RAID1 volume. Reads and writes should both see a 3x improvement over a simple RAID1.

For the RAID6 volume, I estimate that read performance will be 6x that of the RAID1 set but I'm not sure what write performance will be.

Labels: iSCSI, RAID, SAN

So here's my first stab at a SAN unit that can hold 14 or 17 SATA drives. All of the drives will be mounted in hot-swap trays which should make things much easier. Some of the components are a bit overkill (such as the pair of dual-port server NICs) but I'm planning ahead to when we have two gigabit switches and we want to connect multiple gigabit ports together for speed.

For the case, motherboard and misc parts:

$0159 Thermaltake Armor VA8000BNS Black Chassis: 1.0mm SECC
$0190 Thermaltake ToughPower W0117RU ATX12V/ EPS12V 750W
$0035 DVD-RW (BLACK)
$0050 misc parts (fans, cables)
$0182 MB-BA22658 AMD Athlon64 X2 4200+ AM2 (WINDSOR)
$0200 XXXXXXXXXX Asus M2N32-SLI DLX
$0138 XXXXXXXXXX Mwave 2GB DDR2 533 (1GB x 2)
$0009 XXXXXXXXXX Assemble & Test

The motherboard is a ASUS M2N32-SLI with multiple PCIe slots (2 x16, 1 x4, 1 x1), 2 PCI slots, 8 SATA-II ports, and dual-NICs on an nForce 590 chipset. I plan on using the expansion slots as follows:

PCIe x16: Intel Pro/1000 PT Dual-Port PCIe x4
PCIe x4: HP RocketRAID 2320 8-port PCIe x4
PCIe x1: HP RocketRAID 2300 4-port PCIe x4
PCIe x16: Intel Pro/1000 PT Dual-Port PCIe x4
PCI:
PCI: PCI video card

Note: I expect that you will need a fairly recent BIOS version in order to use the first x16 slot for something other then a video card.

Prices for the SATA controllers and NICs:

$0167 INTEL PRO/1000 PT DUAL PORT EXPI9402PT gigabit PCIe x4
$0140 HighPoint RocketRAID 2300 PCIe x1 (4-port SATA-II)
$0260 HighPoint RocketRAID 2320 PCIe x4 (8-port SATA-II)

Note: I'm not 100% sure that I'm going to use the RocketRAID 2320. I still need to do some research to verify that it works properly in Linux without special binary drivers (I want it to work as a regular controller card). Otherwise I may use either the PROMISE SuperTrak EX8350 PCIe x4 (8-port SATA-II) or the 3ware 9590SE-8ML PCIe x4 (8-port SATA-II).

Other components:

$0350 Seagate Barracuda 7200.10 750GB SATA-II
$0110 Athena Power 5:3 SATA-II Backplane SATA3051B (350SATA)

The 5:3 backplane should allow me to fit 15 drives into the front 5.25" expansion bays on the Thermaltake Armor case. If I don't care for the design of the 5:3 backplane there are more conservative 4:3 backplanes that will allow me to fit 12 drives into the front bays.

Or I could even use some 4:3 SCSI SCA hot-swap enclosures along with a PCIe x4 SCSI card. That lets me have both SATA and SCSI drives in the same unit. A pair of those would give me 6 SATA drives and 8 SCSI drives.

Estimated cost for the starter kit is $3200, including a pair of 750GB SATA drives. Since I'll be building two of these, and then gradually expanding them:

$3200 02 - 700GB SAN (2 base disks per SAN) -- $4.57/GB
$3200 02 - redundancy for 700GB SAN (2 base disks per SAN) -- $9.14/GB
$1400 04 - expansion to 1.4TB (2 data disks per SAN added) -- $5.57/GB
$2800 08 - expansion to 2.8TB or reconfig to 3.5TB -- $3.78 to $3.03/GB
$2800 12 - expansion to 4.2TB or reconfig to 6.3TB -- $3.19 to $2.13/GB
$1400 14 - expansion to 4.9TB or reconfig to 7.7TB -- $3.02 to $1.92/GB

The 2nd column is the number of drives installed within a single SAN unit. The first capacity is if I stick with my initial plan of RAID1 within the SAN unit and then RAID1 across the SAN fabric for redundancy. The second capacity is if I RAID5 within the unit and then RAID1 across the SAN fabric for redundancy.

I may also expand the memory in the SAN boxes to 6GB or 8GB down the road to maximize any possible read-caching.

For home use, I wouldn't bother to build a 2nd SAN unit for redundancy. Downtime in case of a blown power-supply would only be about 2 hours (assuming you have one on-hand) to a day or two. I could also cut some costs by going with less expensive NICs or SATA controllers. But that doesn't gain you much.

The other advantage of building out slowly is that you can take advantage of the 1TB and 2TB drives that might appear in 2007-2009.

Labels: iSCSI, SAN