Search:  
Gentoo Wiki

HOWTO_Boot_From_SAN

Note: Updated for Gentoo 2007.0: August 20, 2007 ... Only thing that had to be changed for 2007.0 are version numbers. I also downloaded newest QLogic driver from QLogic's site (qla2xxx-v8.01.07.15-2-dist.tgz) and tested with this configuration. For this new driver, I had to modify some of the source files in QLogic's tar ball; see below for details. Option B for the QLogic HBA driver included in the Linux kernel, still does not work. If anyone has any information on loading external firmware with initrd, please contribute.


Contents

A Few Things

This guide will show you how to install Gentoo directly to a LUN on your SAN and boot from this LUN. I am using a QLogic HBA and I have a Sun StorageTek 6140 disk array, but I'm sure much of this could be applied to a different setup.

Assumptions:

EMC Support:

Note: System configurations using Gentoo is not supported by EMC corp. You can run configurations through the EMC Support matrix at http://powerlink.emc.com to see if they are supported. EMC OS/API Unix will not be able to trouble shoot any Gentoo configuration. ~ EMC OS/API UNIX Engineer
Note: I have an EMC Clariion CX300 and I have not been able to get it to work correctly with QLogic's (very nice w/ built-in MPIO) driver. I'm pretty sure the host initiator type/settings need to be tweaked for it to work. Maybe there is no right settings this will make it work? I've played around with a few different combinations, but I couldn't get any to work. To edit the initiator type in Navisphere: Go to 'Connectivity Status...', then click 'Group Edit...'. Inside this dialog box you can choose the initiators you'd like to modify. You can adjust the Initiator Type, Failover Mode, etc. Please contribute if you get this working.
Note: Let me tell you the Qlogics driver internal failover capability MUST be disabled in order to work properly on Linux. If you need

Install CD

Enable the QLogic BIOS option (CTRL + Q to enter controller configuration at boot time). Also, in the configuration menu, set the LUN you would like to boot . Set the HBA controller card to the first boot device in your system BIOS.

Boot your Gentoo install CD. Configure your network according to the Gentoo Handbook. I am using a 2006.1 install CD, but the firmware doesn't appear to be included:

QLogic Fibre Channel HBA Driver
ACPI: PCI Interrupt 0000:06:09.0[A] -> GSI 36 (level, low) -> IRQ 22
qla2xxx 0000:06:09.0: Found an ISP2312, irq 22, iobase 0xf8904000
qla2xxx 0000:06:09.0: Configuring PCI space...
qla2xxx 0000:06:09.0: Configure NVRAM parameters...
qla2xxx 0000:06:09.0: Verifying loaded RISC code...
qla2xxx 0000:06:09.0: Firmware image unavailable.
qla2xxx 0000:06:09.0: Failed to initialize adapter

Not a problem, we just need to grab the binary blob and put it into /lib/firmware like this:

wget -P /lib/firmware ftp://ftp.qlogic.com/outgoing/linux/firmware/ql2300_fw.bin

Of course you'll need to make sure you get the appropriate file for your card. Now that you have the firmware and its accessible by the qla2xxx driver, simply unload and load the module:

modprobe -r qla2xxx

modprobe qla2xxx

Now our volume should be detected:

QLogic Fibre Channel HBA Driver
ACPI: PCI Interrupt 0000:06:09.0[A] -> GSI 36 (level, low) -> IRQ 22
qla2xxx 0000:06:09.0: Found an ISP2312, irq 22, iobase 0xf8904000
qla2xxx 0000:06:09.0: Configuring PCI space...
qla2xxx 0000:06:09.0: Configure NVRAM parameters...
qla2xxx 0000:06:09.0: Verifying loaded RISC code...
qla2xxx 0000:06:09.0: Waiting for LIP to complete...
qla2xxx 0000:06:09.0: LIP reset occured (f7f7).
qla2xxx 0000:06:09.0: LIP occured (f7f7).
qla2xxx 0000:06:09.0: LOOP UP detected (2 Gbps).
qla2xxx 0000:06:09.0: Topology - (Loop), Host Loop address 0x0
scsi1 : qla2xxx
qla2xxx 0000:06:09.0: 
 QLogic Fibre Channel HBA Driver: 8.01.04-k
  QLogic QLA2340 - 
  ISP2312: PCI-X (133 MHz) @ 0000:06:09.0 hdma-, host#=1, fw=3.03.20 IPX
  Vendor: SUN       Model: CSM200_R          Rev: 0616
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sda: 52428800 512-byte hdwr sectors (26844 MB)
sda: Write Protect is off
sda: Mode Sense: 77 00 10 08
SCSI device sda: drive cache: write back w/ FUA
SCSI device sda: 52428800 512-byte hdwr sectors (26844 MB)
sda: Write Protect is off
sda: Mode Sense: 77 00 10 08
SCSI device sda: drive cache: write back w/ FUA
 sda: unknown partition table
sd 1:0:0:0: Attached scsi disk sda
  Vendor: SUN       Model: Universal Xport   Rev: 0616
  Type:   Direct-Access                      ANSI SCSI revision: 03
SCSI device sdb: 40960 512-byte hdwr sectors (21 MB)
sdb: Write Protect is off
sdb: Mode Sense: 77 00 10 08
SCSI device sdb: drive cache: write through w/ FUA
SCSI device sdb: 40960 512-byte hdwr sectors (21 MB)
sdb: Write Protect is off
sdb: Mode Sense: 77 00 10 08
SCSI device sdb: drive cache: write through w/ FUA
 sdb: unknown partition table
sd 1:0:0:31: Attached scsi disk sdb

I ran into a strange problem when I first started setting this up; I couldn't access every other LUN I added to the host. I would get block error messages. To fix this, I changed the initiator host type to Solaris. I was using host type Linux, which seems to be the obvious choice, but I had that problem. Using host type Solaris seems to work fine.

So, now that we have our SCSI disk (/dev/sda) you can follow the Gentoo Handbook for setting this up (just like any other disk you install Gentoo on). The device node may be different for you, especially if you have other SCSI/RAID disks in your system. Continue the install procedure following the Gentoo Handbook until you get to setting up the kernel.

Setup Driver

Now its time to configure the kernel and HBA driver, the fun stuff. =) We have a couple options here, you can use QLogic's driver (my personal choice) or you can use the qla2xxx driver that comes with the Linux kernel.

If you use QLogic's proprietary driver, you get MPIO (failover) at the driver level and you won't have to install another MPIO solution (such as multipath-tools, EMC's PowerPath, Sun's RDAC driver, etc.). The firmware is also included at compile time, so you won't have to mess around with an external firmware file.

If you use the qla2xxx driver thats included in the kernel source, it will be updated with each kernel revision. You will still have to worry about external firmware for this driver though.

You'll still end having to create an initrd image when you want to update the driver with either choice. Oh, one more thing, no genkernel ladies! All of these directions assume you do manual configuration and configure a mostly monolithic kernel.

Option A – Using QLogic's driver

Configure the kernel manually like usual making sure you enable loading/unloading modules, RAM disk support and initramfs/initrd (loopback device is also helpful), but DO NOT select an HBA driver. Compile and install the kernel/modules like normal. Now, you need to get the driver from QLogic's website and put it into /usr/src. The driver I obtained is qla2xxx-v8.01.07-1-dist.tgz. Make sure you get the newest driver; the previous version of this driver would not work with 2.6.19 kernels. Now lets get to it:

tar xvfz qla2xxx-v8.01.07.15-2-dist.tgz

cd qlogic

./drvrsetup

cd qla2xxx-8.01.07.15

I had to change 2 lines in the extras/build.sh script:

This line: K_VERSION=`uname -r`
Should change to: K_VERSION="2.6.22-gentoo-r2"
This line: KVERSION=`uname -r | sed "s/-.*//" | sed "s/\.//g"`
Should change to: KVERSION=`echo "2.6.22-gentoo-r2" | sed "s/-.*//" | sed "s/\.//g"`

For this new kernel version, I also had to change some lines in the source files. See this note from Linus: http://readlist.com/lists/vger.kernel.org/linux-kernel/67/337925.html You probably just have to change the file that is required for your HBA, but it doesn't hurt to change them all:

grep -r pci_module_init *
qla2xxx-8.01.07.15/ql2300.c:       return pci_module_init(&qla2300_pci_driver);
qla2xxx-8.01.07.15/ql2100.c:       return pci_module_init(&qla2100_pci_driver);
qla2xxx-8.01.07.15/ql2322.c:       return pci_module_init(&qla2322_pci_driver);
qla2xxx-8.01.07.15/ql2200.c:       return pci_module_init(&qla2200_pci_driver);
qla2xxx-8.01.07.15/ql2400.c:       return pci_module_init(&qla24xx_pci_driver);

Change the "pci_module_init" in each file to "pci_register_driver" otherwise you will get a 'Unknown symbol: pci_module_init' when attempting to insert the module.


extras/build.sh install

You should then get something like this:

QLA2XXX -- Building the qla2xxx driver, please wait...
Installing intermodule.ko in /lib/modules/2.6.22-gentoo-r2/kernel/kernel/
QLA2XXX -- Build done.

QLA2XXX -- Installing the qla2xxx modules to 
/lib/modules/2.6.22-gentoo-r2/kernel/drivers/scsi/qla2xxx/...
WARNING: Couldn't open directory /lib/modules/2.6.19-gentoo-r5: No such file or directory
FATAL: Could not open /lib/modules/2.6.19-gentoo-r5/modules.dep.temp for writing: No such file or directory

I think the error at the end comes from depmod. If you select other drivers or options (that the QLogic driver may depend on) as modules, you should probably resolve the error so depmod finds your module dependencies correctly. Also, if you get additional errors, diagnose and fix whats wrong then do a 'extras/build.sh clean' and 'extras/build.sh install' again.

Now continue configuring your system and installing the necessary system tools according to the Gentoo Handbook.

We need mkinitrd:

emerge mkinitrd

Its time to create the initrd image:

mkinitrd -v --with=intermodule --with=qla2xxx --with=qla2300 /boot/initrd-2.6.22-gentoo-r2.gz 2.6.22-gentoo-r2

I'm not sure if this next step is totally necessary, but it gives the modules some time and makes the output a little neater on the screen. I also like to make sure the linuxrc script looks kosher.

gunzip /boot/initrd-2.6.22-gentoo-r2.gz

mount -o loop /boot/initrd-2.6.22-gentoo-r2 /mnt

I just edit the /mnt/linuxrc file and add 3 sleep commands to it after each insmod (there are 3 modules), so my linuxrc now looks like this:

File: /mnt/linuxrc
 #!/bin/nash
 
 echo "Loading intermodule.ko module"
 insmod /lib/intermodule.ko
 sleep 1
 echo "Loading qla2xxx.ko module"
 insmod /lib/qla2xxx.ko
 sleep 1
 echo "Loading qla2300.ko module"
 insmod /lib/qla2300.ko
 sleep 1
 echo Mounting /proc filesystem
 mount -t proc /proc /proc
 echo Creating block devices
 mkdevices /dev
 echo Creating root device
 mkrootdev /dev/root
 echo 0x0100 > /proc/sys/kernel/real-root-dev
 echo Mounting root filesystem
 mount -o noatime --ro -t ext3 /dev/root /sysroot
 pivot_root /sysroot /sysroot/initrd
 umount /initrd/proc
 

Again, its probably not necessary to do that, but I did anyways. =) Now put it back to normal:

umount /mnt

gzip /boot/initrd-2.6.22-gentoo-r2

Make sure you make the initrd mount point (for when pivot_root moves root):

mkdir /initrd

We need to make sure the kernel knows what our root device is:

rdev /boot/kernel-2.6.22-gentoo-r2

Root device /dev/sda3

If it doesn't give you the correct root device you're going to be using, set it with rdev. I've found that initrd basically ignores the 'root=' line in grub.conf or at least it doesn't like it when they're not the same. I had strange problems with this. Anyone got any ideas? Anyone else experience the same weirdness?

Option B – Using Linux kernel driver

Warning: Option B doesn't actually work. I started out trying to get this one to work before, then I realized QLogic's driver has MPIO, so I stopped messing around with it. A recent version of udev is required to load the firmware. It'd be nice if someone could add what else needs to be done to get it working! =)

Configure the kernel manually like usual making sure you enable loading/unloading modules, RAM disk support and initramfs/initrd (loopback device is also helpful), and select the QLA2XXX HBA driver to compile as a module (its in SCSI low-level drivers). When you select the driver to be a module, make sure you take note if it selects any other stuff. If it selects dependencies as modules, be sure to change them to built-in. Compile and install the kernel/modules like normal.

Now continue configuring your system and installing the necessary system tools according to the Gentoo Handbook.

We need mkinitrd:

emerge mkinitrd

Next we'll need to create the initrd image:

mkinitrd -v --with=qla2xxx /boot/initrd-2.6.19-gentoo-r5.gz 2.6.19-gentoo-r5

We need to add the firmware file to our initrd image. I also add a sleep after the insmod, probably not needed, but I do it anyways.

gunzip /boot/initrd-2.6.19-gentoo-r5.gz

mount -o loop /boot/initrd-2.6.19-gentoo-r5 /mnt

I just edit the /mnt/linuxrc file and add the sleep command to it after the insmod, so my linuxrc now looks like this:

File: /mnt/linuxrc
 #!/bin/nash
 
 echo "Loading qla2xxx.ko module"
 insmod /lib/qla2xxx.ko
 sleep 5
 echo Mounting /proc filesystem
 mount -t proc /proc /proc
 echo Creating block devices
 mkdevices /dev
 echo Creating root device
 mkrootdev /dev/root
 echo 0x0100 > /proc/sys/kernel/real-root-dev
 echo Mounting root filesystem
 mount -o noatime --ro -t ext3 /dev/root /sysroot
 pivot_root /sysroot /sysroot/initrd
 umount /initrd/proc
 

Next we can add the firmware file to the initrd image:

mkdir /mnt/lib/firmware

wget -P /mnt/lib/firmware ftp://ftp.qlogic.com/outgoing/linux/firmware/ql2300_fw.bin

Now put it back to normal:

umount /mnt

gzip /boot/initrd-2.6.19-gentoo-r5

Make sure you make the initrd mount point (for when pivot_root moves root):

mkdir /initrd

We need to make sure the kernel knows what our root device is:

rdev /boot/kernel-2.6.19-gentoo-r5

Root device /dev/sda3

I had strange problems with initrd if rdev wasn't set right.

Option B' – Using Linux kernel driver and firmware

File: genkernel-initramfs-with-qla2xxx-firmware.sh
#!/bin/bash

#-- 2008-01-18, Stephane BUNEL (http://stephane.bpf.st/si/os/qla2xxx)
#
# This script is designed to permit Gentoo GNU/Linux using genkernel
# (tested with v3.4.9) to boot from SAN with a QLogic HBA (ISP2xxx)
# controler. You must have installed the sys-block/qla-fc-firmware
# package with emerge before starting.
#
# Genkernel (at least for v3.4.9) is unable to produce an initramfs
# which is able to load firmware needed by qla2xxx driver at boot time.
#
# The script presumes that qla2xxx driver is compiled as a module.
#
# Simply run the script and it will rebuild an initramfs with the
# necessary stuff to load firmware at boot time.
#
# Tested only on x86 architecture (feel free to modify as you need).
#
# See also: http://www.gentoo-wiki.info/HOWTO_Boot_From_SAN


#-- am I root ?
ID=$(id -u)
if test ! "${ID}" -eq '0'
then
    echo "Script must be run as root. Exiting"
    exit 1
fi

#-- X86 architecture ?
ARCH=$(uname -m)
case "${ARCH}" in
    i?86)
	ARCH="x86"
	;;
    *)
	echo "This script was only tested for x86 architecture. Exiting"
	exit 1
	;;
esac

#-- Is qla2xxx in all_modules 
GENKERNEL=/usr/share/genkernel
MODLOAD="${GENKERNEL}/${ARCH}/modules_load"
if test ! -f ${MODLOAD}
then
    echo "Can't find ${MODLOAD}. Is genkernel installed ? Exiting"
    exit 1 
fi

if grep -i qla2xxx ${MODLOAD} >/dev/null
then
    echo "Well qla2xxx is already in SCSI module list"
else
    t=/tmp/modload$$
    cat ${MODLOAD} | sed "s/qla1280/qla1280 qla2xxx/" >${t}
    cp -f ${t} ${MODLOAD}
    rm -f ${t}
    echo "adding qla2xxx in SCSI module list"
fi

overlay=/tmp/initramfs$$

mkdir -p ${overlay}/lib/firmware ${overlay}/sbin
cp -r /lib/firmware/* ${overlay}/lib/firmware/

cat >${overlay}/sbin/hotplug <<EOF
#!/bin/ash
# Both $DEVPATH and $FIRMWARE are already provided in the environment.

HOTPLUG_FW_DIR=/lib/firmware

if test ! -z \${FIRMWARE}
then
  mount -t sysfs sysfs /sys
  echo 1 > "/sys/\${DEVPATH}/loading"
  src="\${HOTPLUG_FW_DIR}/\${FIRMWARE}"
  dst="/sys/\${DEVPATH}/data"
  cp "\${src}" "\${dst}"
  echo 0 > "/sys/\${DEVPATH}/loading"
  umount /sys
fi
EOF

chmod a+x ${overlay}/sbin/hotplug
genkernel initrd --initramfs-overlay=${overlay} --install
rm -rf ${overlay}
 

Configure Bootloader

We will now configure the bootloader:

emerge grub

vi /boot/grub/grub.conf

Your grub.conf should look something like this:

File: /boot/grub/grub.conf
 default 0
 timeout 5
 
 title Gentoo Linux 2.6.19-r5
 root (hd0,0)
 kernel /boot/kernel-2.6.19-gentoo-r5
 initrd /boot/initrd-2.6.19-gentoo-r5.gz
 

You'll notice I didn't set a 'root='; I actually had problems with initrd when I would set 'root=' in grub.conf. If you're kernel has the correct rdev set (as mentioned in step 2) you don't really need one of these anyhow.

With your grub.conf file set, lets move on:

grep -v rootfs /proc/mounts > /etc/mtab

grub-install /dev/sda

You might have to switch some things around; grub-install displays your device nodes along with the BIOS disk (hd0, hd1, etc.). Make sure this corresponds to what you put in grub.conf. If you followed the directions in step 1 (enabled your QLogic BIOS, set boot LUN and set the HBA controller to the first boot device in system BIOS) then it should be hd0.

Reboot according to the Gentoo Handbook and you're done!

Retrieved from "http://www.gentoo-wiki.info/HOWTO_Boot_From_SAN"

Last modified: Fri, 26 Sep 2008 03:04:00 +0000 Hits: 14,095