Search:  
Gentoo Wiki

HOWTO_Custom_Stage4


This article is part of the HOWTO series.
Installation Kernel & Hardware Networks Portage Software System X Server Gaming Non-x86 Emulators Misc

Contents

Introduction

This article will demonstrate how to create a custom stage4 archive. A stage4 archive is an image of your entire root partition. The primary reason to create a stage4 archive is to provide for quick recovery in the event of disk failure. A stage4 archive is the same as a stage3, only you can select the CFLAGS you want to use and what other software you want installed. You can adapt this method to suit your personal uses.

If you want a generic stage4 that you can install on multiple systems, use Genkernel to set up your kernel. This will ensure that on boot, the kernel will work like the one on the livecd. You may also want to use less restrictive CFLAGS (i.e., MCPU instead of MARCH). You may still have to modify /etc/fstab and the USE flags after extracting the stage4, to suit the user.

This article assumes you have already installed Gentoo, if not go to the handbook and do the install first.

This article also assumes that you have already setup any other software you may want to have on your system. For example, on a desktop system you may want X, Xfce4, Sun's JDK, CVS, Emacs, Thunderbird, and Firefox.

(Optional) Tar /boot

Make a copy of your /boot partition:

# mount /boot
# cp -R /boot /bootcpy
# umount /boot

Alternatively, you can just mount /boot before you tar everything up. However, there may be problems with this alternate method if you use the stage4 on systems with different hardware configurations.

Clean Up

If you are cloning machines make sure you use different /etc/ssh/ssh_host_* keys.

rm /etc/ssh/ssh_host*

You can follow the Running out of disk space FAQ to free more space.

Use caution in clearing /var/tmp. On some systems, config information was stored in there for sound and portage. This can cause problems later, when restoring the system. It may be better to have a slightly larger tarball than to find that the restored system is incomplete.

(If you use an exclude file like the one shown in the next paragraph, this step can be omitted.)

Tar the System

Create an exclusion file that tells tar what not to pack into your stage 4 archive. Ending listed directories with /* instead of / will include the directory itself in the tar, but nothing inside the directory, thereby saving you the hassle of recreating them once you've restored.

stage4.excl contents:

.bash_history
/mnt/*
/tmp/*
/proc/*
/sys/*
/dev/*
/etc/ssh/ssh_host_*
/usr/src/*
/usr/portage/*
/path/to/save/at/stage4.tar.bz2

After everything is set up correctly from the previous sections, we will create the archive:

# tar cvjf /path/to/save/at/stage4.tar.bz2 / -X stage4.excl

tar options we used:
 c - create archive
 v - verbosely list files processed
 j - use bzip2 compression
 f - specify file name
 X - use the specified exclusion file
tar option  we DIDN'T use:
 p - preserve file attributes (necessary only during extraction of the tar, not during creation)

Be sure to look at the tar manpage

It will probably take a long time to create the archive depending on how much you have installed. I usually save the archive on a spare disk I have in my system that I use for backups. You can also burn it to a cd or dvd.

If it is too large to fit on a cd you will have to split it up. Large tar files can be split up with 'split'. The parts can be joined later with 'cat'. Read the manpages for these tools before you use them.

Here are some scripts to make this step a little easier.

Install from a stage4

# date MMDDHHmmCCYY
# hwclock --systohc
# cd /mnt/gentoo
# tar -xvjpf stage4.tar.bz2
# cp -R /mnt/gentoo/bootcpy/* /mnt/gentoo/boot/

Double check the boot dir after you copy the files over before deleting /bootcpy!
# rm -rf /mnt/gentoo/bootcpy
# mknod -m 660 /mnt/gentoo/dev/console c 5 1
# mknod -m 660 /mnt/gentoo/dev/null c 1 3
(optional) # mknod -m 660 /mnt/gentoo/dev/tty1 c 4 1
# mount -t proc none /mnt/gentoo/proc
# mount -o bind /dev /mnt/gentoo/dev
# cd /mnt/gentoo/usr/
# wget http://gentoo.osuosl.org/snapshots/portage-latest.tar.bz2
# tar xjf portage-latest.tar.bz2
# chroot /mnt/gentoo /bin/bash
# env-update; source /etc/profile

Partition Table tip

To store partition table info:

dd if=/dev/(your_disk) of=mbr.save count=1 bs=512
sfdisk -d /dev/(your_disk) > partitions.save

The first of those saves the mbr and the second will store all partition info (including logical partitions, which aren't part of the mbr).

To restore partition info:

dd if=mbr.save of=/dev/(your_disk)
sfdisk /dev/(your_disk) < partitions.save

GRUB tip

After chrooting:

root# grub
grub> root (hd0,0)
grub> setup (hd0)
grub> quit

(hd may need to be changed depending on your setup)

udev tip

This took me 2 days to understand so listen carefully. If your eth0 controller doesn't come up after recovery, take a look at the following file:

/etc/udev/rules.d/70-persistent-net.rules

This took me way too long to find out by myself, so I hope this will help. The issue is that the MAC address probably changed on your NIC, so you need to update eth0 with the correct MAC, which is probably automatically setup as eth1.

Sample Stage 4 Build Scripts

Small script, which I use for making backup of my /home. (The /dev/hdX corresponds to the /mnt/tarbackup directory in /etc/fstab.)

lazy_bum removed:

File: backupHome.sh
#! /bin/bash
# Backup script for Gentoo Linux
# Author: fdavid
# Date: 2003.11.29.

# Making backup of home partition to cds

# options for the archive
tarOptions="--create --absolute-names --gzip --file"

# name of the archive
archive=/mnt/tarbackup/$(date +%Y%m%d)home.tar.gz

# mount the backup partition
mount /dev/hdaX
sleep 5

# create the archive
tar ${tarOptions} ${archive} /home/;
echo archive is done

# split the archive to cd size 
# (use: "cat ${archive}.* >> ${archive}" to join the parts)
split --bytes=700000000 ${archive} ${archive}.
echo splitting is done

# unmount
sleep 5
umount /dev/hdaX

odessit added:

Note: test restoration in your environment BEFORE using encryption in production!

BrianW added:

File: mkstage4.sh
#! /bin/bash
##  Backup script for Gentoo Linux
##  Change Log

##  2006.07.20 by odessit
##  Added symmetric & asymmetric encryption options to the script using GnuPG
##  to accomodate secure environments

##  Date: 2006.03.05 by BrianW
##  Initial Script
##  Adapted from backupHome.sh by fdavid
##  Adapted from mkstage4.sh by nianderson

##  This is a script to create a custom stage 4 tarball (System and boot backup)
##  I use this script to make a snapshot of my system. Meant to be done weekly in my case

##  Please check the options and adjust to your specifics.

echo -=- Starting the Backup Script...
echo -=-

echo -=- Setting the variables...

##  The location of the stage 4 tarball.
##  Be sure to include a trailing /
stage4Location=/

##  The name of the stage 4 tarball.
archive=$stage4Location$(hostname)-stage4.tar.bz2

##  Directories/files that will be exluded from the stage 4 tarball.
##
##  Add directories that will be recursively excluded, delimited by a space.
##  Be sure to omit the trailing /
dir_excludes="/mnt/* /dev /proc /sys /tmp /usr/portage /var/tmp"
##
##  Add files that will be excluded, delimited by a space.
##  You can use the * wildcard for multiple matches.
##  There should always be $archive listed or bad things will happen.
file_excludes="$archive"
##
##  Combine the two *-excludes variables into the $excludes variable
excludes="$(for i in $dir_excludes; do if [ -d $i ]; then \
    echo -n " --exclude=$i/*"; fi; done) $(for i in $file_excludes; do \
    echo -n " --exclude=$i"; done)"

##  The options for the stage 4 tarball.
tarOptions="$excludes --create --absolute-names --preserve-permissions --bzip2 --verbose --totals --file"

echo -=- Done!
echo -=-

##  Mounting the boot partition
echo -=- Mounting boot partition, then sleeping for 5 seconds...
mount /boot
sleep 5
echo -=- Done!
echo -=-

##  Creating a copy of the boot partition (copy /boot to /bootcpy).
##  This will allow the archiving of /boot without /boot needing to be mounted.
##  This will aid in restoring the system.
echo -=- Copying /boot to /bootcpy ...
cp -R /boot /bootcpy
echo -=- Done!
echo -=-

##  Unmounting /boot
echo -=- Unmounting /boot then sleeping for 5 seconds...
umount /boot
sleep 5
echo -=- Done!
echo -=-

##  Creating the stage 4 tarball.
echo -=- Creating custom stage 4 tarball \=\=\> $archive
echo -=-
echo -=- Running the following command:
echo -=- tar ${tarOptions} ${archive} /
tar ${tarOptions} ${archive} /;
echo -=- Done!


##  Uncomment this portion to encrypt the stage4 using asymmetric encryption with GnuPG
##  To get a list of available ciphers on your machine do #gpg --version
##  Note: only aes aes192 aes256 and twofish should be used for large files (~1Gig & larger)
##  To Restore: (cat together if split) and #gpg stage4.tar.bz2.gpg
#echo -=- Starting Encryption Process usung asymmetric encryption with a public key  -=-
#cd $stage4Location
#gpg --encrypt --batch --recipient [MY_PUB_KEY] --cipher-algo twofish --output $archive.gpg $archive
#rm $archive
#archive=$stage4Location$(hostname)-stage4.tar.bz2.gpg

##  Uncomment this portion to encrypt the stage4 using symmetric encryption with GnuPG
##  ##  To get a list of ciphers on your machine do #gpg --version
##  Note: only aes aes192 aes256 and Twofish should be used for large files (~1 gig & larger)
##  To Restore: (cat together if split) and #gpg stage4.tar.bz2.gpg
#echo -=- Starting Encryption Process usung symmetric encryption with a password  -=-
#cd $stage4Location
#pass=PLEASE_CHANGE_ME_I_BEG_YOU!
#echo $pass | gpg --batch --cipher-algo twofish --passphrase-fd 0 --symmetric $archive
#rm $archive
#archive=$stage4Location$(hostname)-stage4.tar.bz2.gpg

##  Split the stage 4 tarball in cd size tar files.
##  To combine the tar files after copying them to your
##  chroot do the following: "cat *.tar.bz2 >> stage4.tar.bz2".
##  or if encrypted do the following: "cat *.tar.bz2.gpg >> stage4.tar.bz2.gpg".
##  Uncomment the following lines to enable this feature.
#echo -=- Splitting the stage 4 tarball into CD size tar files...
#split --bytes=700000000 ${archive} ${archive}.
#echo -=- Done!

##  Removing the directory /bootcpy.
##  You may safely uncomment this if you wish to keep /bootcpy.
echo -=- Removing the directory /bootcpy ...
rm -rf /bootcpy
echo -=- Done!
echo -=-

##  This is the end of the line.
echo -=- The Backup Script has completed!

Here is a little script to install from a stage4 created by either of the above two scripts. (Limited testing reported by rpcyan thanks please continue testing) should work though. A few changes were made by rpcyan to get the install script working correctly Please contact me with comments or improvements. nianderson

File: installstage4.sh
#! /bin/bash
# Backup script for Gentoo Linux
# Author: nianderson
# Date: 2004.09.15.
#


#Used to install a stage4 created with mkstage4.sh
#assumes you have already partitioned and formated your drive
#assumes your booted from gentoo live cd
#assumes you already decrypted your *.gpg archive (#gpg stage4.tar.bz2.gpg)

#Define Disk layout
#if using ide disks use hdax if using scsi use sdax

rootPartition=/dev/sda3
bootPartition=/dev/sda1

#where to mount the disk partitions
mntRootPartition=/mnt/gentoo
mntBootPartition=/mnt/gentoo/boot

#URL of stage4
#I put a copy of the tar on a webserver so i can
#easily get it when a reinstall is needed
urlToStage4=http://domain.com/
stage4=hostname-stage4.tar.bz2

#mount root partition
echo mounting root partition $rootPartition to $mntRootPartition
mount $rootPartition $mntRootPartition
sleep 5
echo

#not sure about this part yet
#wget the stage4 to the mounted root partition
cd $mntRootPartition
echo wget $urlToStage4$stage4 to $mntRootPartition
wget $urlToStage4$stage4
sleep 5

#untar the stage4
echo extract stage4
tar xjpf $stage4
sleep 5

echo

#mount boot partiton
echo mounting $bootPartition to $mntBootPartition
mkdir $mntbootPartition
mount $bootPartition $mntBootPartition
sleep 5
echo

#copy boot copy back to boot
echo copy bootcpy back to boot
cp -R $mntRootPartition/bootcpy $mntBootPartition
sleep 5

#remove stage4 file
rm -rf $mntRootPartition/$stage4

echo you need to check your fstab and install grub or lilo then
echo all should be well

echo Removing bootcpy
rm -rf /bootcpy
echo Enjoy


For explanation and further infos refer to the Gentoo Post. A howto (Wiki) of the script can be found here.

File: mkstage4.sh
#!/bin/bash
# Backup script for Gentoo Linux
# Author: Reto Glauser aka blinkeye
# Homepage: http://blinkeye.ch
# Mailto: stage4 at blinkeye dot ch
# Date: 23.03.2005
# If you need further infos check out this post: http://forums.gentoo.org/viewtopic.php?p=1751698#1751698

version=v1.2

# these are the commands we actually need for the backup
command_list="echo tar hostname date split"

# verify that each command we use exists
for command in $command_list; do
	path=`which $command | grep "no $command in"`
	
	if [ ! -x `which $command` -a "$path" ]; then
		echo -e "\n\nERROR: $command not found! Check your commands and/or your \$PATH"
		exit -1
	fi
done

# options for the tar command
tarOptions="--create --absolute-names --preserve-permissions --totals --bzip2 --ignore-failed-read --verbose --file"

# where to put the stage4
stage4Location=/mnt/backups/stage4

# name prefix
stage4prefix=$(hostname)-stage4-`date +\%d.\%m.\%Y`

# these files/directories are always excluded
default_exclude_list="
--exclude=/tmp/*
--exclude=/var/tmp/*
--exclude=/lost+found/*
--exclude=/dev/*
--exclude=/proc/*
--exclude=/mnt/*
--exclude=/sys/*
--exclude=/usr/portage/*
--exclude=/var/log/*
--exclude=$stage4Location"

# depending on your choice these files or directories will additionally be excluded
custom_exclude_list="
--exclude=/usr/src/*
--exclude=/opt/mathematica
--exclude=/usr/share/smssend
--exclude=/home/*"

# check the folder/files stored in $default_exclude_list exist
for exclude in $default_exclude_list; do
	if [ ! -e "`echo "$exclude" | cut -d'=' -f2 | cut -d'*' -f1`"  ]; then
		echo -e "\n\nERROR: `echo "$exclude" | cut -d'=' -f2` not found! Check your \$default_exclude_list"
	fi
done

# check the folder/files stored in $custom_exclude_list exist
for exclude in $custom_exclude_list; do
	if [ ! -e "`echo "$exclude" | cut -d'=' -f2 | cut -d'*' -f1`"  ]; then
		echo -e "\n\nERROR: `echo "$exclude" | cut -d'=' -f2` not found! Check your \$custom_exclude_list"
	fi
done

# print out the version
 echo -e "\nBackup script $version"
 echo -e "==================="


# how do you want to backup?
echo -e "\nWhat do you want to do? (Use CONTROL-C to abort)\n
(1) Minimal backup
(2) Interactive backup"

while [ "$option" != '1' -a "$option" != '2'  ]; do
	echo -en "\nPlease enter your option: "
	read option
done

case $option in
1)
	stage4Name=$stage4Location/$stage4prefix-minimal
	final_command="tar $default_exclude_list $custom_exclude_list $tarOptions $stage4Name.tar.bz2 / /var/log/emerge.log"
	;;
2)
	for folder in $custom_exclude_list; do
		echo -en "Do you want to backup" `echo "$folder" | cut -d'=' -f2`"? (y/n) "
		read answer
		while [ "$answer" != 'y' -a "$answer" != 'n' ]; do
			echo "please enter y or n"
			read answer
		done
		if [ "$answer" == 'n' ]; then
			default_exclude_list="$default_exclude_list $folder"
		fi
	done
	
	stage4Name=$stage4Location/$stage4prefix-custom
	final_command="tar $default_exclude_list $tarOptions $stage4Name.tar.bz2 /  /var/log/emerge.log"
	;;
esac

# show what will be done
echo -e "\n* creating the stage4 at $stage4Location with the following options:\n\n"$final_command

# everything is set, are you sure to continue?
echo -ne "\nDo you want to continue? (y/n) "
read answer
while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
			echo "please enter y or n"
			read answer
done

if [ "$answer" == 'y' ]; then
	# mount boot
	echo -e "\n* mount boot"
	mount /boot >/dev/null 2>&1	
	
	# if necessary, create the stage4Location
	if [ ! -d "$stage4Location" ] ; then
		echo "* creating directory $stage4Location"
		mkdir -p $stage4Location
	fi
	
	# check whether the file already exists
	if [ -a "$stage4Name.tar.bz2" ]; then
		echo -en "\nDo you want to overwrite $stage4Name.tar.bz2? (y/n) "
		read answer
		while [ "$answer" != 'y' ] && [ "$answer" != 'n' ]; do
			echo "please enter y or n"
			read answer
		done
		if [ "$answer" == 'n' ]; then
			echo -e "\n* There's nothing to do ... Exiting"
			exit 0;
		fi
	fi
	
	# do the backup
	time $final_command

	# copy the current world file to the stage4 location
	echo -e "\n* creating stage4 overview $stage4Name.txt"
	cp /var/lib/portage/world $stage4Name.txt >/dev/null 2>&1
	
	# we finished, clean up
	echo "* stage4 is done"
	echo "* umounting boot"
	umount /boot
else
	echo -e "\n* There's nothing to do ... Exiting"
fi

#Uncomment the following command if you want to split the archive in cd size chunks:
#split --suffix-length=1 --bytes=670m $stage4Name.tar.bz2 "$stage4Name".tar.bz2_ && echo "* splitting is done"

Here an alternative mkstage4 shell script:

File: /sbin/mkstage4
#!/bin/sh

# checks if run as root
if ! [ "`whoami`" == "root" ]
then
	echo "`basename $0`: must be root."
	exit 1
fi

# checks for correct cmdline usage                  
if [ "$#" == "0" ]                  
then                  
    echo "`basename $0`: too few arguments."
    echo "syntax: \$ `basename $0` [-q|--quiet] <backup-filename> [custom-tar-options]"
	echo "-q, --quiet: activates quiet mode (no confirmation)."
    exit 1
fi

# checks for quiet mode (no confirmation)
if [ "$1" == "-q" ] || [ "$1" == "--quiet" ]
then
	shift
	AGREE="yes"
fi

# determines if filename was given with relative or absolute path
if [ "`echo $1 | grep -c '\/'`" -gt "0" ] && \
[ "`echo $1 | grep -c '^\/'`" -gt "0" ]
then
	STAGE4_FILENAME="$1"
else
	STAGE4_FILENAME="`pwd`/$1"
fi

# misc vars initialization block
shift;OPTIONS="$@"
EXCLUDES="\
--exclude=/dev/* \
--exclude=/proc/* \
--exclude=/sys/* \
--exclude=/tmp/* \
--exclude=/usr/portage \
--exclude=/var/tmp/* \
--exclude=/var/lock/* \
--exclude=/var/log/* \
--exclude=/var/run/* \
--exclude=.bash_history \
--exclude=lost+found \
--exclude=$STAGE4_FILENAME"

TAR_OPTIONS="-cjpP --ignore-failed-read -f"

# if not in quiet mode, this message will be displayed
if [ "$AGREE" != "yes" ]
then
	echo "Are you sure that you want to make a complete system-backup now?"
	echo ""
	echo "NOTE: external filesystems which should not be saved must be"
	echo "unmounted before backing up or their mountpoints excluded per cmdline."
	echo "example: \$ `basename $0` /my-backup.tar.bz2 --exclude=/mnt/samba/* ..."
	echo ""
	echo "WARNING: since all data is saved by default the user should exclude all"
	echo "security- or privacy-related files and directories manually per cmdline."
	echo "example: \$ `basename $0` /my-backup.tar.bz2 --exclude=/etc/ssh/ssh_host*"
	echo ""
	echo -e "COMMAND LINE PREVIEW:\n\$ tar $TAR_OPTIONS $STAGE4_FILENAME / $EXCLUDES $OPTIONS"
	echo ""
	echo -n "Type \"yes\" to continue or anything else to quit: "
	read AGREE
fi

# start stage4 creation
if [ "$AGREE" == "yes" ]
then
	tar $TAR_OPTIONS $STAGE4_FILENAME / $EXCLUDES $OPTIONS
fi

exit 0

Credits

Original Forum Post by allucid.

Up-to-Date Forum Thread link you to Blinkeye's Wiki by Blinkeye.

Feedback

Concerns or Compliments? Please use the Discussion section.

Retrieved from "http://www.gentoo-wiki.info/HOWTO_Custom_Stage4"

Last modified: Thu, 04 Sep 2008 05:31:00 +0000 Hits: 63,800