Search:  
Gentoo Wiki

HOWTO_Download_Cache_for_LAN-Http-Replicator


This article is part of the HOWTO series.
Installation Kernel & Hardware Networks Portage Software System X Server Gaming Non-x86 Emulators Misc

WARNING - THIS IS OUTDATED - SEE ORIGINAL POST FOR UPDATED HOWTO. PLEASE SOMEONE UPDATE THIS POST!!


Contents

Introduction

Have A Couple Boxes On A LAN?

Be a good Gentoo Netizen and Speed the updates with Http-Replicator! Http-Replicator is a proxy that works with Portage to serve packages from a cache. It saves you bandwidth and helps gentoo grow by minimizing the mirror abuse. The cache download speed is limited only by your disk and LAN Speeds!

Here is how it works. Http-Replictor is running on one of your machines and listens for connections from other gentoo boxes. If the package is in the cache, it sends it out at LAN speeds! If not in the cache, http-replicator will simultaneously download the file, and stream the file to multiple clients! No matter how many machines request the package, only one copy comes down the internet pipe. Multiple copies can stream out the LAN pipe.

This is the easiest, and most reliable way to share packages.

Http-Replicator has been designed with speed and security in mind. Give it a try!!

Download and install

Http-Replicator is now in portage, so install as you would any other package. If you already installed an overlay copy of http-replicator then unmerge it and remove the overlay, your config files won't be touched.

# emerge http-replicator

Configuration

Now modify /etc/make.conf on both the server and your other gentoo boxes. (There is an easier way, but this way makes it obvious and only tells portage about replicator.)

Find the Default fetch command section in /etc/make.conf. If you are already using one of these alternate fetch commands apply the changes to your section. Otherwise, make the following changes:

Warning: http_proxy MUST be lower case.

It will look like this when complete:

File: /etc/make.conf
# Default fetch command (5 tries, passive ftp for firewall compatibility) 
http_proxy="http://YourMirrorHere.com:8080" 
#These commands are no longer necessary in http-replicator 3.0
#FETCHCOMMAND="/usr/bin/wget -t 5  --passive-ftp \${URI} -P \${DISTDIR}" 
RESUMECOMMAND=" /usr/bin/wget -t 5 --passive-ftp  \${URI} -O \${DISTDIR}/\${FILE}"

Repcacheman

Check the config files (2.1=/etc/conf.d/http-replicator - 2.0=/etc/http-replicator.conf ) and then run repcacheman to create the cache dir and transfer files to http-replicators cache.

Most people can just run 'repcacheman' which will complete the setup. You should check the defaults just to be sure.

If using http-replicator 2.1 and you want to change the defaults:

#  repcacheman --user portage --dir /var/cache/http-replicator 

or in http-replicator 2.1 with defaults and with or without changes in 2.0:

#  /usr/bin/repcacheman 

This is because 2.0 will get all changes from the config file, 2.1 doesn't get all changes from the config file yet... fixed in 3.0.

NOTE: I created repcacheman to automate the install and maintenance of Http-Replicator's cache dir. If repcacheman doesn't work for you for some reason, send me the bug reports. Http-replicator will work without repcacheman, but you'll have to create the cache dir and chown the dir yourself to complete the install. repcacheman also checksums your existing /usr/portage/distfiles before moving them to the cache dir. Portage leaves incomplete and corrupt files in the distfile directory and repcacheman will not move those files to the cache.

Activation

Next, activate Http-Replicator on the server:

#  /etc/init.d/http-replicator start

http-replicator in the default runlevel

You should add http-replicator to your default runlevel

# rc-update add http-replicator default

Don't forget that portage needs mirrors! Edit GENTOO_MIRRORS in /etc/make.conf to add more http mirrors and place any ftp mirrors LAST. The default mirrors in gentoo leave something to be desired Use mirrorselect if you need help in selecting mirrors.

Also, some packages in portage have a RESTRICT="nomirror" option which will prevent portage from checking replicator for those packages. The following will override this behavior. Create the file "/etc/portage/mirrors" containing:

File: /etc/portage/mirrors
# Http-Replicator Override for FTP and RESTRICT="nomirror" packages 
local http://gentoo.osuosl.org

If you already have a local setting, don't worry, as long as it is an http mirror this will still be effective.

Then update something and watch http-replicator go!! It doesn't matter which box or how many boxes you update at the same time!! Http-Replicator can handle it!!

I recommend running repcacheman after running emerge's on the server (except emerge sync). It will delete duplicates after the server box fetches any files and also import FTP'd or other files to the cache. The easiest way to do this is to run emerge's on the server like this:

emerge -uDva world && repcacheman

This will run the emerge then runs repcacheman when the emerge is complete.

To keep repcacheman fast and efficient, you should consider deleting any files that remain in your distfile directory after repcacheman runs. They are either:

  1. No longer in Portage
  2. Incomplete or corrupt
  3. Just plain junk

See Also

HOWTO Local Rsync Mirror

Origin

HOWTO Download Cache for LAN-Http-Replicator
Posted by flybynite, 06 Dec 2002
Original Gentoo Forums Post

Feedback

Concerns or Compliments? Please use the Discussion section.

Retrieved from "http://www.gentoo-wiki.info/HOWTO_Download_Cache_for_LAN-Http-Replicator"

Last modified: Fri, 05 Sep 2008 06:39:00 +0000 Hits: 27,128