Mirroring FreeBSD

Daniel Lang

dl@leo.org

$FreeBSD: doc/en_US.ISO8859-1/articles/hubs/article.sgml,v 1.24 2002/08/11 09:20:39 blackend Exp $

An in-progress article on how to mirror FreeBSD, aimed at hub administrators

Table of Contents
1 Requirements for FreeBSD mirrors
2 How to mirror FreeBSD
3 Where to mirror from
4 Official Mirrors
5 Some statistics from mirror sites

1 Requirements for FreeBSD mirrors

Disk space is one of the most important requirements. Depending on the set of releases, architectures, and degree of completeness you want to mirror, a huge amount of disk space may be consumed. Also keep in mind, that official mirrors are probably required to be complete. The CVS repository and the web pages should always be mirrored completely. Also note, that the numbers stated here, are reflecting the current state (at 4.5-RELEASE). Further development and releases will only increase the required amount. Also make sure, to keep some (ca. 10-20%) extra space around, just to be sure. Here are some approximate figures:

Full FTP Distribution: 60 GB
CVS repository: 1.4 GB
CTM deltas: 1.5 GB
Webpages: 150 MB

1.2 Network Connection/Bandwidth

Of course, you need to be connected to the internet. The required bandwidth depends on your intended use of the mirror. If you just want to mirror some parts of FreeBSD for local use at your site/intranet, the demand may be much smaller, than if you want to make the files publicly available or even if you intend to become an official mirror. We can only give rough estimates here:

Local site, no public access: basically no minimum, but < 2 Mbps could make syncing a pain.
Unofficial public site: 34 Mbps is probably a good start.
Official site: > 100 Mbps is recommended, also your host should be connected as close as possible to your border router.

1.3 System Requirements, CPU, RAM

This also depends on the expected amount of clients, which is determined by the servers policy. It is also affected by the types of services you want to offer. Plain FTP or HTTP services may not require a huge amount of resources. Watch out, if you provide CVSup, rsync or even AnonCVS. This can have a huge impact on CPU and memory requirements. Especially rsync is considered a memory hog, and CVSup does indeed consume some CPU. For AnonCVS it might be a nice idea to set up a memory resident filesystem (MFS) of at least 300 MB, so you need to take this into account for your memory requirements. The following are just examples to give you a very rough hint.

For a moderately visited site, that offers rsync, you might consider a current CPU with around 800Mhz - 1 GHz, and at least 512MB RAM. This is probably the minimum you want for an official site.

For a frequently used site you need definitely more RAM (consider 2GB as a good start), and possibly more CPU, which could also mean, that you need to go for a SMP system.

You also want to consider a fast disk subsystem. Operations on the CVS repository require a fast disk subsystem (RAID is greatly advised). A SCSI controller that has a cache of its own can also speed up things, since most of these services incur a very large number of small modifications to the disk.

You can also experiment with enlarging the portion of system memory which is used for the filesystem buffer cache. This will also help to reduce the quantity of disk access. This can be done with the BUFCACHEPERCENT kernel option. The default is to use 5% of system memory.

1.4 Services to offer

Every mirror site is required to have a set of core services available. In addition to these basic services, which mirrors are required to provide, there is a number of optional services that server administrators may choose to offer. This section explains which services you can provide and how to go about implementing them.

1.4.1 FTP (required for FTP fileset)

This is one of the most basic services, and it is required for each mirror, offering public FTP distributions. FTP access must be anonymous, and no upload/download ratios are allowed (a ridiculous thing anyway). Upload capability is not required (and must never be allowed for the FreeBSD file space). Also the FreeBSD archive should be available under the path /pub/FreeBSD.

There is a lot of software available which can be set up to allow anonymous FTP (in alphabetical order).

/usr/libexec/ftpd: FreeBSD's own ftpd can be used. Be sure to read ftpd(8).
ftp/ncftpd: A commercial package, free for educational use.
ftp/oftpd: An ftpd designed with security as a main focus.
ftp/proftpd: A modular and very flexible ftpd.
ftp/pure-ftpd: Another ftpd developed with security in mind.
ftp/twoftpd: As above.
ftp/vsftpd: The ``very secure'' ftpd.
ftp/wu-ftpd: The ftpd from Washington University. It has become infamous, because of the huge amount of security issues that have been found in it. If you do choose to use this software be sure to keep it up to date.

FreeBSD's ftpd, proftpd, wu-ftpd and maybe ncftpd are among the most commonly ones. The others do not have a large userbase among mirror sites.

1.4.2 RSYNC (optional for FTP fileset)

Rsync is often also offered for convenience, for the contents of the FTP area of FreeBSD. The protocol is different from FTP in many ways, and overall, it can be stated, that it is much more bandwidth friendly, as only differences between files are transferred, not whole files. Rsync does require significant amount of memory for each instance. The size depends on the size of the synced module in terms of number of directories and files. Rsync can use rsh and ssh (now default) as a transport, or use it's own protocol for stand-alone access (this is the preferred method for public rsync servers). Authentication, connection limits, and other restrictions may be applied. There is just one software package available:

net/rsync

1.4.3 HTTP (required for webpages, optional for FTP fileset)

If you want to offer the FreeBSD webpages, you need to install a webserver a.k.a. httpd. You may optionally offer the FTP fileset via HTTP. The choice of Webserver software is left up to the mirror administrator. Some of the most popular choices are:

www/apache13: Apache is the most widely deployed Webserver on the Internet. It is used extensively by the FreeBSD Project. You may also wish to use the next generation of the Apache Webserver, available in the ports collection as www/apache2.
www/thttpd: If you are going to be serving a lot amount of static content you may find that using an application such as tHttpd is more efficient than Apache. It is optimized for excellent performance on FreeBSD.
www/boa: Boa is another alternative to tHttpd and Apache. It should provide considerably better performance than Apache for purely static content. It does not, at the time of writing, contain the same set of optimizations for FreeBSD that are found in tHttpd.

1.4.4 CVSup (desired for CVS repository)

CVSup is a very efficient way of distributing files. It works similar as rsync, but was specially designed for the use with CVS repositories. If you want to offer the FreeBSD CVS repository, you really want to consider offering it via CVSup. Still it is possible to offer the CVS repository via AnonCVS, FTP, Rsync or HTTP, but people would benefit much more from CVSup access. CVSup was developed by John Polstra <jdp@FreeBSD.org>. It is a bit tricky to install on non-FreeBSD platforms, since it is written in Modula-3 and therefore requires a Modula-3 environment. John Polstra has built a stripped down version of M3, that is sufficient to run CVSup, and can be installed much easier. See Ezm3 for details. Related ports are:

net/cvsup: The native CVSup port (client and server) which requires lang/ezm3 now.
net/cvsup-mirror: The CVSup mirror kit, which requires net/cvsup, and configures it mirror-ready. Some site administrators may want a different setup, though.

There are a few more like net/cvsupit and net/cvsup-without-gui you might want to have a look at. If you prefer a static binary package, take a look here. This page stil refers to the S1G bug, that was present in CVSup. Maybe John will setup a generic download-site to get static binaries for various platforms.

It is possible to use CVSup to offer any kind of fileset, not just CVS repositories, but configuration can be complex. CVSup is known to eat some CPU on the server as on the client, since it needs to compare lots of files.

1.4.5 AnonCVS (optional for CVS repository)

If you have the CVS repository, you may want to offer anonymous CVS access. A short warning first: There is not that much demand for it, and it requires some experience and you need to know, what you are doing.

Generally there are two ways, how to access a CVS repository remotely: via pserver or via ssh (we don't consider rsh). For anonymous access, pserver is very well suited, but some still offer ssh access as well. There is a custom crafted wrapper in the CVS repository, to be used as a login-shell for the anonymous ssh account. It does a chroot, and therefore requires the CVS repository to be available under the anonymous user's home-directory, which may not be possible for all sites. If you just offer pserver this restriction does not apply, but you may run with more security risks. You don't need to install any special software, since cvs(1) comes with FreeBSD. You need to enable access via inetd, so add an entry into your /etc/inetd.conf like this:

    cvspserver stream tcp nowait root /usr/bin/cvs cvs -f -l -R -T /anoncvstmp --allow-root=/home/ncvs pserver

See the manpage for details of the options. See also the cvs info page, about additional ways to make sure, access is read-only. It is advisable, that you create an unprivileged account, preferably called anoncvs. Also you need to create a file passwd in your /home/ncvs/CVSROOT and assign a CVS password (empty or anoncvs) to that user. The directory /anoncvstmp is a special purpose memory based filesystem. It is not required but advised, since cvs(1) creates a shadow directory structure in your /tmp which is not used after the operation, but slows things dramatically, if real disk operations are required. Here is an excerpt from /etc/fstab, how to set up such a MFS:

    /dev/da0s1b /anoncvstmp mfs rw,-s=786432,-b=4096,-f=512,-i=560,-c=3,-m=0,nosuid,nodev 0 0

This is (of course) tuned a lot, and was suggested by John Polstra <jdp@FreeBSD.org>.