Overview
This post covers building a server that will be used to update both ports and the base system on FreeBSD backend servers that don’t have access to the Internet. For ports it will use poudriere combined with the new pkgng package manager in order to build packages that will be distributed to the other servers using Nginx. Since Nginx is already there it will be used as a forward proxy, as opposed to reverse proxy as it’s usually used, to http://updates.freebsd.org and http://pkgbeta.freebsd.org. This covers freebsd-update and installing pkg on FreeBSD 9.X. For no good reason, just to do something different, there’s a cron job that runs every day and fetches auditfile.tbz from http://portaudit.freebsd.org if it’s changed, so that pkg audit can do it’s job properly. In order to remotely install the servers they are PXE booted into a net install image, mfsbsd, base install files are mirrored on the package repository and served by Nginx. With that, the environment is complete, servers can be installed and kept up to date without giving them any kind of access to the Internet.
Server – poudriere
The Documentation. Poudriere needs ZFS, we installed FreeBSD on a ZFS mirror using this script.
To compile packages for different releases and architectures from different ports trees poudriere uses “jails”, which are actually just chroot, not FreeBSD jails. Each jail or ports tree is a different zfs filesystem so they can be easily added/removed/restored, etc. When compiling packages any combination of jail+ports can be used. A jail represents a base system used to compile the ports. For example, we could have FreeBSD 8 and FreeBSD 9, some of them using the latest php version from a portstree fetched with portsnap while others use an year old version from a portstree checked out by svn. By chrooting inside a jail any modification to the base system can be made to mirror the real server environments, although that’s probably rarely necessary. So yes, it’s very versatile.
Packages are stored in different repositories under /poudriere/data/packages/$JAIL-$PORTS, using our configuration. If WITH_PKGNG is defined each repository will have repo.txz created, which is an sqlite database that pkgng downloads and uses to see what packages are available on the repository and check for updates.
First steps
Won’t go into much detail. Like the documentation says, install poudriere, pkgng if necessary and ccache from ports. Make sure dialog4ports is installed too, it will be needed in order to set options. If it isn’t pulled in as a dependency, install it. Copy poudriere.conf.sample to poudriere.conf and edit as necessary. Our config looks like this, comments filtered out for brevity:
ZPOOL=zroot # CHANGE THIS TO A HOST CLOSE TO YOU FREEBSD_HOST=ftp://ftp.freebsd.org RESOLV_CONF=/etc/resolv.conf BASEFS=/poudriere USE_PORTLINT=yes USE_TMPFS=yes DISTFILES_CACHE=/usr/ports/distfiles CHECK_CHANGED_OPTIONS=yes CCACHE_DIR=/var/cache/ccache
Using this configuration there are two directories of note, /poudriere and /usr/local/etc/poudriere.d. The working directory is /poudriere, logs and packages are written here, while /usr/local/etc/poudriere.d is where various settings are, like make.conf options and ports options for each jail. We also store the lists of packages to be compiled and cron scripts for updates here.
Create a jail:
# poudriere jails -c -j srv_91r_amd64 -v 9.1-RELEASE -a amd64
Create make.conf for this jail:
# echo "WITH_PKGNG=yes" > /usr/local/etc/poudriere.d/srv_91r_amd64-make.conf # echo "WITHOUT_X11=yes" >> /usr/local/etc/poudriere.d/srv_91r_amd64-make.conf
When compiling, the contents of this file will be added to the contents of the /etc/make.conf file that is inside the jail, in this case /poudriere/jails/srv_91r_amd64/etc/make.conf. There is also /usr/local/etc/poudriere.d/make.conf that can be used for options that should be applied to ALL jails.
Create a ports tree:
# poudriere ports -c -p default
Passing -p name creates a tree with that particular name. If -p is not specified, a default ports tree called “default” will be used. This is true for other poudriere commands, too, like poudriere bulk, used to build packages.
The -m parameter stands for “method” and it can be used to specify how that ports tree will be created and updated. For a ports tree that will be downloaded by svn for example, -m svn+http could be used. Obviously, subversion needs to be installed in that case.
Create a list of packages to be compiled by poudriere:
# vim /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist
This file accepts comments marked with a hash tag (#).
Set compilation options for all ports in pkglist:
# poudriere options -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64
Build the packages:
# poudriere bulk -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64
Enjoy watching the damn thing actually do some work for a change!
Daily usage
Adding a package to the list and setting options for the port and it’s dependencies. Like nagios:
# echo "net-mgmt/nagios" >> /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist # poudriere options -j srv_91r_amd64 net-mgmt/nagios
Changing options for a single port:
# poudriere options -c -j srv_91r_amd64 net-mgmt/nagios
Removing options for a port and all dependecies:
# poudriere options -r -j srv_91r_amd64 net-mgmt/nagios
Use -n to set/remove options for nagios only. Note that, at least at the time of this writing, -r removes options for all dependencies, even if they are not direct dependencies of the respective port. For example, for nagios, it also removes options for apache22, because php5 depends on it and nagios depends on php5. But when using -c only options for things nagios directly depends on are set, like php5, but not apache22.
Updating the ports tree and rebuilding ports that were updated:
# poudriere ports -u -p default # poudriere bulk -f /usr/local/etc/poudriere.d/srv_91r_amd64.pkglist -j srv_91r_amd64 -p default
As noted before, if -p is omitted a ports tree named “default” is assumed.
Passing -c or -C to poudriere bulk will clean all built packages or packages from the list respectively before compiling.
Crontab
To have crontab update our repositories we use this script:
#!/bin/sh JAILNAME=$1 PORTSTREE=$2 POUDRIERE_DIR="/usr/local/etc/poudriere.d/" PATH="$PATH:/usr/local/bin" if [ -z "$JAILNAME" ]; then printf "Provide a jail name please.\n" exit 1 fi if [ -z "$PORTSTREE" ]; then PORTSTREE="default" fi # check that the ports tree exists poudriere ports -l | grep "^$PORTSTREE" > /dev/null if [ $? -gt 0 ]; then printf "No such ports tree ($PORTSTREE)\n" exit 2 fi ## check that there's a list of packages for that jail if [ ! -f "$POUDRIERE_DIR$JAILNAME.pkglist" ]; then printf "No such file (list of packages: $POUDRIERE_DIR$PORTSTREE)\n" exit 3 fi # check that the jail is there poudriere jails -l | grep "^$JAILNAME" > /dev/null if [ $? -gt 0 ]; then printf "No such jail ($JAILNAME)\n" exit 4 fi # update the ports tree poudriere ports -u -p $PORTSTREE # build new packages poudriere bulk -f /usr/local/etc/poudriere.d/$JAILNAME.pkglist -j $JAILNAME -p $PORTSTREE |
It’s run from /etc/crontab at 1:30 every night:
# update pkg jails 30 1 * * * root /usr/local/etc/poudriere.d/scripts/repo-update.sh srv_91r_amd64 default
A few hours later the other servers update their local repo.txz and check for updates.
Server – nginx
Nginx is used to make the compiled packages available to the other servers. The default website is set to /poudriere/data/packages and autoindex is turned on, that’s pretty much all there is to it. In addition to serving packages from that website it also mirrors whatever distribution we need from ftp.freebsd.org, like ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/9.1-RELEASE/. auditfile.tbz is also on this website, updated every day using this crontab entry:
# update the audit file for 'pkg audit' # 15 minutes before @daily 45 23 * * * root cd /poudriere/data/packages/ && fetch -mq http://portaudit.freebsd.org/auditfile.tbz
It’s also used as a forward proxy for update.freebsd.org and pkgbeta.freebsd.org. Apparently the guys behind update.freebsd.org don’t like mirrors for security reasons and are likely to block repeated calls from wget and the like, using a proxy is the recommended method. Nginx is already installed and it does the job just fine. Another reason is that freebsd-update uses phttpget and squid&co might not play well with it. pkgbeta.freebsd.org could be mirrored, just like base.
nginx.conf:
#user nobody;
worker_processes 4; # adjust to the number of CPU cores
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
server {
listen 80 default;
server_name pkg-repo.example.org "";
#access_log logs/host.access.log main;
location / {
root /poudriere/data/packages/;
index index.html index.htm;
autoindex on;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/local/www/nginx-dist;
}
}
# proxy connections to update.freebsd.org for
# servers that don't have internet access
server {
listen 80;
server_name fbsd-update.example.org;
location / {
proxy_pass http://update.freebsd.org;
proxy_http_version 1.1;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /var/tmp/nginx/update_temp/;
root /;
}
}
# proxy connections to pkgbeta.freebsd.org for
# servers that don't have internet access
# used for pkg bootstrap on new FBSD9 servers
server {
listen 80;
server_name pkg-bootstrap.example.org;
location / {
proxy_pass http://pkgbeta.freebsd.org;
proxy_http_version 1.1;
proxy_store on;
proxy_store_access user:rw group:rw all:r;
proxy_temp_path /var/tmp/nginx/update_temp/;
root /;
}
}
}
Nginx has three vhosts defined, pkg-repo.example.org, fbsd-update.example.org and pkg-bootstrap.example.org. These names need to be defined in example.org’s DNS and they should point to this server.
Clients – pkgng
FreeBSD 10 will have pkg installed in base by default, but in FreeBSD 9 it’s still in ports. In order to install it, a bare ‘pkg’ is provided which will download the real files from pkgbeta.freebsd.org. Nginx is proxy-ing connections to that site, so in order to bootstrap pkg in FreeBSD 9.1, assuming cshrc, the default root shell:
# setenv PACKAGEROOT "http://pkg-bootstrap.example.org" # pkg
Hopefully it’s clear enough.
Next, edit /usr/local/etc/pkg.conf:
PACKAGESITE: http://pkg-repo-eu.perfectworld.eu/srv_91r_amd64-default/ PORTAUDIT_SITE: http://pkg-repo-eu.perfectworld.eu/auditfile.tbz
Ready to install packages now:
# pkg update Updating repository catalogue repo.txz 100% 1836 1.8KB/s 1.8KB/s 00:00 # pkg install sudo vim-lite (...)
Cron job to check for updates every night:
# check for updated packages 30 5 * * * root pkg update -q && pkg version -vRL=
Check UPDATING before upgrading packages:
# pkg updating
by default it looks for /usr/ports/UPDATING which has no reason to be there on client computers, but it can easily be given out by Nginx on the pkg builder.
When pkgng is installed it also creates the periodic script /usr/local/etc/periodic/security/410.pkg-audit, which runs pkg audit, a replacement for portaudit. This is why we need auditfile in Nginx and the reason for the PORTAUDIT_SITE setting in pkg.conf.
Mandatory link to pkgng page in FreeBSD’s handbook
Clients – freebsd-update
In /etc/freebsd-update.conf the line
ServerName update.FreeBSD.org
should be replaced with
ServerName fbsd-update.example.org
That’s all, freebsd-update should be working now. Provided the DNS entry is in place, of course.
/etc/crontab:
# check for OS updates @daily root freebsd-update cron