Merge branch 'master' of git.mcs.anl.gov:bcfg2

11 files changed, 2551 insertions, 0 deletions

diff --git a/doc/appendix/guides/authentication.txt b/doc/appendix/guides/authentication.txt
new file mode 100644
index 000000000..0a324b718
--- /dev/null
+++ b/doc/appendix/guides/authentication.txt
@@ -0,0 +1,143 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-authentication:
+
+==============
+Authentication
+==============
+
+Scenarios
+=========
+
+1. Cluster nodes that are frequently rebuilt
+
+   Default settings work well; machines do not float, and a per-client
+   password is not required.
+
+2. :ref:`appendix-guides-nat_howto`
+
+ * Build client records in advance with ``bcfg2-admin``, setting a uuid
+   for each new client.
+
+ * Set the address attribute for each to the address of the NAT.
+
+ * Optionally, set a per-client password for each, and set into secure
+   mode.
+
+   .. note::
+
+      This will require the use of the uuid and password from each
+      client, and will require that they come through the NAT address.
+
+Building bcfg2.conf automatically
+=================================
+
+This is a TCheetah template that automatically constructs per-client
+bcfg2.conf from the per-client metadata::
+
+    [communication]
+    protocol = xmlrpc/ssl
+    #if $self.metadata.uuid != None
+    user = $self.metadata.uuid
+    #end if
+    #if $self.metadata.password != None
+    password = $self.metadata.password
+    #else
+    password = my-password-foobat
+    #end if
+
+    [components]
+    bcfg2 = https://localhost:6789
+
+In this setup, this will cause any clients that have uuids established
+to be set to use them in ``bcfg2.conf``. It will also cause any clients
+with passwords set to use them instead of the global password.
+
+How Authentication Works
+========================
+
+#. First, the client is associated with a client record. If the client
+   specifies a uuid, it uses this instead of the results of a dns or
+   address lookup.
+
+#. Next, the ip address is verified against the client record. If the
+   address doesn't match, then the client must be set to
+   location=floating
+
+#. Finally, the password is verified. If the client is set to secure
+   mode, the only its per-client password is accepted. If it is not set
+   to secure mode, then either the global password or per-client password
+   will be accepted
+
+Failure during any of these stages results in authentication
+failure. Note that clients set into secure mode that do not have
+per-client passwords set will not be able to connect.
+
+SSL Cert-based client authentication
+====================================
+
+SSL-based client authentication is supported. This requires several
+things:
+
+#. Certificate Authority (to sign all keys)
+
+#. Server key and cert signed by the CA
+
+#. Client key and cert signed by the CA
+
+A variety of CAs can be used, but these keys can be simply generated
+using the following set of steps:
+
+#. Setup a CA
+
+   http://www.flatmtn.com/article/setting-openssl-create-certificates
+
+#. Create keys for each client and server, signing them with the CA
+   signing cert
+
+   http://www.flatmtn.com/article/setting-ssl-certificates-apache
+
+   .. note::
+       The client CN must be the FQDN of the client (as returned by a
+       reverse DNS lookup of the ip address. Otherwise, you will end up
+       with an error message on the client that looks like::
+
+           Server failure: Protocol Error: 401 Unauthorized
+	   Failed to download probes from bcfg2
+	   Server Failure
+
+       You will also see an error message on the server that looks
+       something like::
+
+           cmssrv01 bcfg2-server[9785]: Got request for cmssrv115 from incorrect address 131.225.206.122
+           cmssrv01 bcfg2-server[9785]: Resolved to cmssrv115.fnal.gov
+
+#. Distribute the keys and certs to the appropriate locations
+
+#. Copy the ca cert to clients, so that the server can be authenticated
+
+Clients authenticating themselves with a certificate will be
+authenticated that way first; clients can be setup to either
+authenticate solely with certs, use certs with a fallback to password,
+or password only. Also a bootstrap mode will be added shortly; this
+will allow a client to authenticate with a password its first time,
+requiring a certificate all subsequent times. This behavior can be
+controlled through the use of the auth attribute in
+``Metadata/clients.xml``::
+
+    <Clients>
+      <Client name='testclient' auth='cert'/>
+    </Clients>
+
+Allowed values are:
+
+    +---------------+------------------------------------------+
+    | **Auth Type** | **Meaning**                              |
+    +---------------+------------------------------------------+
+    | cert          | Certificates must be used                |
+    +---------------+------------------------------------------+
+    | cert+password | Certificate or password may be used      |
+    +---------------+------------------------------------------+
+    | bootstrap     | Password can be used for one client run, |
+    |               | after that certificate is required       |
+    +---------------+------------------------------------------+
diff --git a/doc/appendix/guides/centos.txt b/doc/appendix/guides/centos.txt
new file mode 100644
index 000000000..1e3c90156
--- /dev/null
+++ b/doc/appendix/guides/centos.txt
@@ -0,0 +1,565 @@
+.. -*- mode: rst -*-
+
+.. _EPEL: http://fedoraproject.org/wiki/EPEL
+
+.. _appendix-guides-centos:
+
+=====================
+Quickstart for CentOS
+=====================
+
+This is a complete getting started guide for CentOS. With this document
+you should be able to install a Bcfg2 server and a Bcfg2 client.
+
+Install Bcfg2
+=============
+
+The fastest way to get Bcfg2 onto your system is to use Yum or
+your preferred package management tool. We'll be using the ones
+that are distributed through EPEL_, but depending on your aversion
+to risk you could download an RPM from other places as well.  See
+:ref:`getting_started-using_bcfg2-with-centos` for information about
+building Bcfg2 from source and making your own packages.
+
+Using EPEL
+----------
+
+Make sure EPEL_ is a valid repository on your server. The `instructions
+<http://fedoraproject.org/wiki/EPEL/FAQ#howtouse>`_ on how to do this
+basically say::
+
+    [root@centos ~]# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/x86_64/epel-release-5-4.noarch.rpm
+
+.. note::
+
+    You will have to adjust this command to match your architecture and
+    the current EPEL release.
+
+Install the bcfg2-server and bcfg2 RPMs::
+
+    [root@centos ~]# yum install bcfg2-server bcfg2
+
+Your system should now have the necessary software to use Bcfg2. The
+next step is to set up your Bcfg2 :term:`repository`.
+
+Initialize your repository
+==========================
+
+Now that you're done with the install, you need to initialize your
+repository and setup your ``/etc/bcfg2.conf``. ``bcfg2-admin init``
+is a tool which allows you to automate this::
+
+    [root@centos ~]# bcfg2-admin init
+    Store bcfg2 configuration in [/etc/bcfg2.conf]:
+    Location of bcfg2 repository [/var/lib/bcfg2]:
+    Input password used for communication verification (without echoing; leave blank for a random):
+    What is the server's hostname: [centos]
+    Input the server location [https://centos:6789]:
+    Input base Operating System for clients:
+    1: Redhat/Fedora/RHEL/RHAS/Centos
+    2: SUSE/SLES
+    3: Mandrake
+    4: Debian
+    5: Ubuntu
+    6: Gentoo
+    7: FreeBSD
+    : 1
+    Generating a 2048 bit RSA private key
+    .........................+++
+    ..................+++
+    writing new private key to '/etc/bcfg2.key'
+    -----
+    Signature ok
+    subject=/C=US=ST=Illinois/L=Argonne/CN=centos
+    Getting Private key
+    Repository created successfuly in /var/lib/bcfg2
+
+Change responses as necessary.
+
+Start the server
+================
+
+You are now ready to start your bcfg2 server for the first time::
+
+    [root@centos ~]# /sbin/service bcfg2-server start
+
+To verify that everything started ok, look for the running daemon and check the logs::
+
+    [root@centos ~]# /etc/init.d/service bcfg2-server status
+    [root@centos ~]# tail /var/log/messages
+    Mar 29 12:42:26 centos bcfg2-server[5093]: service available at https://centos:6789
+    Mar 29 12:42:26 centos bcfg2-server[5093]: serving bcfg2-server at https://centos:6789
+    Mar 29 12:42:26 centos bcfg2-server[5093]: serve_forever() [start]
+    Mar 29 12:42:41 centos bcfg2-server[5093]: Handled 16 events in 0.007s
+
+Run bcfg2 to be sure you are able to communicate with the server::
+
+    [root@centos ~]# bcfg2 -vqn
+    No ca is specified. Cannot authenticate the server with SSL.
+    No ca is specified. Cannot authenticate the server with SSL.
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+
+    Phase: initial
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      208
+
+
+    Phase: final
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      208
+
+    No ca is specified. Cannot authenticate the server with SSL.
+
+The ca message is just a warning, meaning that the client does not
+have sufficient information to verify that it is talking to the
+correct server. This can be fixed by distributing the ca certificate
+from the server to all clients. By default, this file is available in
+``/etc/bcfg2.crt`` on the server. Copy this file to the client (with a
+bundle) and add the ca option to ``bcfg2.conf`` pointing at the file,
+and the client will be able to verify it is talking to the correct server
+upon connection::
+
+    [root@centos ~]# cat /etc/bcfg2.conf
+
+
+    [communication]
+    protocol = xmlrpc/ssl
+    password = N41lMNeW
+    ca = /etc/bcfg2.crt
+
+    [components]
+    bcfg2 = https://centos:6789
+
+Now if you run the client, no more warning::
+
+    [root@centos ~]# bcfg2 -vqn
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+
+    Phase: initial
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      208
+
+
+    Phase: final
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      208
+
+Bring your first machine under Bcfg2 control
+============================================
+
+Now it is time to get your first machine's configuration into your
+Bcfg2 :term:`repository`. Let's start with the server itself.
+
+
+Setup the `Packages`_ plugin
+----------------------------
+
+.. _Packages: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages
+
+First, replace **Pkgmgr** with **Packages** in the plugins
+line of ``bcfg2.conf``.  Then create Packages layout (as per
+:ref:`packages-exampleusage`) in ``/var/lib/bcfg2``
+
+.. note:: I am using the RawURL syntax here since we are using `mrepo`_
+          to manage our yum mirrors.
+
+.. _mrepo: http://dag.wieers.com/home-made/mrepo/
+
+.. code-block:: xml
+
+    <Sources>
+            <!-- CentOS (5.4) sources -->
+            <YUMSource>
+                    <Group>centos5.4</Group>
+                    <RawURL>http://mrepo/centos5-x86_64/RPMS.os</RawURL>
+                    <Arch>x86_64</Arch>
+            </YUMSource>
+            <YUMSource>
+                    <Group>centos5.4</Group>
+                    <RawURL>http://mrepo/centos5-x86_64/RPMS.updates</RawURL>
+                    <Arch>x86_64</Arch>
+            </YUMSource>
+            <YUMSource>
+                    <Group>centos5.4</Group>
+                    <RawURL>http://mrepo/centos5-x86_64/RPMS.extras</RawURL>
+                    <Arch>x86_64</Arch>
+            </YUMSource>
+    </Sources>
+
+Due to the `Magic Groups`_, we need to modify our Metadata. Let's
+add a **centos5.4** group which inherits a **centos** group
+(this should replace the existing **redhat** group) present in
+``/var/lib/bcfg2/Metadata/groups.xml``. The resulting file should look
+something like this
+
+.. _Magic Groups: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages#MagicGroups
+
+.. code-block:: xml
+
+    <Groups version='3.0'>
+       <Group profile='true' public='true' default='true' name='basic'>
+          <Group name='centos5.4'/>
+       </Group>
+       <Group name='centos5.4'>
+          <Group name='centos'/>
+       </Group>
+       <Group name='ubuntu'/>
+       <Group name='debian'/>
+       <Group name='freebsd'/>
+       <Group name='gentoo'/>
+       <Group name='centos'/>
+       <Group name='suse'/>
+       <Group name='mandrake'/>
+       <Group name='solaris'/>
+    </Groups>
+
+.. note::
+    When editing your xml files by hand, it is useful to occasionally run
+    `bcfg2-repo-validate` to ensure that your xml validates properly.
+
+The final thing we need is for the client to have the proper
+arch group membership. For this, we will make use of the
+:ref:`unsorted-dynamic_groups` capabilities of the Probes plugin. Add
+Probes to your plugins line in ``bcfg2.conf`` and create the Probe.::
+
+    [root@centos ~]# grep plugins /etc/bcfg2.conf
+    plugins = Base,Bundler,Cfg,Metadata,Packages,Probes,Rules,SSHbase
+    [root@centos ~]# mkdir /var/lib/bcfg2/Probes
+    [root@centos ~]# cat /var/lib/bcfg2/Probes/groups
+    #!/bin/sh
+
+    echo "group:`uname -m`"
+
+Now we restart the bcfg2-server::
+
+    [root@centos ~]# /etc/init.d/bcfg2-server restart
+
+If you tail ``/var/log/syslog`` now, you will see the Packages plugin in
+action, updating the cache.
+
+Start managing packages
+-----------------------
+
+Add a base-packages bundle. Let's see what happens when we just populate
+it with the *yum* package.
+
+.. code-block:: xml
+
+    [root@centos ~]# cat /var/lib/bcfg2/Bundler/base-packages.xml
+    <Bundle name='base-packages'>
+            <Package name='yum'/>
+    </Bundle>
+
+You need to reference the bundle from your Metadata. The resulting
+profile group might look something like this
+
+.. code-block:: xml
+
+    <Group profile='true' public='true' default='true' name='basic'>
+       <Bundle name='base-packages'/>
+       <Group name='centos5.4'/>
+    </Group>
+
+Now if we run the client, we can see what this has done for us.::
+
+    [root@centos ~]# bcfg2 -vqn
+    Running probe groups
+    Probe groups has result:
+    x86_64
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+            Package pam failed verification.
+
+    Phase: initial
+    Correct entries:        94
+    Incorrect entries:      1
+    Total managed entries:  95
+    Unmanaged entries:      113
+
+    In dryrun mode: suppressing entry installation for:
+      Package:pam
+
+    Phase: final
+    Correct entries:        94
+    Incorrect entries:      1
+     Package:pam
+    Total managed entries:  95
+    Unmanaged entries:      113
+
+Interesting, our **pam** package failed verification. What does this
+mean? Let's have a look::
+
+    [root@centos ~]# rpm --verify pam
+    ....L...  c /etc/pam.d/system-auth
+
+Sigh, it looks like the default RPM install for pam fails to verify
+using its own verification process (trust me, it's not the only one). At
+any rate, I was able to get rid of this particular issue by removing the
+symlink and running ``yum reinstall pam``.
+
+As you can see, the Packages plugin has generated the dependencies
+required for the yum package automatically. The ultimate goal should
+be to move all the packages from the **Unmanaged** entries section to
+the **Managed** entries section. So, what exactly *are* those Unmanaged
+entries?::
+
+    [root@centos ~]# bcfg2 -veqn
+    Running probe groups
+    Probe groups has result:
+    x86_64
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+    Extra Package openssh-clients 4.3p2-36.el5_4.4.x86_64.
+    Extra Package libuser 0.54.7-2.1el5_4.1.x86_64.
+    ...
+
+    Phase: initial
+    Correct entries:        95
+    Incorrect entries:      0
+    Total managed entries:  95
+    Unmanaged entries:      113
+
+
+    Phase: final
+    Correct entries:        95
+    Incorrect entries:      0
+    Total managed entries:  95
+    Unmanaged entries:      113
+     Package:at
+     Package:avahi
+     Package:avahi-compat-libdns_sd
+     ...
+
+Now you can go through these and continue adding the packages you want
+to your Bundle. After a while, I ended up with a minimal bundle that
+looks like this
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <Package name='bcfg2-server'/>
+            <Package name='exim'/>
+            <Package name='grub'/>
+            <Package name='kernel'/>
+            <Package name='krb5-workstation'/>
+            <Package name='m2crypto'/>
+            <Package name='openssh-clients'/>
+            <Package name='openssh-server'/>
+            <Package name='prelink'/>
+            <Package name='redhat-lsb'/>
+            <Package name='rpm-build'/>
+            <Package name='rsync'/>
+            <Package name='sysklogd'/>
+            <Package name='vim-enhanced'/>
+            <Package name='yum'/>
+    </Bundle>
+
+Now when I run the client, you can see I have only one unmanaged
+package::
+
+    [root@centos ~]# bcfg2 -veqn
+    Running probe groups
+    Probe groups has result:
+    x86_64
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+    Extra Package gpg-pubkey e8562897-459f07a4.None.
+    Extra Package gpg-pubkey 217521f6-45e8a532.None.
+
+    Phase: initial
+    Correct entries:        187
+    Incorrect entries:      0
+    Total managed entries:  187
+    Unmanaged entries:      16
+
+
+    Phase: final
+    Correct entries:        187
+    Incorrect entries:      0
+    Total managed entries:  187
+    Unmanaged entries:      16
+     Package:gpg-pubkey
+     Service:atd
+     Service:avahi-daemon
+     Service:bcfg2-server
+     ...
+
+The gpg-pubkey packages are special in that they are not really
+packages. Currently, the way to manage them is using :ref:`BoundEntries
+<boundentries>`. So, after adding them, our Bundle now looks like this
+
+.. note:: This does not actually control the contents of the files,
+          you will need to do this part separately (see below).
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <BoundPackage name="gpg-pubkey" type="rpm">
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5" version="e8562897" release="459f07a4"/>
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/>
+            </BoundPackage>
+            <Package name='bcfg2-server'/>
+            <Package name='exim'/>
+            <Package name='grub'/>
+            <Package name='kernel'/>
+            <Package name='krb5-workstation'/>
+            <Package name='m2crypto'/>
+            <Package name='openssh-clients'/>
+            <Package name='openssh-server'/>
+            <Package name='prelink'/>
+            <Package name='redhat-lsb'/>
+            <Package name='rpm-build'/>
+            <Package name='rsync'/>
+            <Package name='sysklogd'/>
+            <Package name='vim-enhanced'/>
+            <Package name='yum'/>
+    </Bundle>
+
+To actually push the gpg keys out via Bcfg2, you will need to manage the
+files as well. This can be done by adding Path entries for each of the
+gpg keys you want to manage
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <Path name='/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5'/>
+            <Path name='/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL'/>
+            <BoundPackage name="gpg-pubkey" type="rpm">
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5" version="e8562897" release="459f07a4"/>
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/>
+            </BoundPackage>
+            <Package name='bcfg2-server'/>
+            <Package name='exim'/>
+            <Package name='grub'/>
+            <Package name='kernel'/>
+            <Package name='krb5-workstation'/>
+            <Package name='m2crypto'/>
+            <Package name='openssh-clients'/>
+            <Package name='openssh-server'/>
+            <Package name='prelink'/>
+            <Package name='redhat-lsb'/>
+            <Package name='rpm-build'/>
+            <Package name='rsync'/>
+            <Package name='sysklogd'/>
+            <Package name='vim-enhanced'/>
+            <Package name='yum'/>
+    </Bundle>
+
+Then add the files to Cfg::
+
+    mkdir -p Cfg/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
+    cp /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5 !$/RPM-GPG-KEY-CentOS-5
+    mkdir -p Cfg/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
+    cp /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL !$/RPM-GPG-KEY-EPEL
+
+Now, running the client shows only unmanaged Service entries. Woohoo!
+
+Manage services
+---------------
+
+Now let's clear up the unmanaged service entries by adding the following
+entries to our bundle.
+
+.. code-block:: xml
+
+    <!-- basic services -->
+    <Service name='atd'/>
+    <Service name='avahi-daemon'/>
+    <Service name='bcfg2-server'/>
+    <Service name='crond'/>
+    <Service name='cups'/>
+    <Service name='gpm'/>
+    <Service name='lvm2-monitor'/>
+    <Service name='mcstrans'/>
+    <Service name='messagebus'/>
+    <Service name='netfs'/>
+    <Service name='network'/>
+    <Service name='postfix'/>
+    <Service name='rawdevices'/>
+    <Service name='sshd'/>
+    <Service name='syslog'/>
+
+...and bind them in Rules
+
+.. code-block:: xml
+
+    [root@centos ~]# cat /var/lib/bcfg2/Rules/services.xml
+    <Rules priority='1'>
+            <!-- basic services -->
+            <Service type='chkconfig' status='on' name='atd'/>
+            <Service type='chkconfig' status='on' name='avahi-daemon'/>
+            <Service type='chkconfig' status='on' name='bcfg2-server'/>
+            <Service type='chkconfig' status='on' name='crond'/>
+            <Service type='chkconfig' status='on' name='cups'/>
+            <Service type='chkconfig' status='on' name='gpm'/>
+            <Service type='chkconfig' status='on' name='lvm2-monitor'/>
+            <Service type='chkconfig' status='on' name='mcstrans'/>
+            <Service type='chkconfig' status='on' name='messagebus'/>
+            <Service type='chkconfig' status='on' name='netfs'/>
+            <Service type='chkconfig' status='on' name='network'/>
+            <Service type='chkconfig' status='on' name='postfix'/>
+            <Service type='chkconfig' status='on' name='rawdevices'/>
+            <Service type='chkconfig' status='on' name='sshd'/>
+            <Service type='chkconfig' status='on' name='syslog'/>
+    </Rules>
+
+Now we run the client and see there are no more unmanaged entries! ::
+
+    [root@centos ~]# bcfg2 -veqn
+    Running probe groups
+    Probe groups has result:
+    x86_64
+    Loaded plugins: fastestmirror
+    Loading mirror speeds from cached hostfile
+    Excluding Packages in global exclude list
+    Finished
+    Loaded tool drivers:
+     Action       Chkconfig  POSIX        YUMng
+
+    Phase: initial
+    Correct entries:        205
+    Incorrect entries:      0
+    Total managed entries:  205
+    Unmanaged entries:      0
+
+
+    Phase: final
+    Correct entries:        205
+    Incorrect entries:      0
+    Total managed entries:  205
+    Unmanaged entries:      0
+
+Dynamic (web) reports
+=====================
+
+See installation instructions at :ref:`server-reports-install`
diff --git a/doc/appendix/guides/converging_rhel5.txt b/doc/appendix/guides/converging_rhel5.txt
new file mode 100644
index 000000000..7581d307f
--- /dev/null
+++ b/doc/appendix/guides/converging_rhel5.txt
@@ -0,0 +1,128 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-converging_rhel5:
+
+======================================
+Converging on Verification with RHEL 5
+======================================
+
+Running verification
+====================
+
+To get complete verification status, run::
+
+    bcfg2 -vqned
+
+Unmanaged entries
+=================
+
+* Package (top-level)
+
+ #. Enable the "Packages" plugin in ``/etc/bcfg2.conf``, and configure
+    the Yum repositories in ``/var/lib/bcfg2/Packages/config.xml``.
+ #. If a package is unwanted, remove it::
+
+        sudo yum remove PACKAGE
+
+ #. Otherwise, add ``<Package name="PACKAGE" />`` to the Base or Bundler configuration.
+
+* Package (dependency)
+
+ #. Ensure the Yum repository sources configured in
+    ``/var/lib/bcfg2/Packages/config.xml`` are correct.
+ #. Ensure the Yum repositories themselves are up-to-date with the main
+    package and dependencies.
+ #. Rebuild the Packages plugin cache::
+
+        bcfg2-admin xcmd Packages.Refresh
+
+* Service
+
+ #. Add ``<Service name="SERVICE" />`` to the Base or Bundler configuration.
+ #. Add ``<Service name="SERVICE" status="on" type="chkconfig" />`` to
+    ``/var/lib/bcfg2/Rules/services.xml``.
+
+Incorrect entries
+=================
+
+For a "Package"
+---------------
+
+* Failed RPM verification
+
+ #. Run ``rpm -V PACKAGE``
+ #. Add configuration files (the ones with "c" next to them in the
+    verification output) to ``/var/lib/bcfg2/Cfg/``.
+
+  * For example, ``/etc/motd`` to ``/var/lib/bcfg2/Cfg/etc/motd/motd``.
+    Yes, there is an extra directory level named after the file.
+
+ #. Specify configuration files as ``<Path name='PATH' />`` in the Base
+    or Bundler configuration.
+ #. Add directories to ``/var/lib/bcfg2/Rules/directories.xml``. For
+    example:
+
+    .. code-block:: xml
+
+        <Rules priority="0">
+          <Directory name="/etc/cron.hourly" group="root" owner="root" perms="0700" />
+          <Directory name="/etc/cron.daily" group="root" owner="root" perms="0700" />
+        </Rules>
+
+* Multiple instances
+
+ * Option A: Explicitly list the instances
+
+  #. Drop the ``<Package />`` from the Base or Bundler configuration.
+  #. Add an explicit ``<BoundPackage>`` and ``<Instance />`` configuration
+     to a new Bundle, like the following:
+
+     .. code-block:: xml
+
+         <Bundle name='keys'>
+           <!-- GPG keys -->
+           <BoundPackage name="gpg-pubkey" type="yum">
+             <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/>
+             <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release" version="37017186" release="45761324"/>
+           </BoundPackage>
+         </Bundle>
+
+  #. Add the bundle to the applicable groups in
+     ``/var/lib/bcfg2/Metadata/groups.xml``.
+
+ * Option B: Disable verification of the package
+
+  #. Add ``pkg_checks="false"`` to the ``<Package />`` tag.
+
+For a "Path"
+-------------------
+
+ * Unclear verification problem (no details from Bcfg2)
+
+  1. Run ``bcfg2 -vqI`` to see detailed verification issues (but deny
+     any suggested actions).
+
+ * Permissions mismatch
+
+  1. Create an ``info.xml`` file in the same directory as the
+     configuration file. Example:
+
+     .. code-block:: xml
+
+         <FileInfo>
+           <Group name='webserver'>
+             <Info owner='root' group='root' perms='0652'/>
+           </Group>
+           <Info owner='root' group='sys' perms='0651'/>
+         </FileInfo>
+
+Other troubleshooting tools
+===========================
+
+ * Generate the physical configuration from the server side::
+
+       bcfg2-info buildfile /test test.example.com
+
+ * Generate the physical configuration from the client side::
+
+       bcfg2 -vqn -c/root/bcfg2-physical.xml
diff --git a/doc/appendix/guides/fedora.txt b/doc/appendix/guides/fedora.txt
new file mode 100644
index 000000000..d8d3b1b34
--- /dev/null
+++ b/doc/appendix/guides/fedora.txt
@@ -0,0 +1,472 @@
+.. -*- mode: rst -*-
+
+.. This guide is based on the Centos guide.
+
+.. _guide-fedora:
+
+======
+Fedora
+======
+
+This guide is work in progess.
+
+
+This is a complete getting started guide for Fedora. With this
+document you should be able to install a Bcfg2 server, a Bcfg2 client, 
+and change the ``/etc/motd`` file on the client.
+
+Install Bcfg2 From RPM
+======================
+
+The fastest way to get Bcfg2 onto your system is to use ``yum`` 
+or PackageKit. ``
+um`` will pull all dependencies of Bcfg2 
+automatically in. ::
+
+  $ su -c 'yum install bcfg2-server bcfg2'
+
+Your system should now have the necessary software to use Bcfg2. 
+The next step is to set up your Bcfg2 :term:`repository`.
+
+Initialize your repository
+==========================
+
+Now that you're done with the install, you need to initialize your
+repository and setup your ``/etc/bcfg2.conf``. ``bcfg2-admin init``
+is a tool which allows you to automate this:
+
+.. code-block:: sh
+
+    # bcfg2-admin init
+    Store bcfg2 configuration in [/etc/bcfg2.conf]: 
+    Location of bcfg2 repository [/var/lib/bcfg2]: 
+    Directory /var/lib/bcfg2 exists. Overwrite? [y/N]:y
+    Input password used for communication verification (without echoing; leave blank for a random): 
+    What is the server's hostname: [config01.local.net]
+    Input the server location [https://config01.local.net:6789]: 
+    Input base Operating System for clients:
+    1: Redhat/Fedora/RHEL/RHAS/Centos
+    2: SUSE/SLES
+    3: Mandrake
+    4: Debian
+    5: Ubuntu
+    6: Gentoo
+    7: FreeBSD
+    : 1
+    Generating a 1024 bit RSA private key
+    .......................................................++++++
+    .....++++++
+    writing new private key to '/etc/bcfg2.key'
+    -----
+    Signature ok
+    subject=/C=US/ST=Illinois/L=Argonne/CN=config01.local.net
+    Getting Private key
+    Repository created successfuly in /var/lib/bcfg2
+
+Change responses as necessary.
+
+Start the server
+================
+
+You are now ready to start your bcfg2 server for the first time::
+
+    $ su -c '/etc/init.d/bcfg2-server start'
+    Starting Configuration Management Server: bcfg2-server     [  OK  ]
+
+To verify that everything started ok, look for the running daemon and
+check the logs:
+
+.. code-block:: sh
+
+    $ su -c 'tail /var/log/messages'
+    May 16 14:14:57 config01 bcfg2-server[2746]: service available at https://config01.local.net:6789
+    May 16 14:14:57 config01 bcfg2-server[2746]: serving bcfg2-server at https://config01.local.net:6789
+    May 16 14:14:57 config01 bcfg2-server[2746]: serve_forever() [start]
+    May 16 14:14:57 config01 bcfg2-server[2746]: Handled 16 events in 0.009s
+
+
+Run ``bcfg2`` to be sure you are able to communicate with the server:
+
+.. code-block:: sh
+
+    $ su -c 'bcfg2 -vqne'
+     
+    /usr/lib/python2.6/site-packages/Bcfg2/Client/Tools/rpmtools.py:23: DeprecationWarning: the md5 module is deprecated; use hashlib instead
+      import md5
+    Loaded plugins: presto, refresh-packagekit
+    Loaded tool drivers:
+     Action     Chkconfig  POSIX      YUMng     
+    Extra Package imsettings-libs 0.108.0-2.fc13.i686.
+    Extra Package PackageKit-device-rebind 0.6.4-1.fc13.i686.
+    ...
+    Extra Package newt-python 0.52.11-2.fc13.i686.
+    Extra Package pulseaudio-gdm-hooks 0.9.21-6.fc13.i686.
+
+    Phase: initial
+    Correct entries:	0
+    Incorrect entries:	0
+    Total managed entries:	0
+    Unmanaged entries:	1314
+
+
+    Phase: final
+    Correct entries:	0
+    Incorrect entries:	0
+    Total managed entries:	0
+    Unmanaged entries:	1314
+     Package:ConsoleKit                         Package:jasper-libs                        Package:pcsc-lite-libs                    
+     Package:ConsoleKit-libs                    Package:java-1.5.0-gcj                     Package:perf                              
+    ...                        
+     Package:iw                                 Package:pcre                               Service:sshd                              
+     Package:jack-audio-connection-kit          Package:pcsc-lite                          Service:udev-post                         
+
+The ``bcfg2.conf`` file contains only standard plugins so far.
+
+.. code-block:: sh
+
+    $ su -c 'cat /etc/bcfg2.conf'
+
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,Metadata,Pkgmgr,Rules,SSHbase
+
+    [statistics]
+    sendmailpath = /usr/lib/sendmail
+    database_engine = sqlite3
+    # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'.
+    database_name =
+    # Or path to database file if using sqlite3.
+    #<repository>/etc/brpt.sqlite is default path if left empty
+    database_user =
+    # Not used with sqlite3.
+    database_password =
+    # Not used with sqlite3.
+    database_host =
+    # Not used with sqlite3.
+    database_port =
+    # Set to empty string for default. Not used with sqlite3.
+    web_debug = True
+
+    [communication]
+    protocol = xmlrpc/ssl
+    password = test1234
+    certificate = /etc/bcfg2.crt
+    key = /etc/bcfg2.key
+    ca = /etc/bcfg2.crt
+
+    [components]
+    bcfg2 = https://config01.local.net:6789
+
+
+Add the machines to Bcfg2
+-------------------------
+
+``bcfg2-admin`` can be used to add a machine to Bcfg2 easily. You 
+need to know the Fully Qualified Domain Name (FQDN) of ever system
+you want to control through Bcfg2. ::  
+
+   bcfg2-admin client add <FQDN machine> 
+
+Bring your first machine under Bcfg2 control
+--------------------------------------------
+
+Now it is time to get the first machine's configuration into the 
+Bcfg2 repository. The server will be the first machine. It's 
+already in the ``Metadata/client.xml``.
+
+
+Setup the `Packages`_ plugin
+++++++++++++++++++++++++++++
+
+.. _Packages: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages
+
+First, replace **Pkgmgr** with **Packages** in the plugins
+line of ``bcfg2.conf``. Then create `Packages/` directory in 
+``/var/lib/bcfg2`` ::
+
+    $ su -c 'mkdir /var/lib/bcfg2/Packages'
+
+Create a ``config.xml`` file for the packages in 
+``/var/lib/bcfg2/Packages`` with the following content.  Choose a 
+mirror near your location according the `Mirror list`_ .
+
+.. _Mirror list: http://mirrors.fedoraproject.org/publiclist/
+
+.. code-block:: xml
+
+	<Sources>
+	    <YUMSource>
+	    <Group>fedora-13</Group>
+	    <URL>ftp://fedora.tu-chemnitz.de/pub/linux/fedora/linux/releases/</URL>
+	    <Version>13</Version>
+	    <Component>Fedora</Component>
+	    <Arch>i386</Arch>
+	    <Arch>x86_64</Arch>
+	    </YUMSource>
+	</Sources>
+
+.. _Magic Groups: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages#MagicGroups
+
+Due to the `Magic Groups`_, we need to modify our Metadata. Let's
+add a **fedora13** group which inherits a **fedora** group
+(this should replace the existing **redhat** group) present in
+``/var/lib/bcfg2/Metadata/groups.xml``. The resulting file should look
+something like this
+
+.. code-block:: xml
+
+   <Groups version='3.0'>
+       <Group profile='true' public='true' default='true' name='basic'>
+          <Group name='fedora13'/>
+       </Group>
+       <Group name='fedora13'/>
+          <Group name='fedora'/>
+       <Group name='ubuntu'/>
+       <Group name='debian'/>
+       <Group name='freebsd'/>
+       <Group name='gentoo'/>
+       <Group name='fedora'/>
+       <Group name='suse'/>
+       <Group name='mandrake'/>
+       <Group name='solaris'/>
+    </Groups>    
+
+.. note::
+    When editing your xml files by hand, it is useful to occasionally 
+    run ``bcfg2-repo-validate`` to ensure that your xml validates 
+    properly.
+
+Add a probe
++++++++++++
+
+The next step for the client will be to have the proper
+arch group membership. For this, we will make use of the
+:ref:`server-plugins-grouping-dynamic_groups` capabilities of 
+the Probes plugin. Add **Probes** to your plugins line in ``bcfg2.conf``
+and create the Probe:
+
+.. code-block:: sh
+
+    $ su -c 'mkdir /var/lib/bcfg2/Probes'
+    $ su -c 'cat /var/lib/bcfg2/Probes/groups'
+    #!/bin/sh
+
+    echo "group:`uname -m`"
+
+Now a restart of ``bcfg2-server`` is needed::
+
+    $ su -c '/etc/init.d/bcfg2-server restart'
+
+To test the Probe just run ``bcfg2 -vqn``.
+
+.. code-block:: xml
+
+    $ su -c 'bcfg2 -vqn'
+    Running probe group
+    Probe group has result:
+    group:i686
+    ...
+
+Start managing packages
++++++++++++++++++++++++
+
+Add a base-packages bundle. Let's see what happens when we just populate
+it with the *yum* package. Create the ``base-packages.xml`` in your 
+``Bundler/`` directory with a entry for ``yum``.
+
+.. code-block:: xml
+
+    $ cat /var/lib/bcfg2/Bundler/base-packages.xml
+    <Bundle name='base-packages'>
+            <Package name='yum'/>
+    </Bundle>
+
+You need to reference the bundle from your ``group.xml``. The resulting
+profile group might look something like this
+
+.. code-block:: xml
+
+    <Group profile='true' public='true' default='true' name='basic'>
+       <Bundle name='base-packages'/>
+       <Group name='fedora13'/>
+    </Group>
+
+Now if we run the client, we can see what this has done for us.::
+
+    output
+
+As you can see, the Packages plugin has generated the dependencies 
+required for the yum package automatically. The ultimate goal should 
+be to move all the packages from the **Unmanaged** entries section 
+to the **Managed** entries section. So, what exactly *are* those 
+Unmanaged entries?::
+
+    output
+
+Now you can go through these and continue adding the packages you 
+want to your Bundle. After a while, I ended up with a minimal bundle 
+that looks like this
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            
+    </Bundle>
+
+Now when I run the client, you can see I have only one unmanaged
+package::
+
+    outout
+
+The gpg-pubkey packages are special in that they are not really
+packages. Currently, the way to manage them is using 
+:ref:`BoundEntries <boundentries>`. So, after adding them, our 
+Bundle now looks like this
+
+.. note:: This does not actually control the contents of the files,
+          you will need to do this part separately (see below).
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <BoundPackage name="gpg-pubkey" type="rpm">
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5" version="e8562897" release="459f07a4"/>
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/>
+            </BoundPackage>
+            <Package name='bcfg2-server'/>
+            <Package name='exim'/>
+            <Package name='grub'/>
+            <Package name='kernel'/>
+            <Package name='krb5-workstation'/>
+            <Package name='m2crypto'/>
+            <Package name='openssh-clients'/>
+            <Package name='openssh-server'/>
+            <Package name='prelink'/>
+            <Package name='redhat-lsb'/>
+            <Package name='rpm-build'/>
+            <Package name='rsync'/>
+            <Package name='sysklogd'/>
+            <Package name='vim-enhanced'/>
+            <Package name='yum'/>
+    </Bundle>
+
+To actually push the gpg keys out via Bcfg2, you will need to manage
+the files as well. This can be done by adding Path entries for each
+of the gpg keys you want to manage 
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <Path name='/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5'/>
+            <Path name='/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL'/>
+            <BoundPackage name="gpg-pubkey" type="rpm">
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5" version="e8562897" release="459f07a4"/>
+                            <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/>
+            </BoundPackage>
+            <Package name='bcfg2-server'/>
+            <Package name='exim'/>
+            <Package name='grub'/>
+            <Package name='kernel'/>
+            <Package name='krb5-workstation'/>
+            <Package name='m2crypto'/>
+            <Package name='openssh-clients'/>
+            <Package name='openssh-server'/>
+            <Package name='prelink'/>
+            <Package name='redhat-lsb'/>
+            <Package name='rpm-build'/>
+            <Package name='rsync'/>
+            <Package name='sysklogd'/>
+            <Package name='vim-enhanced'/>
+            <Package name='yum'/>
+    </Bundle>
+
+Then add the files to Cfg::
+
+    mkdir -p Cfg/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
+    cp /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5 !$/RPM-GPG-KEY-CentOS-5
+    mkdir -p Cfg/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
+    cp /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL !$/RPM-GPG-KEY-EPEL
+
+Now, running the client shows only unmanaged Service entries. Woohoo!
+
+Manage services
++++++++++++++++
+
+Now let's clear up the unmanaged service entries by adding the
+following entries to our bundle...
+
+.. code-block:: xml
+
+    <!-- basic services -->
+    <Service name='atd'/>
+    <Service name='avahi-daemon'/>
+    <Service name='bcfg2-server'/>
+    <Service name='crond'/>
+    <Service name='cups'/>
+    <Service name='gpm'/>
+    <Service name='lvm2-monitor'/>
+    <Service name='mcstrans'/>
+    <Service name='messagebus'/>
+    <Service name='netfs'/>
+    <Service name='network'/>
+    <Service name='postfix'/>
+    <Service name='rawdevices'/>
+    <Service name='sshd'/>
+    <Service name='syslog'/>
+
+...and bind them in Rules
+
+.. code-block:: xml
+
+    [root@centos ~]# cat /var/lib/bcfg2/Rules/services.xml
+    <Rules priority='1'>
+            <!-- basic services -->
+            <Service type='chkconfig' status='on' name='atd'/>
+            <Service type='chkconfig' status='on' name='avahi-daemon'/>
+            <Service type='chkconfig' status='on' name='bcfg2-server'/>
+            <Service type='chkconfig' status='on' name='crond'/>
+            <Service type='chkconfig' status='on' name='cups'/>
+            <Service type='chkconfig' status='on' name='gpm'/>
+            <Service type='chkconfig' status='on' name='lvm2-monitor'/>
+            <Service type='chkconfig' status='on' name='mcstrans'/>
+            <Service type='chkconfig' status='on' name='messagebus'/>
+            <Service type='chkconfig' status='on' name='netfs'/>
+            <Service type='chkconfig' status='on' name='network'/>
+            <Service type='chkconfig' status='on' name='postfix'/>
+            <Service type='chkconfig' status='on' name='rawdevices'/>
+            <Service type='chkconfig' status='on' name='sshd'/>
+            <Service type='chkconfig' status='on' name='syslog'/>
+    </Rules>
+
+Now we run the client and see there are no more unmanaged entries! ::
+
+    $ su -c 'bcfg2 -veqn'
+
+
+Adding Plugins
+++++++++++++++
+
+Git
+---
+
+.. _Git tutorial: http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html
+
+Adding the :ref:`server-plugins-version-git` plugins can preserve
+versioning information. The first step is to add *Git* to your 
+plugin line::
+
+    plugins = Base,Bundler,Cfg,...,Git
+
+For tracking the configuration files in the ``/var/lib/bcfg2`` 
+directory a git repository need to be established::
+
+    git init
+
+For more detail about the setup of git please refer to a `git tutorial`_.
+The first commit can be the empty or the allready populated directory::
+
+    git add . && git commit -a
+
+While running ``bcfg2-info`` the following line will show up::
+
+   Initialized git plugin with git directory = /var/lib/bcfg2/.git
diff --git a/doc/appendix/guides/gentoo.txt b/doc/appendix/guides/gentoo.txt
new file mode 100644
index 000000000..e818364ce
--- /dev/null
+++ b/doc/appendix/guides/gentoo.txt
@@ -0,0 +1,177 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-gentoo:
+
+======
+Gentoo
+======
+
+This document tries to lay out anything Gentoo-specific that you need
+to know in order to use Bcfg2. Mostly that has to do with getting it
+to cooperate with the various pieces of Portage. Services, all things
+POSIX, and just about anything else that Bcfg2 does will work the same
+on Gentoo as on any other distribution.  Bcfg2 is new on Gentoo; please
+let the list know if you find errors or omissions.
+
+Installing Bcfg2
+================
+
+Early in July 2008, Bcfg2 was added to the Gentoo portage tree.  So far
+it's only keyworded for ~x86, but we hope to see it soon in the amd64 and
+x64-solaris ports.  If you're using Gentoo on some other architecture, it
+should still work provided that you have a reasonably up to date Python;
+try adding `app-admin/bcfg2 ~*` to your `/etc/portage/package.keywords`
+file.
+
+If you don't use portage to install Bcfg2, you'll want to make sure you
+have all the prerequisites installed first. For a server, you'll need:
+
+* ``app-admin/gamin`` or ``app-admin/fam``
+* ``dev-python/lxml``
+
+Clients will need at least:
+
+* ``app-portage/gentoolkit``
+
+Package Repository
+==================
+
+You’ll need (to make) at least one archive of binary packages. The
+Portage driver calls ``emerge`` with the ``-getbinpkgonly`` option. See
+:manpage:`make.conf(5)` and :manpage:`emerge(1)` manpages, specifically
+the *PORTAGE_BINHOST* environment variable.
+
+Time Saver: quickpkg
+--------------------
+
+If you have a standing Gentoo machine that you want to preserve or
+propagate, you can generate a complete package archive based on the
+present state of the system by using the quickpkg utility. For example:
+
+.. code-block:: sh
+
+    for pkg in `equery -q l` ; do quickpkg "=$pkg" ; done
+
+...will leave you with a complete archive of all the packages on your
+system in ``/usr/portage/packages/All``, which you can then move to your
+ftp server.
+
+Cataloging Packages In Your Repository
+--------------------------------------
+
+Once you have a set of packages, you will need to create a catalog for
+them in ``/var/lib/bcfg2/Pkgmgr``. Here's a template:
+
+.. code-block:: xml
+
+    <PackageList uri='' type='portage' priority=''>
+      <Group name=''>
+        <Package name='' version=''/>
+      </Group>
+    </PackageList>
+
+...and a partially filled-out example, for our local Gentoo/VMware build:
+
+.. code-block:: xml
+
+    <PackageList uri='ftp://filthy.uchicago.edu/200701-vmware/' type='portage' priority='0'>
+      <Group name='gentoo-200701-vmware'>
+        <Package name='app-admin/bcfg2' version='0.9.1_pre1'/>
+        [...]
+        <Package name='x11-wm/twm' version='1.0.1'/>
+      </Group>
+    </PackageList>
+
+The `<Group>` name (in our example, "gentoo-200701-vmware") should
+be included by any host which will draw its packages from this list. Our
+collection of packages for this class of machines is at the listed URI,
+and we only have one collection of packages for this batch of machines so
+in our case the `priority` doesn’t really matter, we've set it to `0`.
+
+Notice that package name fields are in `CAT/TITLE` format.
+
+Here is a hack which will generate a list of Package lines from
+a system's database of installed packages, especially useful in
+conjunction with the ``quickpkg`` example above:
+
+.. code-block:: sh
+
+    #!/bin/bash
+    for pkg in `equery -q l` ; do
+       title=`echo $pkg | sed -e 's/\(.*\)-\([0-9].*\)/\1/'`
+       version=`echo $pkg | sed -e 's/\(.*\)-\([0-9].*\)/\2/'`
+       echo "    <Package name='${title}' version='${version}'/>"
+    done
+
+Configuring Client Machines
+===========================
+
+Set up ``/etc/bcfg2.conf`` the way you would for any other Bcfg2 client.
+
+In ``make.conf``, set *PORTAGE_BINHOST* to point to the URI of
+your package repository.  You may want to create versions of
+``make.conf`` for each package repository you maintain, with
+appropriate *PORTAGE_BINHOST* URI's in each, and associated with
+that package archive's group under ``Cfg`` -- for example, we have
+``Cfg/etc/make.conf/make.conf.G99_gentoo-200701-vmware``.  If a client
+host switches groups, and the new group needs a different set of packages,
+everything should just fall into place.
+
+Pitfalls
+========
+
+Package Verification Issues
+---------------------------
+
+As of this writing (2007/01/31), we're aware of a number of packages
+marked stable in the Gentoo x86 tree which, for one reason or another,
+consistently fail to verify cleanly under ``equery check``. In some cases
+(pam, openldap), files which don't (ever) exist on the system are
+nonetheless recorded in the package database; in some (python, Bcfg2,
+ahem), whole classes of files (.pyc and .pyo files) consistently fail
+their md5sum checks; and in others, the problem appears to be a
+discrepancy in the way that symlinks are created vs. the way they're
+recorded in the database. For example, in the OpenSSH package,
+/usr/bin/slogin is a symlink to ./ssh, but equery expects it to point to
+an unadorned ssh. An analogous situation exists with their manpages,
+leading to noise like this::
+
+    # equery check openssh
+    [ Checking net-misc/openssh-4.5_p1 ]
+    !!! /etc/ssh/sshd_config has incorrect md5sum
+    !!! /usr/bin/slogin does not point to ssh
+    !!! /usr/share/man/man1/slogin.1.gz does not point to ssh.1.gz
+    !!! /etc/ssh/ssh_config has incorrect md5sum
+     * 62 out of 66 files good
+
+We can ignore the lines for ``ssh_config`` and ``sshd_config``; those will
+be caught by Bcfg2 as registered config files and handled appropriately.
+
+Because Bcfg2 relies on the client system's native package reporting
+tool to judge the state of installed packages, complaints like these
+about trivial or intractable verification failures can trigger unnecessary
+bundle reinstalls when the Bcfg2 client runs. Bcfg2 will catch on after a
+pass or two that the situation isn't getting any better with repeated
+package installs, stop trying, and list those packages as "bad" in
+the client system's statistics.
+
+Aside from filing bugs with the Gentoo package maintainers, your narrator
+has been unable to come up with a good approach to this. Maybe write a
+series of ``Rules`` definitions according to what the package database
+thinks it should find, and/or stage copies of affected files under
+``Cfg``, and associate those rules and files with the affected package in
+a bundle? Annoying but possibly necessary if you want your stats file
+to look good.
+
+/boot
+-----
+
+Gentoo as well as some other distros recommend leaving ``/boot`` unmounted
+during normal runtime. This can lead to trouble during verification and
+package installation, for example when ``/boot/grub/grub.conf`` turns
+up missing. The simplest way around this might just be to ensure that
+``/boot`` is mounted whenever you run Bcfg2, possibly wrapping Bcfg2
+in a script for the purpose. I've also thought about adding *Action*
+clauses to bundles for grub and our kernel packages, which would mount
+``/boot`` before the bundle installs and unmount it afterward, but this
+doesn't get around the problem of those packages flunking verification.
diff --git a/doc/appendix/guides/nat_howto.txt b/doc/appendix/guides/nat_howto.txt
new file mode 100644
index 000000000..818d3e644
--- /dev/null
+++ b/doc/appendix/guides/nat_howto.txt
@@ -0,0 +1,86 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-nat_howto:
+
+=========
+NAT HOWTO
+=========
+
+This page describes how to setup Bcfg2 to properly function with a
+collection of clients behind NAT. It describes the issues, how the
+underlying portions of the Bcfg2 system function, and how to correctly
+setup client metadata to cope with this environment.
+
+Issues
+======
+
+Bcfg2, by default, uses ip address lookup to determine the identity of
+a client that has connected. This process doesn't work properly in the
+case of NAT'ed hosts, because all requests from these clients come from
+the same external address when connecting to the server.
+
+These client identification issues will manifest themselves in a number
+of ways:
+
+* Inability to setup discrete clients with different profiles
+* Incorrect sharing of probe results across clients in the same NAT pool
+* Inability to bootstrap clients properly when client data is not
+  predefined
+
+Architectural Issues
+====================
+
+Client identification is performed at the beginning of each client/server
+interaction. As of 0.9.3, client identification via IP address can be
+completely short-circuited through the use of a client uuid. Setup of
+client uuids requires actions of both the client and server. On the
+server side, the client uuid must be added to the client record in
+``Metadata/clients.xml``. This attribute allows the server to use the
+user part of the authentication to resolve the client's metadata. Also,
+either the location attribute should be set to floating, or the IP address
+of the NAT must be reflected in the address attribute.  Once added,
+the Client entry in clients.xml will look something like this:
+
+.. code-block:: xml
+
+    <Client profile="desktop" name="test1" pingable="N"
+     uuid='9001ec29-1531-4b16-8198-a71bea093d0a' location='floating'/>
+
+Alternatively, the Client entry can be setup like this:
+
+.. code-block:: xml
+
+    <Client profile="desktop" name="test1" pingable="N"
+     uuid='9001ec29-1531-4b16-8198-a71bea093d0a' address='ip-address-of-NAT'/>
+
+The difference between these definitions is explained in detail in the
+:ref:`appendix-guides-authentication` section, but in short, the second
+form requires that the client access the server from the NAT address,
+while the first form allows it to connect from any address provided it
+uses the proper uuid. (Client identification is orthogonal to the use
+of per-client passwords; this can be set in addition to the attributes
+above.)
+
+Once this setup is done, each client must be configured to use the proper
+uuid upon any server interaction. This can be done using either the
+command line argument -u, or the setting "user" in the "communication"
+section of ``/etc/bcfg2.conf``.
+
+UUID Choice
+===========
+
+When determining client UUIDs, one must take care to ensure that UUIDs
+are unique to the client. Any hardware-specific attribute, like a hash
+of a mac address would be sufficient. Alternatively, if a local hostname
+is unique, it may be used as well.
+
+Automated Client Bootstrapping
+==============================
+
+Automated setup of new clients from behind NAT works by using the common
+password. For example::
+
+    /usr/sbin/bcfg2 -u ubik3 -p desktop -x <password>
+
+It is not possible at this time to do automated setup without setting
+up a pre-shared per-client key.
diff --git a/doc/appendix/guides/ubuntu.txt b/doc/appendix/guides/ubuntu.txt
new file mode 100644
index 000000000..595005018
--- /dev/null
+++ b/doc/appendix/guides/ubuntu.txt
@@ -0,0 +1,479 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-ubuntu:
+
+======
+Ubuntu
+======
+
+.. note::
+
+    This particular how to was done on lucid, but should apply to any
+    other `stable`__ version of Ubuntu.
+
+__ ubuntu-releases_
+.. _ubuntu-releases: https://wiki.ubuntu.com/Releases
+
+Install Bcfg2
+=============
+
+We first need to install the server. For this example, we will use the
+bcfg2 server package from the bcfg2 `PPA`_ (note that there is also a
+version available in the ubuntu archives, but it is not as up to date).
+
+.. _PPA: https://launchpad.net/~bcfg2/+archive/ppa
+
+Add the Ubuntu PPA listing to your APT sources
+----------------------------------------------
+
+See http://trac.mcs.anl.gov/projects/bcfg2/wiki/PrecompiledPackages#UbuntuLucid
+
+Install bcfg2-server
+--------------------
+::
+
+    aptitude install bcfg2-server
+
+Remove the default configuration preseeded by the ubuntu package::
+
+    root@lucid:~# rm -rf /etc/bcfg2* /var/lib/bcfg2
+
+Initialize your repository
+==========================
+
+Now that you're done with the install, you need to intialize your
+repository and setup your bcfg2.conf. bcfg2-admin init is a tool which
+allows you to automate this process.::
+
+    root@lucid:~# bcfg2-admin init
+    Store bcfg2 configuration in [/etc/bcfg2.conf]:
+    Location of bcfg2 repository [/var/lib/bcfg2]:
+    Input password used for communication verification (without echoing; leave blank for a random):
+    What is the server's hostname: [lucid]
+    Input the server location [https://lucid:6789]:
+    Input base Operating System for clients:
+    1: Redhat/Fedora/RHEL/RHAS/Centos
+    2: SUSE/SLES
+    3: Mandrake
+    4: Debian
+    5: Ubuntu
+    6: Gentoo
+    7: FreeBSD
+    : 5
+    Generating a 2048 bit RSA private key
+    ......................................................................................+++
+    ...+++
+    writing new private key to '/etc/bcfg2.key'
+    -----
+    Signature ok
+    subject=/C=US/ST=Illinois/L=Argonne/CN=lucid
+    Getting Private key
+    Repository created successfuly in /var/lib/bcfg2
+
+
+Of course, change responses as necessary.
+
+Start the server
+================
+
+You are now ready to start your bcfg2 server for the first time.::
+
+    root@lucid:~# /etc/init.d/bcfg2-server start
+    root@lucid:~# tail /var/log/syslog
+    Dec 17 22:07:02 lucid bcfg2-server[17523]: serving bcfg2-server at https://lucid:6789
+    Dec 17 22:07:02 lucid bcfg2-server[17523]: serve_forever() [start]
+    Dec 17 22:07:02 lucid bcfg2-server[17523]: Processed 16 fam events in 0.502 seconds. 0 coalesced
+
+Run bcfg2 to be sure you are able to communicate with the server::
+
+    root@lucid:~# bcfg2 -vqn
+    Loaded tool drivers:
+     APT      Action   DebInit  POSIX
+
+    Phase: initial
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      382
+
+
+    Phase: final
+    Correct entries:        0
+    Incorrect entries:      0
+    Total managed entries:  0
+    Unmanaged entries:      382
+
+Bring your first machine under Bcfg2 control
+============================================
+
+Now it is time to get your first machine's configuration into your Bcfg2
+repository. Let's start with the server itself.
+
+Setup the `Packages`_ plugin
+----------------------------
+
+.. _Packages: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages
+
+Replace Pkgmgr with Packages in the plugins line of ``bcfg2.conf``::
+
+    root@lucid:~# cat /etc/bcfg2.conf
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,Metadata,Packages,Rules,SSHbase
+
+    [statistics]
+    sendmailpath = /usr/lib/sendmail
+    database_engine = sqlite3
+    # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'.
+    database_name =
+    # Or path to database file if using sqlite3.
+    #<repository>/etc/brpt.sqlite is default path if left empty
+    database_user =
+    # Not used with sqlite3.
+    database_password =
+    # Not used with sqlite3.
+    database_host =
+    # Not used with sqlite3.
+    database_port =
+    # Set to empty string for default. Not used with sqlite3.
+    web_debug = True
+
+    [communication]
+    protocol = xmlrpc/ssl
+    password = secret
+    certificate = /etc/bcfg2.crt
+    key = /etc/bcfg2.key
+    ca = /etc/bcfg2.crt
+
+    [components]
+    bcfg2 = https://lucid:6789
+
+Create Packages layout (as per :ref:`packages-exampleusage`) in
+``/var/lib/bcfg2``
+
+.. code-block:: xml
+
+    root@lucid:~# mkdir /var/lib/bcfg2/Packages
+    root@lucid:~# cat /var/lib/bcfg2/Packages/config.xml
+    <Sources>
+      <APTSource>
+        <Group>ubuntu-lucid</Group>
+        <URL>http://us.archive.ubuntu.com/ubuntu</URL>
+        <Version>lucid</Version>
+        <Component>main</Component>
+        <Component>multiverse</Component>
+        <Component>restricted</Component>
+        <Component>universe</Component>
+        <Arch>amd64</Arch>
+        <Arch>i386</Arch>
+      </APTSource>
+    </Sources>
+
+Due to the `Magic Groups`_, we need to modify our Metadata. Let's add
+an **ubuntu-lucid** group which inherits the **ubuntu** group already
+present in ``/var/lib/bcfg2/Metadata/groups.xml``. The resulting file
+should look something like this
+
+.. _Magic Groups: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/Packages#MagicGroups
+
+.. code-block:: xml
+
+    <Groups version='3.0'>
+       <Group profile='true' public='true' default='true' name='basic'>
+          <Group name='ubuntu-lucid'/>
+       </Group>
+       <Group name='ubuntu-lucid'>
+          <Group name='ubuntu'/>
+       </Group>
+       <Group name='ubuntu'/>
+       <Group name='debian'/>
+       <Group name='freebsd'/>
+       <Group name='gentoo'/>
+       <Group name='redhat'/>
+       <Group name='suse'/>
+       <Group name='mandrake'/>
+       <Group name='solaris'/>
+    </Groups>
+
+.. note::
+    When editing your xml files by hand, it is useful to occasionally run
+    `bcfg2-repo-validate` to ensure that your xml validates properly.
+
+The last thing we need is for the client to have the proper
+arch group membership. For this, we will make use of the
+:ref:`server-plugins-grouping-dynamic_groups` capabilities of the Probes plugin. Add
+Probes to your plugins line in ``bcfg2.conf`` and create the Probe.
+
+.. code-block:: sh
+
+    root@lucid:~# grep plugins /etc/bcfg2.conf
+    plugins = Base,Bundler,Cfg,Metadata,Packages,Probes,Rules,SSHbase
+    root@lucid:~# mkdir /var/lib/bcfg2/Probes
+    root@lucid:~# cat /var/lib/bcfg2/Probes/groups
+    #!/bin/sh
+
+    ARCH=`uname -m`
+    case "$ARCH" in
+        "x86_64")
+            echo "group:amd64"
+        ;;
+        "i686")
+            echo "group:i386"
+        ;;
+    esac
+
+Now we restart the bcfg2-server::
+
+    root@lucid:~# /etc/init.d/bcfg2-server restart
+    Stopping Configuration Management Server:  * bcfg2-server
+    Starting Configuration Management Server:  * bcfg2-server
+    root@lucid:~# tail /var/log/syslog
+    Dec 17 22:36:47 lucid bcfg2-server[17937]: Packages: File read failed; falling back to file download
+    Dec 17 22:36:47 lucid bcfg2-server[17937]: Packages: Updating http://us.archive.ubuntu.com/ubuntu//dists/lucid/main/binary-amd64/Packages.gz
+    Dec 17 22:36:54 lucid bcfg2-server[17937]: Packages: Updating http://us.archive.ubuntu.com/ubuntu//dists/lucid/multiverse/binary-amd64/Packages.gz
+    Dec 17 22:36:55 lucid bcfg2-server[17937]: Packages: Updating http://us.archive.ubuntu.com/ubuntu//dists/lucid/restricted/binary-amd64/Packages.gz
+    Dec 17 22:36:56 lucid bcfg2-server[17937]: Packages: Updating http://us.archive.ubuntu.com/ubuntu//dists/lucid/universe/binary-amd64/Packages.gz
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: Failed to read file probed.xml
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: Loading experimental plugin(s): Packages
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: NOTE: Interfaces subject to change
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: service available at https://lucid:6789
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: serving bcfg2-server at https://lucid:6789
+    Dec 17 22:37:27 lucid bcfg2-server[17937]: serve_forever() [start]
+    Dec 17 22:37:28 lucid bcfg2-server[17937]: Processed 17 fam events in 0.502 seconds. 0 coalesced
+
+Start managing packages
+-----------------------
+
+Add a base-packages bundle. Let's see what happens when we just populate
+it with the ubuntu-standard package.
+
+.. code-block:: xml
+
+    root@lucid:~# cat /var/lib/bcfg2/Bundler/base-packages.xml
+    <Bundle name='base-packages'>
+            <Package name='ubuntu-standard'/>
+    </Bundle>
+
+You need to reference the bundle from your Metadata. The resulting
+profile group might look something like this
+
+.. code-block:: xml
+
+    <Group profile='true' public='true' default='true' name='basic'>
+       <Bundle name='base-packages'/>
+       <Group name='ubuntu-lucid'/>
+    </Group>
+
+Now if we run the client in debug mode (-d), we can see what this has
+done for us.::
+
+    root@lucid:~# bcfg2 -vqdn
+    Running probe groups
+    Probe groups has result:
+    amd64
+    Loaded tool drivers:
+     APT      Action   DebInit  POSIX
+    The following packages are specified in bcfg2:
+     ubuntu-standard
+    The following packages are prereqs added by Packages:
+     adduser             debconf             hdparm              libdevmapper1.02.1  libk5crypto3        libparted1.8-12     libxml2             passwd              upstart
+     apt                 debianutils         info                libdns53            libkeyutils1        libpci3             logrotate           pciutils            usbutils
+     aptitude            dmidecode           install-info        libelf1             libkrb5-3           libpopt0            lsb-base            perl-base           wget
+     at                  dnsutils            iptables            libept0             libkrb5support0     libreadline5        lshw                popularity-contest  zlib1g
+     base-files          dosfstools          libacl1             libgcc1             liblwres50          libreadline6        lsof                psmisc
+     base-passwd         dpkg                libattr1            libgdbm3            libmagic1           libselinux1         ltrace              readline-common
+     bsdmainutils        ed                  libbind9-50         libgeoip1           libmpfr1ldbl        libsigc++-2.0-0c2a  man-db              rsync
+     bsdutils            file                libc-bin            libgmp3c2           libncurses5         libssl0.9.8         memtest86+          sed
+     cpio                findutils           libc6               libgssapi-krb5-2    libncursesw5        libstdc++6          mime-support        sensible-utils
+     cpp                 ftp                 libcap2             libisc50            libpam-modules      libusb-0.1-4        ncurses-bin         strace
+     cpp-4.4             gcc-4.4-base        libcomerr2          libisccc50          libpam-runtime      libuuid1            netbase             time
+     cron                groff-base          libcwidget3         libisccfg50         libpam0g            libxapian15         parted              tzdata
+
+    Phase: initial
+    Correct entries:        101
+    Incorrect entries:      0
+    Total managed entries:  101
+    Unmanaged entries:      281
+
+
+    Phase: final
+    Correct entries:        101
+    Incorrect entries:      0
+    Total managed entries:  101
+    Unmanaged entries:      281
+
+As you can see, the Packages plugin has generated the dependencies
+required for the ubuntu-standard package for us automatically. The
+ultimate goal should be to move all the packages from the **Unmanaged**
+entries section to the **Managed** entries section. So, what exactly *are*
+those Unmanaged entries?::
+
+    root@lucid:~# bcfg2 -vqen
+    Running probe groups
+    Probe groups has result:
+    amd64
+    Loaded tool drivers:
+     APT      Action   DebInit  POSIX
+
+    Phase: initial
+    Correct entries:        101
+    Incorrect entries:      0
+    Total managed entries:  101
+    Unmanaged entries:      281
+
+
+    Phase: final
+    Correct entries:        101
+    Incorrect entries:      0
+    Total managed entries:  101
+    Unmanaged entries:      281
+     Package:apparmor
+     Package:apparmor-utils
+     Package:apport
+     ...
+
+Now you can go through these and continue adding the packages you want to
+your Bundle. Note that ``aptitude why`` is useful when trying to figure
+out the reason for a package being installed. Also, deborphan is helpful
+for removing leftover dependencies which are no longer needed. After a
+while, I ended up with a minimal bundle that looks like this
+
+.. code-block:: xml
+
+    <Bundle name='base-packages'>
+            <Package name='bash-completion'/>
+            <Package name='bcfg2-server'/>
+            <Package name='debconf-i18n'/>
+            <Package name='deborphan'/>
+            <Package name='diffutils'/>
+            <Package name='e2fsprogs'/>
+            <Package name='fam'/>
+            <Package name='grep'/>
+            <Package name='grub-pc'/>
+            <Package name='gzip'/>
+            <Package name='hostname'/>
+            <Package name='krb5-config'/>
+            <Package name='krb5-user'/>
+            <Package name='language-pack-en-base'/>
+            <Package name='linux-generic'/>
+            <Package name='linux-headers-generic'/>
+            <Package name='login'/>
+            <Package name='manpages'/>
+            <Package name='mlocate'/>
+            <Package name='ncurses-base'/>
+            <Package name='openssh-server'/>
+            <Package name='python-fam'/>
+            <Package name='tar'/>
+            <Package name='ubuntu-minimal'/>
+            <Package name='ubuntu-standard'/>
+            <Package name='vim'/>
+            <Package name='vim-runtime'/>
+
+            <!-- PreDepends -->
+            <Package name='dash'/>
+            <Package name='initscripts'/>
+            <Package name='libdbus-1-3'/>
+            <Package name='libnih-dbus1'/>
+            <Package name='lzma'/>
+            <Package name='mountall'/>
+            <Package name='sysvinit-utils'/>
+            <Package name='sysv-rc'/>
+
+            <!-- vim dependencies -->
+            <Package name='libgpm2'/>
+            <Package name='libpython2.6'/>
+    </Bundle>
+
+As you can see below, I no longer have any unmanaged packages. ::
+
+    root@lucid:~# bcfg2 -vqen
+    Running probe groups
+    Probe groups has result:
+    amd64
+    Loaded tool drivers:
+     APT      Action   DebInit  POSIX
+
+    Phase: initial
+    Correct entries:        247
+    Incorrect entries:      0
+    Total managed entries:  247
+    Unmanaged entries:      10
+
+
+    Phase: final
+    Correct entries:        247
+    Incorrect entries:      0
+    Total managed entries:  247
+    Unmanaged entries:      10
+     Service:bcfg2         Service:fam           Service:killprocs     Service:rc.local      Service:single
+     Service:bcfg2-server  Service:grub-common   Service:ondemand      Service:rsync         Service:ssh
+
+Manage services
+---------------
+
+Now let's clear up the unmanaged service entries by adding the following
+entries to our bundle...
+
+.. code-block:: xml
+
+    <!-- basic services -->
+    <Service name='bcfg2'/>
+    <Service name='bcfg2-server'/>
+    <Service name='fam'/>
+    <Service name='grub-common'/>
+    <Service name='killprocs'/>
+    <Service name='ondemand'/>
+    <Service name='rc.local'/>
+    <Service name='rsync'/>
+    <Service name='single'/>
+    <Service name='ssh'/>
+
+
+...and bind them in Rules
+
+.. code-block:: xml
+
+    root@lucid:~# cat /var/lib/bcfg2/Rules/services.xml
+    <Rules priority='1'>
+            <!-- basic services -->
+            <Service type='deb' status='on' name='bcfg2'/>
+            <Service type='deb' status='on' name='bcfg2-server'/>
+            <Service type='deb' status='on' name='fam'/>
+            <Service type='deb' status='on' name='grub-common'/>
+            <Service type='deb' status='on' name='killprocs'/>
+            <Service type='deb' status='on' name='ondemand'/>
+            <Service type='deb' status='on' name='rc.local'/>
+            <Service type='deb' status='on' name='rsync'/>
+            <Service type='deb' status='on' name='single'/>
+            <Service type='deb' status='on' name='ssh'/>
+    </Rules>
+
+Now we run the client and see there are no more unmanaged entries! ::
+
+    root@lucid:~# bcfg2 -vqn
+    Running probe groups
+    Probe groups has result:
+    amd64
+    Loaded tool drivers:
+     APT      Action   DebInit  POSIX
+
+    Phase: initial
+    Correct entries:        257
+    Incorrect entries:      0
+    Total managed entries:  257
+    Unmanaged entries:      0
+
+    All entries correct.
+
+    Phase: final
+    Correct entries:        257
+    Incorrect entries:      0
+    Total managed entries:  257
+    Unmanaged entries:      0
+
+    All entries correct.
+
+Dynamic (web) reports
+=====================
+
+See installation instructions at :ref:`server-reports-install`
diff --git a/doc/appendix/guides/using-bcfg2-info.txt b/doc/appendix/guides/using-bcfg2-info.txt
new file mode 100644
index 000000000..b2354652f
--- /dev/null
+++ b/doc/appendix/guides/using-bcfg2-info.txt
@@ -0,0 +1,132 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-using_bcfg2_info:
+
+================
+Using bcfg2-info
+================
+
+``bcfg2-info`` is a tool for introspecting server functions. It is
+useful for understanding how the server is interpreting your
+repository. It consists of the same logic executed by the server to
+process the repository and produce configuration specifications, just
+without all of the network communication code. Think of ``bcfg2-info``
+as ``bcfg2-server`` on a stick. It is a useful location to do testing
+and staging of new configuration rules, prior to deployment. This is
+particularly useful when developing templates, or developing Bcfg2
+plugins.
+
+Getting Started
+===============
+
+First, fire up the bcfg2-info interpreter.
+
+.. code-block:: none
+
+     [0:464] bcfg2-info
+     Loading experimental plugin(s): Packages
+     NOTE: Interfaces subject to change
+     Handled 8 events in 0.006s
+     Handled 4 events in 0.035s
+     Welcome to bcfg2-info
+     Type "help" for more information
+     >
+
+At this point, the server core has been loaded up, all plugins have
+been loaded, and the ``bcfg2-info`` has both read the initial state of
+the Bcfg2 repository, as well as begun monitoring it for changes. Like
+*bcfg2-server*, ``bcfg2-info`` monitors the repository for changes,
+however, unlike *bcfg2-server*, it does not process change events
+automatically. File modification events can be processed by explicitly
+calling the **update** command. This will process the events,
+displaying the number of events processed and the amount of time taken
+by this processing. If no events are available, no message will be
+displayed. For example, after a change to a file in the repository:
+
+.. code-block:: none
+
+     >update
+     Handled 1 events in 0.001s
+     > update
+     >
+
+This explicit update process allows you to control the update process,
+as well as see the precise changes caused by repository
+modifications.
+
+``bcfg2-info`` has several builtin commands that display the state of
+various internal server core state. These are most useful for
+examining the state of client metadata, either for a single client, or
+for clients overall.
+
+**clients**
+     displays a list of clients, along with their profile groups
+**groups**
+     displays a list of groups, the inheritance hierarchy, profile
+     status, and category name, if there is one.
+**showclient**
+     displays full metadata information for a client, including
+     profile group, group memberships, bundle list, and any connector
+     data, like Probe values or Property info.
+
+Debugging Configuration Rules
+=============================
+
+In addition to the commands listed above for viewing client metadata,
+there are also commands which can shed light on the configuration
+generation process. Recall that configuration generation occurs in
+three major steps:
+
+1) Resolve client metadata
+2) Build list of entries for the configuration
+3) Bind host-specific version of each entry
+
+Step *1* can be viewed with the commands presented in the previous
+section. The latter two steps can be examined using the following
+commands.
+
+**showentries**
+     displays a list of entries (optionally filtered by type) that
+     appear in a client's configuration specification
+
+**buildfile**
+     Perform the entry binding process on a single entry, displaying
+     its results. This command is very useful when developing
+     configuration file templates.
+
+**build**
+     Build the full configuration specification and write it to a
+     file.
+
+**mappings**
+     displays the entries handled by the plugins loaded by the server
+     core. This command is useful when the server reports a bind
+     failure for an entry.
+
+Debugging and Developing Bcfg2
+==============================
+
+``bcfg2-info`` loads a full Bcfg2 server core, so it provides the ideal
+environment for developing and debugging Bcfg2. Because it is hard to
+automate this sort of process, we have only implemented two commands
+in ``bcfg2-info`` to aid in the process.
+
+**profile**
+     The profile command produces python profiling information for
+     other ``bcfg2-info`` commands. This can be used to track
+     performance problems in configuration generation.
+
+**debug**
+     The debug command exits the ``bcfg2-info`` interpreter loop and drops
+     to a python interpreter prompt. The Bcfg2 server core is available
+     in this namespace as "self". Full documentation for the server core
+     is out of scope for this document. This capability is most useful
+     to call into plugin methods, often with setup calls or the enabling
+     of diagnostics.
+
+     It is possible to return to the ``bcfg2-info`` command loop by
+     exiting the python interpreter with ^D.
+
+     There is built-in support for IPython in ``bcfg2-info``. If IPython
+     is installed, dropping into debug mode in ``bcfg2-info`` will use
+     the IPython interpreter by default.
diff --git a/doc/appendix/guides/using-bcfg2-with-centos.txt b/doc/appendix/guides/using-bcfg2-with-centos.txt
new file mode 100644
index 000000000..93875d4c5
--- /dev/null
+++ b/doc/appendix/guides/using-bcfg2-with-centos.txt
@@ -0,0 +1,79 @@
+.. -*- mode: rst -*-
+
+.. _EPEL: http://fedoraproject.org/wiki/EPEL
+.. _RPMForge: https://rpmrepo.org/RPMforge
+
+.. _getting_started-using_bcfg2-with-centos:
+
+=======================
+Using Bcfg2 With CentOS
+=======================
+
+This section covers specific topics for using Bcfg2 with CentOS. Most
+likely the tips on this page also apply to other members of the Red Hat
+family of Linux operating systems.
+
+From Source
++++++++++++
+
+Install Prerequisities
+######################
+
+While you can go about building all these things from source, this
+how to will try and meet the dependencies using packages from EPEL_
+or RPMforge_. The *el5* package should be compatible with CentOS 5.x.
+
+EPEL_::
+
+    [root@centos ~]# rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-3.noarch.rpm
+
+RPMforge_::
+
+    [root@centos ~]# rpm -Uvh http://dag.wieers.com/rpm/packages/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.x86_64.rpm
+
+.. note::
+
+    Be careful with `mixing package repositories
+    <http://fedoraproject.org/wiki/EPEL/FAQ#What_about_compatibility_with_other_third_party_repositories.3F>`_.
+
+Now you can install the rest of the prerequisites::
+
+    [root@centos ~]# yum install python-genshi python-cheetah python-lxml
+
+Build Packages from source
+##########################
+
+* After installing git, clone the master branch::
+
+    [root@centos redhat]# git clone git://git.mcs.anl.gov/bcfg2.git
+
+* Install the ``fedora-packager`` package ::
+
+    [root@centos ~]# yum install fedora-packager
+
+* A directory structure for the RPM build process has to be established. ::
+
+    [you@centos ~]$ rpmdev-setuptree
+
+* Change to the *redhat* directory of the checked out Bcfg2 source::
+
+    [you@centos ~]$ cd bcfg2/redhat/
+
+* In the particular directory is a Makefile which will do the job of building the RPM packages. You can do this as root, but it's not recommanded::
+
+    [you@centos redhat]$ make
+
+* Now the new RPM package can be installed.  Please adjust the path to your RPM package::
+
+    [root@centos ~]# rpm -ihv /home/YOU/rpmbuild/RPMS/noarch/bcfg2-server-1.0.0-0.2r5835.noarch.rpm
+
+Install Packages from Package repository
+########################################
+
+To install the bcfg2-server and bcfg2 from a package repository, just
+use Yum to do it::
+
+    [root@centos ~]# yum install bcfg2-server bcfg2
+
+.. toctree::
+    :hidden:
diff --git a/doc/appendix/guides/vcs.txt b/doc/appendix/guides/vcs.txt
new file mode 100644
index 000000000..6c2879a65
--- /dev/null
+++ b/doc/appendix/guides/vcs.txt
@@ -0,0 +1,106 @@
+.. -*- mode: rst -*-
+
+.. _appendix-guides-vcs:
+
+=======================
+Version control systems
+=======================
+
+The sections in this guide only cover the basics steps in the setup of
+the different version control systems for usage with the Bcfg2.
+
+Git
+===
+
+.. _Git tutorial: http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html
+
+Adding the :ref:`server-plugins-version-git` plugin will allow you to
+store version information in the statistics database. For tracking the
+configuration files in the ``/var/lib/bcfg2`` directory a git repository
+needs to be established::
+
+    git init
+
+For more detail about the setup of git please refer to a `git tutorial`_.
+The first commit can be the empty or the already populated directory::
+
+    git add . && git commit -a
+
+While running ``bcfg2-info`` the following line will show up::
+
+   Initialized git plugin with git directory = /var/lib/bcfg2/.git
+
+Mercurial 
+=========
+
+The :ref:`server-plugins-version-hg` plugin also allows you to store
+version information in the statistics database.
+
+To use mercurial to track your configuration files, the repository must
+be initialized::
+
+    hg init
+
+Mercurial will not commit the files to the repository until a user name
+is defined in ``/var/lib/bcfg2/.hg/``
+
+.. code-block:: sh
+
+    cat <<END_ENTRY >> /var/lib/bcfg2/.hg/hgrc
+    [ui]
+    username = Yor name <you@example.com>
+    END_ENTRY
+
+Now you are able to make submissions to the repository::
+
+    hg commit
+
+While running ``bcfg2-info`` the following line will show up::
+
+   Initialized hg plugin with hg directory = /var/lib/bcfg2/.hg
+
+Darcs 
+=====
+
+The :ref:`server-plugins-version-darcs` plugin also allows you to store
+version information in the statistics database.
+
+To use darcs to track your configuration files, the repository must
+be initialized::
+
+    darcs initialize
+
+To commit to the darcs repository an author must be added to the 
+``_darcs/prefs/author`` file.  If the ``author`` file is missing, 
+darcs will ask you to enter your e-mail address.
+
+.. code-block:: sh
+
+    cat <<END_ENTRY >> /var/lib/bcfg2/_darcs/prefs/author
+    you@example.com
+    END_ENTRY
+
+All files in the ``/var/lib/bcfg2`` directory should be added to darcs
+now::
+
+    darcs add *
+
+After that you can submit them to the repository::
+
+    darcs record
+
+While running ``bcfg2-info`` the following line will show up::
+
+   Initialized Darcs plugin with darcs directory = /var/lib/bcfg2/_darcs
+
+Cvs
+===
+
+The :ref:`server-plugins-version-cvs` plugin also allows you to store
+version information in the statistics database.
+
+    plugins = Base,Bundler,Cfg,...,Cvs
+
+The CVS repository must be initialized::
+
+    cvs -d /var/lib/bcfg2 init
diff --git a/doc/appendix/guides/web-reports-install.txt b/doc/appendix/guides/web-reports-install.txt
new file mode 100644
index 000000000..af2e240fa
--- /dev/null
+++ b/doc/appendix/guides/web-reports-install.txt
@@ -0,0 +1,184 @@
+.. -*- mode: rst -*-
+
+.. _EPEL: http://fedoraproject.org/wiki/EPEL
+
+.. This is combination of the Ubuntu guide and the Centos guide for
+   installing the web reports.
+
+.. _appendix-guides-web-reports-install:
+
+==================================
+Dynamic (web) Reports installation
+==================================
+
+The first step is to install the needed software components like the
+Django framework and the database (SQlite2). All packages for Fedora
+are in the Fedora Package Collection or in EPEL_ for CentOS/RHEL::
+
+    [root@system01 ~]# yum -y install Django python-simplejson python-sqlite2
+
+Of course is a web server needed as well::
+
+    [root@system01 ~]# yum -y install httpd mod_python
+
+The same packages are needed for Ubuntu systems::
+
+    [root@system01 ~]# aptitude install python-django apache2 libapache2-mod-python
+
+Now we need to create the sqlite database. Use the following command on
+Fedora, CentOS, or RHEL.::
+
+    [root@system01 ~]# python /usr/lib/python2.4/site-packages/Bcfg2/Server/Reports/manage.py syncdb
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+
+    You just installed Django's auth system, which means you don't have any superusers defined.
+    Would you like to create one now? (yes/no): no
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+
+.. note:: There are different versions of Python available. If you are
+          unsure about your installed version use the following line instead of
+          the line above.::
+
+              [root@system01 ~]# PYVER=`python -c 'import sys;print(sys.version[0:3])'`; python /usr/lib/python$PYVER/site-packages/Bcfg2/site-packages/Bcfg2/Server/Reports/manage.py syncdb
+
+The path on Ubuntu systems is different. Please use the same path as shown
+in the following command to execute the script on an Ubuntu machine in
+the next steps::
+
+    [root@system01 ~]# python /usr/share/pyshared/Bcfg2/Server/Reports/manage.py syncdb
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+
+    You just installed Django's auth system, which means you don't have any superusers defined.
+    Would you like to create one now? (yes/no): no
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+
+The server should be tested to make sure that there are no mistakes::
+
+    [root@system01 ~]# python /usr/lib/python2.6/site-packages/Bcfg2/Server/Reports/manage.py testserver
+    Creating test database...
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+    Validating models...
+    0 errors found
+
+    Django version 1.1.1, using settings 'Reports.settings'
+    Development server is running at http://127.0.0.1:8000/
+    Quit the server with CONTROL-C.
+
+Add DBStats to the plugins line of ``bcfg2.conf``. The resulting
+**[server]** section should look something like this::
+
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase
+
+Start/restart the Bcfg2 server::
+
+    [root@system01 ~]# /etc/init.d/bcfg2-server restart
+
+Run the Bcfg2 client in order to populate the statistics database
+(this run should take a bit longer since you are uploading the client
+statistics to the database).
+
+Download the static reports content::
+
+    [root@system01 ~]# cd /var/www/
+    [root@system01 ~]# svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2/reports
+
+Configure Apache using :ref:`dynamic-http-install` as a guide
+
+Copy server/statistics sections of ``bcfg2.conf`` to
+``/etc/bcfg2-web.conf`` (make sure it is world-readable). You should
+then have something like this::
+
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase
+
+    [statistics]
+    sendmailpath = /usr/lib/sendmail
+    database_engine = sqlite3
+    # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'.
+    database_name =
+    # Or path to database file if using sqlite3.
+    #<repository>/etc/brpt.sqlite is default path if left empty
+    database_user =
+    # Not used with sqlite3.
+    database_password =
+    # Not used with sqlite3.
+    database_host =
+    # Not used with sqlite3.
+    database_port =
+    # Set to empty string for default. Not used with sqlite3.
+    web_debug = True
+
+Restart apache and point a browser to your Bcfg2 server.
+
+If using sqlite be sure the sql database file and directory containing
+the database are writable to apache.