doc: Massive update

14 files changed, 757 insertions, 300 deletions

diff --git a/doc/development/client-driver.txt b/doc/development/client-driver.txt
new file mode 100644
index 000000000..19da9cb67
--- /dev/null
+++ b/doc/development/client-driver.txt
@@ -0,0 +1,75 @@
+.. -*- mode: rst -*-
+
+.. _development-client-driver:
+
+Writing A Client Tool Driver
+============================
+
+This page describes the step-by-step process of writing a client tool
+driver for a configuration element type. The included example describes
+an existing driver, and the process that was used to create it.
+
+#. Pick a name for the driver. In this case, we picked the name RPM.
+#. Add "RPM" to the ``__all__ list`` in ``src/lib/Client/Tools/__init__.py``
+#. Create a file in ``src/lib/Client/Tools`` with the same name (RPM.py)
+#. Create a class in this file with the same name (``class RPM``)
+
+   * If it handles ``Package`` entries, subclass ``Bcfg2.Client.Tools.PkgTool``
+     (from here referenced as branch [P])
+   * If it handles ``Service`` entries, subclass ``Bcfg2.Client.Tools.SvcTool``
+     (from here referenced as branch [S])
+   * Otherwise, ``subclass Bcfg2.Client.Tools.Tool`` (from here referenced
+     as branch [T])
+
+#. Set ``__name__`` to "RPM"
+#. Add any required executable programs to ``__execs__``
+#. Set ``__handles__`` to a list of (``entry.tag``, ``entry.get('type')``)
+   tuples. This determines which entries the Tool module can be used
+   on. In this case, we set ``__handles__ = [('Package', 'rpm')]``.
+#. Add verification. This method should return True/False depending
+   on current entry installation status.
+
+   * [T] Add a Verify<entry.tag> method.
+   * [P] Add a VerifyPackage method.
+   * [S] Add a VerifyService method.
+   * In the failure path, the current state of failing entry
+     attributes should be set in the entry, to aid in auditing.
+     (For example, if a file should be mode 644, and is currently
+     mode 600, then set attribute current_perms='600' in the input
+     entry)
+
+#. Add installation support. This method should return True/False
+   depending on the results of the installation process.
+
+   * [T,S] Add an Install<entry.tag> method.
+   * [P] The PkgTool baseclass has a generic mechanism for performing
+     all-at-once installations, followed, in the case of failures, 
+     by single installations. To enable this support, set the pkgtype 
+     attribute to the package type handled by this driver. Set the
+     ``pkgtool`` to a tuple ("command string %s", ("per-package string
+     format", [list of package entry fields])). For RPM, we have
+     ``setup pkgtool =  ("rpm --oldpackage --replacepkgs --quiet -U %s", ("%s", ["url"]))``
+
+#. Implement entry removal
+
+   * [T,S] Implement a ``Remove`` method that removes all specified
+     entries (``prototype Remove(self, entries)``)
+   * [P] Implement a ``RemovePackages`` that removes all specified
+     entries (same prototype as Remove)
+
+#. Add a ``FindExtra`` method that locates entries not included in the
+   configuration. This may or may not be required, certain drivers
+   do not have the capability to find extra entries.
+#. [P] Package drivers require a ``RefreshPackages`` method that updates
+   the internal representation of the package database.
+
+Writing Tool Driver Methods
+---------------------------
+
+#. Programs can be run using ``self.cmd.run``. This function returns a
+   (return code, stdout list) tuple.
+#. The configuration is available as ``self.config``
+#. Runtime options are available in a dictionary as ``self.setup``
+#. Informational, error, and debug messages can be produced by 
+   running ``self.logger.info/error/debug``.
+
diff --git a/doc/development/documentation.txt b/doc/development/documentation.txt
new file mode 100644
index 000000000..8223670b8
--- /dev/null
+++ b/doc/development/documentation.txt
@@ -0,0 +1,75 @@
+.. -*- mode: rst -*-
+
+.. _development-documentation:
+
+Documentation
+=============
+
+There are two parts of documentation in the Bcfg2 project:
+
+* The wiki
+* The manual
+
+
+The wiki
+--------
+.. _Wiki: http://trac.mcs.anl.gov/projects/bcfg2/wiki
+.. _Trac: http://trac.edgewall.org/
+.. _OpenID: https://openid.org/
+.. _MCS: http://www.mcs.anl.gov/
+.. _Argonne National Laboratory: http://www.anl.gov/
+
+A python-based Trac_ instance is used for the Bcfg2 website. The
+Wiki_ part of the website can be edited after you have successful
+logged in. For the login is a vaild OpenID provider needed and an 
+interaction from an administrator. Please request your access to
+the Wiki_ on the :ref:`mailinglist` or in the :ref:`ircchannel`.  
+
+
+The manual
+----------
+.. _rst: http://en.wikipedia.org/wiki/ReStructuredText
+.. _Sphinx: http://sphinx.pocoo.org
+.. _Docutils: 
+
+The source for the manual is located in the `doc/` directory in the
+SVN repository or in the source tarball. All files are written in 
+rst_ (ReStructuredText).  For the build process we are using Sphinx_.
+
+Building the Manual
+^^^^^^^^^^^^^^^^^^^
+
+* Install the prerequisites.  Docutils_ and Sphinx_ are needed to build.
+
+ * For Debian (Lenny) the tools are available in the `backports <http://www.backports.org/dokuwiki/doku.php?id=instructionst>`_ repository; installation can be done with the following::
+   
+    apt-get -t lenny-backports install python-sphinx
+
+ * The needed tools for Fedora based systems are in the `Fedora Package Collection <https://admin.fedoraproject.org/pkgdb>`_; installation can be done easily with Yum::
+
+    yum -y install python-sphinx python-docutils
+
+ * Additionally, to build the PDF version:
+
+  * LaTeX
+  * pdftex
+
+* Download the source. Please refer to :ref:`source` for more details.
+
+* Building the HTML version, run the following command in the `doc/` directory. The output will appear in `../build/sphinx/html`::
+
+    python setup.py build_sphinx
+
+* Building the PDF version ::
+  
+   python setup.py build_sphinx --builder=latex 
+   cd build/sphinx/latex
+   make
+
+The latest version of the manual
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The latest version of the manual can always be found
+`on the Four Kitchens server <http://doc.bcfg2.fourkitchens.com/>`_.
+
+This is an auto-updated from the `Launchpad mirror <https://code.launchpad.net/~vcs-imports/bcfg2/trunk>`_.
diff --git a/doc/development/emacs_snippet.txt b/doc/development/emacs_snippet.txt
new file mode 100644
index 000000000..2905a2b45
--- /dev/null
+++ b/doc/development/emacs_snippet.txt
@@ -0,0 +1,60 @@
+.. -*- mode: rst -*-
+
+.. _development-emacs_snippet:
+
+======================
+Emacs + YASnippet mode
+======================
+
+This page describes using emacs with YASnippet mode with a set of
+snippets that allow quick composition of bundles and base files.
+More snippets are under development.
+
+#. Download YASnippet from http://code.google.com/p/yasnippet/
+#. Install it into your emacs load path (typically ~/.emacs.d/site-lisp)
+#. Add YASnippet initialization to your .emacs (remember to re-byte-compile it if needed)
+
+   .. code-block:: cl
+
+       (require 'yasnippet-bundle)
+
+       ;;; Bcfg2 snippet
+
+       (yas/define-snippets 'sgml-mode
+       '(
+         ("<Bundle" "<Bundle name='${1:bundlename}' version='2.0'>
+          $0
+       </Bundle>" nil)
+         ("<Base" "<Base>
+          $0
+       </Base>" nil)
+         ("<Group" "<Group name='${1:groupname}>
+          $0
+       </Group>" nil)
+         ("<Config" "<ConfigFile name='${1:filename}'/>
+       $0" nil)
+         ("<Service" "<Service name='${1:svcname}'/>
+       $0" nil)
+         ("<Package" "<Package name='${1:packagename}'/>
+       $0" nil)
+         ("<Action" "<Action name='${1:name}'/>
+       $0" nil)
+         ("<Directory" "<Directory name='${1:name}'/>
+       $0" nil)
+         ("<SymLink" "<SymLink name='${1:name}'/>
+       $0" nil)
+         ("<Permissions" "<Permissions name='${1:name}'/>
+       $0" nil)
+       )
+       )
+
+#. One quick M-x eval-current-buffer, and this code is enabled
+
+Each of these snippets activates on the opening element, ie <Bundle.
+After this string is entered, but before entering a space, press <TAB>,
+and the snippet will be expanded. The template will be inserted into
+the text with a set of input prompts, which default to overwrite mode
+and can be tabbed through.
+
+The code above only works for bundles and base, but will be expanded
+to support other xml files as well.
diff --git a/doc/development/index.txt b/doc/development/index.txt
index 74621815f..222978c98 100644
--- a/doc/development/index.txt
+++ b/doc/development/index.txt
@@ -6,315 +6,49 @@
 Bcfg2 Development
 =================
 
-There are many ways to get involved in Bcfg2 development. Here we will
-outline some things that can help you get familiar with the various
-areas of the Bcfg2 code.
+There are several ways users can contribute to the Bcfg2 project.
 
-Tips for Bcfg2 Development
---------------------------
+* Developing code
+* Testing prereleases
+* Reporting bugs
+* Adding to the common repository
+* Improving the wiki and writing documentation
 
-#. Focus on either the client or server code. This focuses the development process down to the precise pieces of code that matter for the task at hand.
 
-   * If you are developing a client driver, then write up a small configuration specification that includes the needed characteristics.
-   * If you are working on the server, run ``bcfg2-info`` and use to assess the code.
+This section will outline some things that can help you get familiar
+with the various areas of the Bcfg2 code.
 
-#. Use the python interpreter. One of python's most appealing features is interactive use of the interpreter.
+Send patches to the :ref:`mailinglist` or create a trac 
+`ticket <https://trac.mcs.anl.gov/projects/bcfg2/newticket>`_
+with the patch included. In order to submit a ticket via the
+trac system, you will need to create a session by clicking on the
+`Preferences <https://trac.mcs.anl.gov/projects/bcfg2/prefs>`_ link and
+filling out/saving changes to the form. In order to be considered for
+mainline inclusion, patches need to be BSD licensed. The most convenient
+way to prepare patches is by using ``git diff`` inside of a source tree
+checked out of git.
 
-   * If you are developing for the client-side, run ``python -i /usr/sbin/bcfg2`` with the appropriate bcfg2 options. This will cause the python interpreter to continue running, leaving all variables intact. This can be used to examine data state in a convenient fashion.
-   * If you are developing for the server side, use ``bcfg2-info`` and the "debug" option. This will leave you at a python interpreter prompt, with the server core loaded in the variable "bcore".
+The source tree can be checked out by running::
 
-#. Use ``pylint`` obsessively. It raises a lot of style-related warnings which can be ignored, but most all of the errors are legitimate.
-#. If you are doing anything with Regular Expressions, `Kodos`_ and `re-try`_ are your friends.
-
-
-.. _Kodos: http://kodos.sourceforge.net
-.. _re-try: http://re-try.appspot.com
-
-
-Environment setup for development
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-* Check out a copy of the code::
-
-      git clone git://git.mcs.anl.gov/bcfg2.git
-
-* Create link to src/lib::
-
-      cd bcfg2
-      ln -s src/lib Bcfg2
-
-* Add ``bcfg2/src/sbin`` to your PATH environment variable
-* Add ``bcfg2`` to your PYTHONPATH environment variable
-
-.. _development-index-writingtooldrivers:
-
-Writing A Client Tool Driver
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-This page describes the step-by-step process of writing a client tool
-driver for a configuration element type. The included example describes
-an existing driver, and the process that was used to create it.
-
-#. Pick a name for the driver. In this case, we picked the name RPM.
-#. Add "RPM" to the __all__ list in ``src/lib/Client/Tools/__init__.py``
-#. Create a file in ``src/lib/Client/Tools`` with the same name (RPM.py)
-#. Create a class in this file with the same name (class RPM)
-
-   * If it handles Package entries, subclass Bcfg2.Client.Tools.PkgTool (from here referenced as branch [P])
-   * If it handles Service entries, subclass Bcfg2.Client.Tools.SvcTool (from here referenced as branch [S])
-   * Otherwise, subclass Bcfg2.Client.Tools.Tool (from here referenced as branch [T])
-
-#. Set __name__ to "RPM"
-#. Add any required executable programs to __execs__
-#. Set __handles__ to a list of (entry.tag, entry.get('type')) tuples. This determines which entries the Tool module can be used on. In this case, we set __handles__ = [('Package', 'rpm')].
-#. Add verification. This method should return True/False depending on current entry installation status.
-
-   * [T] Add a Verify<entry.tag> method.
-   * [P] Add a VerifyPackage method.
-   * [S] Add a VerifyService method.
-   * In the failure path, the current state of failing entry attributes should be set in the entry, to aid in auditing. [[BR]] (For example, if a file should be mode 644, and is currently mode  600, then set attribute current_perms='600' in the input entry)
-
-#. Add installation support. This method should return True/False depending on the results of the installation process.
-
-   * [T,S] Add an Install<entry.tag> method.
-   * [P] The PkgTool baseclass has a generic mechanism for performing all-at-once installations, followed, in the case of failures, by single installations. To enable this support, set the pkgtype attribute to the package type handled by this driver. Set the pkgtool to a tuple ("command string %s", ("per-package string format", [list of package entry fields])). For RPM, we have setup pkgtool =  ("rpm --oldpackage --replacepkgs --quiet -U %s", ("%s", ["url"]))
-
-#. Implement entry removal
-
-   * [T,S] Implement a Remove method that removes all specified entries (prototype Remove(self, entries))
-   * [P] Implement a !RemovePackages that removes all specified entries (same prototype as Remove)
-
-#. Add a FindExtra method that locates entries not included in the configuration. This may or may not be required, certain drivers do not have the capability to find extra entries.
-#. [P] Package drivers require a !RefreshPackages method that updates the internal representation of the package database.
-
-Writing Tool Driver Methods
-"""""""""""""""""""""""""""
-
-#. Programs can be run using self.cmd.run. This function returns a (return code, stdout list) tuple.
-#. The configuration is available as self.config
-#. Runtime options are available in a dictionary as self.setup
-#. Informational, error, and debug messages can be produced by running self.logger.info/error/debug.
-
-Bcfg2 Plugin development
-------------------------
-
-While the Bcfg2 server provides a good interface for representing
-general system configurations, its plugin interface offers the ability
-to implement configuration interfaces and representation tailored to
-problems encountered by a particular site. This chapter describes what
-plugins are good for, what they can do, and how to implement them.
-
-Bcfg2 Plugins
-^^^^^^^^^^^^^
-
-Bcfg2 plugins are loadable python modules that the Bcfg2 server loads at
-initialization time. These plugins can contribute to the functions already
-offered by the Bcfg2 server or can extend its functionality. In general,
-plugins will provide some portion of the configuration for clients, with a
-data representation that is tuned for a set of common tasks. Much of the
-core functionality of Bcfg2 is implemented by several plugins, however,
-they are not special in any way; new plugins could easily supplant one
-or all of them.
-
-The following table describes the various functions of bcfg2 plugins.
-
-+--------------------+---------------------------------------------+
-| Name               | Description                                 |
-+====================+=============================================+
-| Probes             | Plugins can issue commands to collect       |
-|                    | client-side state (like hardware inventory) |
-|                    | to include in client configurations         |
-+--------------------+---------------------------------------------+
-| ConfigurationEntry | Plugins can construct a list of per-client  |
-| List               | configuration entry lists to include in     |
-|                    | client configurations.                      |
-+--------------------+---------------------------------------------+
-| ConfigurationEntry | Literal values for configuration entries    |
-| contents           |                                             |
-+--------------------+---------------------------------------------+
-| XML-RPC functions  | Plugins can export function calls that      |
-|                    | expose internal functions.                  |
-+--------------------+---------------------------------------------+
-
-Writing Bcfg2 Plugins
-^^^^^^^^^^^^^^^^^^^^^
-
-Bcfg2 plugins are python classes that subclass from
-Bcfg2.Server.Plugin.Plugin. Several plugin-specific values must be set
-in the new plugin. These values dictate how the new plugin will behave
-with respect to the above four functions.  The following table describes
-all important member fields.
-
-+-----------------+-----------------------------------+--------------------------+
-| Name            | Description                       | Format                   |
-+=================+===================================+==========================+
-| __name__        | The name of the plugin            | string                   |
-+-----------------+-----------------------------------+--------------------------+
-| __version__     | The plugin version (generally     | string                   |
-|                 | tied to revctl keyword expansion) |                          |
-+-----------------+-----------------------------------+--------------------------+
-| __author__      | The plugin author.                | string                   |
-+-----------------+-----------------------------------+--------------------------+
-| __rmi__         | Set of functions to be exposed as | List of function names   |
-|                 | XML-RPC functions                 | (strings)                |
-+-----------------+-----------------------------------+--------------------------+
-| Entries         | Multidimentional dictionary of    | Dictionary of            |
-|                 | keys that point to the function   | ConfigurationEntityType, |
-|                 | used to bind literal contents for | Name keys, and function  |
-|                 | a given configuration entity.     | reference values         |
-+-----------------+-----------------------------------+--------------------------+
-| BuildStructures | Function that returns a list of   | Member function          |
-|                 | the structures for a given client |                          |
-+-----------------+-----------------------------------+--------------------------+
-| GetProbes       | Function that returns a list of   | Member function          |
-|                 | probes that a given client should |                          |
-|                 | execute                           |                          |
-+-----------------+-----------------------------------+--------------------------+
-| ReceiveData     | Function that accepts the probe   | Member function          |
-|                 | results for a given client.       |                          |
-+-----------------+-----------------------------------+--------------------------+
-
-Example Plugin
-^^^^^^^^^^^^^^
-
-.. code-block:: python
-
-    import Bcfg2.Server.Plugin
-    class MyPlugin(Bcfg2.Server.Plugin.Plugin):
-       """An example plugin."""
-       # All plugins need to subclass Bcfg2.Server.Plugin.Plugin
-       __name__ = 'MyPlugin'
-       __version__ = '1'
-       __author__ = 'me@me.com'
-       __rmi__ = ['myfunction']
-       # myfunction is not available remotely as MyPlugin.myfunction
-
-       def __init__(self, core, datastore):
-           Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore)
-           self.Entries = {'Path':{'/etc/foo.conf': self.buildFoo}}
-
-       def myfunction(self):
-           """Function for xmlrpc rmi call."""
-           # Do something
-           return True
-
-       def buildFoo(self, entry, metadata):
-           """Bind per-client information into entry based on metadata."""
-           entry.attrib.update({'type':'file', 'owner':'root', 'group':'root', 'perms':'644'})
-           entry.text = '''contents of foo.conf'''
-
-Example Connector
-^^^^^^^^^^^^^^^^^
-
-.. code-block:: python
-
-    import Bcfg2.Server.Plugin
-
-    class Foo(Bcfg2.Server.Plugin.Plugin,
-             Bcfg2.Server.Plugin.Connector):
-        """The Foo plugin is here to illustrate a barebones connector."""
-        name = 'Foo'
-        version = '$Revision: $'
-        experimental = True
-
-        def __init__(self, core, datastore):
-            Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore)
-            Bcfg2.Server.Plugin.Connector.__init__(self)
-            self.store = XMLFileBacked(self.data, core.fam)
-
-        def get_additional_data(self, metadata):
-
-            mydata = {}
-            for data in self.store.entries['foo.xml'].data.get("foo", []):
-
-                mydata[data] = "bar"
-
-            return  dict([('mydata', mydata)])
-
-        def get_additional_groups(self, meta):
-            return self.cgroups.get(meta.hostname, list())
-
-Server Plugin Types
--------------------
-
-Generator
-^^^^^^^^^
-
-Generator plugins contribute to literal client configurations
-
-Structure
-^^^^^^^^^
-
-Structure Plugins contribute to abstract client configurations
-
-Metadata
-^^^^^^^^
-
-Signal metadata capabilities
-
-Connector
-^^^^^^^^^
-
-Connector Plugins augment client metadata instances
-
-Probing
-^^^^^^^
-
-Signal probe capability
-
-Statistics
-^^^^^^^^^^
-
-Signal statistics handling capability
-
-Decision
-^^^^^^^^
-
-Signal decision handling capability
-
-Version
-^^^^^^^
-
-Interact with various version control systems
-
-Writing Server Plugins
-----------------------
-
-Metadata
-^^^^^^^^
-
-If you would like to define your own Metadata plugin (to extend/change
-functionality of the existing Metadata plugin), here are the steps to
-do so. We will call our new plugin `MyMetadata`.
-
-#. Add MyMetadata.py
-
-   .. code-block:: python
-
-       __revision__ = '$Revision$'
-
-       import Bcfg2.Server.Plugins.Metadata
-
-       class MyMetadata(Bcfg2.Server.Plugins.Metadata.Metadata):
-           '''This class contains data for bcfg2 server metadata'''
-           __version__ = '$Id$'
-           __author__ = 'bcfg-dev@mcs.anl.gov'
-
-       def __init__(self, core, datastore, watch_clients=True):
-           Bcfg2.Server.Plugins.Metadata.Metadata.__init__(self, core, datastore, watch_clients)
-
-#. Add MyMetadata to ``src/lib/Server/Plugins/__init__.py``
-#. Replace Metadata with MyMetadata in the plugins line of bcfg2.conf
-
-
-Documentation
--------------
-
-One of the areas where everyone can help is with the documentation. *Insert verbiage on how people can help.*
+    git clone git://git.mcs.anl.gov/bcfg2.git
 
+Users wishing to contribute on a regular basis can apply for direct
+git access. Mail the :ref:`mailinglist` for details.
 
 
 .. toctree::
    :maxdepth: 1
-
+   
+   tips
+   setup
+   client-driver
+   plugins
+   writing_plugins
+   plugin-types
+   writing_specification
+   server
+   testing
+   documentation
    docstyleguide
+   emacs_snippet
+   vim_snippet
diff --git a/doc/development/plugin-types.txt b/doc/development/plugin-types.txt
new file mode 100644
index 000000000..5f0a4771a
--- /dev/null
+++ b/doc/development/plugin-types.txt
@@ -0,0 +1,46 @@
+.. -*- mode: rst -*-
+
+.. _development-plugin-types:
+
+Server Plugin Types
+-------------------
+
+Generator
+^^^^^^^^^
+
+Generator plugins contribute to literal client configurations
+
+Structure
+^^^^^^^^^
+
+Structure Plugins contribute to abstract client configurations
+
+Metadata
+^^^^^^^^
+
+Signal metadata capabilities
+
+Connector
+^^^^^^^^^
+
+Connector Plugins augment client metadata instances
+
+Probing
+^^^^^^^
+
+Signal probe capability
+
+Statistics
+^^^^^^^^^^
+
+Signal statistics handling capability
+
+Decision
+^^^^^^^^
+
+Signal decision handling capability
+
+Version
+^^^^^^^
+
+Interact with various version control systems
diff --git a/doc/development/plugins.txt b/doc/development/plugins.txt
new file mode 100644
index 000000000..b6debba24
--- /dev/null
+++ b/doc/development/plugins.txt
@@ -0,0 +1,45 @@
+.. -*- mode: rst -*-
+
+.. _development-plugins:
+
+Bcfg2 Plugin development
+------------------------
+
+While the Bcfg2 server provides a good interface for representing
+general system configurations, its plugin interface offers the ability
+to implement configuration interfaces and representation tailored to
+problems encountered by a particular site. This chapter describes what
+plugins are good for, what they can do, and how to implement them.
+
+Bcfg2 Plugins
+^^^^^^^^^^^^^
+
+Bcfg2 plugins are loadable python modules that the Bcfg2 server loads at
+initialization time. These plugins can contribute to the functions already
+offered by the Bcfg2 server or can extend its functionality. In general,
+plugins will provide some portion of the configuration for clients, with a
+data representation that is tuned for a set of common tasks. Much of the
+core functionality of Bcfg2 is implemented by several plugins, however,
+they are not special in any way; new plugins could easily supplant one
+or all of them.
+
+The following table describes the various functions of bcfg2 plugins.
+
++--------------------+---------------------------------------------+
+| Name               | Description                                 |
++====================+=============================================+
+| Probes             | Plugins can issue commands to collect       |
+|                    | client-side state (like hardware inventory) |
+|                    | to include in client configurations         |
++--------------------+---------------------------------------------+
+| ConfigurationEntry | Plugins can construct a list of per-client  |
+| List               | configuration entry lists to include in     |
+|                    | client configurations.                      |
++--------------------+---------------------------------------------+
+| ConfigurationEntry | Literal values for configuration entries    |
+| contents           |                                             |
++--------------------+---------------------------------------------+
+| XML-RPC functions  | Plugins can export function calls that      |
+|                    | expose internal functions.                  |
++--------------------+---------------------------------------------+
+
diff --git a/doc/development/server.txt b/doc/development/server.txt
new file mode 100644
index 000000000..0f594422e
--- /dev/null
+++ b/doc/development/server.txt
@@ -0,0 +1,83 @@
+.. -*- mode: rst -*-
+
+.. _development-server-plugins:
+
+Writing Server Plugins
+----------------------
+
+Metadata
+^^^^^^^^
+
+If you would like to define your own Metadata plugin (to extend/change
+functionality of the existing Metadata plugin), here are the steps to
+do so. We will call our new plugin `MyMetadata`.
+
+#. Add MyMetadata.py
+
+   .. code-block:: python
+
+       __revision__ = '$Revision$'
+
+       import Bcfg2.Server.Plugins.Metadata
+
+       class MyMetadata(Bcfg2.Server.Plugins.Metadata.Metadata):
+           '''This class contains data for bcfg2 server metadata'''
+           __version__ = '$Id$'
+           __author__ = 'bcfg-dev@mcs.anl.gov'
+
+       def __init__(self, core, datastore, watch_clients=True):
+           Bcfg2.Server.Plugins.Metadata.Metadata.__init__(self, core, datastore, watch_clients)
+
+#. Add MyMetadata to ``src/lib/Server/Plugins/__init__.py``
+#. Replace Metadata with MyMetadata in the plugins line of bcfg2.conf
+
+.. _development-server-packages:
+
+Packages
+--------
+
+In order to support a given client package tool driver, that driver
+must support use of the auto value for the version attribute in Package
+entries. In this case, the tool driver views the current state of
+available packages, and uses the underlying package manager's choice of
+correct package version in lieu of an explicit, centrally-specified,
+version. This support enables Packages to provide a list of Package
+entries with version='auto'. Currently, the APT and YUMng drivers support
+this feature. Note that package management systems without any network
+support cannot operate in this fashion, so RPMng and SYSV will never be
+able to use Packages. Emerge, Zypper, IPS, and Blastwave all have the
+needed features to be supported by Packages, but support has not yet
+been written.
+
+Packages fills two major functions in configuration generation. The first
+is to provide entry level binding support for Package entries included
+in client configurations. This function is quite easy to implement;
+Packages determines (based on client group membership) if the package
+is available for the client system, and which type it has. Because
+version='auto' is used, no version determination needs to be done.
+
+The second major function is more complex. Packages ensures that client
+configurations include all package-level prerequisites for package entries
+explicitly included in the configuration. In order to support this,
+Packages needs to directly process network data for package management
+systems (the network sources for apt or yum, for examples), process
+these files, and build data structures describing prerequisites and the
+providers of those functions/paths. To simplify implementations of this,
+there is a generic base class (Bcfg2.Server.Plugins.Packages.Source)
+that provides a framework for fetching network data via HTTP, processing
+those sources (with subclass defined methods for processing the specific
+format provided by the tool), a generic dependency resolution method,
+and a caching mechanism that greatly speeds up server/bcfg2-info startup.
+
+Each source type must define:
+
+* a get_urls attribute (and associated urls property) that describes
+  the URLS where to get data from.
+* a read_files method that reads and processes the downloaded files
+
+Sources may define a get_provides method, if provides are complex. For
+example, provides in rpm can be either rpm names or file paths, so
+multiple data sources need to be multiplexed.
+
+The APT source in ``src/lib/Server/Plugins/Packages.py`` provides a
+relatively simple implementation of a source.
diff --git a/doc/development/setup.txt b/doc/development/setup.txt
new file mode 100644
index 000000000..08a85dac1
--- /dev/null
+++ b/doc/development/setup.txt
@@ -0,0 +1,19 @@
+.. -*- mode: rst -*-
+
+.. _development-setup:
+
+
+Environment setup for development
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+* Check out a copy of the code::
+
+      svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2
+
+* Create link to ``src/lib``::
+
+      cd bcfg2
+      ln -s src/lib Bcfg2
+
+* Add ``bcfg2/src/sbin`` to your PATH environment variable
+* Add ``bcfg2`` to your PYTHONPATH environment variable
diff --git a/doc/development/specification_overview.png b/doc/development/specification_overview.png
new file mode 100644
index 000000000..66774f359
--- /dev/null
+++ b/doc/development/specification_overview.png
diff --git a/doc/development/testing.txt b/doc/development/testing.txt
new file mode 100644
index 000000000..45b1cbefc
--- /dev/null
+++ b/doc/development/testing.txt
@@ -0,0 +1,74 @@
+.. -*- mode: rst -*-
+
+.. _development-testing:
+
+Testing
+=======
+
+Testing Prereleases
+-------------------
+
+Before each release, several prereleases will be tagged. It is 
+helpful to have users test these releases (when feasible) because 
+it is hard to replicate the full range of potential reconfiguration 
+situations; between different operating systems, system management
+tools, and configuration specification variation, there can be 
+large differences between sites.
+
+
+For more details please visit `Tracking Development Releases of Bcfg2 <http://trac.mcs.anl.gov/projects/bcfg2/wiki/TrackingDevelopmentTrunk>`_ .
+
+
+Upgrade Testing
+---------------
+
+This section describes upgrade procedures to completely test the 
+client and server. These procedures can be used for either pre-release 
+testing, or for confidence building in a new release.
+
+
+Server Testing
+^^^^^^^^^^^^^^
+
+1. Ensure that the server produces the same configurations for clients
+
+ * Before the upgrade, generate all client configurations using the buildall subcommand of bcfg2-info. This subcommand takes a directory argument; it will generate one client configuration in each file, naming each according to the client name.
+
+       .. code-block:: sh
+
+          mgt1:~/bcfg# bcfg2-info
+          Filesystem check 1 of 25
+          ...
+          > buildall /path/to/cf-old
+          Generated config for fs2.bgl.mcs.anl.gov in 1.97310400009 seconds
+          Generated config for fs13.bgl.mcs.anl.gov in 1.47958016396 seconds
+          ...
+
+      Take notice of any messages produced during configuration generation. These generally reflect minor issues in the configuration specification. Ideally, they should be fixed. 
+
+ * Upgrade the server software
+ * Generate all client configurations in a second location using the new software. Any tracebacks reflect bugs, and should be filed in the ticketing system. Any new messages should be carefully examined.
+ * Compare each file in the old directory to those in the new directory using ``bcfg2-admin compare -r /old/directory /new/directory``
+
+      .. code-block:: sh
+
+          mgt1:~/bcfg# bcfg2-admin compare -r cf-old/ cf-new/
+          Entry: fs2.bgl.mcs.anl.gov.xml
+          Entry: fs2.bgl.mcs.anl.gov.xml good
+          Entry: fs13.bgl.mcs.anl.gov.xml
+          Entry: fs13.bgl.mcs.anl.gov.xml good
+          Entry: login1.bgl.mcs.anl.gov.xml
+           ConfigFile /bin/whatami contents differ
+           ConfigFile /bin/whatami differs (in bundle softenv)
+          Entry: login1.bgl.mcs.anl.gov.xml bad
+
+      This can be used to compare configurations for single clients, or different clients. 
+
+2. Compare old and new group diagrams (using ``bcfg2-admin viz``)
+
+Client Testing
+^^^^^^^^^^^^^^
+
+Run the client in dry-run and non-dry-run mode; ensure that multiple 
+runs produce consistent results.
+
diff --git a/doc/development/tips.txt b/doc/development/tips.txt
new file mode 100644
index 000000000..03ffb4871
--- /dev/null
+++ b/doc/development/tips.txt
@@ -0,0 +1,23 @@
+.. -*- mode: rst -*-
+
+.. _development-tips:
+
+Tips for Bcfg2 Development
+--------------------------
+
+#. Focus on either the client or server code. This focuses the development process down to the precise pieces of code that matter for the task at hand.
+
+   * If you are developing a client driver, then write up a small configuration specification that includes the needed characteristics.
+   * If you are working on the server, run ``bcfg2-info`` and use to assess the code.
+
+#. Use the python interpreter. One of python's most appealing features is interactive use of the interpreter.
+
+   * If you are developing for the client-side, run ``python -i /usr/sbin/bcfg2`` with the appropriate bcfg2 options. This will cause the python interpreter to continue running, leaving all variables intact. This can be used to examine data state in a convenient fashion.
+   * If you are developing for the server side, use ``bcfg2-info`` and the "debug" option. This will leave you at a python interpreter prompt, with the server core loaded in the variable "bcore".
+
+#. Use ``pylint`` obsessively. It raises a lot of style-related warnings which can be ignored, but most all of the errors are legitimate.
+#. If you are doing anything with Regular Expressions, `Kodos`_ and `re-try`_ are your friends.
+
+
+.. _Kodos: http://kodos.sourceforge.net
+.. _re-try: http://re-try.appspot.com
diff --git a/doc/development/vim_snippet.txt b/doc/development/vim_snippet.txt
new file mode 100644
index 000000000..a01c1cfda
--- /dev/null
+++ b/doc/development/vim_snippet.txt
@@ -0,0 +1,65 @@
+.. -*- mode: rst -*-
+
+.. _development-vim_snippet:
+
+===================
+Vim Snippet Support
+===================
+
+This page describes using vim with snipMate and a set of snippets
+that allow quick composition of bundles and base files.
+
+#. Download snipMate from http://www.vim.org/scripts/script.php?script_id=2540
+#. Install it using the install instructions (unzip snipMate.zip -d ~/.vim or equivalent, e.g. $HOME\vimfiles on Windows)
+#. Add the following to ``~/.vim/snippets/xml.snippets``
+
+   .. code-block:: cl
+
+        # Bundle
+        snippet <Bundle
+                <Bundle name='${1:bundlename}'>
+                        ${2}
+                </Bundle>
+        # Base
+        snippet <Base
+                <Base>
+                        ${1}
+                </Base>
+        # Group
+        snippet <Group
+                <Group name='${1:groupname}'>
+                        ${2}
+                </Group>
+        # ConfigFile
+        snippet <Config
+                <ConfigFile name='${1:filename}'/>
+        # Service
+        snippet <Service
+                <Service name='${1:svcname}'/>
+        # Package
+        snippet <Package
+                <Package name='${1:packagename}'/>
+        # Action
+        snippet <Action
+                <Action name='${1:name}'/>
+        # Directory
+        snippet <Directory
+                <Directory name='${1:name}'/>
+        # SymLink
+        snippet <SymLink
+                <SymLink name='${1:name}'/>
+        # Permissions
+        snippet <Permissions
+                <Permissions name='${1:name}'/>
+
+
+#. Save and start editing away! 
+
+Each of these snippets activates on the opening element, ie <Bundle>. 
+After this string is entered, but before entering a space, press <TAB>, 
+and the snippet will be expanded. The template will be inserted into 
+the text with a set of input prompts, which default to overwrite mode 
+and can be tabbed through.
+
+The code above only works for bundles and base, but will be expanded 
+to support other xml files as well. 
diff --git a/doc/development/writing_plugins.txt b/doc/development/writing_plugins.txt
new file mode 100644
index 000000000..40e077e43
--- /dev/null
+++ b/doc/development/writing_plugins.txt
@@ -0,0 +1,104 @@
+.. -*- mode: rst -*-
+
+.. _development-write-plugins:
+
+Writing Bcfg2 Plugins
+=====================
+
+Bcfg2 plugins are python classes that subclass from
+Bcfg2.Server.Plugin.Plugin. Several plugin-specific values must be set
+in the new plugin. These values dictate how the new plugin will behave
+with respect to the above four functions.  The following table describes
+all important member fields.
+
++-----------------+-----------------------------------+--------------------------+
+| Name            | Description                       | Format                   |
++=================+===================================+==========================+
+| __name__        | The name of the plugin            | string                   |
++-----------------+-----------------------------------+--------------------------+
+| __version__     | The plugin version (generally     | string                   |
+|                 | tied to revctl keyword expansion) |                          |
++-----------------+-----------------------------------+--------------------------+
+| __author__      | The plugin author.                | string                   |
++-----------------+-----------------------------------+--------------------------+
+| __author__      | The plugin author.                | string                   |
++-----------------+-----------------------------------+--------------------------+
+| __rmi__         | Set of functions to be exposed as | List of function names   |
+|                 | XML-RPC functions                 | (strings)                |
++-----------------+-----------------------------------+--------------------------+
+| Entries         | Multidimentional dictionary of    | Dictionary of            |
+|                 | keys that point to the function   | ConfigurationEntityType, |
+|                 | used to bind literal contents for | Name keys, and function  |
+|                 | a given configuration entity.     | reference values         |
++-----------------+-----------------------------------+--------------------------+
+| BuildStructures | Function that returns a list of   | Member function          |
+|                 | the structures for a given client |                          |
++-----------------+-----------------------------------+--------------------------+
+| GetProbes       | Function that returns a list of   | Member function          |
+|                 | probes that a given client should |                          |
+|                 | execute                           |                          |
++-----------------+-----------------------------------+--------------------------+
+| ReceiveData     | Function that accepts the probe   | Member function          |
+|                 | results for a given client.       |                          |
++-----------------+-----------------------------------+--------------------------+
+
+Example Plugin
+--------------
+
+.. code-block:: python
+
+    import Bcfg2.Server.Plugin
+    class MyPlugin(Bcfg2.Server.Plugin.Plugin):
+       '''An example plugin'''
+       # All plugins need to subclass Bcfg2.Server.Plugin.Plugin
+       __name__ = 'MyPlugin'
+       __version__ = '1'
+       __author__ = 'me@me.com'
+       __rmi__ = ['myfunction']
+       # myfunction is not available remotely as MyPlugin.myfunction
+
+       def __init__(self, core, datastore):
+           Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore)
+           self.Entries = {'Path':{'/etc/foo.conf': self.buildFoo}}
+
+       def myfunction(self):
+           '''function for xmlrpc rmi call'''
+           #do something
+           return True
+
+       def buildFoo(self, entry, metadata):
+           '''Bind per-client information into entry based on metadata'''
+           entry.attrib.update({'type':'file', 'owner':'root', 'group':'root', 'perms':'644'})
+           entry.text = '''contents of foo.conf'''
+
+Example Connector
+-----------------
+
+.. code-block:: python
+
+    import Bcfg2.Server.Plugin
+
+    class Foo(Bcfg2.Server.Plugin.Plugin,
+             Bcfg2.Server.Plugin.Connector):
+        '''The Foo plugin is here to illustrate a barebones connector'''
+        name = 'Foo'
+        version = '$Revision: $'
+        experimental = True
+
+        def __init__(self, core, datastore):
+            Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore)
+            Bcfg2.Server.Plugin.Connector.__init__(self)
+            self.store = XMLFileBacked(self.data, core.fam)
+
+        def get_additional_data(self, metadata):
+
+            mydata = {}
+            for data in self.store.entries['foo.xml'].data.get("foo", []):
+
+                mydata[data] = "bar"
+
+            return  dict([('mydata', mydata)])
+
+        def get_additional_groups(self, meta):
+            return self.cgroups.get(meta.hostname, list())
+
diff --git a/doc/development/writing_specification.txt b/doc/development/writing_specification.txt
new file mode 100644
index 000000000..4e1c03483
--- /dev/null
+++ b/doc/development/writing_specification.txt
@@ -0,0 +1,54 @@
+.. -*- mode: rst -*-
+
+.. _development-writing_specification:
+
+===========================
+Writing Bcfg2 Specification
+===========================
+
+Bcfg2 specifications are logically divided in to three areas:
+
+* Metadata
+* Abstract
+* Literal
+
+The metadata portion of the configuration assigns a client to its profile
+group and to its non-profile groups. The profile group is assigned
+in ``Metadata/clients.xml`` and the non profile group assignments are in
+``Metadata/groups.xml``.
+
+The group memberships contained in the metadata are then used to constuct
+an abstract configuration for the client. An abstract configuration for
+a client identifies the configuration entities (packages, configuration
+files, service, etc) that a client requires, but it does not identify
+them explicitly. For instance an abstract configuration may identify
+that a client needs the Bcfg2 package with
+
+.. code-block:: xml
+
+    <Package name=bcfg2/>
+
+but this does not explicitly identify that an RPM package version
+0.9.2 should be loaded from http://rpm.repo.server/bcfg2-1.0.1-0.1.rpm.
+The abstract configuration is defined in the xml configuration files
+for the Base and Bundles plugins.
+
+A combination of a clients metadata (group memberships) and abstract
+configuration is then used to generate the clients literal configuration.
+For instance the above abstract configuration entry may generate a
+literal configuration of
+
+
+.. code-block:: xml
+
+    <Package name='bcfg2' version='1.0.1-0.1' type='yum'/>
+
+A clients literal configuration is generated by a number of plugins that
+handle the different configuration entities.
+
+
+.. figure:: specification_overview.png
+   :width:  60%
+   :alt:    Specification overview
+   :align:  center
+