Merge branch 'options-rewrite'

Conflicts:
	src/lib/Bcfg2/Client/Frame.py
	src/lib/Bcfg2/Options.py
	src/lib/Bcfg2/Server/Admin/Init.py
	src/lib/Bcfg2/Server/Admin/Xcmd.py
	src/lib/Bcfg2/Server/BuiltinCore.py
	src/lib/Bcfg2/Server/Core.py
	src/lib/Bcfg2/Server/MultiprocessingCore.py
	src/lib/Bcfg2/Server/Plugin/base.py
	src/lib/Bcfg2/Server/Plugin/helpers.py
	src/lib/Bcfg2/Server/Plugins/Cfg/__init__.py
	src/lib/Bcfg2/Server/Plugins/Packages/Yum.py
	src/lib/Bcfg2/Server/Plugins/Packages/__init__.py
	src/lib/Bcfg2/Server/SSLServer.py
	src/lib/Bcfg2/Utils.py
	src/lib/Bcfg2/settings.py
	src/sbin/bcfg2-crypt
	src/sbin/bcfg2-info
	src/sbin/bcfg2-lint
	src/sbin/bcfg2-test
	src/sbin/bcfg2-yum-helper
	tools/bcfg2-profile-templates.py

8 files changed, 277 insertions, 95 deletions

diff --git a/doc/development/core.txt b/doc/development/core.txt
index 3953d3402..ecbcbebd3 100644
--- a/doc/development/core.txt
+++ b/doc/development/core.txt
@@ -10,8 +10,10 @@
 
 Bcfg2 1.3 added a pluggable server core system so that the server core
 itself can be easily swapped out to use different technologies.  It
-currently ships with two backends: a builtin core written from scratch
-using the various server tools in the Python standard library; and an
+currently ships with several backends: a builtin core written from
+scratch using the various server tools in the Python standard library;
+a variant on the builtin core that uses Python 2.6's
+:mod:`multiprocessing` library to process requests in parallel; and an
 experimental `CherryPy <http://www.cherrypy.org/>`_ based core.  This
 page documents the server core interface so that other cores can be
 written to take advantage of other technologies, e.g., `Tornado
@@ -20,20 +22,25 @@ written to take advantage of other technologies, e.g., `Tornado
 
 A core implementation needs to:
 
-* Override :func:`Bcfg2.Server.Core.BaseCore._daemonize` to handle
-  daemonization, writing the PID file, and dropping privileges.
-* Override :func:`Bcfg2.Server.Core.BaseCore._run` to handle server
+* Override :func:`Bcfg2.Server.Core.Core._run` to handle server
   startup.
-* Override :func:`Bcfg2.Server.Core.BaseCore._block` to run the
+* Override :func:`Bcfg2.Server.Core.Core._block` to run the
   blocking server loop.
-* Call :func:`Bcfg2.Server.Core.BaseCore.shutdown` on orderly
+* Call :func:`Bcfg2.Server.Core.Core.shutdown` on orderly
   shutdown.
 
+A core that wants to use the network (i.e., a core that isn't used
+entirely for introspection, as in :ref:`bcfg2-info
+<server-bcfg2-info>`, or other local tasks) should inherit from
+:class:`Bcfg2.Server.Core.NetworkCore`, and must also override
+:func:`Bcfg2.Server.Core.NetworkCore._daemonize` to handle daemonization,
+writing the PID file, and dropping privileges.
+
 Nearly all XML-RPC handling is delegated entirely to the core
 implementation.  It needs to:
 
-* Call :func:`Bcfg2.Server.Core.BaseCore.authenticate` to authenticate
-  clients.
+* Call :func:`Bcfg2.Server.Core.NetworkCore.authenticate` to
+  authenticate clients.
 * Handle :exc:`xmlrpclib.Fault` exceptions raised by the exposed
   XML-RPC methods as appropriate.
 * Dispatch XML-RPC method invocations to the appropriate method,
diff --git a/doc/development/lint.txt b/doc/development/lint.txt
index 6a4651f92..685823ab1 100644
--- a/doc/development/lint.txt
+++ b/doc/development/lint.txt
@@ -10,14 +10,14 @@
 lets you easily write your own plugins to verify various parts of your
 Bcfg2 specification.
 
-Plugins are loaded in one of two ways:
+Plugins are included in a module of the same name as the plugin class
+in :mod:`Bcfg2.Server.Lint`, e.g., :mod:`Bcfg2.Server.Lint.Validate`.
 
-* They may be included in a module of the same name as the plugin
-  class in :mod:`Bcfg2.Server.Lint`, e.g.,
-  :mod:`Bcfg2.Server.Lint.Validate`.
-* They may be included directly in a Bcfg2 server plugin, called
-  "<plugin>Lint", e.g.,
-  :class:`Bcfg2.Server.Plugins.Metadata.MetadataLint`.
+.. note::
+
+    It is no longer possible to include lint plugins directly in a
+    Bcfg2 server plugin, e.g.,
+    :class:`Bcfg2.Server.Plugins.Metadata.MetadataLint`.
 
 Plugin Types
 ============
@@ -106,10 +106,10 @@ Basics
 Existing ``bcfg2-lint`` Plugins
 ===============================
 
-BundlerLint
------------
+Bundler
+-------
 
-.. autoclass:: Bcfg2.Server.Plugins.Bundler.BundlerLint
+.. automodule:: Bcfg2.Server.Lint.Bundler
 
 Comments
 --------
@@ -126,10 +126,10 @@ GroupNames
 
 .. automodule:: Bcfg2.Server.Lint.GroupNames
 
-GroupPatternsLint
------------------
+GroupPatterns
+-------------
 
-.. autoclass:: Bcfg2.Server.Plugins.GroupPatterns.GroupPatternsLint
+.. automodule:: Bcfg2.Server.Lint.GroupPatterns
 
 InfoXML
 -------
@@ -141,25 +141,25 @@ MergeFiles
 
 .. automodule:: Bcfg2.Server.Lint.MergeFiles
 
-MetadataLint
-------------
+Metadata
+--------
 
-.. autoclass:: Bcfg2.Server.Plugins.Metadata.MetadataLint
+.. automodule:: Bcfg2.Server.Lint.Metadata
 
-PkgmgrLint
-----------
+Pkgmgr
+------
 
-.. autoclass:: Bcfg2.Server.Plugins.Pkgmgr.PkgmgrLint
+.. automodule:: Bcfg2.Server.Lint.Pkgmgr
 
 RequiredAttrs
 -------------
 
 .. automodule:: Bcfg2.Server.Lint.RequiredAttrs
 
-TemplateHelperLint
-------------------
+TemplateHelper
+--------------
 
-.. autoclass:: Bcfg2.Server.Plugins.TemplateHelper.TemplateHelperLint
+.. automodule:: Bcfg2.Server.Lint.TemplateHelper
 
 Validate
 --------
diff --git a/doc/development/option_parsing.txt b/doc/development/option_parsing.txt
new file mode 100644
index 000000000..52da8fced
--- /dev/null
+++ b/doc/development/option_parsing.txt
@@ -0,0 +1,236 @@
+.. -*- mode: rst -*-
+
+.. _development-option-parsing:
+
+====================
+Bcfg2 Option Parsing
+====================
+
+Bcfg2 uses an option parsing mechanism based on the Python
+:mod:`argparse` module.  It does several very useful things that
+``argparse`` does not:
+
+* Collects options from various places, which lets us easily specify
+  per-plugin options, for example;
+* Automatically loads components (such as plugins);
+* Synthesizes option values from the command line, config files, and
+  environment variables;
+* Can dynamically create commands with many subcommands (e.g.,
+  bcfg2-info and bcfg2-admin); and
+* Supports keeping documentation inline with the option declaration,
+  which will make it easier to generate man pages.
+
+
+Collecting Options
+==================
+
+One of the more important features of the option parser is its ability
+to automatically collect options from loaded components (e.g., Bcfg2
+server plugins).  Given the highly pluggable architecture of Bcfg2,
+this helps ensure two things:
+
+#. We do not have to specify all options in all places, or even in
+   most places. Options are specified alongside the class(es) that use
+   them.
+#. All options needed for a given script to run are guaranteed to be
+   loaded, without the need to specify all components that script uses
+   manually.
+
+For instance, assume a few plugins:
+
+* The ``Foo`` plugin takes one option, ``--foo``
+* The ``Bar`` plugin takes two options, ``--bar`` and ``--force``
+
+The plugins are used by the ``bcfg2-quux`` command, which itself takes
+two options: ``--plugins`` (which selects the plugins) and
+``--test``.  The options would be selected at runtime, so for instance
+these would be valid:
+
+.. code-block:: bash
+
+    bcfg2-quux --plugins Foo --foo --test
+    bcfg2-quux --plugins Foo,Bar --foo --bar --force
+    bcfg2-quux --plugins Bar --force
+
+But this would not:
+
+    bcfg2-quux --plugins Foo --bar
+
+The help message would reflect the options that are available to the
+default set of plugins.  (For this reason, allowing component lists to
+be set in the config file is very useful; that way, usage messages
+reflect the components in the config file.)
+
+Components (in this example, the plugins) can be classes or modules.
+There is no required interface for an option component.  They may
+*optionally* have:
+
+* An ``options`` attribute that is a list of
+  :class:`Bcfg2.Options.Options.Option` objects or option groups.
+* A function or static method, ``options_parsed_hook``, that is called
+  when all options have been parsed.  (This will be called again if
+  :func:`Bcfg2.Options.Parser.Parser.reparse` is called.)
+
+Options are collected through two primary mechanisms:
+
+#. The :class:`Bcfg2.Options.Actions.ComponentAction` class.  When a
+   ComponentAction subclass is used as the action of an option, then
+   options contained in the classes (or modules) given in the option
+   value will be added to the parser.
+#. Modules that are not loaded via a
+   :class:`Bcfg2.Options.Actions.ComponentAction` option may load
+   options at runtime.
+
+Since it is preferred to add components instead of just options,
+loading options at runtime is generally best accomplished by creating
+a container object whose only purpose is to hold options.  For
+instance:
+
+.. code-block:: python
+
+    def foo():
+        # do stuff
+
+    class _OptionContainer(object):
+        options = [
+            Bcfg2.Options.BooleanOption("--foo", help="Enable foo")]
+
+        @staticmethod
+        def options_parsed_hook():
+            if Bcfg2.Options.setup.foo:
+                foo()
+
+    Bcfg2.Options.get_parser().add_component(_OptionContainer)
+
+The Bcfg2.Options module
+========================
+
+.. currentmodule:: Bcfg2.Options
+
+.. autodata:: setup
+
+Options
+-------
+
+The base :class:`Bcfg2.Options.Option` object represents an option.
+Unlike options in :mod:`argparse`, an Option object does not need to
+be associated with an option parser; it exists on its own.
+
+.. autoclass:: Option
+.. autoclass:: PathOption
+.. autoclass:: BooleanOption
+.. autoclass:: PositionalArgument
+
+The Parser
+----------
+
+.. autoclass:: Parser
+.. autofunction:: get_parser
+.. autoexception:: OptionParserException
+
+Option Groups
+-------------
+
+Options can be grouped in various meaningful ways.  This uses a
+variety of :mod:`argparse` functionality behind the scenes.
+
+In all cases, options can be added to groups in-line by simply
+specifying them in the object group constructor:
+
+.. code-block:: python
+
+    options = [
+        Bcfg2.Options.ExclusiveOptionGroup(
+            Bcfg2.Options.Option(...),
+            Bcfg2.Options.Option(...),
+            required=True),
+        ....]
+
+Nesting object groups is supported in theory, but barely tested.
+
+.. autoclass:: OptionGroup
+.. autoclass:: ExclusiveOptionGroup
+.. autoclass:: Subparser
+.. autoclass:: WildcardSectionGroup
+
+Subcommands
+-----------
+
+This library makes it easier to work with programs that have a large
+number of subcommands (e.g., :ref:`bcfg2-info <server-bcfg2-info>` and
+:ref:`bcfg2-admin <server-admin-index>`).
+
+The normal implementation pattern is this:
+
+#. Define all of your subcommands as children of
+   :class:`Bcfg2.Options.Subcommand`.
+#. Define a :class:`Bcfg2.Options.CommandRegistry` object that will be
+   used to register all of the commands.  Registering a command
+   collect its options and adds it as a
+   :class:`Bcfg2.Options.Subparser` option group to the main option
+   parser.
+#. Register your commands with
+   :func:`Bcfg2.Options.register_commands`, parse options, and run.
+
+:mod:`Bcfg2.Server.Admin` provides a fairly simple implementation,
+where the CLI class is itself the command registry:
+
+.. code-block:: python
+
+    class CLI(Bcfg2.Options.CommandRegistry):
+        def __init__(self):
+            Bcfg2.Options.CommandRegistry.__init__(self)
+            Bcfg2.Options.register_commands(self.__class__,
+                                            globals().values(),
+                                            parent=AdminCmd)
+            parser = Bcfg2.Options.get_parser(
+                description="Manage a running Bcfg2 server",
+                components=[self])
+            parser.parse()
+
+In this case, commands are collected from amongst all global variables
+(the most likely scenario), and they must be children of
+:class:`Bcfg2.Server.Admin.AdminCmd`, which itself subclasses
+:class:`Bcfg2.Options.Subcommand`.
+
+Commands are defined by subclassing :class:`Bcfg2.Options.Subcommand`.
+At a minimum, the :func:`Bcfg2.Options.Subcommand.run` method must be
+overridden, and a docstring written.
+
+.. autoclass:: Subcommand
+.. autoclass:: HelpCommand
+.. autoclass:: CommandRegistry
+.. autofunction:: register_commands
+
+Actions
+-------
+
+Several custom argparse `actions
+<http://docs.python.org/dev/library/argparse.html#action>`_ provide
+some of the option collection magic of :mod:`Bcfg2.Options`.
+
+.. autoclass:: ConfigFileAction
+.. autoclass:: ComponentAction
+.. autoclass:: PluginsAction
+
+Option Types
+------------
+
+:mod:`Bcfg2.Options` provides a number of useful types for use as the `type
+<http://docs.python.org/dev/library/argparse.html#type>`_ keyword
+argument to
+the :class:`Bcfg2.Options.Option` constructor.
+
+.. autofunction:: Bcfg2.Options.Types.path
+.. autofunction:: Bcfg2.Options.Types.comma_list
+.. autofunction:: Bcfg2.Options.Types.colon_list
+.. autofunction:: Bcfg2.Options.Types.octal
+.. autofunction:: Bcfg2.Options.Types.username
+.. autofunction:: Bcfg2.Options.Types.groupname
+.. autofunction:: Bcfg2.Options.Types.timeout
+.. autofunction:: Bcfg2.Options.Types.size
+
+Common Options
+--------------
+
+.. autoclass:: Common
diff --git a/doc/server/admin/bundle.txt b/doc/server/admin/bundle.txt
deleted file mode 100644
index e9cb79781..000000000
--- a/doc/server/admin/bundle.txt
+++ /dev/null
@@ -1,34 +0,0 @@
-.. -*- mode: rst -*-
-
-.. _server-admin-bundle:
-
-bundle
-======
-
-For a list of all available xml bundles use ``list-xml``. ``list-genshi``
-will list all available genshi bundles.::
-
-.. code-block:: sh
-
-    # bcfg2-admin bundles list-xml
-    # bcfg2-admin bundles list-genshi
-
-``show`` provides an interactive dialog to get details about the available
-bundles.::
-
-.. code-block:: sh
-
-    # bcfg2-admin bundles show
-    Available bundles (Number of bundles: 4)
-    ----------------------------------------
-    [0]	motd.xml
-    [1]	snmpd.xml
-    [2]	bcfg2.xml
-    [3]	ntp.xml
-    Enter the line number of a bundle for details: 3
-    Details for the "ntp" bundle:
-    Package:	xntp
-    Path:		/etc/sysconfig/xntp
-    Path:		/etc/sysconfig/clock
-    Path:		/etc/ntp.conf
-    Service:	xntpd
diff --git a/doc/server/admin/compare.txt b/doc/server/admin/compare.txt
index 6a770055e..ffe19efdf 100644
--- a/doc/server/admin/compare.txt
+++ b/doc/server/admin/compare.txt
@@ -6,11 +6,10 @@ compare
 =======
 
 Determine differences between files or directories of client
-specification instances.::
+specification instances::
 
     bcfg2-admin compare <file1> <file2>
 
-If you want to compare two directories recursively then use ``-r`` as an
-option. ::
+Or::
 
-    bcfg2-admin compare -r <dir1> <dir2>
+    bcfg2-admin compare <dir1> <dir2>
diff --git a/doc/server/admin/index.txt b/doc/server/admin/index.txt
index 8ea765aac..707f7c724 100644
--- a/doc/server/admin/index.txt
+++ b/doc/server/admin/index.txt
@@ -16,14 +16,11 @@ functionality. Available modes are listed below.
    :maxdepth: 1
 
    backup
-   bundle
    client
    compare
    init
    minestruct
    perf
    pull
-   query
-   tidy
    viz
    xcmd
diff --git a/doc/server/admin/query.txt b/doc/server/admin/query.txt
deleted file mode 100644
index 65851a43d..000000000
--- a/doc/server/admin/query.txt
+++ /dev/null
@@ -1,15 +0,0 @@
-.. -*- mode: rst -*-
-
-.. _server-admin-query:
-
-query
-=====
-
-Query clients.
-
-The default result format is suitable for consumption by `pdsh`_.
-This example queries the server for all clients in the *ubuntu* group::
-
-    bcfg2-admin query g=ubuntu
-
-.. _pdsh: http://sourceforge.net/projects/pdsh/
diff --git a/doc/server/admin/tidy.txt b/doc/server/admin/tidy.txt
deleted file mode 100644
index 816d6cdb3..000000000
--- a/doc/server/admin/tidy.txt
+++ /dev/null
@@ -1,8 +0,0 @@
-.. -*- mode: rst -*-
-
-.. _server-admin-tidy:
-
-tidy
-====
-
-Clean up useless files in the repo.