improving plugin development docs

7 files changed, 183 insertions, 80 deletions

diff --git a/doc/development/index.txt b/doc/development/index.txt
index 2a54bfad8..a8c3b4795 100644
--- a/doc/development/index.txt
+++ b/doc/development/index.txt
@@ -10,20 +10,9 @@ There are many ways to get involved in Bcfg2 development. Here we will
 outline some things that can help you get familiar with the various
 areas of the Bcfg2 code.
 
-
-Send patches to the :ref:`help-mailinglist` or create a trac
-`ticket <https://trac.mcs.anl.gov/projects/bcfg2/newticket>`_
-with the patch included. In order to submit a ticket via the
-trac system, you will need to create a session by clicking on the
-`Preferences <https://trac.mcs.anl.gov/projects/bcfg2/prefs>`_ link and
-filling out/saving changes to the form. In order to be considered for
-mainline inclusion, patches need to be BSD licensed. The most convenient
-way to prepare patches is by using ``git diff`` inside of a source tree
-checked out of git.
-
-The source tree can be checked out by running::
-
-    git clone git://git.mcs.anl.gov/bcfg2.git
+The easiest way to submit a patch is to submit a pull request on
+Github.  You can fork and clone the source tree at
+https://github.com/bcfg2/Bcfg2
 
 Users wishing to contribute on a regular basis can apply for direct
 git access. Mail the :ref:`help-mailinglist` for details.
diff --git a/doc/development/plugins.txt b/doc/development/plugins.txt
index b2b70f553..96469602d 100644
--- a/doc/development/plugins.txt
+++ b/doc/development/plugins.txt
@@ -23,53 +23,169 @@ core functionality of Bcfg2 is implemented by several plugins, however,
 they are not special in any way; new plugins could easily supplant one
 or all of them.
 
-The following table describes the various functions of bcfg2 plugins.
-
-+--------------------+---------------------------------------------+
-| Name               | Description                                 |
-+====================+=============================================+
-| Probes             | Plugins can issue commands to collect       |
-|                    | client-side state (like hardware inventory) |
-|                    | to include in client configurations         |
-+--------------------+---------------------------------------------+
-| ConfigurationEntry | Plugins can construct a list of per-client  |
-| List               | configuration entry lists to include in     |
-|                    | client configurations.                      |
-+--------------------+---------------------------------------------+
-| ConfigurationEntry | Literal values for configuration entries    |
-| contents           |                                             |
-+--------------------+---------------------------------------------+
-| XML-RPC functions  | Plugins can export function calls that      |
-|                    | expose internal functions.                  |
-+--------------------+---------------------------------------------+
-
 Server Plugin Types
 -------------------
 
+A plugin must implement at least one of the interfaces described
+below.  Each interface is available as a class in
+``Bcfg2.Server.Plugin``.  In most cases, a plugin must also inherit
+from ``Bcfg2.Server.Plugin.Plugin``, which is the base Plugin object
+(described below).  Some of the interfaces listed below are themselves
+Plugin objects, so your custom plugin would only need to inherit from
+the plugin type.
+
 Generator
 ^^^^^^^^^
 
-Generator plugins contribute to literal client configurations
+Generator plugins contribute to literal client configurations.  That
+is, they generate entry contents.  Examples are
+:ref:`server-plugins-generators-cfg` and
+:ref:`server-plugins-generators-sshbase`.
+
+An entry is generated in one of two ways:
+
+#.  First, the Bcfg2 core looks in the ``Entries`` dict attribute of
+ the plugin object.  ``Entries`` is expected to be a dict whose keys
+ are entry tags (e.g., ``"Path"``, ``"Service"``, etc.) and whose
+ values are dicts; those dicts should map the ``name`` attribute of an
+ entry to a callable that will be called to generate the content.  The
+ callable will receive two arguments: the abstract entry (as an
+ lxml.etree._Element object), and the client metadata object the entry
+ is being generated for.
+#.  Second, if the entry is not listed in ``Entries``, the Bcfg2 core
+ calls ``HandlesEntry(entry, metadata)``; if that returns True, then
+ it calls ``HandleEntry(entry, metadata)``.
+
+The Generator plugin should provide one or both methods to bind
+entries, but does not have to provide both.
+
+Both ``HandleEntry()`` and the callable objects in the ``Entries``
+dict should return an lxml.etree._Element object representing the
+fully-bound entry.  They should raise
+``Bcfg2.Server.Plugin.PluginExecutionError`` with a sensible error
+message on failure.
 
 Structure
 ^^^^^^^^^
 
-Structure Plugins contribute to abstract client configurations
+Structure Plugins contribute to abstract client configurations.  That
+is, they produce lists of entries that will be generated for a client.
+:ref:`server-plugins-structure-bundler-index` is a Structure plugin.
+
+Structure plugins must implement one method: ``BuildStructures()``,
+which is called with a single argument, a client metadata object
+structures should be built for.  It should return a list of
+lxml.etree._Element objects that will be added to the top-level
+``<Configuration>`` tag of the client configuration.  Consequently,
+each object in the list must consist of a container tag (e.g.,
+``<Bundle>`` or ``<Independent>``) which contains the entry tags.  It
+must not return a list of entry tags.
 
 Metadata
 ^^^^^^^^
 
-Signal metadata capabilities
+Metadata plugins provide client metadata.
+:ref:`server-plugins-grouping-metadata` is a Metadata plugin.
+
+Metadata plugins **must** implement the following methods:
+
+* ``AuthenticateConnection(cert, user, password, addresspair)``:
+  Authenticate the given client.  Arguments are:
+  * ``cert``: an x509 certificate dict;
+  * ``user``: The username of the user trying to authenticate;
+  * ``password``: The password supplied by the client;
+  * ``addresspair``: A tuple of ``(<ip address>, <hostname>)``
+  ``AuthenticateConnection()`` should return True if the authenticate
+  succeeds, False otherwise.  Failures should be logged at the error
+  level.
+* ``get_initial_metadata(client_name)``: Return a ClientMetadata
+  object that fully describes everything the Metadata plugin knows
+  about the named client.  See :file:``Metadata.py`` for a reference
+  implementation of the ClientMetadata object.
+* ``merge_additional_data(metadata, source, data)``: Add data from the
+  Connector plugin named by ``<source>`` (a string giving the name of
+  the Connector plugin) to the given metadata object.
+  ``merge_additional_data()`` should modify the ``metadata`` object in
+  place; it doesn't need to return anything.
+* ``merge_additional_groups(metadata, groups)``: Add groups from an
+  anonymous Connector plugin to the given metadata object.
+  ``merge_additional_groups()`` should modify the ``metadata`` object in
+  place; it doesn't need to return anything.
+
+Metadata plugins **may** implement the following methods:
+
+* ``viz(hosts, bundles, key, only_client, colors)``: Return a string
+  containing a graphviz document that maps out the Metadata.  The
+  first three options are boolean, and describe whether or not the
+  named item(s) should be included in the graphviz document.
+  ``only_client`` is the name of a client which, if included, will be
+  the only client whose metadata will be included on the map.
+  ``colors`` is a list of graphviz color names to use.  If
+  unimplemented, the empty string will be returned.
+* ``set_version(client, version)``: Set the version for the named
+  client to the specified version string.  If unimplemented, setting
+  the version of a client will fail silently.
+* ``set_profile(client, profile, addresspair)``: Set the profile for
+  the named client to the named profile group.  If unimplemented,
+  setting a client profile will fail silently.
+* ``resolve_client(addresspair, cleanup_cache=False)``: Given a tuple
+  of ``(<ip address>, <hostname>)``, resolve the canonical name of
+  this client.  If this method is not implemented, the hostname
+  claimed by the client is used.  (This may be a security risk; it's
+  highly recommended that you implement ``resolve_client`` if you are
+  writing a Metadata plugin.)
 
 Connector
 ^^^^^^^^^
 
-Connector Plugins augment client metadata instances
+Connector plugins augment client metadata instances with additional
+data, additional groups, or both.  Connector plugins include
+:ref:`server-plugins-grouping-grouppatterns`,
+:ref:`server-plugins-connectors-properties`, and
+:ref:`server-plugins-probes-index`.
+
+Connector plugins should implement one or all of the following
+methods:
+
+* ``get_additional_groups(metadata)``: Return a list of additional
+  groups for the given ClientMetadata object.  If unimplemented, the
+  empty list is returned.
+* ``get_additional_data(metadata)``: Return arbitrary additional data
+  for the given ClientMetadata object.  By convention this is usually
+  a dict object, but doesn't need to be.  If unimplemented, the empty
+  dict is returned.
 
 Probing
 ^^^^^^^
 
-Signal probe capability
+Probing plugins can collect data from clients and process it.
+Examples include :ref:`server-plugins-probes-index` and
+:ref:`server-plugins-probes-fileprobes`.
+
+Probing plugins must implement the following methods:
+
+* ``GetProbes(metadata)``: Return a list of probes for the given
+  ClientMetadata object.  Each probe should be an XML document,
+  described below.
+* ``ReceiveData(metadata, datalist)``: Process data returned from the
+  probes for the given ClientMetadata object.  ``datalist`` is a list
+  of lxml.etree._Element objects, each of which is a single tag; the
+  ``name`` attribute holds the unique name of the probe that was run,
+  and the text contents of the tag hold the results of the probe.
+
+``GetProbes()`` returns a list of probes, each of which is an
+``lxml.etree._Element`` object that adheres to the following
+specification.  Each probe must the following attributes:
+
+* ``name``: The unique name of the probe.
+* ``source``: The origin of the probe; probably the name of the plugin
+  that supplies the probe.
+* ``interpreter``: The command that will be run on the client to
+  interpret the probe script.  Compiled (i.e., non-interpreted) probes
+  are not supported.
+
+The text of the XML tag should be the contents of the probe, i.e., the
+code that will be run on the client.
 
 Statistics
 ^^^^^^^^^^
diff --git a/doc/development/setup.txt b/doc/development/setup.txt
index b04bce3fe..05ad4157f 100644
--- a/doc/development/setup.txt
+++ b/doc/development/setup.txt
@@ -10,7 +10,7 @@ Checking Out a Copy of the Code
 
 * Check out a copy of the code::
 
-      git clone git://git.mcs.anl.gov/bcfg2.git
+      git clone https://github.com/Bcfg2/bcfg2.git
 
 * Add :file:`bcfg2/src/sbin` to your :envvar:`PATH` environment variable
 * Add :file:`bcfg2/src/lib` to your :envvar:`PYTHONPATH` environment variable
diff --git a/src/lib/Bcfg2/Proxy.py b/src/lib/Bcfg2/Proxy.py
index 5a5129939..9bd239e26 100644
--- a/src/lib/Bcfg2/Proxy.py
+++ b/src/lib/Bcfg2/Proxy.py
@@ -299,6 +299,7 @@ class XMLRPCTransport(xmlrpclib.Transport):
 
     def request(self, host, handler, request_body, verbose=0):
         """Send request to server and return response."""
+        print "len(request_body) = %s" % len(request_body)
         try:
             conn = self.send_request(host, handler, request_body, False)
             response = conn.getresponse()
@@ -331,27 +332,6 @@ class XMLRPCTransport(xmlrpclib.Transport):
             self.send_content(conn, request_body)
             return conn
 
-    def _get_response(self, fd, length):
-        # read response from input file/socket, and parse it
-        recvd = 0
-
-        p, u = self.getparser()
-
-        while recvd < length:
-            rlen = min(length - recvd, 1024)
-            response = fd.read(rlen)
-            recvd += len(response)
-            if not response:
-                break
-            if self.verbose:
-                print("body:", repr(response), len(response))
-            p.feed(response)
-
-        fd.close()
-        p.close()
-
-        return u.close()
-
 
 def ComponentProxy(url, user=None, password=None, key=None, cert=None, ca=None,
                    allowedServerCNs=None, timeout=90, retries=3, delay=1):
diff --git a/src/lib/Bcfg2/Server/Plugin.py b/src/lib/Bcfg2/Server/Plugin.py
index c6476eb90..80537c200 100644
--- a/src/lib/Bcfg2/Server/Plugin.py
+++ b/src/lib/Bcfg2/Server/Plugin.py
@@ -168,9 +168,12 @@ class PluginDatabaseModel(object):
 
 
 class Generator(object):
-    """Generator plugins contribute to literal client configurations."""
+    """Generator plugins contribute to literal client
+    configurations."""
+
     def HandlesEntry(self, entry, metadata):
-        """This is the slow path method for routing configuration binding requests."""
+        """This is the slow path method for routing configuration
+        binding requests."""
         return False
 
     def HandleEntry(self, entry, metadata):
@@ -187,20 +190,21 @@ class Structure(object):
 
 class Metadata(object):
     """Signal metadata capabilities for this plugin"""
-    def add_client(self, client_name):
-        """Add client."""
-        pass
+    def viz(self, hosts, bundles, key, only_client, colors):
+        """Create viz str for viz admin mode."""
+        return ''
 
-    def remove_client(self, client_name):
-        """Remove client."""
+    def set_version(self, client, version):
         pass
 
-    def viz(self, hosts, bundles, key, colors):
-        """Create viz str for viz admin mode."""
+    def set_profile(self, client, profile, address):
         pass
 
-    def _handle_default_event(self, event):
-        pass
+    def resolve_client(self, address, cleanup_cache=False):
+        return address[1]
+
+    def AuthenticateConnection(self, cert, user, password, address):
+        raise NotImplementedError
 
     def get_initial_metadata(self, client_name):
         raise NotImplementedError
@@ -225,13 +229,13 @@ class Connector(object):
 
 class Probing(object):
     """Signal probe capability for this plugin."""
-    def GetProbes(self, _):
+    def GetProbes(self, metadata):
         """Return a set of probes for execution on client."""
-        return []
+        raise NotImplementedError
 
-    def ReceiveData(self, _, dummy):
+    def ReceiveData(self, metadata, datalist):
         """Receive probe results pertaining to client."""
-        pass
+        raise NotImplementedError
 
 
 class Statistics(Plugin):
diff --git a/src/lib/Bcfg2/Server/Plugins/Probes.py b/src/lib/Bcfg2/Server/Plugins/Probes.py
index 056521ce7..fd80cfbe4 100644
--- a/src/lib/Bcfg2/Server/Plugins/Probes.py
+++ b/src/lib/Bcfg2/Server/Plugins/Probes.py
@@ -141,7 +141,7 @@ class ProbeSet(Bcfg2.Server.Plugin.EntrySet):
 
         for (name, entry) in list(build.items()):
             probe = lxml.etree.Element('probe')
-            probe.set('name', name.split('/')[-1])
+            probe.set('name', os.path.basename(name))
             probe.set('source', self.plugin_name)
             probe.text = entry.data
             match = self.bangline.match(entry.data.split('\n')[0])
diff --git a/testsuite/Testsrc/Testlib/TestServer/TestPlugin.py b/testsuite/Testsrc/Testlib/TestServer/TestPlugin.py
index 0581fb1ea..cbeec965c 100644
--- a/testsuite/Testsrc/Testlib/TestServer/TestPlugin.py
+++ b/testsuite/Testsrc/Testlib/TestServer/TestPlugin.py
@@ -215,6 +215,12 @@ class TestMetadata(Bcfg2TestCase):
     def get_obj(self):
         return self.test_obj()
 
+    def test_AuthenticateConnection(self):
+        m = self.get_obj()
+        self.assertRaises(NotImplementedError,
+                          m.AuthenticateConnection,
+                          None, None, None, (None, None))
+
     def test_get_initial_metadata(self):
         m = self.get_obj()
         self.assertRaises(NotImplementedError,
@@ -241,12 +247,20 @@ class TestConnector(Bcfg2TestCase):
 
 
 class TestProbing(Bcfg2TestCase):
-    """ placeholder """
+    test_obj = Probing
+
+    def get_obj(self):
+        return self.test_obj()
+
     def test_GetProbes(self):
-        pass
+        p = self.get_obj()
+        self.assertRaises(NotImplementedError,
+                          p.GetProbes, None)
 
     def test_ReceiveData(self):
-        pass
+        p = self.get_obj()
+        self.assertRaises(NotImplementedError,
+                          p.ReceiveData, None, None)
 
 
 class TestStatistics(TestPlugin):