Rewrote arbitrary data cache system

The caching facilities in Bcfg2.Server.Cache provided basically no
features.  This rewrites that to allow for much more powerful cache
expiration, with a particular focus on interoperation between
different components and plugins to let caches be expired as
necessary.  (E.g., the Probes plugin can expire the Metadata cache.)

This does not affect any of the file data cached by Bcfg2, only the
caches that are populated with arbitrary data (Metadata, Packages,
Probes, etc.).

1 files changed, 73 insertions, 0 deletions

diff --git a/doc/development/caching.txt b/doc/development/caching.txt
new file mode 100644
index 000000000..47d627278
--- /dev/null
+++ b/doc/development/caching.txt
@@ -0,0 +1,73 @@
+.. -*- mode: rst -*-
+
+.. _development-cache:
+
+============================
+ Server-side Caching System
+============================
+
+.. versionadded:: 1.4.0
+
+Bcfg2 caches two kinds of data:
+
+* The contents of all files that it reads in, including (often) an
+  optimized representation.  E.g., XML files are cached both in their
+  raw (text) format, and also as :class:`lxml.etree._Element` objects.
+* Arbitrary data, in the server-side caching system documented on this
+  page.
+
+The caching system keeps a single unified cache with all cache data in
+it.  Each individual datum stored in the cache is associated with any
+number of "tags" -- simple terms that uniquely identify the datum.
+This lets you very easily expire related data from multiple caches at
+once; for isntance, for expiring all data related to a host:
+
+.. code-block:: python
+
+    Bcfg2.Server.Cache.expire("foo.example.com")
+
+This would expire *all* data related to ``foo.example.com``,
+regardless of which plugin cached it, and so on.
+
+This permits a high level of interoperation between different plugins
+and the cache, which is necessary due to the wide distribution of data
+in Bcfg2 and the many different data sources that can be incorported.
+More technical details about writing code that uses the caches is below.
+
+Currently known caches are:
+
+.. currentmodule:: Bcfg2.Server.Plugins.Packages.Collection
+
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Tags        | Key(s)                                | Values                                          | Use                                                  |
++=============+=======================================+=================================================+======================================================+
+| Metadata    | Hostname                              | :class:`ClientMetadata                          | The :ref:`Metadata cache <server-caching>`           |
+|             |                                       | <Bcfg2.Server.Plugins.Metadata.ClientMetadata>` |                                                      |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Probes,     | Hostname                              | ``list`` of group names                         | Groups set by :ref:`server-plugins-probes-index`     |
+| probegroups |                                       |                                                 |                                                      |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Probes,     | Hostname                              | ``dict`` of ``<probe name>``:                   | Other data set by :ref:`server-plugins-probes-index` |
+| probedata   |                                       | :class:`ProbeData                               |                                                      |
+|             |                                       | <Bcfg2.Server.Plugins.Probes.ProbeData>`        |                                                      |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Packages,   | :attr:`Packages Collection cache key  | :class:`Collection`                             | Kept by :ref:`server-plugins-generators-packages` in |
+| collections | <Collection.cachekey>`                |                                                 | order to expire repository metadata cached on disk   |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Packages,   | Hostname                              | :attr:`Packages Collection cache key            | Used by the Packages plugin to return Collection     |
+| clients     |                                       | <Collection.cachekey>`                          | objects for clients.  This is cross-referenced with  |
+|             |                                       |                                                 | the ``Packages, collections`` cache                  |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Packages,   | :attr:`Packages Collection cache key  | ``set`` of package names                        | Cached results from looking up                       |
+| pkg_groups  | <Collection.cachekey>`,               |                                                 | ``<Package group="..."/>`` entries                   |
+|             | hash of the selected package groups   |                                                 |                                                      |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+| Packages,   | :attr:`Packages Collection cache key  | ``set`` of package names                        | Cached results from resolving complete package sets  |
+| pkg_sets    | <Collection.cachekey>`,               |                                                 | for clients                                          |
+|             | hash of the initial package selection |                                                 |                                                      |
++-------------+---------------------------------------+-------------------------------------------------+------------------------------------------------------+
+
+These are enumerated so that they can be expired as needed by other
+plugins or other code points.
+
+.. automodule:: Bcfg2.Server.Cache