diff options
| author | Sol Jerome <solj@ices.utexas.edu> | 2010-01-24 18:47:00 +0000 |
|---|---|---|
| committer | Sol Jerome <solj@ices.utexas.edu> | 2010-01-24 18:47:00 +0000 |
| commit | 7c6c1e10f3cc62acc919123d0067a8a9720aeffa (patch) | |
| tree | a6f6cb0cc17d41e6ccbc27c2e9f986df2e37c67d /doc/unsorted | |
| parent | e3bd887d601dad1be9352768206cbb45a91a8cc7 (diff) | |
| download | bcfg2-7c6c1e10f3cc62acc919123d0067a8a9720aeffa.tar.gz bcfg2-7c6c1e10f3cc62acc919123d0067a8a9720aeffa.tar.bz2 bcfg2-7c6c1e10f3cc62acc919123d0067a8a9720aeffa.zip | |
doc: Add a lot more unsorted stuff from the wiki
Signed-off-by: Sol Jerome <solj@ices.utexas.edu>
git-svn-id: https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2@5703 ce84e21b-d406-0410-9b95-82705330c041
Diffstat (limited to 'doc/unsorted')
31 files changed, 1642 insertions, 44 deletions
diff --git a/doc/unsorted/altsrc.txt b/doc/unsorted/altsrc.txt new file mode 100644 index 000000000..248405ebc --- /dev/null +++ b/doc/unsorted/altsrc.txt @@ -0,0 +1,71 @@ +.. -*- mode: rst -*- + +.. _unsorted-altsrc: + +=========================== +Fun and Profit using altsrc +=========================== + +Altsrc is a generic, bcfg2-server-side mechanism for performing configuration entry name remapping for the purpose of data binding. + +Use Cases +========= + +* Equivalent configuration entries on different architectures with different names +* Mapping entries with the same name to different bind results in a configuration (two packages with the same name but different types) +* A single configuration entry across multiple specifications (multi-plugin, or multi-repo) + +Examples +======== + +* Consider the case of /etc/hosts on linux and /etc/inet/hosts on solaris. These files contain the same data in the same format, and should typically be synchronized, however, exist in different locations. Classically, one would need to create one entry for each in Cfg or TCheetah and perform manual synchronization. Or, you could use symlinks and pray. Altsrc is driven from the bundle side. For example: + + .. code-block:: xml + + <Bundle name='netinfo'> + <Group name='solaris'> + <ConfigFile name='/etc/inet/hosts' altsrc='/etc/hosts'/> + </Group> + <Group name='linux'> + <ConfigFile name='/etc/hosts'/> + </Group> + </Bundle> + + In this case, when a solaris host gets the 'netinfo' bundle, it will get the first !ConfigFile entry, which includes an altsrc parameter. This will cause the server to bind the entry as if it were a !ConfigFile called '/etc/hosts'. This configuration entry is still called '/etc/inet/hosts', and is installed as such. + +* On encap systems, frequently multiple packages of the same name, but of different types will exist. For example, there might be an openssl encap package, and an openssl rpm package. This can be dealt with using a bundle like: + + .. code-block:: xml + + <Bundle name='openssl'> + <Package name='openssl' altsrc='openssl-encap'/> + <Package name='openssl' altsrc='openssl-rpm'/> + </Bundle> + + This bundle will bind data for the packages "openssl-encap" and "openssl-rpm", but will be delivered to the client with both packages named "openssl" with different types. + +* Finally, consider the case where there exist complicated, but completely independent specifications for the same configuration entry but different groups of clients. The following bundle will allow the use of two different TCheetah templates /etc/firewall-rules-external and /etc/firewall-rules-internal for different clients based on their group membership. + + .. code-block:: xml + + <Bundle name='firewall'> + ... + <Group name='conduit'> + <ConfigFile name='/etc/firewall-rules' altsrc='/etc/firewall-rules-external'/> + </Group> + <Group name='internal'> + <ConfigFile name='/etc/firewall-rules' altsrc='/etc/firewall-rules-internal'/> + </Group> + </Bundle> + +* Consider the case where a variety of files can be constructed by a single template (TCheetah or TGenshi). It would be possible to copy this template into the proper location for each file, but that requires proper synchronization upon modification and knowing up front what the files will all be called. Instead, the following bundle allows the use of a single template for all proper config file instances. + + .. code-block:: xml + + <Bundle name='netconfig'> + <ConfigFile name='/etc/sysconfig/network-scripts/ifcfg-eth0' altsrc='/etc/ifcfg-template'/> + <ConfigFile name='/etc/sysconfig/network-scripts/ifcfg-eth1' altsrc='/etc/ifcfg-template'/> + <ConfigFile name='/etc/sysconfig/network-scripts/ifcfg-eth2' altsrc='/etc/ifcfg-template'/> + </Bundle> + + altsrc can be used as a parameter for any entry type, and can be used in any structure, including Bundler, Base, and SGenshi. diff --git a/doc/unsorted/boundentry.txt b/doc/unsorted/boundentry.txt new file mode 100644 index 000000000..6dfa4b9bd --- /dev/null +++ b/doc/unsorted/boundentry.txt @@ -0,0 +1,15 @@ +.. -*- mode: rst -*- + +.. _unsorted-boundentry: + +============= +Bound Entries +============= + +This feature is a mechanism to specify a full entry at once from a bundle. Traditionally, entries are defined in two stages. First, an abstract entry is defined in a bundle. This entry includes a type (the XML tag) and a name attribute. Then this entry is bound for a client, providing the appropriate instance of that entry for the client. Specifying a bound entry short-circuits this process; the only second stage processing on Bound entries is to remove the "Bound" prefix from the element tag. The use of a bound entry allows the single stage definition of a complete entry. Bound entries can be used for any type. + +.. code-block:: xml + + <Bundle name='ntp'> + <BoundPackage name='ntp' type='deb' version='1:4.2.4p4+dfsg-3ubuntu2.1'/> + </Bundle> diff --git a/doc/unsorted/configurationentries.txt b/doc/unsorted/configurationentries.txt new file mode 100644 index 000000000..0234dc0f0 --- /dev/null +++ b/doc/unsorted/configurationentries.txt @@ -0,0 +1,32 @@ +.. -*- mode: rst -*- + +.. _unsorted-configurationentries: + +===================== +Configuration Entries +===================== + +This page describes the names and semantics of each of the configuration entries used by Bcfg2. + +Common non-POSIX entries +======================== + +|| '''!TagName''' || '''Description''' || '''Attributes''' || +|| Action || Command || name, command, when, timing || +|| Package || Software Packages || name, type, version, url || +|| !PostInstall || !PostInstall command || name || +|| Service || System Services || name, type, status, reload || + +Bcfg2 >= 1.0.0: New unified POSIX entries +========================================= + +In 1.0.0, there are new unified POSIX Path entries which prevent inconsistent configuration specifications of multiple entries for a given path. The following table describes the various types available for new `Path` entries. + +|| '''Type''' || '''Replacement/New''' || '''Description''' || '''Attributes''' || +|| device || New || Create block, character, and fifo devices || name, dev_type (block, char, fifo), owner, group, major/minor (for block or char devices) || +|| directory || Replaces Directory entries || Directories || name, owner, group, perms, prune || +|| file || Replaces !ConfigFile entries || Configuration File || name, owner, group, perms, encoding, empty || +|| hardlink || New || Create hardlinks || name, to || +|| nonexistent (not yet implemented) || New || Specify a path that should not exist || name || +|| permissions || Replaces Permissions entries || Permissions of POSIX entities || name, owner, group, perms || +|| symlink || Replaces !SymLink entries || !SymLinks || name, to || diff --git a/doc/unsorted/contribute.txt b/doc/unsorted/contribute.txt new file mode 100644 index 000000000..e43ed4ad3 --- /dev/null +++ b/doc/unsorted/contribute.txt @@ -0,0 +1,60 @@ +.. -*- mode: rst -*- + +.. _unsorted-contribute: + +===================== +Contributing to Bcfg2 +===================== + +There are several ways users can contribute to the Bcfg2 project. + +* Developing code +* Testing prereleases +* Adding to the common repository +* Improving the wiki + +Development +=========== + +Send patches to the [wiki:MailingList bcfg mailing list] or create a trac [https://trac.mcs.anl.gov/projects/bcfg2/newticket ticket] with the patch included. In order to submit a ticket via the trac system, you will need to create a session by clicking on the [https://trac.mcs.anl.gov/projects/bcfg2/prefs Preferences] link and filling out/saving changes to the form. In order to be considered for mainline inclusion, patches need to be BSD licensed. The most convenient way to prepare patches is by using svn diff inside of a source tree checked out of subversion. The source tree can be checked out by running:: + + svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2 + +Users wishing to contribute on a regular basis can apply for direct subversion access. Mail the mailing list for details. + +Several resources for developers exist in the wiki: + +* [wiki:DevelopmentTips] +* [wiki:Development/WritingPlugins Writing Bcfg2 Server Plugins] +* [wiki:Architecture] +* [wiki:WritingClientToolDrivers] +* [wiki:Bcfg2SubversionHowto] +* [wiki:LearningPython] +* [wiki:UsingRcache] + +Bcfg2 is the result of a lot of work by many different people. They are listed on the [wiki:Contributors contributors page.] + +Feel free to drop in during a [wiki:CodeSprintIdeas code sprint] if you would like to help out with some easier problems. + +Testing Prereleases +=================== + +Before each release, several prereleases will be tagged. It is helpful to have users test these releases (when feasible) because it is hard to replicate the full range of potential reconfiguration situations; between different operating systems, system management tools, and configuration specification variation, there can be large differences between sites. + +See the [wiki:TrackingDevelopmentTrunk] page for a better view of changes in the prereleases. + +Adding to the Common Repository +=============================== + +The Bcfg2 common repository is a set of portable examples that new repositories can be based on. This repo has several parts. The first is a series of group definitions that describe a wide range of client systems. The second is a set of portable bundles that have been ported to use these groups. This makes these bundles transparently portable across new client architectures. + +This approach has several benefits to users + +* Configuration specification can be shared across sites where appropriate +* This common configuration specification can be reused, allowing sites to migrate to new systems that other sites have already ported the common repository to +* Setup of new systems becomes a lot easier. + +Improving the wiki +================== + +Mail the [wiki:MailingList mailing list] for an account on the wiki. diff --git a/doc/unsorted/contributors.txt b/doc/unsorted/contributors.txt new file mode 100644 index 000000000..b9dd7cd77 --- /dev/null +++ b/doc/unsorted/contributors.txt @@ -0,0 +1,32 @@ +.. -*- mode: rst -*- + +.. _unsorted-contributors: + +============ +Contributors +============ + +A large number of people have contributed to Bcfg2. + +* Narayan Desai - wrote and designed much of the bcfg1/2 +* Andrew Lusk - early development and hardening +* Brian Pellin - early development and hardening +* Rick Bradshaw +* Cory Lueninghoener - Bugfixes and change management code +* Scott Matott +* Ti Leggett - ebuild packaging and bugfixes, RPM packaging +* Joey Hagedorn - Mac OS X support in bcfg1, launchd support in Bcfg2, and both versions of the reporting system +* Zach Lowry - Solaris support and general hardening +* Jos Catnook - Bugfixes +* Jason Pepas - Package manager source generation scripts +* Patrick Ruckstuhl - Templating bugfixes +* Chris Vuletich - statistics annotations and client features +* Ken Raffenetti - Hostbase +* Sami Haahtinen - Debian Packaging +* Danny Clark - Encap Packaging +* Michael Jinks - Gentoo support +* David Dahl - Hostbase +* Pedro Flores - Reporting system design help +* David Strauss - CentOS, RHEL, Yum, and Bazaar VCS support + +The entire MCS systems team has provided invaluable help in the design process and refinement of the user interface. In particular, Gene Rackow and Sandra Bittner have provided great assistance throughout this project. Philip Steinbachs provided detailed feedback as an early external user. diff --git a/doc/unsorted/converging_rhel5.txt b/doc/unsorted/converging_rhel5.txt new file mode 100644 index 000000000..e41c46f8e --- /dev/null +++ b/doc/unsorted/converging_rhel5.txt @@ -0,0 +1,116 @@ +.. -*- mode: rst -*- + +.. _unsorted-converging_rhel5: + +====================================== +Converging on Verification with RHEL 5 +====================================== + +Running verification +==================== + +To get complete verification status, run:: + + bcfg2 -vqned + +Unmanaged entries +================= + +* Package (top-level) + + #. Enable the "Packages" plugin in {{{/etc/bcfg2.conf}}}, and configure the Yum repositories in {{{/var/lib/bcfg2/Packages/config.xml}}}. + #. If a package is unwanted, remove it:: + + sudo yum remove PACKAGE + + #. Otherwise, add {{{<Package name="PACKAGE" />}}} to the Base or Bundler configuration. + +* Package (dependency) + + #. Ensure the Yum repository sources configured in {{{/var/lib/bcfg2/Packages/config.xml}}} are correct. + #. Ensure the Yum repositories themselves are up-to-date with the main package and dependencies. + #. Rebuild the Packages plugin cache:: + + bcfg2-admin xcmd Packages.Refresh + +* Service + + #. Add {{{<Service name="SERVICE" />}}} to the Base or Bundler configuration. + #. Add {{{<Service name="SERVICE" status="on" type="chkconfig" />}}} to {{{/var/lib/bcfg2/Rules/services.xml}}}. + +Incorrect entries +================= + +For a "Package" +--------------- + +* Failed RPM verification + + #. Run {{{rpm -V PACKAGE}}} + #. Add configuration files (the ones with "c" next to them in the verification output) to {{{/var/lib/bcfg2/Cfg/}}}. + + * For example, {{{/etc/motd}}} to {{{/var/lib/bcfg2/Cfg/etc/motd/motd}}}. Yes, there is an extra directory level named after the file. + + #. Specify configuration files as {{{<ConfigFile name='PATH' />}}} in the Base or Bundler configuration. + #. Add directories to {{{/var/lib/bcfg2/Rules/directories.xml}}}. For example: + + .. code-block:: xml + + <Rules priority="0"> + <Directory name="/etc/cron.hourly" group="root" owner="root" perms="0700" /> + <Directory name="/etc/cron.daily" group="root" owner="root" perms="0700" /> + </Rules> + +* Multiple instances + + * Option A: Explicitly list the instances + + #. Drop the {{{<Package />}}} from the Base or Bundler configuration. + #. Add an explicit {{{<BoundPackage>}}} and {{{<Instance />}}} configuration to a new Bundle, like the following: + + .. code-block:: xml + + <Bundle name='keys'> + <!-- GPG keys --> + <BoundPackage name="gpg-pubkey" type="yum"> + <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL" version="217521f6" release="45e8a532"/> + <Instance simplefile="/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release" version="37017186" release="45761324"/> + </BoundPackage> + </Bundle> + + #. Add the bundle to the applicable groups in {{{/var/lib/bcfg2/Metadata/groups.xml}}}. + + * Option B: Disable verification of the package + + #. Add {{{pkg_checks="false"}}} to the {{{<Package />}}} tag. + +For a "ConfigFile" +------------------- + + * Unclear verification problem (no details from BCFG2) + + 1. Run {{{bcfg2 -vqI}}} to see detailed verification issues (but deny any suggested actions). + + * Permissions mismatch + + 1. Create an {{{info.xml}}} file in the same directory as the configuration file. Example: + + .. code-block:: xml + + <FileInfo> + <Group name='webserver'> + <Info owner='root' group='root' perms='0652'/> + </Group> + <Info owner='root' group='sys' perms='0651'/> + </FileInfo> + +Other troubleshooting tools +=========================== + + * Generate the physical configuration from the server side:: + + bcfg2-info buildfile /test test.example.com + + * Generate the physical configuration from the client side:: + + bcfg2 -vqn -c/root/bcfg2-physical.xml diff --git a/doc/unsorted/developing_for_bcfg2.txt b/doc/unsorted/developing_for_bcfg2.txt new file mode 100644 index 000000000..39582b520 --- /dev/null +++ b/doc/unsorted/developing_for_bcfg2.txt @@ -0,0 +1,111 @@ +.. -*- mode: rst -*- + +.. _unsorted-developing_for_bcfg2: + +==================== +Developing For Bcfg2 +==================== + +While the Bcfg2 server provides a good interface for representing +general system configurations, its plugin interface offers the +ability to implement configuration interfaces and representation +tailored to problems encountered by a particular site. This +chapter describes what plugins are good for, what they can do, and +how to implement them. + +Bcfg2 Plugins +============= + +Bcfg2 plugins are loadable python modules that the Bcfg2 server +loads at initialization time. These plugins can contribute to +the functions already offered by the Bcfg2 server or can extend +its functionality. In general, plugins will provide some portion +of the configuration for clients, with a data representation +that is tuned for a set of common tasks. Much of the core +functionality of Bcfg2 is implemented by several plugins, +however, they are not special in any way; new plugins could +easily supplant one or all of them. + +The following table describes the various functions of bcfg2 plugins. + +|| '' '''Name''' '' || '' '''Description''' '' || +|| Probes || Plugins can issue commands to collect client-side state (like hardware inventory) to include in client configurations || +|| !ConfigurationEntry List || Plugins can construct a list of per-client configuration entry lists to include in client configurations. || +|| !ConfigurationEntry contents || Literal values for configuration entries. || +|| XML-RPC functions || Plugins can export function calls that expose internal functions. || + +Writing Bcfg2 Plugins +===================== + +Bcfg2 plugins are python classes that subclass from +Bcfg2.Server.Plugin.Plugin. Several plugin-specific values must +be set in the new plugin. These values dictate how the new +plugin will behave with respect to the above four functions. +The following table describes all important member fields. + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Format''' '' || +|| __name__ || The name of the plugin || string || +|| __version__ || The plugin version (generally tied to revctl keyword expansion). || string || +|| __author__ || The plugin author. || string || +|| __rmi__ || Set of functions to be exposed as XML-RPC functions || List of function names (strings) || +|| Entries || Multidimentional dictionary of keys that point to the function [[BR]] used to bind literal contents for a given configuration entity. || Dictionary of !ConfigurationEntityType, Name keys and function reference values || +|| !BuildStructures || Function that returns a list of the structures for a given client || Member function || +|| !GetProbes || Function that returns a list of probes that a given client should execute || Member function || +|| !ReceiveData || Function that accepts the probe results for a given client. || Member function || + +Example Plugin +============== + +.. code-block:: python + + import Bcfg2.Server.Plugin + class MyPlugin(Bcfg2.Server.Plugin.Plugin): + '''An example plugin''' + # All plugins need to subclass Bcfg2.Server.Plugin.Plugin + __name__ = 'MyPlugin' + __version__ = '1' + __author__ = 'me@me.com' + __rmi__ = ['myfunction'] + # myfunction is not available remotely as MyPlugin.myfunction + + def __init__(self, core, datastore): + Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore) + self.Entries = {'ConfigFile':{'/etc/foo.conf': self.buildFoo}} + + def myfunction(self): + '''function for xmlrpc rmi call''' + #do something + return True + + def buildFoo(self, entry, metadata): + '''Bind per-client information into entry based on metadata''' + entry.attrib.update({'owner':'root', 'group':'root', 'perms':'644'}) + entry.text = '''contents of foo.conf''' + +Example Connector +================= + +.. code-block:: python + + import Bcfg2.Server.Plugin + + class Foo(Bcfg2.Server.Plugin.Plugin, + Bcfg2.Server.Plugin.Connector): + '''The Foo plugin is here to illustrate a barebones connector''' + name = 'Foo' + version = '$Revision: $' + experimental = True + + def __init__(self, core, datastore): + Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore) + Bcfg2.Server.Plugin.Connector.__init__(self) + self.store = XMLFileBacked(self.data, core.fam) + + def get_additional_data(self, metadata): + mydata = {} + for data in self.store.entries['foo.xml'].data.get("foo", []): + mydata[data] = "bar" + return dict([('mydata', mydata)]) + + def get_additional_groups(self, meta): + return self.cgroups.get(meta.hostname, list()) diff --git a/doc/unsorted/development_tips.txt b/doc/unsorted/development_tips.txt new file mode 100644 index 000000000..29924e645 --- /dev/null +++ b/doc/unsorted/development_tips.txt @@ -0,0 +1,20 @@ +.. -*- mode: rst -*- + +.. _unsorted-development_tips: + +========================== +Tips for Bcfg2 Development +========================== + +#. Focus on either the client or server code. This focuses the development process down the the precise pieces of code that matter for the task at hand. + + * If you are developing a client driver, then write up a small configuration specification that includes the needed characteristics. + * If you are working on the server, run bcfg2-info and use to assess the code + +#. Use the python interpreter. One of python's most appealing features is interactive use of the interpreter. + + * If you are developing for the client-side, run "python -i /usr/sbin/bcfg2" with the appropriate bcfg2 options. This will cause the python interpreter to continue running, leaving all variables intact. This can be used to examine data state in a convenient fashion. + * If you are developing for the server side, use bcfg2-info and the "debug" option. This will leave you at a python interpreter prompt, with the server core loaded in the variable "bcore". + +#. Use pylint obsessively. It raises a lot of style-related warnings which can be ignored, but most all of the errors are legitimate. +#. If you are doing anything with Regular Expressions, [http://kodos.sourceforge.net/ Kodos - The Python Regular Expression Debugger] and [http://re-try.appspot.com/ re-try] are your friends. diff --git a/doc/unsorted/development_writing_plugins.txt b/doc/unsorted/development_writing_plugins.txt new file mode 100644 index 000000000..89b1af28a --- /dev/null +++ b/doc/unsorted/development_writing_plugins.txt @@ -0,0 +1,77 @@ +.. -*- mode: rst -*- + +.. _unsorted-development_writing_plugins: + +=============== +Writing Plugins +=============== + +Server Plugin Types +=================== + +Generator +--------- + +Generator plugins contribute to literal client configurations + +Structure +--------- + +Structure Plugins contribute to abstract client configurations + +Metadata +-------- + +Signal metadata capabilities + +Connector +--------- + +Connector Plugins augment client metadata instances + +Probing +------- + +Signal probe capability + +Statistics +---------- + +Signal statistics handling capability + +Decision +-------- + +Signal decision handling capability + +Version +------- + +Interact with various version control systems + +Writing Server Plugins +====================== + +Metadata +-------- + +If you would like to define your own Metadata plugin (to extend/change functionality of the existing Metadata plugin), here are the steps to do so. We will call our new plugin `MyMetadata`. + +#. Add MyMetadata.py + + .. code-block:: python + + __revision__ = '$Revision$' + + import Bcfg2.Server.Plugins.Metadata + + class MyMetadata(Bcfg2.Server.Plugins.Metadata.Metadata): + '''This class contains data for bcfg2 server metadata''' + __version__ = '$Id$' + __author__ = 'bcfg-dev@mcs.anl.gov' + + def __init__(self, core, datastore, watch_clients=True): + Bcfg2.Server.Plugins.Metadata.Metadata.__init__(self, core, datastore, watch_clients) + +#. Add MyMetadata to `src/lib/Server/Plugins/__init__.py` +#. Replace Metadata with MyMetadata in the plugins line of bcfg2.conf diff --git a/doc/unsorted/doc_architecture.txt b/doc/unsorted/doc_architecture.txt new file mode 100644 index 000000000..0aca0ef10 --- /dev/null +++ b/doc/unsorted/doc_architecture.txt @@ -0,0 +1,128 @@ +.. -*- mode: rst -*- + +.. _unsorted-doc_architecture: + +================== +Bcfg2 Architecture +================== + +Bcfg2 is based on a client-server architecture. The client is responsible for interpreting (but not processing) the configuration served by the server. This configuration is literal, so no local process is required. After completion of the configuration process, the client uploads a set of statistics to the server. This section will describe the goals and then the architecture motivated by it. + +Goals +===== + +* '''Model configurations using declarative semantics.'''[[BR]] + Declarative semantics maximize the utility of configuration management tools; they provide the most flexibility for the tool to determine the right course of action in any given situation. This means that users can focus on the task of describing the desired configuration, while leaving the task of transitioning clients states to the tool. +* '''Configuration descriptions should be comprehensive.'''[[BR]] + This means that configurations served to the client should be sufficient to reproduce all desired functionality. This assumption allows the use of heuristics to detect extra configuration, aiding in reliable, comprehensive configuration definitions. +* '''Provide a flexible approach to user interactions.'''[[BR]] + Most configuration management systems take a rigid approach to user interactions; that is, either the client system is always correct, or the central system is. This means that users are forced into an overly proscribed model where the system asserts where correct data is. Configuration data modification is frequently undertaken on both the configuration server and clients. Hence, the existence of a single canonical data location can easily pose a problem during normal tool use. Bcfg2 takes a different approach. + +The default assumption is that data on the server is correct, however, the client has option to run in another mode where local changes are catalogued for server-side integration. If the Bcfg2 client is run in dry run mode, it can help to reconcile differences between current client state and the configuration described on the server. +The Bcfg2 client also searches for extra configuration; that is, configuration that is not specified by the configuration description. When extra configuration is found, either configuration has been removed from the configuration description on the server, or manual configuration has occurred on the client. Options related to two-way verification and removal are useful for configuration reconciliation when interactive access is used. + +* Plugins and administrative applications. +* Incremental operations. + +The Bcfg2 Client +================ + +The Bcfg2 client performs all client configuration or reconfiguration operations. It renders a declarative configuration specification, provided by the Bcfg2 server, into a set of configuration operations which will, if executed, attempt to change the client's state into that described by the configuration specification. Conceptually, the Bcfg2 client serves to isolate the Bcfg2 server and specification from the imperative operations required to implement configuration changes. + +This isolation allows declarative specifications to be manipulated symbolically on the server, without needing to understand the properties of the underlying system tools. In this way, the Bcfg2 client acts as a sort of expert system that "knows" how to implement declarative configuration changes. + +The operation of the Bcfg2 client is intended to be as simple as possible. The normal configuration process consists of four main steps: + +* Probe Execution + During the probe execution stage, the client connects to the server and downloads a series of probes to execute. These probes reveal local facts to the Bcfg2 server. For example, a probe could discover the type of video card in a system. The Bcfg2 client returns this data to the server, where it can influence the client configuration generation process. +* Configuration Download and Inventory + The Bcfg2 client now downloads a configuration specification from the Bcfg2 server. The configuration describes the complete target state of the machine. That is, all aspects of client configuration should be represented in this specification. For example, all software packages and services should be represented in the configuration specification. The client now performs a local system inventory. This process consists of verifying each entry present in the configuration specification. After this check is completed, heuristic checks for configuration not included in the configuration specification. We refer to this inventory process as 2-way validation, as first we verify that the client contains all configuration that is included in the specification, then we check if the client has any extra configuration that isn't present. This provides a fairly rigorous notion of client configuration congruence. Once the 2-way verification process has been performed, the client has built a list of all configuration entries that are out of spec. This list has two parts: specified configuration that is incorrect (or missing) and unspecified configuration that should be removed. +* Configuration Update + The client now attempts to update its configuration to match the specification. Depending on options, changes may not (or only partially) be performed. First, if extra configuration correction is enabled, extra configuration can be removed. Then the remaining changes are processed. The Bcfg2 client loops while progress is made in the correction of these incorrect configuration entries. This loop results in the client being able to accomplish all it will be able to during one execution. Once all entries are fixed, or no progress is being made, the loop terminates.Once all configuration changes that can be performed have been, bundle dependencies are handled. Bundle groupings result in two different behaviors. Contained entries are assumed to be inter-dependant. To address this, the client re-verifies each entry in any bundle containing an updates configuration entry. Also, services contained in modified bundles are restarted. +* Statistics Upload + Once the reconfiguration process has concluded, the client reports information back to the server about the actions it performed during the reconfiguration process. Statistics function as a detailed return code from the client. The server stores statistics information. Information included in this statistics update includes (but is not limited to): + + * Overall client status (clean/dirty) + * List of modified configuration entries + * List of uncorrectable configuration entries + +Architecture Abstraction +------------------------ + +The Bcfg2 client internally supports the administrative tools available on different architectures. For example, rpm and apt-get are both supported, allowing operation of Debian, Redhat, SUSE, and Mandriva systems. The client toolset is specified in the configuration specification. The client merely includes a series of libraries which describe how to interact with the system tools on a particular platform. + +Three of the libraries exist. There is a base set of functions, which contain definitions describing how to perform POSIX operations. Support for configuration files, directories, and symlinks are included here. Two other libraries subclass this one, providing support for Debian and rpm-based systems. + +The Debian toolset includes support for apt-get and update-rc.d. These tools provide the ability to install and remove packages, and to install and remove services. + +The Redhat toolset includes support for rpm and chkconfig. Any other platform that uses these tools can also use this toolset. Hence, all of the other familiar rpm-based distributions can use this toolset without issue. + +Other platforms can easily use the POSIX toolset, ignoring support for packages or services. Alternatively, adding support for new toolsets isn't difficult. Each toolset consists of about 125 lines of python code. + + +The Bcfg2 Server +================ + +The Bcfg2 server is responsible for taking a network description and turning it into a series of configuration specifications for particular clients. It also manages probed data and tracks statistics for clients. + +The Bcfg2 server takes information from two sources when generating client configuration specifications. The first is a pool of metadata that describes clients as members of an aspect-based classing system. That is, clients are defined in terms of aspects of their behavior. The other is a file system repository that contains mappings from metadata to literal configuration. These are combined to form the literal configuration specifications for clients. + + +The Configuration Specification Construction Process +---------------------------------------------------- + +As we described in the previous section, the client connects to the server to request a configuration specification. The server uses the client's metadata and the file system repository to build a specification that is tailored for the client. This process consists of the following steps: + +* Metadata Lookup + The server uses the client's IP address to initiate the metadata lookup. This initial metadata consists of a (profile, image) tuple. If the client already has metadata registered, then it is used. If not, then default values are used and stored for future use. This metadata tuple is expanded using some profile and class definitions also included in the metadata. The end result of this process is metadata consisting of hostname, profile, image, a list of classes, a list of attributes and a list of bundles. +* Abstract Configuration Construction + Once the server has the client metadata, it is used to create an abstract configuration. An abstract configuration contains all of the configuration elements that will exist in the final specification without any specifics. All entries will be typed (ie the tagname will be one of Package, !ConfigurationFile, Service, !Symlink, or Directory) and will include a name. These configuration entries are grouped into bundles, which document installation time interdependencies. +* Configuration Binding + The abstract configuration determines the structure of the client configuration, however, it doesn't contain literal configuration information. After the abstract configuration is created, each configuration entry must be bound to a client-specific value. The Bcfg2 server uses plugins to provide these client-specific bindings. The Bcfg2 server core contains a dispatch table that describes which plugins can handle requests of a particular type. The responsible plugin is located for each entry. It is called, passing in the configuration entry and the client's metadata. The behavior of plugins is explicitly undefined, so as to allow maximum flexibility. The behaviours of the stock plugins are documented elsewhere in this manual. Once this binding process is completed, the server has a literal, client-specific configuration specification. This specification is complete and comprehensive; the client doesn't need to process it at all in order to use it. It also represents the totality of the configuration specified for the client. + +The Literal Configuration Specification +======================================= + +Literal configuration specifications are served to clients by the Bcfg2 server. This is a differentiating factor for Bcfg2; all other major configuration management systems use a non-literal configuration specification. That is, the clients receive a symbolic configuration that they process to implement target states. We took the literal approach for a few reasons: + +* A small list of configuration element types can be defined, each of which can have a set of defined semantics. This allows the server to have a well-formed model of client-side operations. Without a static lexicon with defined semantics, this isn't possible. This allows the server, for example, to record the update of a package as a coherent event. +* Literal configurations do not require client-side processing. Removing client-side processing reduces the critical footprint of the tool. That is, the Bcfg2 client (and the tools it calls) need to be functional, but the rest of the system can be in any state. Yet, the client will receive a correct configuration. +* Having static, defined element semantics also requires that all operations be defined and implemented in advance. The implementation can maximize reliability and robustness. In more ad-hoc setups, these operations aren't necessarily safely implemented. + +The Structure of Specifications +------------------------------- + +Configuration specifications contain some number of clauses. Two types of clauses exist. Bundles are groups of inter-dependent configuration entities. The purpose of bundles is to encode installation-time dependencies such that all new configuration is properly activated during reconfiguration operations. T hat is, if a daemon configuration file is changed, its daemon should be restarted. Another example of bundle usage is the reconfiguration of a software package. If a package contains a default configuration file, but it gets overwritten by an environment-specific one, then that updated configuration file should survive package upgrade. The purpose of bundles is to describe services, or reconfigured software packages. Independent clauses contains groups of configuration entities that aren't related in any way. This provides a convenient mechanism that can be used for bulk installations of software. + +Each of these clauses contains some number of configuration entities. Five types of configuration entities exist: !ConfigurationFile, Package, !SymLink, Directory, and Service. Each of these correspond to the obvious system item. Configuration specifications can get quite large; many systems have specifications that top one megabyte in size. An example of one is included in an appendix. These configurations can be written by hand, or generated by the server. The easiest way to start using Bcfg2 is to write small static configurations for clients. Once configurations get larger, this process gets unwieldy; at this point, using the server makes more sense. + +Design Considerations +===================== + +This section will discuss several aspects of the design of bcfg2, and the particular use cases that motivated them. Initially, this will consist of a discussion of the system metadata, and the intended usage model for package indices as well. + +System Metadata +--------------- + +Bcfg2 system metadata describes the underlying patterns in system configurations. It describes commonalities and differences between these specifications in a rigorous way. The groups used by bcfg2's metadata are responsible for differentiating clients from one another, and building collections of allocatable configuration. + +The Bcfg2 metadata system has been designed with several high-level goals in mind. Flexibility and precision are paramount concerns; no configuration should be undescribable using the constructs present in the bcfg2 repository. We have found (generally the hard way) that any assumptions about the inherent simplicity of configuration patterns tend to be wrong, so obscenely complex configurations must be representable, even if these requirements seem illogical during the implementation. + +In particular, we wanted to streamline several operations that commonly occurred in our environment. + +* Copying one node's profile to another node. + In many environments, many nodes are instances of a common configuration specification. They all have similar roles and software. In our environment, desktop machines were the best example of this. Other than strictly per-host configuration like SSH keys, all desktop machines use a common configuration specification. This trivializes the process of creating a new desktop machine. +* Creating a specialized version of a currently existing profile. + In environments with highly varied configurations, departmental infrastructure being a good example, "another machine like X but with extra software" is a common requirement. For this reason, it must be trivially possible to inherit most of a configuration specification from some more generic source, while being able to describe overriding aspects in a convenient fashion. +* 3. Compose several pre-existing configuration aspects to create a new profile. + The ability to compose configuration aspects allows the easy creation of new profiles based on a series of predefined set of configuration specification fragments. The end result is more agility in environments where change is the norm. + + In order for a classing system to be comprehensive, it must be usable in complex ways. The Bcfg2 metadata system has constructs that map cleanly to first-order logic. This implies that any complex configuration pattern can be represented (at all) by the metadata, as first-order logic is provably comprehensive. (There is a discussion later in the document describing the metadata system in detail, and showing how it corresponds to first-order logic) + +These use cases motivate several of the design decisions that we made: +* There must be a many to one correspondence between clients and groups. Membership in a given profile group must imbue a client with all of its configuration properties. + +Package Management +------------------ + +The interface provided in the bcfg2 repository for package specification was designed with automation in mind. The goal was to support an append only interface to the repository, so that users do not need to continuously re-write already existing bits of specification. diff --git a/doc/unsorted/dynamic_groups.txt b/doc/unsorted/dynamic_groups.txt new file mode 100644 index 000000000..87e858fec --- /dev/null +++ b/doc/unsorted/dynamic_groups.txt @@ -0,0 +1,18 @@ +.. -*- mode: rst -*- + +.. _unsorted-dynamic_groups: + +============== +Dynamic Groups +============== + +Bcfg2 supports the use of dynamic groups. These groups are not included in a client's profile group, but instead are derived from the results of probes executed on the client. These dynamic groups need not already exist in Metadata/groups.xml. If a dynamic group is defined in Metadata/groups.xml, clients that include this group will also get all included groups and bundles. + +Setting up dynamic groups +========================= + +In order to define a dynamic group, setup a probe that outputs the text based on system properties:: + + group:groupname + +This output is processed by the bcfg2 server, and results in dynamic group membership in groupname for the client. See the [wiki:Plugins/Probes Probes] page for a more thorough description of probes. diff --git a/doc/unsorted/emacs_snippet.txt b/doc/unsorted/emacs_snippet.txt new file mode 100644 index 000000000..471a3b244 --- /dev/null +++ b/doc/unsorted/emacs_snippet.txt @@ -0,0 +1,53 @@ +.. -*- mode: rst -*- + +.. _unsorted-emacs_snippet: + +======================================= +Using Bcfg2 with Emacs + YASnippet mode +======================================= + +This page describes using emacs with YASnippet mode with a set of snippets that allow quick composition of bundles and base files. More snippets are under development. + +#. Download YASnippet from http://code.google.com/p/yasnippet/ +#. Install it into your emacs load path (typically ~/.emacs.d/site-lisp) +#. Add YASnippet initialization to your .emacs (remember to re-byte-compile it if needed) + + .. code-block:: cl + + (require 'yasnippet-bundle) + + ;;; Bcfg2 snippet + + (yas/define-snippets 'sgml-mode + '( + ("<Bundle" "<Bundle name='${1:bundlename}' version='2.0'> + $0 + </Bundle>" nil) + ("<Base" "<Base> + $0 + </Base>" nil) + ("<Group" "<Group name='${1:groupname}> + $0 + </Group>" nil) + ("<Config" "<ConfigFile name='${1:filename}'/> + $0" nil) + ("<Service" "<Service name='${1:svcname}'/> + $0" nil) + ("<Package" "<Package name='${1:packagename}'/> + $0" nil) + ("<Action" "<Action name='${1:name}'/> + $0" nil) + ("<Directory" "<Directory name='${1:name}'/> + $0" nil) + ("<SymLink" "<SymLink name='${1:name}'/> + $0" nil) + ("<Permissions" "<Permissions name='${1:name}'/> + $0" nil) + ) + ) + +#. One quick M-x eval-current-buffer, and this code is enabled + +Each of these snippets activates on the opening element, ie <Bundle. After this string is entered, but before entering a space, press <TAB>, and the snippet will be expanded. The template will be inserted into the text with a set of input prompts, which default to overwrite mode and can be tabbed through. + +The code above only works for bundles and base, but will be expanded to support other xml files as well. diff --git a/doc/unsorted/gentoo.txt b/doc/unsorted/gentoo.txt new file mode 100644 index 000000000..2375fcb17 --- /dev/null +++ b/doc/unsorted/gentoo.txt @@ -0,0 +1,128 @@ +.. -*- mode: rst -*- + +.. _unsorted-gentoo: + +====== +Gentoo +====== + +''This document tries to lay out anything Gentoo-specific that you need to know in order to use bcfg2. Mostly that has to do with getting it to cooperate with the various pieces of Portage. Services, all things POSIX, and just about anything else that bcfg2 does will work the same on Gentoo as on any other distribution. bcfg2 is new on Gentoo; please let the list know if you find errors or omissions.'' + +Installing bcfg2 +================ + +Early in July 2008, bcfg2 was added to the Gentoo portage tree. So far it's only keyworded for ~x86, but we hope to see it soon in the amd64 and x64-solaris ports. If you're using Gentoo on some other architecture, it should still work provided that you have a reasonably up to date Python; try adding `app-admin/bcfg2 ~*` to your `/etc/portage/package.keywords` file. + +If you don’t use emerge to install bcfg2, you’ll want to make sure you have all the prerequisites installed first. For a server, you’ll need: + +* `app-admin/gamin` or `app-admin/fam`; +* `dev-python/pyopenssl`; +* `dev-python/lxml` + +Clients will need at least: + +* `app-portage/gentoolkit`; +* `dev-python/lxml` or `dev-python/elementtree` (if you’re running python-2.4 or below); + +Package Repository +================== + +You’ll need (to make) at least one archive of binary packages. The +Portage driver calls `emerge` with the `–getbinpkgonly` option. See +:manpage:`make.conf(5)` and :manpage:`emerge(1)` manpages, specifically the `PORTAGE_BINHOST` environment variable. + +Time Saver: quickpkg +-------------------- + +If you have a standing Gentoo machine that you want to preserve or propagate, you can generate a complete package archive based on the present state of the system by using the quickpkg utility. For example: + +.. code-block:: sh + + for pkg in `equery -q l` ; do quickpkg "=$pkg" ; done + +…will leave you with a complete archive of all the packages on your system in `/usr/portage/packages/All`, which you can then move to your ftp server. + +Cataloging Packages In Your Repository +-------------------------------------- + +Once you have a set of packages, you’ll need to create a catalog for them in /var/lib/bcfg2/Pkgmgr. Here’s a template: + +.. code-block:: xml + + <PackageList uri='' type='portage' priority=''> + <Group name=''> + <Package name='' version=''/> + </Group> + </PackageList> + +…and a partially filled-out example, for our local Gentoo/VMware build: + +.. code-block:: xml + + <PackageList uri='ftp://filthy.uchicago.edu/200701-vmware/' type='portage' priority='0'> + <Group name='gentoo-200701-vmware'> + <Package name='app-admin/bcfg2' version='0.9.1_pre1'/> + [...] + <Package name='x11-wm/twm' version='1.0.1'/> + </Group> + </PackageList> + +The `<Group>` name (in our example, “gentoo-200701-vmware”) should be included by any host which will draw its packages from this list. Our collection of packages for this class of machines is at the listed URI, and we only have one collection of packages for this batch of machines so in our case the `priority` doesn’t really matter, we’ve set it to `0`. + +Notice that package name fields are in `CAT/TITLE` format. + +Here’s a hack which will generate a list of Package lines from a system’s database of installed packages, especially useful in conjunction with the `quickpkg` example above: + +.. code-block:: sh + + #!/bin/bash + for pkg in `equery -q l` ; do + title=`echo $pkg | sed -e 's/\(.*\)-\([0-9].*\)/\1/'` + version=`echo $pkg | sed -e 's/\(.*\)-\([0-9].*\)/\2/'` + echo " <Package name='${title}' version='${version}'/>" + done + +Configuring Client Machines +=========================== + +Set up `/etc/bcfg2.conf` the way you would for any other bcfg2 client. + +In `make.conf`, set `PORTAGE_BINHOST` to point to the URI of your package repository. You may want to create versions of `make.conf` for each package repository you maintain, with appropriate `PORTAGE_BINHOST` URI's in each, and associated with that package archive's group under `Cfg` -- for example, we have `Cfg/etc/make.conf/make.conf.G99_gentoo-200701-vmware`. If a client host switches groups, and the new group needs a different set of packages, everything should just fall into place. + +Pitfalls +======== + +Package Verification Issues +--------------------------- + +As of this writing (2007/01/31), we’re aware of a number of packages +marked stable in the Gentoo x86 tree which, for one reason or another, +consistently fail to verify cleanly under `equery check`. In some cases +(pam, openldap), files which don’t (ever) exist on the system are +nonetheless recorded in the package database; in some (python, bcfg2, +ahem), whole classes of files (.pyc and .pyo files) consistently fail +their md5sum checks; and in others, the problem appears to be a +discrepancy in the way that symlinks are created vs. the way they’re +recorded in the database. For example, in the OpenSSH package, +/usr/bin/slogin is a symlink to ./ssh, but equery expects it to point to +an unadorned ssh. An analogous situation exists with their manpages, +leading to noise like this:: + + # equery check openssh + [ Checking net-misc/openssh-4.5_p1 ] + !!! /etc/ssh/sshd_config has incorrect md5sum + !!! /usr/bin/slogin does not point to ssh + !!! /usr/share/man/man1/slogin.1.gz does not point to ssh.1.gz + !!! /etc/ssh/ssh_config has incorrect md5sum + * 62 out of 66 files good + +''We can ignore the lines for `ssh_config` and `sshd_config`; those will be caught by bcfg2 as registered config files and handled appropriately.'' + +Because bcfg2 relies on the client system’s native package reporting tool to judge the state of installed packages, complaints like these about trivial or intractable verification failures can trigger unnecessary bundle reinstalls when the bcfg2 client runs. bcfg2 will catch on after a pass or two that the situation isn’t getting any better with repeated package installs, stop trying, and list those packages as “bad” in the client system’s statistics. + +Aside from filing bugs with the Gentoo package maintainers, your narrator has been unable to come up with a good approach to this. Maybe write a series of `Rules` definitions according to what the package database thinks it should find, and/or stage copies of affected files under `Cfg`, and associate those rules and files with the affected package in a bundle? Annoying but possibly necessary if you want your stats file to look good. + +/boot +----- + +Gentoo as well as some other distros recommend leaving `/boot` unmounted during normal runtime. This can lead to trouble during verification and package installation, for example when `/boot/grub/grub.conf` turns up missing. The simplest way around this might just be to ensure that `/boot` is mounted whenever you run bcfg2, possibly wrapping bcfg2 in a script for the purpose. I’ve also thought about adding `Action` clauses to bundles for grub and our kernel packages, which would mount `/boot` before the bundle installs and unmount it afterward, but this doesn’t get around the problem of those packages flunking verification. diff --git a/doc/unsorted/groupnames.txt b/doc/unsorted/groupnames.txt new file mode 100644 index 000000000..7df223387 --- /dev/null +++ b/doc/unsorted/groupnames.txt @@ -0,0 +1,17 @@ +.. -*- mode: rst -*- + +.. _unsorted-groupnames: + +=========== +Group Names +=========== + +This pages describes a set of common group names. The goal is to build a small set of common group names that can be used across installations. With any luck, this will increase portability of repository contents. Use of these names is, and will continue to be entirely optional. None of these groups are profile groups. This will result in a highly functional example repository, hosted in [source:/trunk/repository svn] + +|| '''Group Name''' || '''Category''' || '''Semantic Content''' || +|| linux || os || Clients running the Linux kernel || +|| solaris || os || Clients running the Solaris kernel || + +The following image describes the base compatibility OS groups for bcfg2. This file is available from [source:/trunk/repository/Metadata/bcfg2-os.xml]. + +.. image:: viz.png diff --git a/doc/unsorted/help.txt b/doc/unsorted/help.txt new file mode 100644 index 000000000..155dbf77c --- /dev/null +++ b/doc/unsorted/help.txt @@ -0,0 +1,45 @@ +.. -*- mode: rst -*- + +.. _unsorted-help: + +================ +Ways to get help +================ + +Interactive Help +================ + +* [wiki:IRCChannel IRC Channel, with indexed archives] +* [wiki:MailingList Mailing list, with indexed archives] + +Note that the IRC channel tends to be much busier than the mailing list; use whichever seems most appropriate for your query, but don't let the lack of mailing list activity make you think the project isn't active. + +Frequently Asked Questions +========================== + +* [wiki:FAQ The FAQ] + +Examples +======== + +* There are examples sprinkled throughout this wiki; we should link to them from here. +* The [http://www.fsf.org Free Software Foundation] is (very slowly) working towards having configurations for the majority of the machines it administers available via [http://config.fsf.org config.fsf.org]. This is a tie-in with the [http://autonomo.us/2008/07/franklin-street-statement/ Franklin Street Statement on Freedom and Network Services] (FSF offices are on Franklin Street). Documentation on how to have a public access Bcfg2 configuration repository will be at PublicRepository. + +Manuals +======= + +* The current canonical source of documentation are pages on this wiki ([wiki:UsingBcfg2]). Please mail the MailingList for editor access to this wiki. +* There is a printed manual in the SAGE short topics series, "!#19: Configuration Management with Bcfg2", that you can [https://db.usenix.org/cgi-bin/sage/booklets/order.cgi order] for $20 (or get for free if you are a [http://www.sage.org/index.html SAGE] member and haven't gotten a booklet yet during your current membership year). The book includes documentation up to and including most features in Bcfg2 0.9.6. Note that all proceeds from the sale of this book go to SAGE. + +FLOSS Manual Project +==================== + +A project is getting started to make a user-contributed manual using the [http://en.flossmanuals.net/ FLOSS Manuals] web site and tools. The intention is for this manual to be based on but not a verbatim copy of the information on the wiki, formatted in a way that is easier for new users to read, and written mostly by users of Bcfg2, rather than the authors of Bcfg2. This manual will also be free (as in freedom and price). + +One important point is that new contributors can get edit access to the manual in about a minute, and the manual is edited via WYSIWYG tools, so there should be pretty much no barrier for new manual authors to get started. + +There will be an announcement to the mailing list about this soon. + +If you are seriously interested in dedicating time to this manual, it would make sense to read the [http://en.flossmanuals.net/FLOSSManuals FLOSS Manuals Manual] (free online) and the [https://db.usenix.org/cgi-bin/sage/booklets/order.cgi Configuration Management with Bcfg2] manual ($20). If you are willing to commit time to manual writing, would like physical copies of these manuals, and purchasing them would be a financial hardship for you, email [http://pobox.com/~dclark Danny Clark] at dclark@pobox.com (ping djbclark on [wiki:IRCChannel #bcfg2 irc] if you don't get a reply) with your postal address (don't be shy, I already bought a bunch of these, and they aren't doing much good sitting on my shelf :-). + +You can get to the Bcfg2 FLOSS Manual at http://docs.bcfg2.org (which just redirects to http://en.flossmanuals.net/bin/view/BCFG2). diff --git a/doc/unsorted/howtos.txt b/doc/unsorted/howtos.txt new file mode 100644 index 000000000..d8af1d36b --- /dev/null +++ b/doc/unsorted/howtos.txt @@ -0,0 +1,22 @@ +.. -*- mode: rst -*- + +.. _unsorted-howtos: + +====== +HOWTOs +====== + +Here are several howtos that describe different aspects of Bcfg2 deployment + +* [wiki:Authentication] - a description of the bcfg2 authentication infrastructure +* AnnotatedExamples - a description of basic bcfg2 specification operations +* EncapPackages, EncapReadme, EncapInstall, EncapHowto - building encap packages +* NewDynamicReports - Howto set up new reporting system first released in 0.8.2 +* BuildingDebianPackages - How to build debian packages +* [wiki:Gentoo] - Issues specific to running bcfg2 on Gentoo +* [wiki:Plugins/TCheetah TCheetah] - Howto use the TCheetah template plugin +* [wiki:Hostbase] - How to use the Hostbase plugin and web interface +* [wiki:Plugins/Probes Probes] - How to use Probes to gather information from a client machine. +* [wiki:Plugins/Actions Actions] - How to use the new Action facility in Bcfg2-0.9.0 +* [wiki:DynamicGroups] - Using new dynamic groups in Bcfg2-0.9.0 +* [wiki:ParanoidMode] - How to run an update in paranoid mode diff --git a/doc/unsorted/index.txt b/doc/unsorted/index.txt index 4984ebdbd..63426c774 100644 --- a/doc/unsorted/index.txt +++ b/doc/unsorted/index.txt @@ -2,7 +2,6 @@ .. _unsorted-index: -============= Unsorted Docs ============= @@ -15,46 +14,7 @@ list below. .. _TitleIndex: https://trac.mcs.anl.gov/projects/bcfg2/wiki/TitleIndex * `Annotated Examples` -* `Audio/Video` -* `Bcfg2Doc` -* `Bcfg2 Subversion How To` -* `Bound Entry` -* `Building Debian Packages` -* `Building RPM Packages` -* `Client/Debugging` -* `Client/Metadata` -* `ClientTools/RPMng` -* `Comparison` -* `Configuration Entries` -* `Contribute` -* `Contributors` -* `Converging on verification with RHEL 5` -* `Debian Pkgmgr creation script` -* `Developing for Bcfg2` -* `Development/WritingPlugins` -* `Development Tips` -* `Doc:Architecture` -* `Documentation` -* `Download` -* `Dynamic groups` -* `Emacs snippet support` -* `Encap` -* `FAQ` -* `FeatureWindows` -* `Gentoo` -* `Group names` -* `Help` -* `Hostbase` -* `HostbaseInstallation` -* `Howtos` -* `IRCChannel` -* `Install` -* `IptablesDeadmanScript` -* `LearningPython` -* `MailingList` * `ManualPages` -* `NAT_HOWTO` -* `NISAuth` * `NewDynamicReports` * `NewDynamicReportsInstallation` * `News` @@ -126,7 +86,9 @@ list below. * `UsingRcache` * `VimSnippetSupport` * `WritingClientToolDrivers` -* `WritingSpecification` -* `altsrc` -* `mrepo` -* `ssl` + +.. toctree:: + :maxdepth: 2 + :glob: + + * diff --git a/doc/unsorted/install.txt b/doc/unsorted/install.txt new file mode 100644 index 000000000..56a4b9ebf --- /dev/null +++ b/doc/unsorted/install.txt @@ -0,0 +1,47 @@ +.. -*- mode: rst -*- + +.. _unsorted-install: + +============ +Installation +============ + +Prerequisites +============= + +First, install the prerequisite libraries. See the [wiki:Prereqs] page for more information. + +Bcfg2 +===== + +Before installing, you will need to choose a machine to be the Bcfg2 server. We recommend a Linux-based machine for this purpose, but the server will work on any supported operating system. Note that you may eventually want to run a web server on this machine for reporting and serving up package repositories. + +* '''From packages:''' The easiest way to install Bcfg2 is from packages for your operating system. You can grab packages (and source packages) for various OSes from the [wiki:Download] page. Install them as you would any other packages. The server package only needs to be installed on your designated Bcfg2 server machine; the clients package needs to be installed on any machine you plan to manage by Bcfg2. +* '''From source:''' The Bcfg2 source tarball can also be grabbed from the [wiki:Download] page. After untarring the file, you can build Bcfg2 with {{{python setup.py install --prefix=/install/prefix}}} This will install both the client and server on that machine. + +Configuration +============= + +Once Bcfg2 is installed, head over to the [wiki:QuickStart] to get it configured and up-and-running. + + +OS X +==== + +Using native OS X python +------------------------ + +First, make sure you have Xcode installed as you need `packagemaker` which comes bundled in the Developer tools. + +Check out the source from subversion:: + + svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2 + +Change to the osx directory and type make. Your new package should be located at bcfg2-'''$VERSION'''.pkg (where '''$VERSION''' is that which is specified in setup.py). + +Using macports +-------------- + +Once macports is installed:: + + port install bcfg2 diff --git a/doc/unsorted/ircchannel.txt b/doc/unsorted/ircchannel.txt new file mode 100644 index 000000000..4f4efa6e5 --- /dev/null +++ b/doc/unsorted/ircchannel.txt @@ -0,0 +1,32 @@ +.. -*- mode: rst -*- + +.. _unsorted-ircchannel: + +=========== +IRC Channel +=========== + +The Bcfg2 IRC channel is [irc://chat.freenode.net/bcfg2 #bcfg2 on chat.freenode.net]. It is home to both support and development discussions. If you have a question, suggestion, or just want to know about Bcfg2, please drop in and say hi. + +Archives are available at: [http://colabti.org/irclogger/irclogger_logs/bcfg2]. + +.. raw:: html + + <form method="get" action="http://colabti.org/irclogger/irclogger_log_search/bcfg2"> + <input type=text name=search value='' size=48> + <input type=submit value="Search #bcfg2 irc logs"> + <input type=hidden name=action value=search> <br> + in the timespan + <input type=text name=timespan value='' size=17> + (yyyymmdd-yyyymmdd) + <p>Options:<br> + <label><input type=checkbox name=nick value="checked">Search in the nick field</label><br> + <label><input type=checkbox name=text checked value="checked">Search in the text field</label><br> + </form> + +Administrative Note +=================== + +If the IRC logging stops working for a while, coordinate on #bcfg2 and then bug `feb` on #irclogger (freenode), and stick around on that channel until you get an answer (`feb` is great, but busy and in a non-US time zone). Actually as long as `ilogger2` is logged in you should be okay (`feb` looks at the logs). + +If you have private logs for the period of time `ilogger2` was off the channel, you can convert them to the [http://colabti.org/irclogger/irclogger_log/bcfg2?date=2008-03-21,Fri;raw=on format shown here], and `feb` will incorporate them into the online logs. Be sure to strip out any private messages in your logs first :-) diff --git a/doc/unsorted/learningpython.txt b/doc/unsorted/learningpython.txt new file mode 100644 index 000000000..3810d3899 --- /dev/null +++ b/doc/unsorted/learningpython.txt @@ -0,0 +1,23 @@ +.. -*- mode: rst -*- + +.. _unsorted-learningpython: + +=============== +Learning Python +=============== + +Learning +======== + +* MIT OpenCourseWare [http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-189January--IAP--2008/CourseHome/index.htm A Gentle Introduction to Programming Using Python] which primarily uses [http://www.greenteapress.com/thinkpython/thinkpython.html Think Python: How to Think Like a Computer Scientist]. +* [http://www.diveintopython.org/ Dive into Python], available in (free) online and dead trees formats. Good introduction, but a bit dated at this point (2004). +* djbclark recommends [http://hetland.org/writing/beginning-python/ Beginning Python: From Novice to Professional] +* djbclark recommends [http://www.oreilly.com/catalog/pythoncook2/ Python Cookbook, 2nd Edition] + +References +========== + +* desai recommends "Python: Essential Reference, 3rd Edition", which is available [http://safari.samspublishing.com/0672328623 online], in [http://www.samspublishing.com/bookstore/product.asp?isbn=0768664985&rl=1 pdf], and as a [http://www.samspublishing.com/bookstore/product.asp?isbn=0672328623&rl=1 book]. +* djbclark recommends [http://www.oreilly.com/catalog/pythonpr3/ Python Pocket Reference, 3rd Edition] +* [http://www.oreilly.com/catalog/pythonian2/ Python in a Nutshell, 2nd Edition] +* lueningh recommends the official [http://docs.python.org/lib/lib.html library reference] for, say, when you just need to know what all of a string object's methods are. diff --git a/doc/unsorted/mailinglist.txt b/doc/unsorted/mailinglist.txt new file mode 100644 index 000000000..d7cf8a8ca --- /dev/null +++ b/doc/unsorted/mailinglist.txt @@ -0,0 +1,19 @@ +.. -*- mode: rst -*- + +.. _unsorted-mailinglist: + +============ +Mailing List +============ + +To subscribe to the mailing list for bcfg2 please visit: [https://lists.mcs.anl.gov/mailman/listinfo/bcfg-dev bcfg-dev mailman page] + +[http://dir.gmane.org/gmane.comp.sysutils.bcfg2.devel Searchable archives] are available from Gmane. You can also read the mailing list from any NNTP client via Gmane. + +.. raw:: html + + <form method="get" action="http://search.gmane.org/"> + <input type="text" name="query"> + <input type="hidden" name="group" value="gmane.comp.sysutils.bcfg2.devel"> + <input type="submit" value="Search gmane.comp.sysutils.bcfg2.devel"> + </form> diff --git a/doc/unsorted/mrepo.txt b/doc/unsorted/mrepo.txt new file mode 100644 index 000000000..cae6e1dc0 --- /dev/null +++ b/doc/unsorted/mrepo.txt @@ -0,0 +1,65 @@ +.. -*- mode: rst -*- + +.. _unsorted-mrepo: + +===== +mrepo +===== + +This page describes how to setup an [http://dag.wieers.com/home-made/mrepo/ mrepo] mirror. + +mrepo builds a local APT/Yum RPM repository from local ISO files, downloaded updates, and extra packages from 3rd party repositories. It takes care of setting up the ISO files, downloading the RPMs, configuring HTTP access and providing PXE/TFTP resources for remote network installations. + +Sample mrepo configuration +========================== + +:: + + ### Configuration file for mrepo + + ### The [main] section allows to override mrepo's default settings + ### The mrepo-example.conf gives an overview of all the possible settings + [main] + srcdir = /var/mrepo/src + wwwdir = /var/www/mrepo + confdir = /etc/mrepo.conf.d + arch = x86_64 + + mailto = <youremail> + smtp-server = localhost + + hardlink = yes + shareiso = yes + + rsync-timeout = 3600 + + [centos5] + name = CentOS Server $release ($arch) + release = 5 + arch = x86_64 + metadata = yum repomd + + # ISO images + iso = centos-$release-server-$arch-DVD.iso + + #addons = rsync://mirrors.kernel.org/centos/$release/addons/$arch/RPMS + centosplus = rsync://mirrors.kernel.org/centos/$release/centosplus/$arch/RPMS + extras = rsync://mirrors.kernel.org/centos/$release/extras/$arch/RPMS + #fasttrack = rsync://mirrors.kernel.org/centos/$release/fasttrack/$arch/RPMS + os = rsync://mirrors.kernel.org/centos/$release/os/$arch/CentOS + updates = rsync://mirrors.kernel.org/centos/$release/updates/$arch/RPMS + dag = http://apt.sw.be/redhat/el$release/en/$arch/RPMS.dag + dries = http://apt.sw.be/redhat/el$release/en/$arch/RPMS.dries + rpmforge = http://apt.sw.be/redhat/el$release/en/$arch/RPMS.rpmforge + + ### Any other section is considered a definition for a distribution + ### You can put distribution sections in /etc/mrepo.conf.d/ + ### Examples can be found in the documentation at: + ### /usr/share/doc/mrepo-0.8.6/dists/. + +Run mrepo to update the repositories +==================================== + +:: + + mrepo -ug diff --git a/doc/unsorted/nat_howto.txt b/doc/unsorted/nat_howto.txt new file mode 100644 index 000000000..131c0c533 --- /dev/null +++ b/doc/unsorted/nat_howto.txt @@ -0,0 +1,56 @@ +.. -*- mode: rst -*- + +.. _unsorted-nat_howto: + +========= +NAT HOWTO +========= + +This page describes how to setup bcfg2 to properly function with a collection of clients behind NAT. It describes the issues, how the underlying portions of the bcfg2 system function, and how to correctly setup client metadata to cope with this environment. + +Issues +====== + +Bcfg2, by default, uses ip address lookup to determine the identity of a client that has connected. This process doesn't work properly in the case of NATted hosts, because all requests from these clients come from the same external address when connecting to the server. + +These client identification issues will manifest themselves in a number of ways: + +* Inability to setup discrete clients with different profiles +* Incorrect sharing of probe results across clients in the same NAT pool +* Inability to bootstrap clients properly when client data is not predefined + +Architectural Issues +==================== + +Client identification is performed as the beginning of each client/server interaction. As of 0.9.3pre3, client identification via IP address can be completely short-circuited through the use of a client uuid. Setup of client uuids requires actions of both the client and server. On the server side, the client uuid must be added to the client record in Metadata/clients.xml. This attribute allows the server to use the user part of the authentication to resolve the client's metadata. Also, either the location attribute should be set to floating, or the IP address of the NAT must be reflected in the address attribute. +Once added, the Client entry in clients.xml will look like: + +.. code-block:: xml + + <Client profile="desktop" name="test1" pingable="N" + uuid='9001ec29-1531-4b16-8198-a71bea093d0a' location='floating'/> + +Alternatively, the Client entry can be setup like: + +.. code-block:: xml + + <Client profile="desktop" name="test1" pingable="N" + uuid='9001ec29-1531-4b16-8198-a71bea093d0a' address='ip-address-of-NAT'/> + +The difference between these definitions is explained in detail on the [wiki:Authentication] page, but in short, the second form requires that the client access the server from the NAT address, while the first form allows it to connect from any address provided it uses the proper uuid. (Client identification is orthogonal to the use of per-client passwords; this can be set in addition to the attributes above.) + +Once this setup is done, each client must be configured to use the proper uuid upon any server interaction. This can be done using either the command line argument -u, or the setting "user" in the "communication" section of /etc/bcfg2.conf. + +UUID Choice +=========== + +When determining client UUIDs, one must take care to ensure that UUIDs are unique to the client. Any hardware-specific attribute, like a hash of a mac address would be sufficient. Alternatively, if a local hostname is unique, it may be used as well. + +Automated Client Bootstrapping +============================== + +Automated setup of new clients from behind NAT works by using the common password. For example:: + + /usr/sbin/bcfg2 -u ubik3 -p desktop -x <password> + +It is not possible at this time to do automated setup without setting up a pre-shared per-client key. diff --git a/doc/unsorted/nisauth.txt b/doc/unsorted/nisauth.txt new file mode 100644 index 000000000..3dfad1d26 --- /dev/null +++ b/doc/unsorted/nisauth.txt @@ -0,0 +1,19 @@ +.. -*- mode: rst -*- + +.. _unsorted-nisauth: + +================== +NIS Authentication +================== + +'''Edit Django settings''' + +Django allows for custom authentication backends to its login procedure. Hostbase has an NIS authentication backend that verifies a user to be in the unix group allowed to modify Hostbase. + +To enable this feature: + +* first edit your {{{Hostbase/settings.py}}} file and uncomment the line {{{'Hostbase.backends.NISBackend',}}} in the list of {{{AUTHENTICATION_BACKENDS}}} +* enter the name of the unix group you want to give access to Hostbase in the {{{AUTHORIZED_GROUP}}} variable +* in your {{{Hostbase/hostbase/views.py}}} file at the very bottom, uncomment the block(s) of lines that give you the desired level of access + +Hostbase will now direct the user to a login page if he or she is not authorized to view a certain page. Users should log in with their regular Unix username and password. diff --git a/doc/unsorted/python-ssl_1.14-1_amd64.deb b/doc/unsorted/python-ssl_1.14-1_amd64.deb Binary files differnew file mode 100644 index 000000000..e6c8ad137 --- /dev/null +++ b/doc/unsorted/python-ssl_1.14-1_amd64.deb diff --git a/doc/unsorted/python-stdeb_0.3-1_all.deb b/doc/unsorted/python-stdeb_0.3-1_all.deb Binary files differnew file mode 100644 index 000000000..5cee96e3c --- /dev/null +++ b/doc/unsorted/python-stdeb_0.3-1_all.deb diff --git a/doc/unsorted/specification_overview.png b/doc/unsorted/specification_overview.png Binary files differnew file mode 100644 index 000000000..66774f359 --- /dev/null +++ b/doc/unsorted/specification_overview.png diff --git a/doc/unsorted/ssl.txt b/doc/unsorted/ssl.txt new file mode 100644 index 000000000..6189fdd3c --- /dev/null +++ b/doc/unsorted/ssl.txt @@ -0,0 +1,53 @@ +.. -*- mode: rst -*- + +.. _unsorted-ssl: + +========== +Python SSL +========== + +The ssl module can be found `here <http://pypi.python.org/pypi/ssl>`_. + +With this change, SSL certificate based client authentication is supported. In order to use this, based CA-type capabilities are required. A central CA needs to be created, with each server and all clients getting a signed cert. See [wiki:Authentication] for details. + +Setting up keys is accomplished with three settings, each in the "`[communication]`" section of bcfg2.conf:: + + key = /path/to/ssl private key + certificate = /path/to/signed cert for that key + ca = /path/to/cacert.pem + + +Python SSL Backport Packaging +============================= + +Both the Bcfg2 server and client are able to use the in-tree ssl module included with python 2.6. The client is also able to still use M2Crypto. A python ssl backport exists for 2.3, 2.4, and 2.5. With this, M2Crypto is not needed, and tlslite is no longer included with bcfg2 sources. See [wiki:Authentication] for details. + +To build a package of the ssl backport for .deb based distributions that don't ship with python 2.6, you can follow these instructions, which use [http://github.com/astraw/stdeb/tree/master stdeb]. Alternatively if you happen to have .deb packaging skills, it would be great to get policy-complaint .debs into the major deb-based distributions. + +The following commands were used to generate :download:`this +<python-ssl_1.14-1_amd64.deb>` debian package ('''NOTE:''' Version +numbers for the SSL module have changed). The `easy_install` command can +be found in the `python-setuptools` package.:: + + sudo aptitude install python-all-dev fakeroot + sudo easy_install stdeb + wget http://pypi.python.org/packages/source/s/ssl/ssl-1.14.tar.gz#md5=4e08aae0cd2c7388d1b4bbb7f374b14a + tar xvfz ssl-1.14.tar.gz + cd ssl-1.14 + stdeb_run_setup + cd deb_dist/ssl-1.14 + dpkg-buildpackage -rfakeroot -uc -us + sudo dpkg -i ../python-ssl_1.14-1_amd64.deb + +For complete bcfg2 goodness, you'll also want to package stdeb using stdeb. +The completed debian package can be grabbed from :download:`here +<python-stdeb_0.3-1_all.deb>`, which was generated using the following:: + + sudo aptitude install apt-file + wget http://pypi.python.org/packages/source/s/stdeb/stdeb-0.3.tar.gz#md5=e692f745597dcdd9343ce133e3b910d0 + tar xvfz stdeb-0.3.tar.gz + cd stdeb-0.3 + stdeb_run_setup + cd deb_dist/stdeb-0.3 + dpkg-buildpackage -rfakeroot -uc -us + sudo dpkg -i ../python-stdeb_0.3-1_all.deb diff --git a/doc/unsorted/viz.png b/doc/unsorted/viz.png Binary files differnew file mode 100644 index 000000000..56f7cb03c --- /dev/null +++ b/doc/unsorted/viz.png diff --git a/doc/unsorted/windows.txt b/doc/unsorted/windows.txt new file mode 100644 index 000000000..33b45d67d --- /dev/null +++ b/doc/unsorted/windows.txt @@ -0,0 +1,51 @@ +.. -*- mode: rst -*- + +.. _unsorted-windows: + +================================= +Notes on possible Windows support +================================= + +* Windows Management Instrumentation (WMI) should be used wherever possible; there is an excellent [http://tgolden.sc.sabren.com/python/wmi.html WMI Python Module] available, which also comes with a [http://tgolden.sc.sabren.com/python/wmi_cookbook.html WMI Cookbook]. +* Before Windows 2003 SP1, on 64-bit machines there are [http://msdn2.microsoft.com/en-us/library/aa393067.aspx no API or WMI calls] to get to many 32-bit windows functions (such as the 32-bit registry) from 64-bit programs, and vice versa. There also is no (official) x86_64 native python distributions for Windows pre-Python 2.5. So the choice would be: + + #. Only support Windows in Python 2.5+ (which wouldn't be that bad because part of the build process would probably be to create stand-alone bcfg2 executables using [http://www.py2exe.org/ py2exe]). For 64-bit support there would have to be some kind of convoluted py2exe build process that built some things with 32-bit python and some things with 64-bit python. + #. Wrap external command-line programs such as winreg, which is part of [http://dmst.aueb.gr/dds/sw/outwit/ outwit], and screen scrape. Each external command-line program would need to be compiled into 32 and 64 bit versions. This approach might lead to licensing annoyances and having binary blobs in source control. + +Services +======== + +With the exception of 32/64 bit issues, Windows Services support should be pretty trivial; it would differ from *nix services in that it would be done via WMI API calls and not a 3rd party python module or wrapping a binary. + +Registry +======== + +The best way of handling the registry may be to map it into a file-based representation on the server end. The Cfg and TCheetah plugins could then be used to set registry values as needed. + +Files +===== + +For a first run there may be some way of utilizing [http://cygwin.com/ cygwin] to make use of the existing *nix POSIX module for manipulating files. There would probably need to be some changes to deal with the fact that open files can't be manipulated/moved/deleted at all in Windows (other than to do some registry magic that makes the changes on the next reboot). + +Packages +======== + +Listing and removal of packages should be pretty easy via WMI. For installation in most cases the admin would need to figure out the correct silent install flags (there is a [http://www.appdeploy.com/ web site] that catalogs a lot of this information), and include that in the bcfg2 server-side XML along with a URL (like with the RPM plugin); the bcfg2 client itself would need to take care of download, perhaps via the [http://linux.duke.edu/projects/urlgrabber/ urlgrabber python module]. + +Another option would be to utilize one of the existing FLOSS tools for dealing with Windows packages, such as [http://wpkg.org/ WPKG]. + +Prior FLOSS Art +=============== + +* [http://www.autoitscript.com/autoit3/ AutoIt] - For dealing with packages that don't have a silent install option +* [http://www.opensysadmin.com/trac/ticket/4 French Stuff] + * [http://ocsinventory.sourceforge.net/ Open Computers and Software Inventory - Next Generation] + * [http://www.glpi-project.org/spip.php?lang=en GLPI - Gestionnaire libre de parc informatique] +* Javascript thing a colleague of Desai's at ANL wrote - Desai was going to see if this can be released +* [http://sial.org/howto/cfengine/windows/ Managing Windows with CFEngine and Perl] +* [http://www.dmst.aueb.gr/dds/sw/outwit/ Outwit] - Small unixy utilities for Windows stuff like the registry and clipboard +* [http://www.cfengine.org/docs/cfengine-NT/ Porting cfengine to Windows NT] +* [http://isg.ee.ethz.ch/tools/realmen/ Real Men Don't Click] - Tobi Oetiker's stuff + * [http://isg.ee.ethz.ch/tools/realmen/res/index.en.html More Prior FLOSS Art] +* [http://unattended.sourceforge.net/ Unattended] - Bare Metal Installs, Package Management +* [http://wpkg.org/ WPKG] - Package Management diff --git a/doc/unsorted/writing_specification.txt b/doc/unsorted/writing_specification.txt new file mode 100644 index 000000000..083065f76 --- /dev/null +++ b/doc/unsorted/writing_specification.txt @@ -0,0 +1,326 @@ +.. -*- mode: rst -*- + +.. _unsorted-writing_specification: + +=========================== +Writing Bcfg2 Specification +=========================== + +Bcfg2 specifications are logically divided in to three areas: + +* Metadata +* Abstract +* Literal + +The metadata portion of the configuration assigns a client to its profile group and to its non profle groups. The profile group is assigned in Metadata/clients.xml and the non profile group assignments are in Metadata/groups.xml. + +The group memberships contained in the metadata are then used to constuct an abstract configuration for the client. An abstract configuration for a client identifies the configuration entities (packages, configuration files, service, etc) that a client requires, but it does not identify them explicitly. For instance an abstract configuration may identify that a client needs the bcfg2 package with + +.. code-block:: xml + + <Package name=bcfg2/> + +but this does not explicitly identify that an RPM package version 0.9.2 should be loaded from http://rpm.repo.server/bcfg2-0.9.2-0.1.rpm. The abstract configuration is defined in the xml configuration files for the Base and Bundles plugins. + +A combination of a clients metadata (group memberships) and abstract configuration is then used to generate the clients literal configuration. For instance the above abstract configuration entry may generate a literal configuration of + + +.. code-block:: xml + + <Package name='bcfg2' version='0.9.2-0.1' type='yum'/> + +A clients literal configuration is generated by a number of plugins that handle the different configuration entities. + +.. image:: specification_overview.png + +Dynamic Groups +============== + +Dynamic groups are likewise complex, and are covered on their own [wiki:DynamicGroups page] + +Abstract Configuration (Structures) +=================================== + +A clients Abstract Configuration is the inventory of configuration entities that should be installed on a client. Two plugins provide the basis for the abstract configuration, the Bundler and Base. + +The plugin Bundler builds descriptions of interrelated configuration entities. These are typically used for the representation of services, or other complex groups of entities. + +The Base provides a laundry list of configuration entities that need to be installed on hosts. These entities are independent from one another, and can be installed individually without worrying about the impact on other entities. + +Usage of Groups in Base and Bundles +----------------------------------- + +Groups are used by the Base and Bundles plugins for selecting +Configuration Entity Types for inclusion in a clients abstract +configuration. They can be thought of as:: + + if client is a member of group1 then + assign to abstract config + +Nested groups are conjunctive (logical and).:: + + if client is a member of group1 and group2 then + assign to abstract config + +Group membership maybe negated. See "Writing Bundles" for an example. + +Configuration Entity Types +-------------------------- + +Entities in the abstract configuration (and correspondingly in the literal configuration) can have one of several types. In the abstract configuration, each of these entities only has a tag and the name attribute set. + +The types of Configuration Entities that maybe assigned to the abstract configuration are as follows: + +|| '' '''Tag''' '' || '' '''Description''' '' || +|| Package || Software Package || +|| !ConfigFile || Configuration File || +|| Service || Persistent system services and daemons || +|| Directory || File system Directories || +|| !SymLink || Symbolic links || +|| Permissions || The permissions (not contents) of a POSIX path || +|| Action || Actions to be performed as part of a bundle installation || + +An example of each entity type is below. + +.. code-block:: xml + + <Package name='bcfg2'/> + <ConfigFile name='/etc/bcfg2.conf'/> + <Service name='ntpd'/> + <Directory name='/var/tmp'/> + <SymLink name='/dev/MAKEDEV'/> + <Permissions name='/etc/passwd'/> + <Action name='action_name'/> + +Configuration Entity Name Attribute +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +With the exception of the Action Entity type, the value of the configuration entity name attribute is not arbitary. It must map to entities of the same type in the Literal Configuration, which in turn map to physical entities on the client. + +For the most part the physical entity being mapped to is intuitive and are listed below. + +|| '' '''Tag''' '' || '' '''Description''' '' || +|| Package || name of the software package || +|| !ConfigFile || Full absolute path of the configuration file || +|| Service || Name of the init script for the service which may not be that same as the name of the daemon started or the package name || +|| Directory || Full absolute path of the directories || +|| !SymLink || Full absolute path of the symbolic linkss || +|| Permissions || Full absolute path of the file or directory || +|| Action || Arbitary name || + +Writing Bundles +--------------- + +Bundles consist of a set of configuration entities. These entities are grouped together due to a configuration-time interdependency. Basic services tend to be the simplest example of these. They normally consist of + +* some software package(s) +* some configuration files +* an indication that some service should be activated + +If any of these pieces are installed or updated, all should be rechecked and any associated services should be restarted. + +All files in the Bundles/ subdirectory of the repository are processed. +Each bundle must be defined in its own file and the filename must be the +same as the bundle name with a .xml suffix.:: + + # ls Bundler + Glide3.xml + LPRng.xml + Tivoli-backup.xml + Tivoli.xml + a2ps.xml + abiword.xml + account.xml + adsm-client.xml + amihappy.xml + apache-basic.xml + apache.xml + apache2-basic.xml + apt-proxy.xml + at.xml + atftp-server.xml + atftp.xml + .... + +Groups can be used inside of bundles to differentiate which entries particular clients will receive. This is useful for the case where entries are named differently across systems; for example, one linux distro may have a package called openssh while another uses the name ssh. Configuration entries nested inside of Group elements only apply to clients who are a member of those groups; multiply nested groups must all apply. + +Also, groups may be negated; entries included in such groups will only apply to clients who are not a member of said group. + +When packages in a bundle are verified by the client toolset, the !ConfigFiles included in the same bundle are taken into consideration. That is, a package will not fail verification from a bcfg2 perspective if the package verification only failed because of configuration files that are defined in the same bundle. + +The following is an annotated copy of a bundle: + +.. code-block:: xml + + <Bundle revision='$Revision: 2668 $' name='ssh' version='2.0' + origin='https://svn.mcs.anl.gov/repos/bcfg/trunk/repository/Bundler/ssh.xml'> + <Path name='/etc/ssh/ssh_host_dsa_key'/> + <Path name='/etc/ssh/ssh_host_rsa_key'/> + <Path name='/etc/ssh/ssh_host_dsa_key.pub'/> + <Path name='/etc/ssh/ssh_host_rsa_key.pub'/> + <Path name='/etc/ssh/ssh_host_key'/> + <Path name='/etc/ssh/ssh_host_key.pub'/> + <Path name='/etc/ssh/sshd_config'/> + <Path name='/etc/ssh/ssh_config'/> + <Path name='/etc/ssh/ssh_known_hosts'/> + <Group name='rpm'> + <Package name='openssh'/> + <Package name='openssh-askpass'/> + <Service name='sshd'/> + <Group name='fedora' > + <Group name='fc4' negate='true'> + <Package name='openssh-clients'/> + </Group> + <Package name='openssh-server'/> + </Group> + </Group> + <Group name='deb'> + <Package name='ssh'/> + <Service name='ssh'/> + </Group> + </Bundle> + +In this bundle, most of the entries are common to all systems. Clients in group "deb" get one extra package and service, while clients in group "rpm" get two extra packages and an extra service. In addition, clients in group "fedora" and group "rpm" get one extra package entries, unless they are not in the fc4 group, in which case, they get an extra package. Notice that this file doesn't describe which versions of these entries that clients should get, only that they should get them. (Admittedly, this example is slightly contrived, but demonstrates how group entries can be used in bundles) + +|| '' '''Group ''' '' || '' '''Entry ''' '' || +|| all || /etc/ssh/ssh_host_dsa_key || +|| all || /etc/ssh/ssh_host_rsa_key || +|| all || /etc/ssh/ssh_host_dsa_key.pub || +|| all || /etc/ssh/ssh_host_rsa_key.pub || +|| all || /etc/ssh/ssh_host_key || +|| all || /etc/ssh/ssh_host_key.pub || +|| all || /etc/ssh/sshd_config || +|| all || /etc/ssh/ssh_config || +|| all || /etc/ssh/ssh_known_hosts || +|| rpm || Package openssh || +|| rpm || Package openssh-askpass || +|| rpm || Service sshd || +|| rpm and fedora || Package openssh-server || +|| rpm and fedora and not fc4 || Package openssh-clients || +|| deb || Package ssh || +|| deb || Service ssh || + +Bundle Tag +^^^^^^^^^^ + +The Bundle Tag has the following possible attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || The name of the bundle || String || +|| version || Bundle schema version || String || +|| origin || URL of master version (for common repo) || String || +|| revision || Master version control revision || String || + +As mentioned above the Configuration Entity Tags may only have the name attribute in Bundle definitions. + +Abstract Group Tag +^^^^^^^^^^^^^^^^^^ + +In the Abstract Configuration plugins (Base and Bundle) the Group Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Name of group. || String || +|| negate || Negate the group association (is not a member of) || (True|False*) || + +An abstract group may contain any of the Configuration Entity types and other groups. + +Using Base +^^^^^^^^^^ + +The Base plugin provides a mechanism to add independent configuration +entities to a client's abstract configuration. All files in the Base/ +subdirectory of the repository are processed, and all entries that fall +within the scope of the client metadata are included in its abstract +configuration.:: + + $ ls Base/ + centos-4-x86.xml + fedora-core-4-x86.xml + rhel-as-4-x86.xml + rhel-es-4-x86.xml + rhel-ws-4-x86_64.xml + rhel-ws-4-x86.xml + +.. code-block:: xml + + <Base> + <Group name='Centos4.4-Standard'> + <Package name='audit'/> + <Package name='audit-libs'/> + <Package name='basesystem'/> + <Package name='bash'/> + <Package name='bcfg2'/> + <Package name='beecrypt'/> + .... + <Package name='yum'/> + <Package name='zlib'/> + <Group name='x86_64'> + <Package name='systemimager-x86_64initrd_template'/> + </Group> + </Group> + </Base> + +The format of the Base files are similar to those used by the Bundler. The majority of the elements are usually Packages, but !ConfigFiles, Directories, Services, Permissions, and !SymLinks may all be defined. A partial example is below: + +Base Tag +^^^^^^^^ + +The Base Tag has no attributes + +As mentioned above the Configuration Entity Tags contained in a Base definition may only have the name attribute in Base definitions. + +Abstract Group Tag +^^^^^^^^^^^^^^^^^^ + +In the Abstract Configuration plugins (Base and Bundle) the Group Tag may have the following attributes: + +|| '' '''Name''' '' || '' '''Description''' '' || '' '''Values''' '' || +|| name || Name of group. || String || +|| negate || Negate the group association (is not a member of) || (True|False*) || + +An abstract group may contain any of the Configuration Entity types and other groups. + +Literal Configuration (Generators) +================================== + +A Generator is a bcfg2 piece of code that is run to generate the literal configuration for a host using a combination of the hosts metadata and abstract configuration. + +A Generator can take care of a particular configuration element. Any time this element is requested by the client, the server dynamically generates it either by crunching data and creating new information or by reading a file off of disk and passes it down to the client for installation. + +Usage of Groups in Generators +----------------------------- + +Similar to Abstract Configuration plugins, groups are used by generator +plugins for selecting Configuration Entities for inclusion in a clients +literal configuration. They can be thought of as:: + + if client is a member of group1 then + assign to abstract config + +Nested groups are conjunctive (logical and).:: + + if client is a member of group1 and group2 then + assign to abstract config + +How the groups are configured is specific to the plugin, but here are two common methods: + +* xml configuration file (Pkgmgr, Rules) +* file name encoding (Cfg, TCheetah, SSHBase) + +Details are included on each plugin's page. + +Generators +---------- + +Each of the generators is covered on their own page. + +|| '' '''Plugin''' '' || '' '''Description''' '' || +|| [wiki:Plugins/Actions Actions] || Action entries are commands that are executed either before bundle installation, after bundle installation or both. || +|| [wiki:Plugins/Cfg Cfg] || The Cfg plugin provides a repository to describe configuration file contents for clients. In its simplest form, the Cfg repository is just a directory tree modeled off of the directory tree on your client machines. || +|| [wiki:Plugins/Deps Deps] || The deps plugin allows inter-entry dependencies (like package prerequisites) to be mapped independent of client roles. || +|| [wiki:Plugins/Hostbase Hostbase] || Hostbase is an IP management system built on top of Bcfg2. || +|| [wiki:Plugins/Pkgmgr Pkgmgr] || The Pkgmgr plugin define what packages are available for a host or image and where to find those packages. || +|| [wiki:Plugins/Rules Rules] || The Rules plugin is responsible for defining the details about the Directory, Permissions, !SymLink and Service elements. || +|| [wiki:Plugins/SSHbase SSHbase] || SSHbase is a purpose build bcfg2 plugin for managing ssh host keys. It is responsible for making ssh keys persist beyond a client rebuild and building a proper ssh_known_hosts file, including a correct localhost record for the current system. || +|| [wiki:Plugins/TCheetah TCheetah] || The TCheetah plugin allows you to use the [http://www.cheetahtemplate.org/ cheetah templating system] to create files, instead of the various diff-based methods offered by the Cfg plugin. It also allows you to include the results of probes executed on the client in the created files. || +|| [wiki:Plugins/TGenshi TGenshi] || The TGenshi plugin uses the [http://genshi.edgewall.org/ genshi toolkit] for generating files and functions much like the TCheetah plugin || |
