path: root/doc/server/plugins/generators/packages.txt

.. -*- mode: rst -*-

.. _server-plugins-generators-packages:

========
Packages
========

.. versionadded:: 1.0.0

This page documents the Packages plugin. Packages is an alternative to
:ref:`Pkgmgr <server-plugins-generators-pkgmgr>` for specifying package
entries for clients. Where Pkgmgr explicitly specifies package entry
information, Packages delegates control of package version information to
the underlying package manager, installing the latest version available
through those channels.

.. _server-plugins-generators-packages-magic-groups:

"Magic Groups"
==============

Packages is the only plugin that uses "magic groups". Most plugins
operate based on client group memberships, without any concern for the
particular names chosen for groups by the user. The Packages plugin is
the sole exception to this rule. Packages needs to "know" two
different sorts of facts about clients. The first is the basic
OS/distro of the client, enabling classes of sources. The second is
the architecture of the client, enabling sources for a given
architecture. In addition to these magic groups, each source may also
specify non-magic groups to limit the source's applicability to group
member clients.

+--------+----------+--------------+
| Source | OS Group | Architecture |
+========+==========+==============+
| Apt    | debian   | i386         |
+--------+----------+--------------+
| Apt    | ubuntu   | amd64        |
+--------+----------+--------------+
| Apt    | nexenta  |              |
+--------+----------+--------------+
| Apt    | apt      |              |
+--------+----------+--------------+
| Yum    | redhat   | i386         |
+--------+----------+--------------+
| Yum    | centos   | x86_64       |
+--------+----------+--------------+
| Yum    | fedora   |              |
+--------+----------+--------------+
| Yum    | yum      |              |
+--------+----------+--------------+

.. note:: 

   .. versionadded:: 1.2.0

   Magic OS groups can be disabled in Bcfg2 1.2 and greater by setting
   ``magic_groups`` to ``0`` in ``Packages/packages.conf``.  This may
   give you greater flexibility in determining which source types to
   use for which OSes.  Magic architecture groups cannot be disabled.


Limiting sources to groups
==========================

``Packages/sources.xml`` processes ``<Group>`` and ``<Client>`` tags
just like Bundles.  In addition to any groups or clients specified
that way, clients must
be a member of the appropriate architecture group as specified in a
Source stanza.  In total, in order for a source to be associated with
a client, the client must be in one of the magic groups (debian,
ubuntu, or nexenta), any explicit groups or clients specified in
``sources.xml``, and any specified architecture groups.

Memberships in architecture groups is needed so that Packages can map
software sources to clients. There is no other way to handle this than
to impose membership in the appropriate architecture group.

When multiple sources are specified, clients are associated with each
source to which they apply (based on group memberships, as described
above). Packages and dependencies are resolved from all applicable
sources.

.. note:: To recap, a client needs to be a member of the **OS Group**,
          **Architecture** group, and any other groups defined in your
          ``Packages/sources.xml`` file in order for the client to be
          associated to the proper sources.

Setup
=====

Three basic steps are required for Packages to work properly.

#. Create ``Packages/sources.xml``. This file should look
   approximately like the example below, and describes both which
   software repositories should be used, and which clients are eligible
   to use each one.
#. Ensure that clients are members of the proper groups. Each client
   should be a member of one of the magic groups listed above, all of
   the groups listed in the ``sources.xml`` (like ubuntu-intrepid or
   centos-5.2 in the following examples), and one of the architecture
   groups listed in the source configuration (i386, amd64 or x86_64 in
   the following examples). '''Failure to do this will result in the
   source either not applying to the client, or only architecture
   independent packages being made available to the client.'''
#. Add Package entries to bundles.
#. Sit back and relax, as dependencies are resolved, and automatically
   added to client configurations.

Prerequisite Resolution
=======================

Packages provides a prerequisite resolution mechanism which has no
analogue in Pkgmgr. During configuration generation, all structures are
processed. After this phase, but before entry binding, a list of packages
and the client metadata instance is passed into Packages' resolver. This
process determines a superset of packages that will fully satisfy
dependencies of all package entries included in structures, and reports
any prerequisites that cannot be satisfied. This facility should largely
remove the need to use the :ref:`Base <server-plugins-structures-base>`
plugin.

Disabling dependency resolution
-------------------------------

.. versionadded:: 1.1.0

Dependency resolution can be disabled by adding this to
``Packages/packages.conf`` in the ``global`` section::

    [global]
    resolver=disabled

All metadata processing can be disabled as well::

    [global]
    metadata=disabled

Blacklisting faulty dependencies
--------------------------------

If you encounter an issue with faulty dependency resolution due to
Packages, please file a bug report so that we can fix the problem in
future releases. In the meantime, you can work around this issue by
blacklisting the offending Package in your Sources. The blacklist
element should immediately follow the Component section of your source
and should look like the following::

    <Blacklist>unwanted-packagename</Blacklist>

If you use the built-in :ref:`Yum config generator
<generating-client-configs>`, blacklisted packages will be added to
the ``exclude`` list for the source.

Handling GPG Keys
-----------------

.. versionadded:: 1.2.0

Packages can automatically handle GPG signing keys for Yum and Pulp
repositories. Simply specify the URL to the GPG key(s) for a
repository in ``sources.xml``::

    <Source type="yum"
            rawurl="http://mirror.example.com/centos6-x86_64/RPMS.os">
      <Arch>x86_64</Arch>
      <GPGKey>http://mirror.example.com/keys/RPM-GPG-KEY-CentOS-6</GPGKey>
    </Source>

More than one ``<GPGKey>`` tag can be specified per Source.

With the keys specified thusly, Packages will include the keys in the
generated yum config file, and will ensure that the keys are imported
on the client.

There is no need to specify ``<GPGKey>`` tags for :ref:``Pulp sources
<pulp-source-support>``; that data is pulled directly from the Pulp
REST API.

.. _packages-exampleusage:

Example usage
=============

Create a ``sources.xml`` file in the Packages directory that looks
something like this::

    <Sources>
      <Group name="ubuntu-intrepid">
        <Source type="apt"
                url="http://us.archive.ubuntu.com/ubuntu"
                version="intrepid">
          <Component>main</Component>
          <Component>universe</Component>
          <Arch>i386</Arch>
          <Arch>amd64</Arch>
        </Source>
      </Group>
    </Sources>

.. note::

    .. versionadded:: 1.1.0

    The default behavior of the Packages plugin is to not make
    any assumptions about which packages you want to have added
    automatically. For that reason, neither **Recommended** nor
    **Suggested** packages are added as dependencies by default. You
    will notice that the default behavior for apt is to add Recommended
    packages as dependencies. You can configure the Packages plugin to
    add recommended packages by adding the ``recommended`` attribute,
    e.g.:

    .. code-block:: xml

        <Source type="apt" recommended="true" ...>

Yum sources can be similarly specified::

    <Sources>
      <Group name="centos-5.2">
        <Source type="yum"
                url="http://mirror.centos.org/centos/"
                version="5.2">
          <Component>os</Component>
          <Component>updates</Component>
          <Component>extras</Component>
          <Arch>i386</Arch>
          <Arch>x86_64</Arch>
          <GPGKey>http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5</GPGKey>
        </Source>
      </Group>
    </Sources>

For sources with a **URL** attribute, the **Version** attribute is
also necessary.

:ref:``Pulp sources <pulp-source-support>`` are very simple to specify
due to the amount of data that can be queried from Pulp itself::

    <Sources>
      <Group name="centos-6-x86_64">
        <Source type="yum" pulp_id="centos-6-x86_64-os"/>
        <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
        <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
      </Group>
    </Sources>

.. note:: There is also a rawurl attribute for specifying sources that
          don't follow the conventional layout.

          .. code-block:: xml

              <Sources>
                <Group name="centos5.4">
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.os">
                    <Arch>x86_64</Arch>
                  </Source>
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.updates">
                    <Arch>x86_64</Arch>
                  </Source>
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.extras">
                    <Arch>x86_64</Arch>
                  </Source>
                </Group>
              </Sources>

          .. code-block:: xml

              <Sources>
                <Group name="ubuntu-lucid">
                  <Source type="apt"
                          rawurl="http://hudson-ci.org/debian/binary">
                    <Arch>amd64</Arch>
                  </Source>
                  <Source type="apt"
                          rawurl=http://hudson-ci.org/debian/binary">
                    <Arch>i386</Arch>
                  </Source>
                </Group>
              </Sources>

Configuration Updates
=====================

Packages will reload its configuration upon an explicit command via
bcfg2-admin::

    [0:3711] bcfg2-admin xcmd Packages.Refresh
    True

During this command (which will take some time depending on the quantity
and size of the sources listed in the configuration file), the server
will report information like::

    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz
    ...
    Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz
    Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz

Once line per file download needed. ``Packages/sources.xml`` will
be reloaded at this time, so any source specification changes (new
or modified sources in this file) will be reflected by the server at
this point.

This process is much, much faster if you use the :ref:`native yum
library support <native-yum-libraries>`.

Soft reload
-----------

.. versionadded:: 1.2.0

A soft reload can be performed to reread the configuration file and
download only missing sources.::

    [0:3711] bcfg2-admin xcmd Packages.Reload
    True

This is done automatically any time ``Packages/sources.xml`` is
updated.

Availability
============

Support for clients using yum and apt is currently available.  Support for
other package managers (Portage, Zypper, IPS, etc) remain to be added.

Validation
==========

A schema for ``Packages/sources.xml`` is included; ``sources.xml`` can
be validated using ``bcfg2-lint``.

.. note:: The schema requires that elements be specified in the above order.

Limitations
===========

Packages does not do traditional caching as other plugins do. Changes
to the Packages config file require a server restart for the time being.

Package Checking and Verification
=================================

In order to do disable per-package verification Pkgmgr style, you will
need to use :ref:`BoundEntries <boundentries>`, e.g.::

    <BoundPackage name="mem-agent" priority="1" version="auto"
                  type="yum" verify="false"/>


.. _generating-client-configs:

Generating Client APT/Yum Configurations
========================================

.. versionadded:: 1.2.0

The Packages plugin has native support for generating Yum configs.
You must set ``yum_config`` in ``Packages/packages.conf`` to the path
to the yum config file you want to generate::

    [global]
    yum_config=/etc/yum.repos.d/all.repo

Then add the corresponding Path entry to your Yum bundle.

.. versionadded:: 1.1.0

APT repository information can be generated automatically from
software sources using :doc:`./tgenshi/index` or :doc:`./tcheetah`.  A
list of source urls are exposed in the client's metadata as
``metadata.Packages.sources``.  E.g.::

    # bcfg2 maintained apt

    {% for s in metadata.Packages.sources %}\
    deb ${s.url}${s.version} ${s.groups[0]} {% for comp in s.components %}$comp {% end %}

    {% end %}\

.. _native-yum-libraries:

Using Native Yum Libraries
==========================

.. versionadded:: 1.2.0

By default, Bcfg2 uses an internal implementation of Yum's dependency
resolution and other routines so that the Bcfg2 server can be run on a
host that does not support Yum itself.  If you run the Bcfg2 server on
a machine that does have Yum libraries, however, you can enable use of
those native libraries in Bcfg2 by setting ``use_yum_libraries`` to
``1`` in the ``[yum]`` section of ``Packages/packages.conf``.

Benefits to this include:

* Much lower memory usage by the ``bcfg2-server`` process.
* Much faster ``Packages.Refresh`` behavior.
* More accurate dependency resolution.
* Support for package groups.

Drawbacks include:

* More disk I/O.  In some cases, you may have to raise the open file
  limit for the user who runs your Bcfg2 server process, particularly
  if you have a lot of repositories.
* Resolution of package dependencies is slower in some cases,
  particularly after running ``Packages.Refresh``.

Setting Yum Options
-------------------

In ``Packages/packages.conf``, any options you set in the ``[yum]``
section other than ``use_yum_libraries`` will be passed along verbatim
to the configuration of the Yum objects used in the Bcfg2 server.  The
following options are set by default, and should not generally be
overridden:

* ``cachedir`` is set to a hashed value unique to each distinct Yum
  configuration.  Don't set this unless you know what you're doing.
* ``keepcache`` is set to ``0``; there is no benefit to changing this.
* ``sslverify`` is set to ``0``; change this if you know what you're
  doing.
* ``reposdir`` is set to ``/dev/null`` to prevent the server's Yum
  configuration from being read; do not change this.

Package Groups
--------------

Yum package groups are supported by the native Yum libraries.  To
include a package group, use the ``group`` attribute of the
``Package`` tag.  You can use either the short group ID or the long
group name::

    <Package group="SNMP Support"/>
    <Package group="system-management-snmp"/>

.. _pulp-source-support:

Pulp Support
============

.. versionadded:: 1.2.0

Bcfg2 contains explicit support for repositories managed by Pulp
(http://pulpproject.org/).  Due to the amount of data about a
repository that can be retrieved directly from Pulp, the only thing
necessary to configure a Pulp repo is the repo ID::

    <Sources>
      <Group name="centos-6-x86_64">
        <Source type="yum" pulp_id="centos-6-x86_64-os"/>
        <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
        <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
      </Group>
    </Sources>

Pulp sources require some additional configuration.  First, the Bcfg2
server must have a valid ``/etc/pulp/consumer/consumer.conf`` that is
readable by the user your Bcfg2 server runs as; the Pulp server ,
URLs, and so on, are determined from this.

Secondly, in ``Packages/packages.conf`` you must set the following
options in the ``[pulp]`` section:

* ``username`` and ``password``: The username and password of a Pulp
  user that will be used to register new clients and bind them to
  repositories.  Membership in the default ``consumer-users`` role is
  sufficient.

Bcfg2 clients using Pulp sources will be registered to the Pulp server
as consumers, and will be bound to the appropriate repositories.

Debugging unexpected behavior
=============================

Using bcfg2-info
----------------

The dependency resolver used in Packages can be run in debug mode::


    $ bcfg2-info
    ...
    Handled 20 events in 0.004s
    > debug
    dropping to python interpreter; press ^D to resume
    ...
    (debug_shell)
    >>> m = self.build_metadata('ubik3')
    >>> self.plugins['Packages'].complete(m, ['ssh'], debug=True)
    Package ssh: adding new deps ['openssh-client', 'openssh-server']
    Package openssh-server: adding new deps ['libc6', 'libcomerr2', 'libkrb53', 'libpam0g', 'libselinux1', 'libssl0.9.8
    ', 'libwrap0', 'zlib1g', 'debconf', 'libpam-runtime', 'libpam-modules', 'adduser', 'dpkg', 'lsb-base']
    Package debconf: adding new deps ['debconf-i18n']
    Package libpam-modules: adding new deps ['libdb4.7']
    Package openssh-client: adding new deps ['libedit2', 'libncurses5', 'passwd']
    Package lsb-base: adding new deps ['sed', 'ncurses-bin']
    Package adduser: adding new deps ['perl-base']
    Package debconf-i18n: adding new deps ['liblocale-gettext-perl', 'libtext-iconv-perl', 'libtext-wrapi18n-perl', 'libtext-charwidth-perl']
    Package passwd: adding new deps ['debianutils']
    Package libtext-charwidth-perl: adding new deps ['perlapi-5.10.0']
    VPackage perlapi-5.10.0: got provides ['perl-base']
    Package libkrb53: adding new deps ['libkeyutils1']
    Package libtext-iconv-perl: adding new deps ['perlapi-5.10.0']
    Package libc6: adding new deps ['libgcc1', 'findutils']
    Package libgcc1: adding new deps ['gcc-4.3-base']
    (set(['debconf', 'libgcc1', 'lsb-base', 'libtext-wrapi18n-perl', 'libtext-iconv-perl', 'sed', 'passwd', 'findutils', 'libpam0g', 'openssh-client', 'debconf-i18n', 'libselinux1', 'zlib1g', 'adduser', 'libwrap0', 'ncurses-bin', 'libssl0.9.8', 'liblocale-gettext-perl', 'libkeyutils1', 'libpam-runtime', 'libpam-modules', 'openssh-server', 'libkrb53', 'ssh', 'libncurses5', 'libc6', 'libedit2', 'libcomerr2', 'dpkg', 'perl-base', 'libdb4.7', 'libtext-charwidth-perl', 'gcc-4.3-base', 'debianutils']), set([]), 'deb')

This will show why the resolver is acting as it is. Replace
``"ubik3"`` and ``['ssh']`` with a client name and list of packages,
respectively. Also, a more polished interface to this functionality is
coming as well.

Each line starting with Package: <name> describes a set of new
prerequisites pulled in by a package. Lines starting with VPackage <vname>
describe provides entries and their mappings to required names. The last
line describes the overall results of the resolver, with three fields:
a list of packages that should be installed, a list of unresolved
requirements, and a type for these packages.

Using bcfg2-server
------------------

Once the server is started, enable debugging via bcfg2-admin::

    $ bcfg2-admin xcmd Packages.toggle_debug

TODO list
=========

* Zypper support
* Portage support
* Explicit version pinning (a la Pkgmgr)

Developing for Packages
=======================

In order to support a given client package tool driver, that driver
must support use of the auto value for the version attribute in Package
entries. In this case, the tool driver views the current state of
available packages, and uses the underlying package manager's choice of
correct package version in lieu of an explicit, centrally-specified,
version. This support enables Packages to provide a list of Package
entries with version='auto'. Currently, the APT and YUMng drivers support
this feature. Note that package management systems without any network
support cannot operate in this fashion, so RPMng and SYSV will never be
able to use Packages. Emerge, Zypper, IPS, and Blastwave all have the
needed features to be supported by Packages, but support has not yet
been written.

Packages fills two major functions in configuration generation. The first
is to provide entry level binding support for Package entries included
in client configurations. This function is quite easy to implement;
Packages determines (based on client group membership) if the package
is available for the client system, and which type it has. Because
version='auto' is used, no version determination needs to be done.

The second major function is more complex. Packages ensures that client
configurations include all package-level prerequisites for package entries
explicitly included in the configuration. In order to support this,
Packages needs to directly process network data for package management
systems (the network sources for apt or yum, for examples), process
these files, and build data structures describing prerequisites and the
providers of those functions/paths. To simplify implementations of this,
there is a generic base class (Bcfg2.Server.Plugins.Packages.Source)
that provides a framework for fetching network data via HTTP, processing
those sources (with subclass defined methods for processing the specific
format provided by the tool), a generic dependency resolution method,
and a caching mechanism that greatly speeds up server/bcfg2-info startup.

Each source type must define:

* a get_urls attribute (and associated urls property) that describes
  the URLS where to get data from.
* a read_files method that reads and processes the downloaded files

Sources may define a get_provides method, if provides are complex. For
example, provides in rpm can be either rpm names or file paths, so
multiple data sources need to be multiplexed.

The APT source in ``src/lib/Server/Plugins/Packages.py`` provides a
relatively simple implementation of a source.

packages.conf
=============

``packages.conf`` contains miscellaneous configuration options for the
Packages plugin.  It understands the following directives:

[global] section
----------------

* ``resolver``: Disable dependency resolution.  Default is "enabled".
* ``metadata``: Disable metadata processing.  Default is "enabled".
* ``yum_config``: The path at which to generate Yum configs.  No
  default.
* ``apt_config``: The path at which to generate APT configs.  No
  default.
* ``gpg_keypath``: The path on the client RPM GPG keys will be copied
  to before they are imported on the client.  Default is
  "/etc/pki/rpm-gpg".
* ``import_gpg_keys``: The RPM release of an RPM GPG key cannot be
  reliably and automatically determined without importing the key into
  the server's key chain.  If ``import_gpg_keys`` is "false" (the
  default), the release of automatically-generated RPM GPG key entries
  in the specification will be set to "any", which disables
  verification of the release.  (Version will still be verified.)  In
  practice, this is unlikely to be an issue, as the RPM version of a
  GPG key is the key's fingerprint, and collisions are rare.  If you
  do encounter a GPG key version collision, you will need to set this
  to "true", whereupon Packages will import the keys into the server's
  key chain.  Python RPM libraries must be installed for this to work.

[yum] section
-------------

* ``use_yum_libraries``: Whether or not to use the :ref:`native yum
  library support <native-yum-libraries>`.  Default is ``0`` (false).

All other options in the ``[yum]`` section will be passed along
verbatim to the Yum configuration if you are using the native Yum
library support.

[pulp] section
--------------

* ``username`` and ``password``: The username and password of a Pulp
  user that will be used to register new clients and bind them to
  repositories.  Membership in the default ``consumer-users`` role is
  sufficient.