path: root/doc/server/plugins/generators/packages.txt

.. -*- mode: rst -*-

.. _server-plugins-generators-packages:

========
Packages
========

.. versionadded:: 1.0.0

This page documents the Packages plugin. Packages is an alternative to
:ref:`Pkgmgr <server-plugins-generators-pkgmgr>` for specifying package
entries for clients. Where Pkgmgr explicitly specifies package entry
information, Packages delegates control of package version information to
the underlying package manager, installing the latest version available
through those channels.

Limiting sources to groups
==========================

``Packages/sources.xml`` processes ``<Group>`` and ``<Client>`` tags
just like Bundles. In addition to any groups or clients specified that
way, clients must be a member of the appropriate architecture group as
specified in a Source stanza.

Memberships in architecture groups is needed so that Packages can map
software sources to clients. There is no other way to handle this than
to impose membership in the appropriate architecture group.

When multiple sources are specified, clients are associated with each
source to which they apply (based on group memberships, as described
above). Packages and dependencies are resolved from all applicable
sources.

Setup
=====

Three basic steps are required for Packages to work properly.

#. Create Packages/`sources.xml`_. This file should look
   approximately  like the  example  below, and  describes both  which
   software  repositories  should  be  used,  and  which  clients  are
   eligible to use each one.
#. Ensure that clients are members of the proper groups. Each client
   should be a member of all of the groups listed in the
   ``sources.xml`` (like ubuntu-intrepid or centos-5.2 in the
   following examples), and one of the architecture groups listed in
   the source configuration (i386, amd64 or x86_64 in the following
   examples). '''Failure to do this will result in the source either
   not applying to the client, or only architecture independent
   packages being made available to the client.'''
#. Add Package entries to bundles.
#. Sit back and relax, as dependencies are resolved, and automatically
   added to client configurations.

sources.xml
-----------

``sources.xml`` is where all package sources are configured for the
Packages plugin.  It processes ``<Group>`` and ``<Client>`` tags just like
Bundles. The primary element in ``sources.xml`` is the Source tag:

.. xml:element:: Source
   :noautodep: py:genshiElements

Handling GPG Keys
-----------------

.. versionadded:: 1.2.0

If you have yum libraries installed, Packages can automatically handle
GPG signing keys for Yum and Pulp repositories. (You do not need to
use the native yum resolver; if yum libraries are available, GPG
signing keys can be handled automatically.) Simply specify the URL to
the GPG key(s) for a repository with :xml:element:`GPGKey` elements:

.. code-block:: xml

    <Source type="yum"
            rawurl="http://mirror.example.com/centos6-x86_64/RPMS.os">
      <Arch>x86_64</Arch>
      <GPGKey>http://mirror.example.com/keys/RPM-GPG-KEY-CentOS-6</GPGKey>
    </Source>

More than one ``<GPGKey>`` tag can be specified per Source.

With the keys specified thusly, Packages will include the keys in the
generated yum config file, and will ensure that the keys are imported
on the client.

There is no need to specify ``<GPGKey>`` tags for :ref:`Pulp sources
<pulp-source-support>`; that data is pulled directly from the Pulp
REST API.

Arbitrary Repo Options
----------------------

.. versionadded:: 1.2.3

You can specify arbitrary options to be added to the repository config
on the server side, if you are using the native yum libraries, and on
the client side if you are using the ability of Packages to
automatically generate your Yum config.  To do this, add an
:xml:element:`Options` tag to a :xml:element:`Source`; all of its
attributes will be added verbatim to the repository in the generated
config.  For instance:

.. code-block:: xml

    <Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
      <Arch>x86_64</Arch>
      <Options proxy="http://proxy.example.com"/>
    </Source>

If you are using native yum libraries and need to set options only on
the Bcfg2 server, you can set the
:xml:attribute:`RepoOptionsType:serveronly` attribute to "true"; or,
if you need to set options only on the client, you can set the
:xml:attribute:`RepoOptionsType:clientonly` attribute to "true".  For
instance, if your Bcfg2 server needed to use a proxy to access a repo,
and you wanted to expire metadata caches very quickly on the client,
you could do:

.. code-block:: xml

    <Source type="yum" rawurl="http://mirror.example.com/centos-6-os">
      <Arch>x86_64</Arch>
      <Options serveronly="true" proxy="http://proxy.example.com"/>
      <Options clientonly="true" metadata_expire="0"/>
    </Source>

Prerequisite Resolution
=======================

Packages provides a prerequisite resolution mechanism which has no
analogue in Pkgmgr. During configuration generation, all structures are
processed. After this phase, but before entry binding, a list of packages
and the client metadata instance is passed into Packages' resolver. This
process determines a superset of packages that will fully satisfy
dependencies of all package entries included in structures, and reports
any prerequisites that cannot be satisfied.

Disabling dependency resolution
-------------------------------

.. versionadded:: 1.1.0

Dependency resolution can be disabled by adding the following setting
to ``bcfg2.conf`` in the ``packages`` section::

    [packages]
    resolver=0

All metadata processing can be disabled as well::

    [packages]
    metadata=0

This setting implies disabling the resolver.

Blacklisting faulty dependencies
--------------------------------

If you encounter an issue with faulty dependency resolution due to
Packages, please file a bug report so that we can fix the problem in
future releases. In the meantime, you can work around this issue by
blacklisting the offending Package in your Sources. The
:xml:element:`Blacklist` element should immediately follow the
Component section of your source and should look like the following:

.. code-block:: xml

    <Blacklist>unwanted-packagename</Blacklist>

If you use the built-in :ref:`Yum config generator
<generating-client-configs>`, blacklisted packages will be added to
the ``exclude`` list for the source.

.. _packages-exampleusage:

Example usage
=============

Create a _`sources.xml` file in the Packages directory that looks
something like this:

.. code-block:: xml

    <Sources>
      <Group name="ubuntu-intrepid">
        <Source type="apt"
                url="http://us.archive.ubuntu.com/ubuntu"
                version="intrepid"
                debsrc="true">
          <Component>main</Component>
          <Component>universe</Component>
          <Arch>i386</Arch>
          <Arch>amd64</Arch>
        </Source>
      </Group>
    </Sources>

.. note::

    .. versionadded:: 1.1.0

    The default behavior of the Packages plugin is to not make any
    assumptions about which packages you want to have added
    automatically [#f1]_. For that reason, neither **Recommended** nor
    **Suggested** packages are added as dependencies by default. You
    will notice that the default behavior for apt is to add
    Recommended packages as dependencies. You can configure the
    Packages plugin to add recommended packages by adding the
    :xml:attribute:`SourceType:recommended` attribute, e.g.:

    .. code-block:: none

        <Source type="apt" recommended="true" ...>

    .. [#f1] Bcfg2 will by default add **Essential** packages to the
             client specification. You can disable this behavior by
             setting the :xml:attribute:`SourceType:essential`
             attribute to *false*:

        .. code-block:: none

            <Source type="apt" essential="false" ...>

Yum sources can be similarly specified:

.. code-block:: xml

    <Sources>
      <Group name="centos-5.2">
        <Source type="yum"
                url="http://mirror.centos.org/centos/"
                version="5.2">
          <Component>os</Component>
          <Component>updates</Component>
          <Component>extras</Component>
          <Arch>i386</Arch>
          <Arch>x86_64</Arch>
          <GPGKey>http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5</GPGKey>
        </Source>
      </Group>
    </Sources>

For sources with a :xml:attribute:`SourceType:url` attribute, the
:xml:attribute:`SourceType:version` attribute is also necessary.

:ref:`Pulp sources <pulp-source-support>` are very simple to specify
due to the amount of data that can be queried from Pulp itself:

.. code-block:: xml

    <Sources>
      <Group name="centos-6-x86_64">
        <Source type="yum" pulp_id="centos-6-x86_64-os"/>
        <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
        <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
      </Group>
    </Sources>

.. note:: There is also a rawurl attribute for specifying sources that
          don't follow the conventional layout.

          .. code-block:: xml

              <Sources>
                <Group name="centos5.4">
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.os">
                    <Arch>x86_64</Arch>
                  </Source>
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.updates">
                    <Arch>x86_64</Arch>
                  </Source>
                  <Source type="yum"
                          rawurl="http://mrepo.ices.utexas.edu/centos5-x86_64/RPMS.extras">
                    <Arch>x86_64</Arch>
                  </Source>
                </Group>
              </Sources>

          .. code-block:: xml

              <Sources>
                <Group name="ubuntu-lucid">
                  <Source type="apt"
                          rawurl="http://hudson-ci.org/debian/binary">
                    <Arch>amd64</Arch>
                  </Source>
                  <Source type="apt"
                          rawurl=http://hudson-ci.org/debian/binary">
                    <Arch>i386</Arch>
                  </Source>
                </Group>
              </Sources>

Configuration Updates
=====================

Packages will reload its configuration upon an explicit command via
bcfg2-admin::

    [0:3711] bcfg2-admin xcmd Packages.Refresh
    True

During this command (which will take some time depending on the quantity
and size of the sources listed in the configuration file), the server
will report information like::

    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz
    Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz
    ...
    Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz
    Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz

One line per file download needed. ``Packages/sources.xml`` will be
reloaded at this time, so any source specification changes (new or
modified sources in this file) will be reflected by the server at
this point.

This process is much, much faster if you use the :ref:`native yum
library support <native-yum-libraries>`.

Soft reload
-----------

.. versionadded:: 1.2.0

A soft reload can be performed to reread the configuration file and
download only missing sources.::

    [0:3711] bcfg2-admin xcmd Packages.Reload
    True

This is done automatically any time `sources.xml`_ is updated.

Availability
============

Support for the following clients is currently available. Support for
other package managers (Portage, Zypper, IPS, etc) remain to be added.

apt
---

All dpkg based clients (for example Debian, Ubuntu or Nexenta) could be
handled with the apt module:

.. code-block:: xml

    <Source type="apt"
            url="http://us.archive.ubuntu.com/ubuntu"
            version="intrepid">
      <Component>main</Component>
      <Component>universe</Component>
      <Arch>i386</Arch>
      <Arch>amd64</Arch>
    </Source>


pac
---

For Arch Linux or Parabola GNU/Linux-libre you could use the pac module
for packages. You do not need to supply a version attribute as the mirrors
are rolling release and does not supply different versions.

.. code-block:: xml

    <Source type="pac"
            url="http://mirrors.kernel.org/archlinux/">
      <Component>core</Component>
      <Component>extra</Component>
      <Component>community</Component>
      <Arch>i686</Arch>
      <Arch>x86_64</Arch>
    </Source>


pkgng
-----

The support for the Next Generation package management tool for FreeBSD
is called pkgng. It downloads the packagesite file from the mirror
and parses the dependencies out of it. It currently does not use the
DNS SRV record lookup mechanism to get the correct mirror and does
not verify the signature inside the packagesite file.

.. code-block:: xml

    <Source type="pkgng"
            url="http://pkg.freebsd.org/"
            version="10">
      <Component>latest</Component>
      <Arch>x86:64</Arch>
      <Arch>x86:32</Arch>
    </Source>


yum
---

Rpm based clients (for example RedHat, CentOS or Fedora) could be handled
with the yum module:

.. code-block:: xml

    <Source type="yum"
            url="http://mirror.centos.org/centos/"
            version="5.2">
      <Component>os</Component>
      <Component>updates</Component>
      <Component>extras</Component>
      <Arch>i386</Arch>
      <Arch>x86_64</Arch>
    </Source>


Package Checking and Verification
=================================

In order to do disable per-package verification, you will need to use
:ref:`BoundEntries <boundentries>`, e.g.:

.. code-block:: xml

    <BoundPackage name="mem-agent" priority="1" version="auto"
                  type="yum" verify="false"/>


.. _generating-client-configs:

Generating Client APT/Yum Configurations
========================================

The Packages plugin has native support for generating Yum and Apt
configs.  Simply add entries like these to the appropriate bundles:

.. code-block:: xml

    <Path name="/etc/yum.repos.d/bcfg2.repo"/>
    <Path name="/etc/apt/sources.list.d/bcfg2-packages-generated-sources.list"/>

If you want to change the path to either of those files, you can set
``yum_config`` or ``apt_config`` in ``bcfg2.conf`` to the path to the
config files you want to generate::

    [packages]
    yum_config=/etc/yum.repos.d/all.repo
    apt_config=/etc/apt/sources.d/all

If you need to distribute a config to different places on different
hosts, you can use the :ref:`server-plugins-structures-altsrc`
attribute, e.g.:

.. code-block:: xml

    <Bundle>
      <Group name="sles">
        <Path name="/etc/yum/yum.repos.d/bcfg2.repo"
              altsrc="/etc/yum.repos.d/bcfg2.repo"/>
      </Group>
      <Group name="sles" negate="true">
        <Path name="/etc/yum.repos.d/bcfg2.repo"/>
      </Group>
    </Bundle>

See :ref:`configuration` for more details on these options.

.. note::

    Support for generating Yum configs was added in 1.2.0, and Apt
    configs was added in 1.3.0.  Before that, you could use
    :ref:`server-plugins-generators-cfg-genshi` or
    :ref:`server-plugins-generators-cfg-cheetah` to generate your
    configs.

.. _native-yum-libraries:

Using Native Yum Libraries
==========================

.. versionadded:: 1.2.0

By default, Bcfg2 uses an internal implementation of Yum's dependency
resolution and other routines so that the Bcfg2 server can be run on a
host that does not support Yum itself. If you run the Bcfg2 server on
a machine that does have Yum libraries, however, you can enable use of
those native libraries in Bcfg2 by setting ``use_yum_libraries`` to
``1`` in the ``[packages:yum]`` section of ``bcfg2.conf``.

Benefits to this include:

* Much lower memory usage by the ``bcfg2-server`` process.
* Much faster ``Packages.Refresh`` behavior.
* More accurate dependency resolution.
* Better use of multiple processors/cores.

Drawbacks include:

* Resolution of package dependencies is slower and more
  resource-intensive.  At times it can be much slower, particularly
  after running ``Packages.Refresh``.
* More disk I/O. This can be alleviated by putting
  ``/var/lib/bcfg2/Packages/cache`` on tmpfs, but that offsets the
  lower memory usage.  In some cases, you may have to raise the open file
  limit for the user who runs your Bcfg2 server process, particularly
  if you have a lot of repositories.

Configuring the Yum Helper
--------------------------

Due to poor memory management by the Yum API, the long-lived
bcfg2-server process uses an external short-lived helper,
``bcfg2-yum-helper``, to do the actual Yum API calls for native yum
library support.  By default, Bcfg2 looks for this helper in
``$PATH``, or, failing that, at ``/usr/sbin/bcfg2-yum-helper``.  If
you have installed the helper elsewhere, you will need to configure
that location with the ``helper`` option in the ``[packages:yum]``
section, e.g.::

    [packages:yum]
    use_yum_libraries = 1
    helper = /usr/local/sbin/bcfg2-yum-helper

Setting Yum Options
-------------------

In ``bcfg2.conf``, any options you set in the ``[packages:yum]``
section other than ``use_yum_libraries`` and ``helper`` will be passed
along verbatim to the configuration of the Yum objects used in the
Bcfg2 server. The following options are set by default, and should not
generally be overridden:

* ``cachedir`` is set to a hashed value unique to each distinct Yum
  configuration. Don't set this unless you know what you're doing.
* ``keepcache`` is set to ``0``; there is no benefit to changing this.
* ``sslverify`` is set to ``0``; change this if you know what you're
  doing.
* ``reposdir`` is set to ``/dev/null`` to prevent the server's Yum
  configuration from being read; do not change this.

Package Groups
--------------

Yum package groups are supported by both the native Yum libraries and
Bcfg2's internal dependency resolver. To include a package group, use
the :xml:attribute:`PackageStructure:group` attribute of the
:xml:element:`Package` tag. You can use either the short group ID or
the long group name:

.. code-block:: xml

    <Package group="SNMP Support"/>
    <Package group="system-management-snmp"/>

By default, only those packages considered the "default" packages in a
group will be installed. You can change this behavior using the
:xml:attribute:`PackageStructure:type` attribute:

.. code-block:: xml

    <Package group="development" type="optional"/>
    <Package group="Administration Tools" type="mandatory"/>

Valid values of "type" are:

* ``mandatory``: Only install mandatory packages in the group.
* ``default``: Install default packages from the group (the default).
* ``optional`` or ``all``: Install all packages in the group,
  including mandatory, default, and optional packages.

See :xml:type:`PackageStructure` for details.

You can view the packages in a group by category with the ``yum
groupinfo`` command. More information about the different levels can
be found at
http://fedoraproject.org/wiki/How_to_use_and_edit_comps.xml_for_package_groups#Installation

Abstract Package Tags
---------------------

If you are using the native Yum libraries, the abstract Package tag
supports several attributes in addition to the standard
:xml:attribute:`PackageStructure:name`:

.. xml:type:: PackageStructure

.. _pulp-source-support:

Pulp Support
============

.. versionadded:: 1.2.0

Bcfg2 contains explicit support for repositories managed by Pulp
(http://pulpproject.org/).

.. note::

    Only the Pulp 1.x API is supported at this time.  When the Pulp
    2.x API is finalized support will be added for it.

Due to the amount of data about a repository that can be retrieved
directly from Pulp, the only thing necessary to configure a Pulp repo
is the repo ID, in :xml:attribute:`SourceType:pulp_id`:

.. code-block:: xml

    <Sources>
      <Group name="centos-6-x86_64">
        <Source type="yum" pulp_id="centos-6-x86_64-os"/>
        <Source type="yum" pulp_id="centos-6-x86_64-updates"/>
        <Source type="yum" pulp_id="centos-6-x86_64-extras"/>
      </Group>
    </Sources>

Pulp sources require some additional configuration. First, the Bcfg2
server must have a valid ``/etc/pulp/consumer/consumer.conf`` that is
readable by the user your Bcfg2 server runs as; the Pulp server,
URLs, and so on, are determined from this.

Secondly, in ``bcfg2.conf`` you must set the following
options in the ``[packages:pulp]`` section:

* ``username`` and ``password``: The username and password of a Pulp
  user that will be used to register new clients and bind them to
  repositories. Membership in the default ``consumer-users`` role is
  sufficient.

Bcfg2 clients using Pulp sources will be registered to the Pulp server
as consumers, and will be bound to the appropriate repositories.

Debugging unexpected behavior
=============================

.. versionadded:: 1.2.1

Using bcfg2-info
----------------

The dependency resolver used in Packages can be run in debug mode::

    $ bcfg2-info packageresolve foo.example.com bcfg2-server zlib
    ...
    2 initial packages
        bcfg2-server
        zlib
    54 new packages added
        sqlite
        less
        libxml2
        expat
        ...
    1 unknown packages
        libglib-2.0.so.0()(64bit)

This will show why the resolver is acting as it is. Replace
``foo.example.com`` and ``bcfg2-server`` with a client name and list
of packages, respectively.

Note that resolving a partial package list (as above) may result in
more unknown entries than you'd have otherwise; some of the package
drivers (Yum in particular) consider the full package list when
resolving multiple providers, and will not be able to properly resolve
some dependencies without a full package list.

You can also view the sources applicable to a client::

    $ bcfg2-info packagesources foo.example.com
    ...
    Name: centos-6-x86_64-updates
      Type: yum
      URL: http://mirror.example.com/centos-6-x86_64-updates
      GPG Key(s): http://mirror.example.com/centos-6-x86_64-updates/RPM-GPG-KEY-CentOS-6

    Name: centos-6-x86_64-os
      Type: yum
      URL: http://mirror.example.com/centos-6-x86_64-os
      GPG Key(s): http://mirror.example.com/centos-6-x86_64-os/RPM-GPG-KEY-CentOS-6

Using bcfg2-server
------------------

Once the server is started, enable debugging via bcfg2-admin::

    $ bcfg2-admin xcmd Packages.toggle_debug

TODO list
=========

* Zypper support
* Portage support

.. _configuration:

Configuration
=============

``bcfg2.conf`` contains miscellaneous configuration options for the
Packages plugin. Any booleans in the config file accept the values
"1", "yes", "true", and "on" for True, and "0", "no", "false", and
"off" for False.  For historical reasons, ``resolver`` and
``metadata`` also accept "enabled" and "disabled".

It understands the following directives:

[packages] section
------------------

+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| Name        | Description                                          | Values   | Default                                                           |
+=============+======================================================+==========+===================================================================+
| backends    | List of backends that should be loaded for the       | List     | Yum,Apt,Pac,Pkgng                                                 |
|             | dependency resolution.                               |          |                                                                   |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| resolver    | Enable dependency resolution                         | Boolean  | True                                                              |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| metadata    | Enable metadata processing. Disabling ``metadata``   | Boolean  | True                                                              |
|             | implies disabling ``resolver`` as well.              |          |                                                                   |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| yum_config  | The path at which to generate Yum configs.           | String   | /etc/yum.repos.d/bcfg2.repo                                       |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| apt_config  | The path at which to generate APT configs.           | String   | /etc/apt/sources.list.d/bcfg2-packages-generated-sources.list     |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| gpg_keypath | The path on the client RPM GPG keys will be copied   | String   | /etc/pki/rpm-gpg                                                  |
|             | to before they are imported on the client.           |          |                                                                   |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| version     | Set the version attribute used when binding Packages | any|auto | auto                                                              |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+
| cache       | Path where Packages will store its cache             | String   | <repo>/Packages/cache                                             |
+-------------+------------------------------------------------------+----------+-------------------------------------------------------------------+


[packages:yum] section
----------------------

+-------------------+----------------------------------------------------------+---------+-----------+
| Name              | Description                                              | Values  | Default   |
+===================+==========================================================+=========+===========+
| use_yum_libraries | Whether or not to use the                                | Boolean | False     |
|                   | :ref:`native yum library support <native-yum-libraries>` |         |           |
+-------------------+----------------------------------------------------------+---------+-----------+
| helper            | Path to ``bcfg2-yum-helper``                             | String  | See below |
+-------------------+----------------------------------------------------------+---------+-----------+

To find ``bcfg2-yum-helper`` if none is specified, Bcfg2 looks first
in ``$PATH`` and then in ``/usr/sbin/bcfg2-yum-helper`` for the
helper.

All other options in the ``[packages:yum]`` section will be passed
along verbatim to the Yum configuration if you are using the native
Yum library support.

[packages:pulp] section
-----------------------

+----------+-----------------------------------------------------+--------+---------+
| Name     | Description                                         | Values | Default |
+==========+=====================================================+========+=========+
| username | The username of a Pulp user that will be used to    | String | None    |
|          | register new clients and bind them to repositories. |        |         |
+----------+-----------------------------------------------------+--------+---------+
| password | The password of the Pulp user                       | String | None    |
+----------+-----------------------------------------------------+--------+---------+

The user should be a member of the default ``consumer-users`` role.