1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
|
.. -*- mode: rst -*-
.. _plugins-generators-packages:
========
Packages
========
This page documents the Packages plugin. Packages is an alternative to [wiki:Plugins/Pkgmgr Pkgmgr] for specifying package entries for clients. Where Pkgmgr explicitly specifies package entry information, Packages delegates control of package version information to the underlying package manager, installing the latest version available through those channels.
"Magic Groups"
==============
Packages is the only plugin that uses "magic groups". Most plugins operate based on client group memberships, without any concern for the particular names chosen for groups by the user. The Packages plugin is the sole exception to this rule. Packages needs to "know" two different sorts of facts about clients. The first is the basic OS/distro of the client, enabling classes of sources. The second is the architecture of the client, enabling sources for a given architecture. In addition to these magic groups, each source may also specify a non-magic group to limit the source's applicability to group member clients.
+-----------+----------+--------------+
| Source | OS Group | Architecture |
+===========+==========+==============+
| APTSource | debian | i386 |
+-----------+----------+--------------+
| APTSource | ubuntu | amd64 |
+-----------+----------+--------------+
| APTSource | nexenta | |
+-----------+----------+--------------+
| YUMSource | redhat | i386 |
+-----------+----------+--------------+
| YUMSource | centos | x86_64 |
+-----------+----------+--------------+
Limiting sources to groups
==========================
Each source can also specify explicit group memberships. In the following example, the ubuntu-hardy group is also required. Finally, clients must be a member of the appropriate architecture group, in this case, either i386 or amd64. In total, in order for this source to be associated with a client is for the client to be in one of the sentinel groups (debian, ubuntu, or nexenta), the explicit group ubuntu-hardy, and any of the architecture groups (i386 or amd64).
Memberships in architecture groups is needed so that Packages can map software sources to clients. There is no other way to handle this than to impose
When multiple sources are specified, clients are associated with each source to which they apply (based on group memberships, as described above). Packages and dependencies are resolved from all applicable sources.
Setup
=====
Three basic steps are required for Packages to work properly.
1. Create Packages/config.xml. This file should look approximately like the example below, and describes both which software repositories should be used, and which clients are eligible to use each one.
2. Ensure that clients are members of the proper groups. Each client should be a member of one of the sentinel groups listed above (debian/ubuntu/redhat/centos/nexenta), all of the groups listed in the source (like ubuntu-intrepid or centos-5.2 in the following examples), and one of the architecture groups listed in the source configuration (i386, amd64 or x86_64 in the following examples). '''Failure to do this will result in the source either not applying to the client, or only architecture independent packages being made available to the client.'''
3. Add Package entries to bundles.
4. Sit back and relax, as dependencies are resolved, and automatically added to client configurations.
Prerequisite Resolution
=======================
Packages provides a prerequisite resolution mechanism which has no analogue in Pkgmgr. During configuration generation, all structures are processed. After this phase, but before entry binding, a list of packages and the client metadata instance is passed into Packages' resolver. This process determines a superset of packages that will fully satisfy dependencies of all package entries included in structures, and reports any prerequisites that cannot be satisfied. This facility should largely remove the need to use the [wiki:Plugins/Base Base] plugin.
Example usage
=============
Create a config.xml file in the Packages directory that looks something like this:
.. code-block:: xml
<Sources>
<APTSource>
<Group>ubuntu-intrepid</Group>
<URL>http://us.archive.ubuntu.com/ubuntu</URL>
<Version>intrepid</Version>
<Component>main</Component>
<Component>universe</Component>
<Arch>i386</Arch>
<Arch>amd64</Arch>
</APTSource>
</Sources>
Yum sources can be similarly specified:
.. code-block:: xml
<Sources>
<YUMSource>
<Group>centos-5.2</Group>
<URL>http://mirror.centos.org/centos/</URL>
<Version>5.2</Version>
<Component>os</Component>
<Component>updates</Component>
<Component>extras</Component>
<Arch>i386</Arch>
<Arch>x86_64</Arch>
</YUMSource>
</Sources>
Configuration Updates
=====================
Packages will reload its configuration upon an explicit command via bcfg2-admin.::
[0:3711] bcfg2-admin xcmd Packages.Refresh
True
During this command (which will take some time depending on the quantity and size of the sources listed in the configuration file), the server will report information like::
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/main/binary-amd64/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-i386/Packages.gz
Packages: Updating http://mirror.anl.gov/ubuntu//dists/jaunty/universe/binary-amd64/Packages.gz
...
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/filelists.xml.gz
Packages: Updating http://mirror.centos.org/centos/5/extras/x86_64/repodata/primary.xml.gz
Once line per file download needed. Packages/config.xml will be reloaded at this time, so any source specification changes (new or modified sources in this file) will be reflected by the server at this point.
Availability
============
Packages was added in 1.0pre1, with APT support. YUM support was added for 1.0pre2. Support for other package managers (Portage, Zypper, IPS, etc) remain to be added.
Validation
==========
A schema for Packages/config.xml is included; config.xml can be validated using bcfg2-repo-validate. Note that the schema requires that elements be specified in the above order.
Limitations
===========
Packages does not do traditional caching as other plugins do. Changes to the Packages config file require a server restart for the time being.
Package Verification
====================
In order to do disable per-package verification Pkgmgr style, you will need to use [wiki:BoundEntry BoundEntries] like below
.. code-block:: xml
<BoundPackage name="mem-agent" priority="1" version="auto" type="yum" verify="false"/>
Debugging unexpected behavior
=============================
Using bcfg2-info
----------------
The dependency resolver used in Packages can be run in debug mode::
$ bcfg2-info
...
Handled 20 events in 0.004s
> debug
dropping to python interpreter; press ^D to resume
...
(debug_shell)
>>> m = self.build_metadata('ubik3')
>>> self.plugins['Packages'].complete(m, ['ssh'], debug=True)
Package ssh: adding new deps ['openssh-client', 'openssh-server']
Package openssh-server: adding new deps ['libc6', 'libcomerr2', 'libkrb53', 'libpam0g', 'libselinux1', 'libssl0.9.8
', 'libwrap0', 'zlib1g', 'debconf', 'libpam-runtime', 'libpam-modules', 'adduser', 'dpkg', 'lsb-base']
Package debconf: adding new deps ['debconf-i18n']
Package libpam-modules: adding new deps ['libdb4.7']
Package openssh-client: adding new deps ['libedit2', 'libncurses5', 'passwd']
Package lsb-base: adding new deps ['sed', 'ncurses-bin']
Package adduser: adding new deps ['perl-base']
Package debconf-i18n: adding new deps ['liblocale-gettext-perl', 'libtext-iconv-perl', 'libtext-wrapi18n-perl', 'libtext-charwidth-perl']
Package passwd: adding new deps ['debianutils']
Package libtext-charwidth-perl: adding new deps ['perlapi-5.10.0']
VPackage perlapi-5.10.0: got provides ['perl-base']
Package libkrb53: adding new deps ['libkeyutils1']
Package libtext-iconv-perl: adding new deps ['perlapi-5.10.0']
Package libc6: adding new deps ['libgcc1', 'findutils']
Package libgcc1: adding new deps ['gcc-4.3-base']
(set(['debconf', 'libgcc1', 'lsb-base', 'libtext-wrapi18n-perl', 'libtext-iconv-perl', 'sed', 'passwd', 'findutils', 'libpam0g', 'openssh-client', 'debconf-i18n', 'libselinux1', 'zlib1g', 'adduser', 'libwrap0', 'ncurses-bin', 'libssl0.9.8', 'liblocale-gettext-perl', 'libkeyutils1', 'libpam-runtime', 'libpam-modules', 'openssh-server', 'libkrb53', 'ssh', 'libncurses5', 'libc6', 'libedit2', 'libcomerr2', 'dpkg', 'perl-base', 'libdb4.7', 'libtext-charwidth-perl', 'gcc-4.3-base', 'debianutils']), set([]), 'deb')
This will show why the resolver is acting as it is. Replace "ubik3" and ['ssh'] with a client name and list of packages, respectively. Also, a more polished interface to this functionality is coming as well.
Each line starting with Package: <name> describes a set of new prerequisites pulled in by a package. Lines starting with VPackage <vname> describe provides entries and their mappings to required names. The last line describes the overall results of the resolver, with three fields: a list of packages that should be installed, a list of unresolved requirements, and a type for these packages.
Using bcfg2-server
------------------
Once the server is started, enable debugging via bcfg2-admin::
$ bcfg2-admin xcmd Packages.toggle_debug
TODO list
=========
* Zypper support
* Portage support
* Explicit version pinning (a la Pkgmgr)
Developing for Packages
=======================
In order to support a given client package tool driver, that driver must support use of the auto value for the version attribute in Package entries. In this case, the tool driver views the current state of available packages, and uses the underlying package manager's choice of correct package version in lieu of an explicit, centrally-specified, version. This support enables Packages to provide a list of Package entries with version='auto'. Currently, the APT and YUMng drivers support this feature. Note that package management systems without any network support cannot operate in this fashion, so RPMng and SYSV will never be able to use Packages. Emerge, Zypper, IPS, and Blastwave all have the needed features to be supported by Packages, but support has not yet been written.
Packages fills two major functions in configuration generation. The first is to provide entry level binding support for Package entries included in client configurations. This function is quite easy to implement; Packages determines (based on client group membership) if the package is available for the client system, and which type it has. Because version='auto' is used, no version determination needs to be done.
The second major function is more complex. Packages ensures that client configurations include all package-level prerequisites for package entries explicitly included in the configuration. In order to support this, Packages needs to directly process network data for package management systems (the network sources for apt or yum, for examples), process these files, and build data structures describing prerequisites and the providers of those functions/paths. To simplify implementations of this, there is a generic base class (Bcfg2.Server.Plugins.Packages.Source) that provides a framework for fetching network data via HTTP, processing those sources (with subclass defined methods for processing the specific format provided by the tool), a generic dependency resolution method, and a caching mechanism that greatly speeds up server/bcfg2-info startup.
Each source type must define:
* a get_urls attribute (and associated urls property) that describes the URLS where to get data from.
* a read_files method that reads and processes the downloaded files
Sources may define a get_provides method, if provides are complex. For example, provides in rpm can be either rpm names or file paths, so multiple data sources need to be multiplexed.
The APT source in src/lib/Server/Plugins/Packages.py provides a relatively simple implementation of a source.
|
