diff options
| author | Chris St. Pierre <chris.a.st.pierre@gmail.com> | 2012-07-30 10:24:12 -0400 |
|---|---|---|
| committer | Chris St. Pierre <chris.a.st.pierre@gmail.com> | 2012-07-30 10:24:12 -0400 |
| commit | 8b438fda3ae2d9516dbfb6014c280b68036c17e1 (patch) | |
| tree | b8acb22b313e4b57797a227b42f69b95b54bd976 | |
| parent | 7a008a0b1b4d3b819da5a6544ac15faab3cbb28a (diff) | |
| download | bcfg2-8b438fda3ae2d9516dbfb6014c280b68036c17e1.tar.gz bcfg2-8b438fda3ae2d9516dbfb6014c280b68036c17e1.tar.bz2 bcfg2-8b438fda3ae2d9516dbfb6014c280b68036c17e1.zip | |
Metadata and other improvements:
* Added support for Client tag in groups.xml
* Added support for nested Group tags in groups.xml
* Added support for negated groups in groups.xml
* Added DatabaseBacked plugin mixin to easily allow plugins to connect
to a database specified in global database settings in bcfg2.conf
* Added DBMetadata plugin that uses relational DB to store client
records instead of writing to clients.xml
29 files changed, 1737 insertions, 605 deletions
diff --git a/doc/appendix/guides/authentication.txt b/doc/appendix/guides/authentication.txt index dab122f80..68a232f6f 100644 --- a/doc/appendix/guides/authentication.txt +++ b/doc/appendix/guides/authentication.txt @@ -62,7 +62,7 @@ How Authentication Works #. Next, the ip address is verified against the client record. If the address doesn't match, then the client must be set to - location=floating + floating='true' #. Finally, the password is verified. If the client is set to secure mode, the only its per-client password is accepted. If it is not set diff --git a/doc/appendix/guides/nat_howto.txt b/doc/appendix/guides/nat_howto.txt index 5bd3f7b13..b3492e871 100644 --- a/doc/appendix/guides/nat_howto.txt +++ b/doc/appendix/guides/nat_howto.txt @@ -44,7 +44,7 @@ the Client entry in clients.xml will look something like this: .. code-block:: xml <Client profile="desktop" name="test1" - uuid='9001ec29-1531-4b16-8198-a71bea093d0a' location='floating'/> + uuid='9001ec29-1531-4b16-8198-a71bea093d0a' floating='true'/> Alternatively, the Client entry can be setup like this: diff --git a/doc/server/backends.txt b/doc/server/backends.txt index 49bfe3b96..71ecac10b 100644 --- a/doc/server/backends.txt +++ b/doc/server/backends.txt @@ -2,9 +2,9 @@ .. _server-backends: -======== -Backends -======== +=============== +Server Backends +=============== .. versionadded:: 1.3.0 diff --git a/doc/server/database.txt b/doc/server/database.txt new file mode 100644 index 000000000..8094e9c5e --- /dev/null +++ b/doc/server/database.txt @@ -0,0 +1,45 @@ +.. -*- mode: rst -*- + +.. _server-database: + +======================== +Global Database Settings +======================== + +.. versionadded:: 1.3.0 + +Several Bcfg2 plugins, including +:ref:`server-plugins-grouping-dbmetadata` and +:ref:`server-plugins-probes-index`, can connect use a relational +database to store data. They use the global database settings in +``bcfg2.conf``, described in this document, to connect. + +.. note:: + + The :ref:`server-plugins-statistics-dbstats` plugin and the + :ref:`reports-dynamic` do *not* currently use the global database + settings. They use their own separate database configuration. + +Configuration Options +===================== + +All of the following options should go in the ``[database]`` section +of ``/etc/bcfg2.conf``. + ++-------------+------------------------------------------------------------+-------------------------------+ +| Option name | Description | Default | ++=============+============================================================+===============================+ +| engine | The full name of the Django database backend to use. See | "django.db.backends.sqlite3" | +| | https://docs.djangoproject.com/en/dev/ref/settings/#engine | | +| | for available options | | ++-------------+------------------------------------------------------------+-------------------------------+ +| name | The name of the database | "/var/lib/bcfg2/bcfg2.sqlite" | ++-------------+------------------------------------------------------------+-------------------------------+ +| user | The user to connect to the database as | None | ++-------------+------------------------------------------------------------+-------------------------------+ +| password | The password to connect to the database with | None | ++-------------+------------------------------------------------------------+-------------------------------+ +| host | The host to connect to | "localhost" | ++-------------+------------------------------------------------------------+-------------------------------+ +| port | The port to connect to | None | ++-------------+------------------------------------------------------------+-------------------------------+ diff --git a/doc/server/index.txt b/doc/server/index.txt index 6c2b7b889..1b832dbee 100644 --- a/doc/server/index.txt +++ b/doc/server/index.txt @@ -30,3 +30,4 @@ clients. bcfg2-info selinux backends + database diff --git a/doc/server/plugins/generators/tgenshi/clientsxml.txt b/doc/server/plugins/generators/tgenshi/clientsxml.txt index 7a8d1fcc4..87d6d728a 100644 --- a/doc/server/plugins/generators/tgenshi/clientsxml.txt +++ b/doc/server/plugins/generators/tgenshi/clientsxml.txt @@ -65,7 +65,7 @@ Possible improvements: name="${name}" uuid="${name}" password="${metadata.Properties['passwords.xml'].xdata.find('password').find('bcfg2-client').find(name).text}" - location="floating" + floating="true" secure="true" />\ {% end %}\ diff --git a/doc/server/plugins/grouping/dbmetadata.txt b/doc/server/plugins/grouping/dbmetadata.txt new file mode 100644 index 000000000..292367f0c --- /dev/null +++ b/doc/server/plugins/grouping/dbmetadata.txt @@ -0,0 +1,39 @@ +.. -*- mode: rst -*- + +.. _server-plugins-grouping-dbmetadata: + +========== +DBMetadata +========== + +.. versionadded:: 1.3.0 + +The DBMetadata plugin is an alternative to the +:ref:`server-plugins-grouping-metadata` plugin that stores client +records in a database rather than writing back to ``clients.xml``. +This provides several advantages: + +* ``clients.xml`` will never be written by the server, removing an + area of contention between the user and server. +* ``clients.xml`` can be removed entirely for many sites. +* The Bcfg2 client list can be queried by other machines without + obtaining and parsing ``clients.xml``. +* A single client list can be shared amongst multiple Bcfg2 servers. + +In general, DBMetadata works almost the same as Metadata. +``groups.xml`` is parsed identically. If ``clients.xml`` is present, +it is parsed, but ``<Client>`` tags in ``clients.xml`` *do not* assert +client existence; they are only used to set client options *if* the +client exists (in the database). That is, the two purposes of +``clients.xml`` -- to track which clients exist, and to set client +options -- have been separated. + +With the improvements in ``groups.xml`` parsing in 1.3, client groups +can now be set directly in ``groups.xml`` with ``<Client>`` tags. (See +:ref:`metadata-client-tag` for more details.) As a result, +``clients.xml`` is only necessary with DBMetadata if you need to set +options (e.g., aliases, floating clients, per-client passwords, etc.) +on clients. + +DBMetadata uses the :ref:`Global Server Database Settings +<server-database>` to connect to its database. diff --git a/doc/server/plugins/grouping/metadata.txt b/doc/server/plugins/grouping/metadata.txt index 2c05e9e7e..1ab3b9c05 100644 --- a/doc/server/plugins/grouping/metadata.txt +++ b/doc/server/plugins/grouping/metadata.txt @@ -6,11 +6,11 @@ Metadata ======== -The metadata mechanism has two types of information, client metadata and -group metadata. The client metadata describes which top level group a -client is associated with.The group metadata describes groups in terms -of what bundles and other groups they include. Each aspect grouping -and clients' memberships are reflected in the ``Metadata/groups.xml`` and +The metadata mechanism has two types of information, client metadata +and group metadata. The client metadata describes which top level +group a client is associated with.The group metadata describes groups +in terms of what bundles and other groups they include. Group data and +clients' memberships are reflected in the ``Metadata/groups.xml`` and ``Metadata/clients.xml`` files, respectively. Usage of Groups in Metadata @@ -85,9 +85,9 @@ Additionally, the following properties can be specified: | address | Establishes an extra IP address that | ip address | | | resolves to this client. | | +----------+----------------------------------------+----------------+ -| location | Requires requests to come from an IP | fixed|floating | -| | address that matches the client | | -| | record. | | +| floating | Allows requests to come from any IP, | true|false | +| | rather than requiring requests to come | | +| | from an IP associated with the client | | +----------+----------------------------------------+----------------+ | password | Establishes a per-node password that | String | | | can be used instead of the global | | @@ -101,6 +101,9 @@ Additionally, the following properties can be specified: | | resolution. | | +----------+----------------------------------------+----------------+ +Floating can also be configured by setting ``location="floating"``, +but that is deprecated. + For detailed information on client authentication see :ref:`appendix-guides-authentication` @@ -112,31 +115,88 @@ definitions. Here's a simple ``Metadata/groups.xml`` file: .. code-block:: xml - <Groups version='3.0'> + <Groups> <Group name='mail-server' profile='true' - public='false' comment='Top level mail server group' > <Bundle name='mail-server'/> <Bundle name='mailman-server'/> <Group name='apache-server'/> - <Group name='rhel-as-5-x86'/> <Group name='nfs-client'/> <Group name='server'/> + <Group name='rhel5'> + <Group name='sendmail-server'/> + </Group> + <Group name='rhel6'> + <Group name='postfix-server'/> + </Group> + </Group> + <Group name='rhel'> + <Group name='selinux-enabled'/> </Group> - <Group name='rhel-as-5-x86'> - <Group name='rhel'/> + <Group name='oracle-server'> + <Group name='selinux-enabled' negate='true'/> </Group> - <Group name='apache-server'/> - <Group name='nfs-client'/> - <Group name='server'/> - <Group name='rhel'/> + <Client name='foo.eample.com'> + <Group name='oracle-server'/> + <Group name='apache-server'/> + </Client> </Groups> +A Group or Client tag that does not contain any child tags is a +declaration of membership; a Group or Client tag that does contain +children is a conditional. So the example above does not assign +either the ``rhel5`` or ``rhel6`` groups to machines in the +``mail-server`` group, but conditionally assigns the +``sendmail-server`` or ``postfix-server`` groups depending on the OS +of the client. (Presumably in this example the OS groups are set by a +probe.) + +Consequently, a client that is RHEL 5 and a member of the +``mail-server`` profile group would also be a member of the +``apache-server``, ``nfs-client``, ``server``, and ``sendmail-server`` +groups; a RHEL 6 client that is a member of the ``mail-server`` +profile group would be a member of the ``apache-server``, +``nfs-client``, ``server``, and ``postfix-server`` groups. + +Client tags in ``groups.xml`` allow you to supplement the profile +group declarations in ``clients.xml`` and/or client group assignments +with the :ref:`server-plugins-grouping-grouppatterns` plugin. They +should be used sparingly. (They are more useful with the +:ref:`server-plugins-grouping-dbmetadata` plugin.) + +You can also declare that a group should be negated; this allows you +to set defaults and override them efficiently. Negation is applied +after other group memberships are calculated, so it doesn't matter how +many times a client is assigned to a group or how many times it is +negated; a single group negation is sufficient to remove a client from +that group. For instance, in the following example, +``foo.example.com`` is **not** a member of ``selinux-enabled``, even +though it is a member of the ``foo-server`` and ``every-server`` +groups: + +.. code-block:: xml + + <Groups> + <Group name="foo-server"> + <Group name="apache-server"/> + <Group name="selinux-enabled"/> + </Group> + <Group name="apache-server"> + <Group name="selinux-enabled"/> + </Group> + <Group name="every-server"> + <Group name="selinux-enabled"/> + </Group> + <Client name="foo.example.com"> + <Group name="selinux-enabled" negate="true"/> + </Client> + +.. note:: -Nested/chained groups definitions are conjunctive (logical and). For -instance, in the above example, a client associated with the Profile -Group ``mail-server`` is also a member of the ``apache-server``, -``rhel-as-5-x86``, ``nfs-client``, ``server``, and ``rhel`` groups. + Nested Group conditionals, Client tags, and negated Group tags are + all new in 1.3.0. + +Order of ``groups.xml`` does not matter. Groups describe clients in terms for abstract, disjoint aspects. Groups can be combined to form complex descriptions of clients that use @@ -165,33 +225,63 @@ Metadata Group Tag The Group Tag has the following possible attributes: -+----------+------------------------------------------+--------------+ -| Name | Description | Values | -+==========+==========================================+==============+ -| name | Name of the group | String | -+----------+------------------------------------------+--------------+ -| profile | If a client can be directly associated | True|False | -| | with this group | | -+----------+------------------------------------------+--------------+ -| public | If a client can freely associate itself | True|False | -| | with this group. For use with the | | -| | *bcfg2 -p* option on the client. | | -+----------+------------------------------------------+--------------+ -| category | A group can only contain one instance of | String | -| | a group in any one category. This | | -| | provides the basis for representing | | -| | groups which are conjugates of one | | -| | another in a rigorous way. It also | | -| | provides the basis for negation. | | -+----------+------------------------------------------+--------------+ -| default | Set as the profile to use for clients | True|False | -| | that are not associated with a profile | | -| | in ``clients.xml`` | | -+----------+------------------------------------------+--------------+ -| comment | English text description of group | String | -+----------+------------------------------------------+--------------+ - -Groups can also contain other groups and bundles. ++----------+----------------------------------------------+--------------+ +| Name | Description | Values | ++==========+==============================================+==============+ +| name | Name of the group | String | ++----------+----------------------------------------------+--------------+ +| profile | If a client can be directly associated with | True|False | +| | this group | | ++----------+----------------------------------------------+--------------+ +| public | If a client can freely associate itself with | True|False | +| | this group. For use with the ``bcfg2 -p`` | | +| | option on the client. | | ++----------+----------------------------------------------+--------------+ +| category | A group can only contain one instance of a | String | +| | group in any one category. This provides the | | +| | basis for representing groups which are | | +| | conjugates of one another in a rigorous way. | | +| | way. | ++----------+----------------------------------------------+--------------+ +| default | Set as the profile to use for clients that | True|False | +| | are not associated with a profile in | | +| | ``clients.xml`` | | ++----------+----------------------------------------------+--------------+ +| comment | English text description of group | String | ++----------+----------------------------------------------+--------------+ +| negate | When used as a conditional, only apply the | True|False | +| | children if the named group does not match. | | +| | When used as a declaration, do not apply | | +| | the named group to matching clients. | | ++----------+----------------------------------------------+--------------+ + +The ``profile``, ``public``, ``category``, ``default``, and +``comment`` attributes are only parsed if a Group tag either a) is the +direct child of a Groups tag (i.e., at the top level of an XML file); +or b) has no children. This matches legacy behavior in Bcfg2 1.2 and +earlier. + +Groups can also contain other groups, clients, and bundles. + +.. _metadata-client-tag: + +Metadata Client Tag +------------------- + +The Client Tag has the following possible attributes: + ++----------+-----------------------------------------------+--------------+ +| Name | Description | Values | ++==========+===============================================+==============+ +| name | Name of the client | String | ++----------+-----------------------------------------------+--------------+ +| negate | Only apply the child tags if the named client | True|False | +| | does not match. | | ++----------+-----------------------------------------------+--------------+ + +Clients can also contain groups, other clients (although that's likely +nonsensical), and bundles. + Use of XInclude =============== diff --git a/schemas/clients.xsd b/schemas/clients.xsd index d50f3626e..3b98c5fc3 100644 --- a/schemas/clients.xsd +++ b/schemas/clients.xsd @@ -26,7 +26,8 @@ <xsd:attribute type='xsd:string' name='uuid'/> <xsd:attribute type='xsd:string' name='password'/> <xsd:attribute type='xsd:string' name='location'/> - <xsd:attribute type='xsd:string' name='secure'/> + <xsd:attribute type='xsd:boolean' name='floating'/> + <xsd:attribute type='xsd:boolean' name='secure'/> <xsd:attribute type='xsd:string' name='pingtime' use='optional'/> <xsd:attribute type='xsd:string' name='address'/> <xsd:attribute type='xsd:string' name='version'/> diff --git a/schemas/metadata.xsd b/schemas/metadata.xsd index f79039d25..c3cb46b28 100644 --- a/schemas/metadata.xsd +++ b/schemas/metadata.xsd @@ -1,6 +1,6 @@ <xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xi="http://www.w3.org/2001/XInclude" xml:lang="en"> - + <xsd:annotation> <xsd:documentation> metadata schema for bcfg2 @@ -13,38 +13,51 @@ <xsd:import namespace="http://www.w3.org/2001/XInclude" schemaLocation="xinclude.xsd"/> + <xsd:complexType name='bundleDeclaration'> + <xsd:attribute type='xsd:string' name='name' use='required'/> + </xsd:complexType> + <xsd:complexType name='groupType'> <xsd:choice minOccurs='0' maxOccurs='unbounded'> - <xsd:element name='Bundle'> - <xsd:complexType> - <xsd:attribute type='xsd:string' name='name' use='required'/> - </xsd:complexType> - </xsd:element> - <xsd:element name='Group' > - <xsd:complexType> - <xsd:attribute name='name' use='required'/> - </xsd:complexType> - </xsd:element> + <xsd:element name='Bundle' type='bundleDeclaration'/> + <xsd:element name='Group' type='groupType'/> + <xsd:element name='Client' type='clientType'/> + <xsd:element name='Groups' type='groupsType'/> + <xsd:element name='Options' type='optionsType'/> + </xsd:choice> + <xsd:attribute type='xsd:string' name='name' use='required'/> + <xsd:attribute type='xsd:boolean' name='profile'/> + <xsd:attribute type='xsd:boolean' name='public'/> + <xsd:attribute type='xsd:boolean' name='default'/> + <xsd:attribute type='xsd:string' name='auth'/> + <xsd:attribute type='xsd:string' name='category'/> + <xsd:attribute type='xsd:string' name='comment'/> + <xsd:attribute type='xsd:string' name='negate'/> + </xsd:complexType> + + <xsd:complexType name='clientType'> + <xsd:choice minOccurs='0' maxOccurs='unbounded'> + <xsd:element name='Bundle' type='bundleDeclaration'/> + <xsd:element name='Group' type='groupType'/> + <xsd:element name='Client' type='clientType'/> + <xsd:element name='Groups' type='groupsType'/> + <xsd:element name='Options' type='optionsType'/> </xsd:choice> - <xsd:attribute type='xsd:boolean' name='profile' use='optional'/> - <xsd:attribute type='xsd:boolean' name='public' use='optional'/> - <xsd:attribute type='xsd:boolean' name='default' use='optional'/> <xsd:attribute type='xsd:string' name='name' use='required'/> - <xsd:attribute type='xsd:string' name='auth' use='optional'/> - <xsd:attribute type='xsd:string' name='category' use='optional'/> - <xsd:attribute type='xsd:string' name='comment' use='optional'/> + <xsd:attribute type='xsd:string' name='negate'/> </xsd:complexType> <xsd:complexType name='groupsType'> <xsd:choice minOccurs='0' maxOccurs='unbounded'> <xsd:element name='Group' type='groupType'/> + <xsd:element name='Client' type='clientType'/> <xsd:element name='Groups' type='groupsType'/> <xsd:element ref="xi:include"/> </xsd:choice> <xsd:attribute name='version' type='xsd:string'/> - <xsd:attribute name='origin' type='xsd:string'/> - <xsd:attribute name='revision' type='xsd:string'/> - <xsd:attribute ref='xml:base'/> + <xsd:attribute name='origin' type='xsd:string'/> + <xsd:attribute name='revision' type='xsd:string'/> + <xsd:attribute ref='xml:base'/> </xsd:complexType> <xsd:element name='Groups' type='groupsType'/> diff --git a/src/lib/Bcfg2/Options.py b/src/lib/Bcfg2/Options.py index fe1bad110..fb36a985b 100644 --- a/src/lib/Bcfg2/Options.py +++ b/src/lib/Bcfg2/Options.py @@ -224,6 +224,7 @@ def get_bool(s): return False else: raise ValueError + """ Options: @@ -424,6 +425,32 @@ SERVER_BACKEND = \ default='best', cf=('server', 'backend')) +# database options +DB_ENGINE = \ + Option('Database engine', + default='django.db.backends.sqlite3', + cf=('database', 'engine')) +DB_NAME = \ + Option('Database name', + default=os.path.join(SERVER_REPOSITORY.default, "bcfg2.sqlite"), + cf=('database', 'name')) +DB_USER = \ + Option('Database username', + default=None, + cf=('database', 'user')) +DB_PASSWORD = \ + Option('Database password', + default=None, + cf=('database', 'password')) +DB_HOST = \ + Option('Database host', + default='localhost', + cf=('database', 'host')) +DB_PORT = \ + Option('Database port', + default='', + cf=('database', 'port'),) + # Client options CLIENT_KEY = \ Option('Path to SSL key', @@ -898,12 +925,15 @@ class OptionParser(OptionSet): OptionParser bootstraps option parsing, getting the value of the config file """ - def __init__(self, args, argv=None): + def __init__(self, args, argv=None, quiet=False): if argv is None: argv = sys.argv[1:] + # the bootstrap is always quiet, since it's running with a + # default config file and so might produce warnings otherwise self.Bootstrap = OptionSet([('configfile', CFILE)], quiet=True) self.Bootstrap.parse(argv, do_getopt=False) - OptionSet.__init__(self, args, configfile=self.Bootstrap['configfile']) + OptionSet.__init__(self, args, configfile=self.Bootstrap['configfile'], + quiet=quiet) self.optinfo = copy.copy(args) def HandleEvent(self, event): diff --git a/src/lib/Bcfg2/Server/Admin/Bundle.py b/src/lib/Bcfg2/Server/Admin/Bundle.py index 89c099602..ab07e29b3 100644 --- a/src/lib/Bcfg2/Server/Admin/Bundle.py +++ b/src/lib/Bcfg2/Server/Admin/Bundle.py @@ -8,12 +8,11 @@ from Bcfg2.Server.Plugins.Metadata import MetadataConsistencyError class Bundle(Bcfg2.Server.Admin.MetadataCore): - __shorthelp__ = "Create or delete bundle entries" - # TODO: add/del functions + __shorthelp__ = "List and view bundle entries" __longhelp__ = (__shorthelp__ + "\n\nbcfg2-admin bundle list-xml" "\nbcfg2-admin bundle list-genshi" "\nbcfg2-admin bundle show\n") - __usage__ = ("bcfg2-admin bundle [options] [add|del] [group]") + __usage__ = ("bcfg2-admin bundle [options] [list-xml|list-genshi|show]") def __call__(self, args): Bcfg2.Server.Admin.MetadataCore.__call__(self, args) @@ -28,18 +27,6 @@ class Bundle(Bcfg2.Server.Admin.MetadataCore): if len(args) == 0: self.errExit("No argument specified.\n" "Please see bcfg2-admin bundle help for usage.") -# if args[0] == 'add': -# try: -# self.metadata.add_bundle(args[1]) -# except MetadataConsistencyError: -# print("Error in adding bundle.") -# raise SystemExit(1) -# elif args[0] in ['delete', 'remove', 'del', 'rm']: -# try: -# self.metadata.remove_bundle(args[1]) -# except MetadataConsistencyError: -# print("Error in deleting bundle.") -# raise SystemExit(1) # Lists all available xml bundles elif args[0] in ['list-xml', 'ls-xml']: bundle_name = [] diff --git a/src/lib/Bcfg2/Server/Admin/Client.py b/src/lib/Bcfg2/Server/Admin/Client.py index 734e9573d..34dfd7550 100644 --- a/src/lib/Bcfg2/Server/Admin/Client.py +++ b/src/lib/Bcfg2/Server/Admin/Client.py @@ -4,50 +4,23 @@ from Bcfg2.Server.Plugins.Metadata import MetadataConsistencyError class Client(Bcfg2.Server.Admin.MetadataCore): - __shorthelp__ = "Create, delete, or modify client entries" + __shorthelp__ = "Create, delete, or list client entries" __longhelp__ = (__shorthelp__ + "\n\nbcfg2-admin client add <client> " - "attr1=val1 attr2=val2" - "\nbcfg2-admin client update <client> " - "attr1=val1 attr2=val2" "\nbcfg2-admin client list" "\nbcfg2-admin client del <client>\n") - __usage__ = ("bcfg2-admin client [options] [add|del|update|list] [attr=val]") + __usage__ = ("bcfg2-admin client [options] [add|del|list] [attr=val]") def __call__(self, args): Bcfg2.Server.Admin.MetadataCore.__call__(self, args) if len(args) == 0: self.errExit("No argument specified.\n" - "Please see bcfg2-admin client help for usage.") + "Usage: %s" % self.usage) if args[0] == 'add': - attr_d = {} - for i in args[2:]: - attr, val = i.split('=', 1) - if attr not in ['profile', 'uuid', 'password', - 'location', 'secure', 'address', - 'auth']: - print("Attribute %s unknown" % attr) - raise SystemExit(1) - attr_d[attr] = val try: - self.metadata.add_client(args[1], attr_d) + self.metadata.add_client(args[1]) except MetadataConsistencyError: print("Error in adding client") raise SystemExit(1) - elif args[0] in ['update', 'up']: - attr_d = {} - for i in args[2:]: - attr, val = i.split('=', 1) - if attr not in ['profile', 'uuid', 'password', - 'location', 'secure', 'address', - 'auth']: - print("Attribute %s unknown" % attr) - raise SystemExit(1) - attr_d[attr] = val - try: - self.metadata.update_client(args[1], attr_d) - except MetadataConsistencyError: - print("Error in updating client") - raise SystemExit(1) elif args[0] in ['delete', 'remove', 'del', 'rm']: try: self.metadata.remove_client(args[1]) @@ -55,7 +28,9 @@ class Client(Bcfg2.Server.Admin.MetadataCore): print("Error in deleting client") raise SystemExit(1) elif args[0] in ['list', 'ls']: - tree = lxml.etree.parse(self.metadata.data + "/clients.xml") - tree.xinclude() - for node in tree.findall("//Client"): - print(node.attrib["name"]) + for client in self.metadata.list_clients(): + print(client.hostname) + else: + print("No command specified") + raise SystemExit(1) + diff --git a/src/lib/Bcfg2/Server/Admin/Group.py b/src/lib/Bcfg2/Server/Admin/Group.py deleted file mode 100644 index 16a773d6f..000000000 --- a/src/lib/Bcfg2/Server/Admin/Group.py +++ /dev/null @@ -1,63 +0,0 @@ -import lxml.etree -import Bcfg2.Server.Admin -from Bcfg2.Server.Plugins.Metadata import MetadataConsistencyError - - -class Group(Bcfg2.Server.Admin.MetadataCore): - __shorthelp__ = "Create, delete, or modify group entries" - __longhelp__ = (__shorthelp__ + "\n\nbcfg2-admin group add <group> " - "attr1=val1 attr2=val2" - "\nbcfg2-admin group update <group> " - "attr1=val1 attr2=val2" - "\nbcfg2-admin group list" - "\nbcfg2-admin group del <group>\n") - __usage__ = ("bcfg2-admin group [options] [add|del|update|list] [attr=val]") - - def __call__(self, args): - Bcfg2.Server.Admin.MetadataCore.__call__(self, args) - if len(args) == 0: - self.errExit("No argument specified.\n" - "Please see bcfg2-admin group help for usage.") - if args[0] == 'add': - attr_d = {} - for i in args[2:]: - attr, val = i.split('=', 1) - if attr not in ['profile', 'public', 'default', - 'name', 'auth', 'toolset', 'category', - 'comment']: - print("Attribute %s unknown" % attr) - raise SystemExit(1) - attr_d[attr] = val - try: - self.metadata.add_group(args[1], attr_d) - except MetadataConsistencyError: - print("Error in adding group") - raise SystemExit(1) - elif args[0] in ['update', 'up']: - attr_d = {} - for i in args[2:]: - attr, val = i.split('=', 1) - if attr not in ['profile', 'public', 'default', - 'name', 'auth', 'toolset', 'category', - 'comment']: - print("Attribute %s unknown" % attr) - raise SystemExit(1) - attr_d[attr] = val - try: - self.metadata.update_group(args[1], attr_d) - except MetadataConsistencyError: - print("Error in updating group") - raise SystemExit(1) - elif args[0] in ['delete', 'remove', 'del', 'rm']: - try: - self.metadata.remove_group(args[1]) - except MetadataConsistencyError: - print("Error in deleting group") - raise SystemExit(1) - elif args[0] in ['list', 'ls']: - tree = lxml.etree.parse(self.metadata.data + "/groups.xml") - for node in tree.findall("//Group"): - print(node.attrib["name"]) - else: - print("No command specified") - raise SystemExit(1) diff --git a/src/lib/Bcfg2/Server/Admin/Init.py b/src/lib/Bcfg2/Server/Admin/Init.py index 8d0c2a4a9..30603bddc 100644 --- a/src/lib/Bcfg2/Server/Admin/Init.py +++ b/src/lib/Bcfg2/Server/Admin/Init.py @@ -308,9 +308,8 @@ class Init(Bcfg2.Server.Admin.Mode): for plugin in self.plugins: if plugin == 'Metadata': Bcfg2.Server.Plugins.Metadata.Metadata.init_repo(self.repopath, - groups, - self.os_sel, - clients) + groups_xml=groups % self.os_sel, + clients_xml=clients) else: try: module = __import__("Bcfg2.Server.Plugins.%s" % plugin, '', diff --git a/src/lib/Bcfg2/Server/Admin/Syncdb.py b/src/lib/Bcfg2/Server/Admin/Syncdb.py new file mode 100644 index 000000000..73dc5b8b2 --- /dev/null +++ b/src/lib/Bcfg2/Server/Admin/Syncdb.py @@ -0,0 +1,33 @@ +import Bcfg2.settings +import Bcfg2.Options +import Bcfg2.Server.Admin +from django.core.management import setup_environ + +class Syncdb(Bcfg2.Server.Admin.Mode): + __shorthelp__ = ("Sync the Django ORM with the configured database") + __longhelp__ = __shorthelp__ + "\n\nbcfg2-admin syncdb" + __usage__ = "bcfg2-admin syncdb" + options = {'configfile': Bcfg2.Options.CFILE, + 'repo': Bcfg2.Options.SERVER_REPOSITORY} + + def __call__(self, args): + Bcfg2.Server.Admin.Mode.__call__(self, args) + + # Parse options + self.opts = Bcfg2.Options.OptionParser(self.options) + self.opts.parse(args) + + # we have to set up the django environment before we import + # the syncdb command, but we have to wait to set up the + # environment until we've read the config, which has to wait + # until we've parsed options. it's a windy, twisting road. + Bcfg2.settings.read_config(cfile=self.opts['configfile'], + repo=self.opts['repo']) + setup_environ(Bcfg2.settings) + import Bcfg2.Server.models + Bcfg2.Server.models.load_models(cfile=self.opts['configfile']) + + from django.core.management.commands import syncdb + + cmd = syncdb.Command() + cmd.handle_noargs(interactive=False) diff --git a/src/lib/Bcfg2/Server/Admin/__init__.py b/src/lib/Bcfg2/Server/Admin/__init__.py index 0c9158351..3a7ba45cf 100644 --- a/src/lib/Bcfg2/Server/Admin/__init__.py +++ b/src/lib/Bcfg2/Server/Admin/__init__.py @@ -11,6 +11,7 @@ __all__ = [ 'Query', 'Reports', 'Snapshots', + 'Syncdb', 'Tidy', 'Viz', 'Xcmd' diff --git a/src/lib/Bcfg2/Server/Core.py b/src/lib/Bcfg2/Server/Core.py index 1ee01585c..20eee2d7f 100644 --- a/src/lib/Bcfg2/Server/Core.py +++ b/src/lib/Bcfg2/Server/Core.py @@ -1,5 +1,6 @@ """Bcfg2.Server.Core provides the runtime support for Bcfg2 modules.""" +import os import atexit import logging import select @@ -9,6 +10,11 @@ import time import inspect import lxml.etree from traceback import format_exc + +# this must be set before we import the Metadata plugin +os.environ['DJANGO_SETTINGS_MODULE'] = 'Bcfg2.settings' + +import Bcfg2.settings import Bcfg2.Server import Bcfg2.Logger import Bcfg2.Server.FileMonitor @@ -95,6 +101,10 @@ class BaseCore(object): # Create an event to signal worker threads to shutdown self.terminate = threading.Event() + # generate Django ORM settings. this must be done _before_ we + # load plugins + Bcfg2.settings.read_config(cfile=self.cfile, repo=self.datastore) + if '' in setup['plugins']: setup['plugins'].remove('') @@ -195,8 +205,7 @@ class BaseCore(object): try: self.plugins[plugin] = plug(self, self.datastore) except PluginInitError: - self.logger.error("Failed to instantiate plugin %s" % plugin, - exc_info=1) + logger.error("Failed to instantiate plugin %s" % plugin, exc_info=1) except: self.logger.error("Unexpected instantiation failure for plugin %s" % plugin, exc_info=1) @@ -526,8 +535,6 @@ class BaseCore(object): def RecvProbeData(self, address, probedata): """Receive probe data from clients.""" client, metadata = self.resolve_client(address) - # clear dynamic groups - self.metadata.cgroups[metadata.hostname] = [] try: xpdata = lxml.etree.XML(probedata.encode('utf-8'), parser=Bcfg2.Server.XMLParser) diff --git a/src/lib/Bcfg2/Server/Plugin.py b/src/lib/Bcfg2/Server/Plugin.py index 6b4276444..51d1b1cdb 100644 --- a/src/lib/Bcfg2/Server/Plugin.py +++ b/src/lib/Bcfg2/Server/Plugin.py @@ -102,6 +102,16 @@ class Debuggable(object): self.logger.error(message) +class DatabaseBacked(object): + def __init__(self): + pass + + +class PluginDatabaseModel(object): + class Meta: + app_label = "Server" + + class Plugin(Debuggable): """This is the base class for all Bcfg2 Server plugins. Several attributes must be defined in the subclass: @@ -139,8 +149,7 @@ class Plugin(Debuggable): @classmethod def init_repo(cls, repo): - path = "%s/%s" % (repo, cls.name) - os.makedirs(path) + os.makedirs(os.path.join(repo, cls.name)) def shutdown(self): self.running = False @@ -169,7 +178,7 @@ class Structure(object): class Metadata(object): """Signal metadata capabilities for this plugin""" - def add_client(self, client_name, attribs): + def add_client(self, client_name): """Add client.""" pass @@ -181,6 +190,9 @@ class Metadata(object): """Create viz str for viz admin mode.""" pass + def _handle_default_event(self, event): + pass + def get_initial_metadata(self, client_name): raise PluginExecutionError @@ -650,7 +662,7 @@ class XMLFileBacked(FileBacked): def add_monitor(self, fpath, fname): self.extras.append(fname) - if self.fam: + if self.fam and self.should_monitor: self.fam.AddMonitor(fpath, self) def __iter__(self): @@ -666,22 +678,13 @@ class StructFile(XMLFileBacked): def _include_element(self, item, metadata): """ determine if an XML element matches the metadata """ + negate = item.get('negate', 'false').lower() == 'true' if item.tag == 'Group': - if ((item.get('negate', 'false').lower() == 'true' and - item.get('name') not in metadata.groups) or - (item.get('negate', 'false').lower() == 'false' and - item.get('name') in metadata.groups)): - return True - else: - return False + return ((negate and item.get('name') not in metadata.groups) or + (not negate and item.get('name') in metadata.groups)) elif item.tag == 'Client': - if ((item.get('negate', 'false').lower() == 'true' and - item.get('name') != metadata.hostname) or - (item.get('negate', 'false').lower() == 'false' and - item.get('name') == metadata.hostname)): - return True - else: - return False + return ((negate and item.get('name') != metadata.hostname) or + (not negate and item.get('name') == metadata.hostname)) elif isinstance(item, lxml.etree._Comment): return False else: diff --git a/src/lib/Bcfg2/Server/Plugins/DBMetadata.py b/src/lib/Bcfg2/Server/Plugins/DBMetadata.py new file mode 100644 index 000000000..16a6e0dcc --- /dev/null +++ b/src/lib/Bcfg2/Server/Plugins/DBMetadata.py @@ -0,0 +1,128 @@ +import os +import sys +from UserDict import DictMixin +from django.db import models +import Bcfg2.Server.Lint +import Bcfg2.Server.Plugin +from Bcfg2.Server.Plugins.Metadata import * + +class MetadataClientModel(models.Model, + Bcfg2.Server.Plugin.PluginDatabaseModel): + hostname = models.CharField(max_length=255, primary_key=True) + version = models.CharField(max_length=31, null=True) + + +class ClientVersions(DictMixin): + def __getitem__(self, key): + try: + return MetadataClientModel.objects.get(hostname=key).version + except MetadataClientModel.DoesNotExist: + raise KeyError(key) + + def __setitem__(self, key, value): + client = MetadataClientModel.objects.get_or_create(hostname=key)[0] + client.version = value + client.save() + + def keys(self): + return [c.hostname for c in MetadataClientModel.objects.all()] + + def __contains__(self, key): + try: + client = MetadataClientModel.objects.get(hostname=key) + return True + except MetadataClientModel.DoesNotExist: + return False + + +class DBMetadata(Metadata, Bcfg2.Server.Plugin.DatabaseBacked): + __files__ = ["groups.xml"] + experimental = True + conflicts = ['Metadata'] + + def __init__(self, core, datastore, watch_clients=True): + Metadata.__init__(self, core, datastore, watch_clients=watch_clients) + Bcfg2.Server.Plugin.DatabaseBacked.__init__(self) + if os.path.exists(os.path.join(self.data, "clients.xml")): + self.logger.warning("DBMetadata: clients.xml found, parsing in " + "compatibility mode") + self._handle_file("clients.xml") + self.versions = ClientVersions() + + def add_group(self, group_name, attribs): + msg = "DBMetadata does not support adding groups" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def add_bundle(self, bundle_name): + msg = "DBMetadata does not support adding bundles" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def add_client(self, client_name): + """Add client to clients database.""" + client = MetadataClientModel(hostname=client_name) + client.save() + self.clients = self.list_clients() + return client + + def update_group(self, group_name, attribs): + msg = "DBMetadata does not support updating groups" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def update_bundle(self, bundle_name): + msg = "DBMetadata does not support updating bundles" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def update_client(self, client_name, attribs): + msg = "DBMetadata does not support updating clients" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def list_clients(self): + """ List all clients in client database """ + return set([c.hostname for c in MetadataClientModel.objects.all()]) + + def remove_group(self, group_name, attribs): + msg = "DBMetadata does not support removing groups" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def remove_bundle(self, bundle_name): + msg = "DBMetadata does not support removing bundles" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def remove_client(self, client_name): + """Remove a client""" + try: + client = MetadataClientModel.objects.get(hostname=client_name) + except MetadataClientModel.DoesNotExist: + msg = "Client %s does not exist" % client_name + self.logger.warning(msg) + raise MetadataConsistencyError(msg) + client.delete() + self.clients = self.list_clients() + + def _set_profile(self, client, profile, addresspair): + if client not in self.clients: + # adding a new client + self.add_client(client) + if client not in self.clientgroups: + self.clientgroups[client] = [profile] + else: + msg = "DBMetadata does not support asserting client profiles" + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginExecutionError(msg) + + def _handle_clients_xml_event(self, event): + # clients.xml is parsed and the options specified in it are + # understood, but it does _not_ assert client existence. + Metadata._handle_clients_xml_event(self, event) + self.clients = self.list_clients() + + +class DBMetadataLint(MetadataLint): + pass diff --git a/src/lib/Bcfg2/Server/Plugins/Metadata.py b/src/lib/Bcfg2/Server/Plugins/Metadata.py index 4f6e82128..447a7cd05 100644 --- a/src/lib/Bcfg2/Server/Plugins/Metadata.py +++ b/src/lib/Bcfg2/Server/Plugins/Metadata.py @@ -2,6 +2,7 @@ This file stores persistent metadata for the Bcfg2 Configuration Repository. """ +import re import copy import fcntl import lxml.etree @@ -10,8 +11,9 @@ import socket import sys import time import Bcfg2.Server -import Bcfg2.Server.FileMonitor +import Bcfg2.Server.Lint import Bcfg2.Server.Plugin +import Bcfg2.Server.FileMonitor from Bcfg2.version import Bcfg2VersionInfo def locked(fd): @@ -38,10 +40,10 @@ class MetadataRuntimeError(Exception): class XMLMetadataConfig(Bcfg2.Server.Plugin.XMLFileBacked): """Handles xml config files and all XInclude statements""" def __init__(self, metadata, watch_clients, basefile): - # we tell XMLFileBacked _not_ to add a monitor for this - # file, because the main Metadata plugin has already added - # one. then we immediately set should_monitor to the proper - # value, so that XIinclude'd files get properly watched + # we tell XMLFileBacked _not_ to add a monitor for this file, + # because the main Metadata plugin has already added one. + # then we immediately set should_monitor to the proper value, + # so that XInclude'd files get properly watched fpath = os.path.join(metadata.data, basefile) Bcfg2.Server.Plugin.XMLFileBacked.__init__(self, fpath, fam=metadata.core.fam, @@ -210,7 +212,8 @@ class ClientMetadata(object): class MetadataQuery(object): - def __init__(self, by_name, get_clients, by_groups, by_profiles, all_groups, all_groups_in_category): + def __init__(self, by_name, get_clients, by_groups, by_profiles, + all_groups, all_groups_in_category): # resolver is set later self.by_name = by_name self.names_by_groups = by_groups @@ -229,6 +232,36 @@ class MetadataQuery(object): return [self.by_name(name) for name in self.all_clients()] +class MetadataGroup(tuple): + def __new__(cls, name, bundles=None, category=None, + is_profile=False, is_public=False, is_private=False): + if bundles is None: + bundles = set() + return tuple.__new__(cls, (bundles, category)) + + def __init__(self, name, bundles=None, category=None, + is_profile=False, is_public=False, is_private=False): + if bundles is None: + bundles = set() + tuple.__init__(self) + self.name = name + self.bundles = bundles + self.category = category + self.is_profile = is_profile + self.is_public = is_public + self.is_private = is_private + + def __str__(self): + return repr(self) + + def __repr__(self): + return "%s %s (bundles=%s, category=%s)" % \ + (self.__class__.__name__, self.name, self.bundles, + self.category) + + def __hash__(self): + return hash(self.name) + class Metadata(Bcfg2.Server.Plugin.Plugin, Bcfg2.Server.Plugin.Metadata, Bcfg2.Server.Plugin.Statistics): @@ -236,69 +269,80 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, __author__ = 'bcfg-dev@mcs.anl.gov' name = "Metadata" sort_order = 500 + __files__ = ["groups.xml", "clients.xml"] def __init__(self, core, datastore, watch_clients=True): Bcfg2.Server.Plugin.Plugin.__init__(self, core, datastore) Bcfg2.Server.Plugin.Metadata.__init__(self) Bcfg2.Server.Plugin.Statistics.__init__(self) + self.watch_clients = watch_clients self.states = dict() - if watch_clients: - for fname in ["groups.xml", "clients.xml"]: - self.states[fname] = False - try: - core.fam.AddMonitor(os.path.join(self.data, fname), self) - except: - err = sys.exc_info()[1] - msg = "Unable to add file monitor for %s: %s" % (fname, err) - print(msg) - raise Bcfg2.Server.Plugin.PluginInitError(msg) - - self.clients_xml = XMLMetadataConfig(self, watch_clients, 'clients.xml') - self.groups_xml = XMLMetadataConfig(self, watch_clients, 'groups.xml') - self.addresses = {} + self.extra = dict() + self.handlers = [] + for fname in self.__files__: + self._handle_file(fname) + + # mapping of clientname -> authtype self.auth = dict() - self.clients = {} - self.aliases = {} - self.groups = {} - self.cgroups = {} - self.versions = {} - self.public = [] - self.private = [] - self.profiles = [] - self.categories = {} - self.bad_clients = {} - self.uuid = {} + # list of clients required to have non-global password self.secure = [] + # list of floating clients self.floating = [] + # mapping of clientname -> password self.passwords = {} + self.addresses = {} + self.raddresses = {} + # mapping of clientname -> [groups] + self.clientgroups = {} + # list of clients + self.clients = [] + self.aliases = {} + self.raliases = {} + # mapping of groupname -> MetadataGroup object + self.groups = {} + # mappings of predicate -> MetadataGroup object + self.group_membership = dict() + self.negated_groups = dict() + # mapping of hostname -> version string + self.versions = dict() + self.uuid = {} self.session_cache = {} self.default = None self.pdirty = False - self.extra = {'groups.xml': [], - 'clients.xml': []} self.password = core.password self.query = MetadataQuery(core.build_metadata, - lambda: list(self.clients.keys()), + lambda: list(self.clients), self.get_client_names_by_groups, self.get_client_names_by_profiles, self.get_all_group_names, self.get_all_groups_in_category) @classmethod - def init_repo(cls, repo, groups, os_selection, clients): - path = os.path.join(repo, cls.name) - os.makedirs(path) - open(os.path.join(repo, "Metadata", "groups.xml"), - "w").write(groups % os_selection) - open(os.path.join(repo, "Metadata", "clients.xml"), - "w").write(clients % socket.getfqdn()) - - def get_groups(self): - '''return groups xml tree''' - groups_tree = lxml.etree.parse(os.path.join(self.data, "groups.xml"), - parser=Bcfg2.Server.XMLParser) - root = groups_tree.getroot() - return root + def init_repo(cls, repo, **kwargs): + # must use super here; inheritance works funny with class methods + super(Metadata, cls).init_repo(repo) + + for fname in cls.__files__: + aname = re.sub(r'[^A-z0-9_]', '_', fname) + if aname in kwargs: + open(os.path.join(repo, cls.name, fname), + "w").write(kwargs[aname]) + + def _handle_file(self, fname): + if self.watch_clients: + try: + self.core.fam.AddMonitor(os.path.join(self.data, fname), self) + except: + err = sys.exc_info()[1] + msg = "Unable to add file monitor for %s: %s" % (fname, err) + self.logger.error(msg) + raise Bcfg2.Server.Plugin.PluginInitError(msg) + self.states[fname] = False + aname = re.sub(r'[^A-z0-9_]', '_', fname) + xmlcfg = XMLMetadataConfig(self, self.watch_clients, fname) + setattr(self, aname, xmlcfg) + self.handlers.append(xmlcfg.HandleEvent) + self.extra[fname] = [] def _search_xdata(self, tag, name, tree, alias=False): for node in tree.findall("//%s" % tag): @@ -325,9 +369,8 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, def _add_xdata(self, config, tag, name, attribs=None, alias=False): node = self._search_xdata(tag, name, config.xdata, alias=alias) if node != None: - msg = "%s \"%s\" already exists" % (tag, name) - self.logger.error(msg) - raise MetadataConsistencyError(msg) + self.logger.error("%s \"%s\" already exists" % (tag, name)) + raise MetadataConsistencyError element = lxml.etree.SubElement(config.base_xdata.getroot(), tag, name=name) if attribs: @@ -352,15 +395,14 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, def _update_xdata(self, config, tag, name, attribs, alias=False): node = self._search_xdata(tag, name, config.xdata, alias=alias) if node == None: - msg = "%s \"%s\" does not exist" % (tag, name) - self.logger.error(msg) - raise MetadataConsistencyError(msg) + self.logger.error("%s \"%s\" does not exist" % (tag, name)) + raise MetadataConsistencyError xdict = config.find_xml_for_xpath('.//%s[@name="%s"]' % (tag, node.get('name'))) if not xdict: - msg = "Unexpected error finding %s \"%s\"" % (tag, name) - self.logger.error(msg) - raise MetadataConsistencyError(msg) + self.logger.error("Unexpected error finding %s \"%s\"" % + (tag, name)) + raise MetadataConsistencyError for key, val in list(attribs.items()): xdict['xquery'][0].set(key, val) config.write_xml(xdict['filename'], xdict['xmltree']) @@ -377,17 +419,16 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, def _remove_xdata(self, config, tag, name, alias=False): node = self._search_xdata(tag, name, config.xdata) if node == None: - msg = "%s \"%s\" does not exist" % (tag, name) - self.logger.error(msg) - raise MetadataConsistencyError(msg) + self.logger.error("%s \"%s\" does not exist" % (tag, name)) + raise MetadataConsistencyError xdict = config.find_xml_for_xpath('.//%s[@name="%s"]' % (tag, node.get('name'))) if not xdict: - msg = "Unexpected error finding %s \"%s\"" % (tag, name) - self.logger.error(msg) - raise MetadataConsistencyError(msg) + self.logger.error("Unexpected error finding %s \"%s\"" % + (tag, name)) + raise MetadataConsistencyError xdict['xquery'][0].getparent().remove(xdict['xquery'][0]) - self.groups_xml.write_xml(xdict['filename'], xdict['xmltree']) + config.write_xml(xdict['filename'], xdict['xmltree']) def remove_group(self, group_name): """Remove a group.""" @@ -397,12 +438,16 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, """Remove a bundle.""" return self._remove_xdata(self.groups_xml, "Bundle", bundle_name) + def remove_client(self, client_name): + """Remove a bundle.""" + return self._remove_xdata(self.clients_xml, "Client", client_name) + def _handle_clients_xml_event(self, event): xdata = self.clients_xml.xdata - self.clients = {} + self.clients = [] + self.clientgroups = {} self.aliases = {} self.raliases = {} - self.bad_clients = {} self.secure = [] self.floating = [] self.addresses = {} @@ -423,9 +468,10 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, 'cert+password') if 'uuid' in client.attrib: self.uuid[client.get('uuid')] = clname - if client.get('secure', 'false') == 'true': + if client.get('secure', 'false').lower() == 'true': self.secure.append(clname) - if client.get('location', 'fixed') == 'floating': + if (client.get('location', 'fixed') == 'floating' or + client.get('floating', 'false').lower() == 'true'): self.floating.append(clname) if 'password' in client.attrib: self.passwords[clname] = client.get('password') @@ -445,106 +491,157 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, if clname not in self.raddresses: self.raddresses[clname] = set() self.raddresses[clname].add(alias.get('address')) - self.clients.update({clname: client.get('profile')}) + self.clients.append(clname) + try: + self.clientgroups[clname].append(client.get('profile')) + except KeyError: + self.clientgroups[clname] = [client.get('profile')] self.states['clients.xml'] = True def _handle_groups_xml_event(self, event): - xdata = self.groups_xml.xdata - self.public = [] - self.private = [] - self.profiles = [] self.groups = {} - grouptmp = {} - self.categories = {} - groupseen = list() - for group in xdata.xpath('//Groups/Group'): - if group.get('name') not in groupseen: - groupseen.append(group.get('name')) + + # get_condition and aggregate_conditions must be separate + # functions in order to ensure that the scope is right for the + # closures they return + def get_condition(element): + negate = element.get('negate', 'false').lower() == 'true' + pname = element.get("name") + if element.tag == 'Group': + return lambda c, g, _: negate != (pname in g) + elif element.tag == 'Client': + return lambda c, g, _: negate != (pname == c) + + def aggregate_conditions(conditions): + return lambda client, groups, cats: \ + all(cond(client, groups, cats) for cond in conditions) + + # first, we get a list of all of the groups declared in the + # file. we do this in two stages because the old way of + # parsing groups.xml didn't support nested groups; in the old + # way, only Group tags under a Groups tag counted as + # declarative. so we parse those first, and then parse the + # other Group tags if they haven't already been declared. + # this lets you set options on a group (e.g., public="false") + # at the top level and then just use the name elsewhere, which + # is the original behavior + for grp in self.groups_xml.xdata.xpath("//Groups/Group") + \ + self.groups_xml.xdata.xpath("//Groups/Group//Group"): + if grp.get("name") in self.groups: + continue + self.groups[grp.get("name")] = \ + MetadataGroup(grp.get("name"), + bundles=[b.get("name") + for b in grp.findall("Bundle")], + category=grp.get("category"), + is_profile=grp.get("profile", "false") == "true", + is_public=grp.get("public", "false") == "true", + is_private=grp.get("public", "true") == "false") + if grp.get('default', 'false') == 'true': + self.default = grp.get('name') + + self.group_membership = dict() + self.negated_groups = dict() + self.options = dict() + # confusing loop condition; the XPath query asks for all + # elements under a Group tag under a Groups tag; that is + # infinitely recursive, so "all" elements really means _all_ + # elements. We then manually filter out non-Group elements + # since there doesn't seem to be a way to get Group elements + # of arbitrary depth with particular ultimate ancestors in + # XPath. We do the same thing for Client tags. + for el in self.groups_xml.xdata.xpath("//Groups/Group//*") + \ + self.groups_xml.xdata.xpath("//Groups/Client//*"): + if ((el.tag != 'Group' and el.tag != 'Client') or + el.getchildren()): + continue + + conditions = [] + for parent in el.iterancestors(): + cond = get_condition(parent) + if cond: + conditions.append(cond) + + gname = el.get("name") + if el.get("negate", "false").lower() == "true": + self.negated_groups[aggregate_conditions(conditions)] = \ + self.groups[gname] else: - self.logger.error("Metadata: Group %s defined multiply" % - group.get('name')) - grouptmp[group.get('name')] = \ - ([item.get('name') for item in group.findall('./Bundle')], - [item.get('name') for item in group.findall('./Group')]) - grouptmp[group.get('name')][1].append(group.get('name')) - if group.get('default', 'false') == 'true': - self.default = group.get('name') - if group.get('profile', 'false') == 'true': - self.profiles.append(group.get('name')) - if group.get('public', 'false') == 'true': - self.public.append(group.get('name')) - elif group.get('public', 'true') == 'false': - self.private.append(group.get('name')) - if 'category' in group.attrib: - self.categories[group.get('name')] = group.get('category') - - for group in grouptmp: - # self.groups[group] => (bundles, groups, categories) - self.groups[group] = (set(), set(), {}) - tocheck = [group] - group_cat = self.groups[group][2] - while tocheck: - now = tocheck.pop() - self.groups[group][1].add(now) - if now in grouptmp: - (bundles, groups) = grouptmp[now] - for ggg in groups: - if ggg in self.groups[group][1]: - continue - if (ggg not in self.categories or \ - self.categories[ggg] not in self.groups[group][2]): - self.groups[group][1].add(ggg) - tocheck.append(ggg) - if ggg in self.categories: - group_cat[self.categories[ggg]] = ggg - elif ggg in self.categories: - self.logger.info("Group %s: %s cat-suppressed %s" % \ - (group, - group_cat[self.categories[ggg]], - ggg)) - [self.groups[group][0].add(bund) for bund in bundles] + if self.groups[gname].category and gname in self.groups: + category = self.groups[gname].category + + def in_cat(client, groups, categories): + if category in categories: + self.logger.warning("%s: Group %s suppressed by " + "category %s; %s already a " + "member of %s" % + (self.name, gname, category, + client, categories[category])) + return False + return True + conditions.append(in_cat) + + self.group_membership[aggregate_conditions(conditions)] = \ + self.groups[gname] self.states['groups.xml'] = True def HandleEvent(self, event): """Handle update events for data files.""" - if self.clients_xml.HandleEvent(event): - self._handle_clients_xml_event(event) - elif self.groups_xml.HandleEvent(event): - self._handle_groups_xml_event(event) - - if False not in list(self.states.values()): - # check that all client groups are real and complete - real = list(self.groups.keys()) - for client in list(self.clients.keys()): - if self.clients[client] not in self.profiles: - self.logger.error("Client %s set as nonexistent or " - "incomplete group %s" % - (client, self.clients[client])) - self.logger.error("Removing client mapping for %s" % client) - self.bad_clients[client] = self.clients[client] - del self.clients[client] - for bclient in list(self.bad_clients.keys()): - if self.bad_clients[bclient] in self.profiles: - self.logger.info("Restored profile mapping for client %s" % - bclient) - self.clients[bclient] = self.bad_clients[bclient] - del self.bad_clients[bclient] - - def set_profile(self, client, profile, addresspair): + for hdlr in self.handlers: + aname = re.sub(r'[^A-z0-9_]', '_', os.path.basename(event.filename)) + if hdlr(event): + try: + proc = getattr(self, "_handle_%s_event" % aname) + except AttributeError: + proc = self._handle_default_event + proc(event) + + if False not in list(self.states.values()) and self.debug_flag: + # check that all groups are real and complete. this is + # just logged at a debug level because many groups might + # be probed, and we don't want to warn about them. + for client, groups in list(self.clientgroups.items()): + for group in groups: + if group not in self.groups: + self.debug_log("Client %s set as nonexistent group %s" % + (client, group)) + for gname, ginfo in list(self.groups.items()): + for group in ginfo.groups: + if group not in self.groups: + self.debug_log("Group %s set as nonexistent group %s" % + (gname, group)) + + + def set_profile(self, client, profile, addresspair, force=False): """Set group parameter for provided client.""" - self.logger.info("Asserting client %s profile to %s" % (client, - profile)) + self.logger.info("Asserting client %s profile to %s" % + (client, profile)) if False in list(self.states.values()): - raise MetadataRuntimeError("Metadata has not been read yet") - if profile not in self.public: - msg = "Failed to set client %s to private group %s" % (client, - profile) + raise MetadataRuntimeError + if not force and profile not in self.groups: + msg = "Profile group %s does not exist" % profile + self.logger.error(msg) + raise MetadataConsistencyError(msg) + group = self.groups[profile] + if not force and not group.is_public: + msg = "Cannot set client %s to private group %s" % (client, profile) self.logger.error(msg) raise MetadataConsistencyError(msg) + self._set_profile(client, profile, addresspair) + + def _set_profile(self, client, profile, addresspair): if client in self.clients: - self.logger.info("Changing %s group from %s to %s" % - (client, self.clients[client], profile)) + profiles = [g for g in self.clientgroups[client] + if g in self.groups and self.groups[g].is_profile] + self.logger.info("Changing %s profile from %s to %s" % + (client, profiles, profile)) self.update_client(client, dict(profile=profile)) + if client in self.clientgroups: + for p in profiles: + self.clientgroups[client].remove(p) + self.clientgroups[client].append(profile) + else: + self.clientgroups[client] = [profile] else: self.logger.info("Creating new client: %s, profile %s" % (client, profile)) @@ -555,7 +652,8 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, address=addresspair[0])) else: self.add_client(client, dict(profile=profile)) - self.clients[client] = profile + self.clients.append(client) + self.clientgroups[client] = [profile] self.clients_xml.write() def set_version(self, client, version): @@ -614,6 +712,31 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, self.logger.warning(warning) raise MetadataConsistencyError(warning) + def _merge_groups(self, client, groups, categories=None): + """ set group membership based on the contents of groups.xml + and initial group membership of this client. Returns a tuple + of (allgroups, categories)""" + numgroups = -1 # force one initial pass + if categories is None: + categories = dict() + while numgroups != len(groups): + numgroups = len(groups) + for predicate, group in self.group_membership.items(): + if group.name in groups: + continue + if predicate(client, groups, categories): + groups.add(group.name) + if group.category: + categories[group.category] = group.name + for predicate, group in self.negated_groups.items(): + if group.name not in groups: + continue + if predicate(client, groups, categories): + groups.remove(group.name) + if group.category: + del categories[group.category] + return (groups, categories) + def get_initial_metadata(self, client): """Return the metadata for a given client.""" if False in list(self.states.values()): @@ -621,25 +744,66 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, client = client.lower() if client in self.aliases: client = self.aliases[client] - if client in self.clients: - profile = self.clients[client] - (bundles, groups, categories) = self.groups[profile] - else: - if self.default == None: - msg = "Cannot set group for client %s; no default group set" % \ - client + + groups = set() + categories = dict() + profile = None + + if client not in self.clients: + pgroup = None + if client in self.clientgroups: + pgroup = self.clientgroups[client][0] + elif self.default: + pgroup = self.default + + if pgroup: + self.set_profile(client, pgroup, (None, None), force=True) + groups.add(pgroup) + category = self.groups[pgroup].category + if category: + categories[category] = pgroup + if (pgroup in self.groups and self.groups[pgroup].is_profile): + profile = pgroup + else: + msg = "Cannot add new client %s; no default group set" % client self.logger.error(msg) raise MetadataConsistencyError(msg) - self.set_profile(client, self.default, (None, None)) - profile = self.default - [bundles, groups, categories] = self.groups[self.default] + + if client in self.clientgroups: + for cgroup in self.clientgroups[client]: + if cgroup in groups: + continue + if cgroup not in self.groups: + self.groups[cgroup] = MetadataGroup(cgroup) + category = self.groups[cgroup].category + if category and category in categories: + self.logger.warning("%s: Group %s suppressed by " + "category %s; %s already a member " + "of %s" % + (self.name, cgroup, category, + client, categories[category])) + continue + if category: + categories[category] = cgroup + groups.add(cgroup) + # favor client groups for setting profile + if not profile and self.groups[cgroup].is_profile: + profile = cgroup + + groups, categories = self._merge_groups(client, groups, + categories=categories) + + bundles = set() + for group in groups: + try: + bundles.update(self.groups[group].bundles) + except KeyError: + self.logger.warning("%s: %s is a member of undefined group %s" % + (self.name, client, group)) + aliases = self.raliases.get(client, set()) addresses = self.raddresses.get(client, set()) version = self.versions.get(client, None) - newgroups = set(groups) - newbundles = set(bundles) - newcategories = {} - newcategories.update(categories) if client in self.passwords: password = self.passwords[client] else: @@ -650,36 +814,41 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, uuid = uuids[0] else: uuid = None - for group in self.cgroups.get(client, []): - if group in self.groups: - nbundles, ngroups, ncategories = self.groups[group] - else: - nbundles, ngroups, ncategories = ([], [group], {}) - [newbundles.add(b) for b in nbundles if b not in newbundles] - [newgroups.add(g) for g in ngroups if g not in newgroups] - newcategories.update(ncategories) - return ClientMetadata(client, profile, newgroups, newbundles, aliases, - addresses, newcategories, uuid, password, version, + if not profile: + # one last ditch attempt at setting the profile + profiles = [g for g in groups + if g in self.groups and self.groups[g].is_profile] + if len(profiles) >= 1: + profile = profiles[0] + + return ClientMetadata(client, profile, groups, bundles, aliases, + addresses, categories, uuid, password, version, self.query) def get_all_group_names(self): all_groups = set() - [all_groups.update(g[1]) for g in list(self.groups.values())] + all_groups.update(self.groups.keys()) + all_groups.update([g.name for g in self.group_membership.values()]) + all_groups.update([g.name for g in self.negated_groups.values()]) + for grp in self.clientgroups.values(): + all_groups.update(grp) return all_groups def get_all_groups_in_category(self, category): - all_groups = set() - [all_groups.add(g) for g in self.categories \ - if self.categories[g] == category] - return all_groups + return set([g.name for g in self.groups.values() + if g.category == category]) def get_client_names_by_profiles(self, profiles): - return [client for client, profile in list(self.clients.items()) \ - if profile in profiles] + rv = [] + for client in list(self.clients): + mdata = self.get_initial_metadata(client) + if mdata.profile in profiles: + rv.append(client) + return rv def get_client_names_by_groups(self, groups): mdata = [self.core.build_metadata(client) - for client in list(self.clients.keys())] + for client in list(self.clients)] return [md.hostname for md in mdata if md.groups.issuperset(groups)] def get_client_names_by_bundles(self, bundles): @@ -689,27 +858,26 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, def merge_additional_groups(self, imd, groups): for group in groups: - if (group in self.categories and - self.categories[group] in imd.categories): + if group in imd.groups or group not in self.groups: continue - newbundles, newgroups, _ = self.groups.get(group, - (list(), - [group], - dict())) - for newbundle in newbundles: - if newbundle not in imd.bundles: - imd.bundles.add(newbundle) - for newgroup in newgroups: - if newgroup not in imd.groups: - if (newgroup in self.categories and - self.categories[newgroup] in imd.categories): - continue - if newgroup in self.private: - self.logger.error("Refusing to add dynamic membership " - "in private group %s for client %s" % - (newgroup, imd.hostname)) - continue - imd.groups.add(newgroup) + category = self.groups[group].category + if category: + if self.groups[group].category in imd.categories: + self.logger.warning("%s: Group %s suppressed by category " + "%s; %s already a member of %s" % + (self.name, group, category, + imd.hostname, + imd.categories[category])) + continue + imd.categories[group] = category + imd.groups.add(group) + + self._merge_groups(imd.hostname, imd.groups, + categories=imd.categories) + + for group in imd.groups: + if group in self.groups: + imd.bundles.update(self.groups[group].bundles) def merge_additional_data(self, imd, source, data): if not hasattr(imd, source): @@ -728,8 +896,8 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, (client, address)) return True else: - self.logger.error("Got request for non-float client %s from %s" % - (client, address)) + self.logger.error("Got request for non-float client %s from %s" + % (client, address)) return False resolved = self.resolve_client(addresspair) if resolved.lower() == client.lower(): @@ -853,20 +1021,26 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, del categories[None] if hosts: instances = {} - clients = self.clients - for client, profile in list(clients.items()): + for client in list(self.clients): if include_client(client): continue - if profile in instances: - instances[profile].append(client) + if client in self.clientgroups: + groups = self.clientgroups[client] + elif self.default: + groups = [self.default] else: - instances[profile] = [client] - for profile, clist in list(instances.items()): + continue + for group in groups: + try: + instances[group].append(client) + except KeyError: + instances[group] = [client] + for group, clist in list(instances.items()): clist.sort() viz_str.append('"%s-instances" [ label="%s", shape="record" ];' % - (profile, '|'.join(clist))) + (group, '|'.join(clist))) viz_str.append('"%s-instances" -> "group-%s";' % - (profile, profile)) + (group, group)) if bundles: bundles = [] [bundles.append(bund.get('name')) \ @@ -907,3 +1081,35 @@ class Metadata(Bcfg2.Server.Plugin.Plugin, viz_str.append('"%s" [label="%s", shape="record", style="filled", fillcolor="%s"];' % (category, category, categories[category])) return "\n".join("\t" + s for s in viz_str) + + +class MetadataLint(Bcfg2.Server.Lint.ServerPlugin): + def Run(self): + self.nested_clients() + self.deprecated_options() + + @classmethod + def Errors(cls): + return {"nested-client-tags": "warning", + "deprecated-clients-options": "warning"} + + def deprecated_options(self): + groupdata = self.metadata.clients_xml.xdata + for el in groupdata.xpath("//Client"): + loc = el.get("location") + if loc: + if loc == "floating": + floating = True + else: + floating = False + self.LintError("deprecated-clients-options", + "The location='%s' option is deprecated. " + "Please use floating='%s' instead: %s" % + (loc, floating, self.RenderXML(el))) + + def nested_clients(self): + groupdata = self.metadata.groups_xml.xdata + for el in groupdata.xpath("//Client//Client"): + self.LintError("nested-client-tags", + "Client %s nested within Client tag: %s" % + (el.get("name"), self.RenderXML(el))) diff --git a/src/lib/Bcfg2/Server/Plugins/Packages/Collection.py b/src/lib/Bcfg2/Server/Plugins/Packages/Collection.py index ac78ea0fc..9cea9da48 100644 --- a/src/lib/Bcfg2/Server/Plugins/Packages/Collection.py +++ b/src/lib/Bcfg2/Server/Plugins/Packages/Collection.py @@ -375,15 +375,7 @@ def factory(metadata, sources, basepath, debug=False): ",".join([s.__name__ for s in sclasses])) cclass = Collection elif len(sclasses) == 0: - # you'd think this should be a warning, but it happens all the - # freaking time if you have a) machines in your clients.xml - # that do not have the proper groups set up yet (e.g., if you - # have multiple Bcfg2 servers and Packages-relevant groups set - # by probes); and b) templates that query all or multiple - # machines (e.g., with metadata.query.all_clients()) - if debug: - logger.error("Packages: No sources found for %s" % - metadata.hostname) + logger.error("Packages: No sources found for %s" % metadata.hostname) cclass = Collection else: cclass = get_collection_class(sclasses.pop().__name__.replace("Source", @@ -398,4 +390,3 @@ def factory(metadata, sources, basepath, debug=False): clients[metadata.hostname] = ckey collections[ckey] = collection return collection - diff --git a/src/lib/Bcfg2/Server/Reports/settings.py b/src/lib/Bcfg2/Server/Reports/settings.py index b27348aee..26138cddb 100644 --- a/src/lib/Bcfg2/Server/Reports/settings.py +++ b/src/lib/Bcfg2/Server/Reports/settings.py @@ -43,7 +43,7 @@ try: db_engine = c.get('statistics', 'database_engine') except ConfigParser.NoSectionError: e = sys.exc_info()[1] - raise ImportError("Failed to determine database engine: %s" % e) + raise ImportError("Failed to determine database engine for reports: %s" % e) db_name = '' if c.has_option('statistics', 'database_name'): db_name = c.get('statistics', 'database_name') diff --git a/src/lib/Bcfg2/Server/models.py b/src/lib/Bcfg2/Server/models.py new file mode 100644 index 000000000..ba9ea761c --- /dev/null +++ b/src/lib/Bcfg2/Server/models.py @@ -0,0 +1,62 @@ +import sys +import logging +import Bcfg2.Options +import Bcfg2.Server.Plugins +from django.db import models +from Bcfg2.Bcfg2Py3k import ConfigParser + +logger = logging.getLogger('Bcfg2.Server.models') + +MODELS = [] + +def load_models(plugins=None, cfile='/etc/bcfg2.conf', quiet=True): + global MODELS + + if plugins is None: + # we want to provide a different default plugin list -- + # namely, _all_ plugins, so that the database is guaranteed to + # work, even if /etc/bcfg2.conf isn't set up properly + plugin_opt = Bcfg2.Options.SERVER_PLUGINS + plugin_opt.default = Bcfg2.Server.Plugins.__all__ + + setup = Bcfg2.Options.OptionParser(dict(plugins=plugin_opt, + configfile=Bcfg2.Options.CFILE), + quiet=quiet) + setup.parse([Bcfg2.Options.CFILE.cmd, cfile]) + plugins = setup['plugins'] + + if MODELS: + # load_models() has been called once, so first unload all of + # the models; otherwise we might call load_models() with no + # arguments, end up with _all_ models loaded, and then in a + # subsequent call only load a subset of models + for model in MODELS: + delattr(sys.modules[__name__], model) + MODELS = [] + + for plugin in plugins: + try: + mod = getattr(__import__("Bcfg2.Server.Plugins.%s" % + plugin).Server.Plugins, plugin) + except ImportError: + try: + mod = __import__(plugin) + except: + if plugins != Bcfg2.Server.Plugins.__all__: + # only produce errors if the default plugin list + # was not used -- i.e., if the config file was set + # up. don't produce errors when trying to load + # all plugins, IOW + err = sys.exc_info()[1] + logger.error("Failed to load plugin %s: %s" % (plugin, err)) + continue + for sym in dir(mod): + obj = getattr(mod, sym) + if hasattr(obj, "__bases__") and models.Model in obj.__bases__: + print("Adding %s to models" % sym) + setattr(sys.modules[__name__], sym, obj) + MODELS.append(sym) + +# basic invocation to ensure that a default set of models is loaded, +# and thus that this module will always work. +load_models(quiet=True) diff --git a/src/lib/Bcfg2/manage.py b/src/lib/Bcfg2/manage.py new file mode 100755 index 000000000..3e4eedc9f --- /dev/null +++ b/src/lib/Bcfg2/manage.py @@ -0,0 +1,14 @@ +#!/usr/bin/env python +from django.core.management import execute_manager +import imp +try: + imp.find_module('settings') # Assumed to be in the same directory. +except ImportError: + import sys + sys.stderr.write("Error: Can't find the file 'settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n" % __file__) + sys.exit(1) + +import settings + +if __name__ == "__main__": + execute_manager(settings) diff --git a/src/lib/Bcfg2/settings.py b/src/lib/Bcfg2/settings.py new file mode 100644 index 000000000..5de590fec --- /dev/null +++ b/src/lib/Bcfg2/settings.py @@ -0,0 +1,71 @@ +import sys +import django +import Bcfg2.Options + +DATABASES = dict() + +# Django < 1.2 compat +DATABASE_ENGINE = None +DATABASE_NAME = None +DATABASE_USER = None +DATABASE_PASSWORD = None +DATABASE_HOST = None +DATABASE_PORT = None + +def read_config(cfile='/etc/bcfg2.conf', repo=None, quiet=False): + global DATABASE_ENGINE, DATABASE_NAME, DATABASE_USER, DATABASE_PASSWORD, \ + DATABASE_HOST, DATABASE_PORT + + setup = \ + Bcfg2.Options.OptionParser(dict(repo=Bcfg2.Options.SERVER_REPOSITORY, + configfile=Bcfg2.Options.CFILE, + db_engine=Bcfg2.Options.DB_ENGINE, + db_name=Bcfg2.Options.DB_NAME, + db_user=Bcfg2.Options.DB_USER, + db_password=Bcfg2.Options.DB_PASSWORD, + db_host=Bcfg2.Options.DB_HOST, + db_port=Bcfg2.Options.DB_PORT), + quiet=quiet) + setup.parse([Bcfg2.Options.CFILE.cmd, cfile]) + + if repo is None: + repo = setup['repo'] + + DATABASES['default'] = \ + dict(ENGINE=setup['db_engine'], + NAME=setup['db_name'], + USER=setup['db_user'], + PASSWORD=setup['db_password'], + HOST=setup['db_host'], + PORT=setup['db_port']) + + if django.VERSION[0] == 1 and django.VERSION[1] < 2: + DATABASE_ENGINE = setup['db_engine'] + DATABASE_NAME = DATABASES['default']['NAME'] + DATABASE_USER = DATABASES['default']['USER'] + DATABASE_PASSWORD = DATABASES['default']['PASSWORD'] + DATABASE_HOST = DATABASES['default']['HOST'] + DATABASE_PORT = DATABASES['default']['PORT'] + +# initialize settings from /etc/bcfg2.conf, or set up basic defaults. +# this lets manage.py work in all cases +read_config(quiet=True) + +if django.VERSION[0] == 1 and django.VERSION[1] > 2: + TIME_ZONE = None + +DEBUG = False +TEMPLATE_DEBUG = DEBUG + +ADMINS = (('Root', 'root')) +MANAGERS = ADMINS + +# Language code for this installation. All choices can be found here: +# http://www.w3.org/TR/REC-html40/struct/dirlang.html#langcodes +# http://blogs.law.harvard.edu/tech/stories/storyReader$15 +LANGUAGE_CODE = 'en-us' + +SITE_ID = 1 + +INSTALLED_APPS = ('Bcfg2.Server') + diff --git a/src/sbin/bcfg2-info b/src/sbin/bcfg2-info index 8754fb066..28f4d17ac 100755 --- a/src/sbin/bcfg2-info +++ b/src/sbin/bcfg2-info @@ -102,7 +102,7 @@ def getClientList(hostglobs): """ given a host glob, get a list of clients that match it """ # special cases to speed things up: if '*' in hostglobs: - return list(self.metadata.clients.keys()) + return self.metadata.clients has_wildcards = False for glob in hostglobs: # check if any wildcard characters are in the string @@ -113,7 +113,7 @@ def getClientList(hostglobs): return hostglobs rv = set() - clist = set(self.metadata.clients.keys()) + clist = set(self.metadata.clients) for glob in hostglobs: for client in clist: if fnmatch.fnmatch(client, glob): @@ -328,7 +328,7 @@ class infoCore(cmd.Cmd, Bcfg2.Server.Core.BaseCore): if len(alist) > 1: clients = getClientList(alist[1:]) else: - clients = list(self.metadata.clients.keys()) + clients = self.metadata.clients for client in clients: self.do_build("%s %s" % (client, os.path.join(destdir, client + ".xml"))) @@ -360,7 +360,7 @@ class infoCore(cmd.Cmd, Bcfg2.Server.Core.BaseCore): if len(args) > 2: clients = getClientList(args[1:]) else: - clients = list(self.metadata.clients.keys()) + clients = self.metadata.clients if altsrc: args = "--altsrc %s -f %%s %%s %%s" % altsrc else: @@ -448,10 +448,11 @@ class infoCore(cmd.Cmd, Bcfg2.Server.Core.BaseCore): def do_clients(self, _): """Print out client info.""" data = [('Client', 'Profile')] - clist = list(self.metadata.clients.keys()) + clist = self.metadata.clients clist.sort() for client in clist: - data.append((client, self.metadata.clients[client])) + imd = self.metadata.get_initial_metadata(client) + data.append((client, imd.profile)) printTabular(data) def do_config(self, _): diff --git a/testsuite/Testlib/TestServer/TestPlugins/TestDBMetadata.py b/testsuite/Testlib/TestServer/TestPlugins/TestDBMetadata.py new file mode 100644 index 000000000..99cbf1962 --- /dev/null +++ b/testsuite/Testlib/TestServer/TestPlugins/TestDBMetadata.py @@ -0,0 +1,407 @@ +import os +import sys +import unittest +import lxml.etree +from mock import Mock, patch +from django.core.management import setup_environ + +os.environ['DJANGO_SETTINGS_MODULE'] = "Bcfg2.settings" + +import Bcfg2.settings +Bcfg2.settings.DATABASE_NAME = \ + os.path.join(os.path.dirname(os.path.abspath(__file__)), "test.sqlite") +Bcfg2.settings.DATABASES['default']['NAME'] = Bcfg2.settings.DATABASE_NAME + +import Bcfg2.Server.Plugin +from Bcfg2.Server.Plugins.DBMetadata import * + +from TestMetadata import datastore, groups_test_tree, clients_test_tree, \ + TestMetadata + +def test_syncdb(): + # create the test database + setup_environ(Bcfg2.settings) + from django.core.management.commands import syncdb + cmd = syncdb.Command() + cmd.handle_noargs(interactive=False) + assert os.path.exists(Bcfg2.settings.DATABASE_NAME) + + # ensure that we a) can connect to the database; b) start with a + # clean database + MetadataClientModel.objects.all().delete() + assert list(MetadataClientModel.objects.all()) == [] + + +class TestClientVersions(unittest.TestCase): + test_clients = dict(client1="1.2.0", + client2="1.2.2", + client3="1.3.0pre1", + client4="1.1.0", + client5=None, + client6=None) + + def setUp(self): + test_syncdb() + for client, version in self.test_clients.items(): + MetadataClientModel(hostname=client, version=version).save() + + def test__contains(self): + v = ClientVersions() + self.assertIn("client1", v) + self.assertIn("client5", v) + self.assertNotIn("client__contains", v) + + def test_keys(self): + v = ClientVersions() + self.assertItemsEqual(self.test_clients.keys(), v.keys()) + + def test__setitem(self): + v = ClientVersions() + + # test setting version of existing client + v["client1"] = "1.2.3" + self.assertIn("client1", v) + self.assertEqual(v['client1'], "1.2.3") + client = MetadataClientModel.objects.get(hostname="client1") + self.assertEqual(client.version, "1.2.3") + + # test adding new client + new = "client__setitem" + v[new] = "1.3.0" + self.assertIn(new, v) + self.assertEqual(v[new], "1.3.0") + client = MetadataClientModel.objects.get(hostname=new) + self.assertEqual(client.version, "1.3.0") + + # test adding new client with no version + new2 = "client__setitem_2" + v[new2] = None + self.assertIn(new2, v) + self.assertEqual(v[new2], None) + client = MetadataClientModel.objects.get(hostname=new2) + self.assertEqual(client.version, None) + + def test__getitem(self): + v = ClientVersions() + + # test getting existing client + self.assertEqual(v['client2'], "1.2.2") + self.assertIsNone(v['client5']) + + # test exception on nonexistent client. can't use assertRaises + # for this because assertRaises requires a callable + try: + v['clients__getitem'] + assert False + except KeyError: + assert True + except: + assert False + + +class TestDBMetadataBase(TestMetadata): + __test__ = False + + def __init__(self, *args, **kwargs): + TestMetadata.__init__(self, *args, **kwargs) + test_syncdb() + + def load_clients_data(self, metadata=None, xdata=None): + if metadata is None: + metadata = get_metadata_object() + for client in clients_test_tree.findall("Client"): + metadata.add_client(client.get("name")) + return metadata + + def get_metadata_object(self, core=None, watch_clients=False): + if core is None: + core = Mock() + metadata = DBMetadata(core, datastore, watch_clients=watch_clients) + return metadata + + def get_nonexistent_client(self, _, prefix="client"): + clients = [o.hostname for o in MetadataClientModel.objects.all()] + i = 0 + client_name = "%s%s" % (prefix, i) + while client_name in clients: + i += 1 + client_name = "%s%s" % (prefix, i) + return client_name + + @patch('os.path.exists') + def test__init(self, mock_exists): + core = Mock() + core.fam = Mock() + mock_exists.return_value = False + metadata = self.get_metadata_object(core=core, watch_clients=True) + self.assertIsInstance(metadata, Bcfg2.Server.Plugin.DatabaseBacked) + core.fam.AddMonitor.assert_called_once_with(os.path.join(metadata.data, + "groups.xml"), + metadata) + + mock_exists.return_value = True + core.fam.reset_mock() + metadata = self.get_metadata_object(core=core, watch_clients=True) + core.fam.AddMonitor.assert_any_call(os.path.join(metadata.data, + "groups.xml"), + metadata) + core.fam.AddMonitor.assert_any_call(os.path.join(metadata.data, + "clients.xml"), + metadata) + + def test_add_group(self): + pass + + def test_add_bundle(self): + pass + + def test_add_client(self): + metadata = self.get_metadata_object() + hostname = self.get_nonexistent_client(metadata) + client = metadata.add_client(hostname) + self.assertIsInstance(client, MetadataClientModel) + self.assertEqual(client.hostname, hostname) + self.assertIn(hostname, metadata.clients) + self.assertIn(hostname, metadata.list_clients()) + self.assertItemsEqual(metadata.clients, + [c.hostname + for c in MetadataClientModel.objects.all()]) + + def test_update_group(self): + pass + + def test_update_bundle(self): + pass + + def test_update_client(self): + pass + + def test_list_clients(self): + metadata = self.get_metadata_object() + self.assertItemsEqual(metadata.list_clients(), + [c.hostname + for c in MetadataClientModel.objects.all()]) + + def test_remove_group(self): + pass + + def test_remove_bundle(self): + pass + + def test_remove_client(self): + metadata = self.get_metadata_object() + client_name = self.get_nonexistent_client(metadata) + + self.assertRaises(MetadataConsistencyError, + metadata.remove_client, + client_name) + + metadata.add_client(client_name) + metadata.remove_client(client_name) + self.assertNotIn(client_name, metadata.clients) + self.assertNotIn(client_name, metadata.list_clients()) + self.assertItemsEqual(metadata.clients, + [c.hostname + for c in MetadataClientModel.objects.all()]) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("Bcfg2.Server.Plugins.DBMetadata.DBMetadata._set_profile") + def test_set_profile(self, mock_set_profile): + TestMetadata.test_set_profile(self, + inherited_set_profile=mock_set_profile) + + def test__set_profile(self): + metadata = self.get_metadata_object() + profile = "group1" + client_name = self.get_nonexistent_client(metadata) + metadata._set_profile(client_name, profile, None) + self.assertIn(client_name, metadata.list_clients()) + self.assertIn(client_name, metadata.clientgroups) + self.assertItemsEqual(metadata.clientgroups[client_name], [profile]) + + self.assertRaises(Bcfg2.Server.Plugin.PluginExecutionError, + metadata._set_profile, + client_name, profile, None) + + def test_process_statistics(self): + pass + + +class TestDBMetadata_NoClientsXML(TestDBMetadataBase): + """ test DBMetadata without a clients.xml. we have to disable or + override tests that rely on client options """ + __test__ = True + + def __init__(self, *args, **kwargs): + TestMetadata.__init__(self, *args, **kwargs) + + for client in self.clients_test_tree.findall("Client"): + newclient = lxml.etree.SubElement(self.groups_test_tree.getroot(), + "Client", name=client.get("name")) + lxml.etree.SubElement(newclient, "Group", + name=client.get("profile")) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.write_xml", Mock()) + @patch("Bcfg2.Server.Plugins.Metadata.ClientMetadata") + def test_get_initial_metadata(self, mock_clientmetadata): + metadata = self.get_metadata_object() + if 'clients.xml' in metadata.states: + metadata.states['clients.xml'] = False + self.assertRaises(MetadataRuntimeError, + metadata.get_initial_metadata, None) + + self.load_groups_data(metadata=metadata) + self.load_clients_data(metadata=metadata) + + # test basic client metadata + metadata.get_initial_metadata("client1") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client1", "group1", set(["group1"]), set(), set(), + set(), dict(category1='group1'), None, None)) + + # test bundles, category suppression + metadata.get_initial_metadata("client2") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client2", "group2", set(["group2"]), + set(["bundle1", "bundle2"]), set(), set(), + dict(category1="group2"), None, None)) + + # test new client creation + new1 = self.get_nonexistent_client(metadata) + imd = metadata.get_initial_metadata(new1) + self.assertEqual(mock_clientmetadata.call_args[0][:9], + (new1, "group1", set(["group1"]), set(), set(), set(), + dict(category1="group1"), None, None)) + + # test nested groups, per-client groups + imd = metadata.get_initial_metadata("client8") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client8", "group1", + set(["group1", "group8", "group9", "group10"]), set(), + set(), set(), dict(category1="group1"), None, None)) + + # test per-client groups, group negation, nested groups + imd = metadata.get_initial_metadata("client9") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client9", "group2", + set(["group2", "group8", "group11"]), + set(["bundle1", "bundle2"]), set(), set(), + dict(category1="group2"), None, None)) + + # test exception on new client with no default profile + metadata.default = None + new2 = self.get_nonexistent_client(metadata) + self.assertRaises(MetadataConsistencyError, + metadata.get_initial_metadata, + new2) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("Bcfg2.Server.Plugins.Metadata.Metadata.resolve_client") + def test_validate_client_address(self, mock_resolve_client): + metadata = self.load_clients_data(metadata=self.load_groups_data()) + # this is upper case to ensure that case is folded properly in + # validate_client_address() + mock_resolve_client.return_value = "CLIENT4" + self.assertTrue(metadata.validate_client_address("client4", + ("1.2.3.7", None))) + mock_resolve_client.assert_called_with(("1.2.3.7", None)) + + mock_resolve_client.reset_mock() + self.assertFalse(metadata.validate_client_address("client5", + ("1.2.3.5", None))) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("Bcfg2.Server.Plugins.Metadata.Metadata.validate_client_address") + @patch("Bcfg2.Server.Plugins.Metadata.Metadata.resolve_client") + def test_AuthenticateConnection(self, mock_resolve_client, + mock_validate_client_address): + metadata = self.load_clients_data(metadata=self.load_groups_data()) + metadata.password = "password1" + + cert = dict(subject=[[("commonName", "client1")]]) + mock_validate_client_address.return_value = False + self.assertFalse(metadata.AuthenticateConnection(cert, "root", None, + "1.2.3.1")) + mock_validate_client_address.return_value = True + self.assertTrue(metadata.AuthenticateConnection(cert, "root", + metadata.password, + "1.2.3.1")) + + cert = dict(subject=[[("commonName", "client8")]]) + + mock_resolve_client.return_value = "client5" + self.assertTrue(metadata.AuthenticateConnection(None, "root", + "password1", "1.2.3.8")) + + mock_resolve_client.side_effect = MetadataConsistencyError + self.assertFalse(metadata.AuthenticateConnection(None, "root", + "password1", + "1.2.3.8")) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("socket.gethostbyaddr") + def test_resolve_client(self, mock_gethostbyaddr): + metadata = self.load_clients_data(metadata=self.load_groups_data()) + metadata.session_cache[('1.2.3.3', None)] = (time.time(), 'client3') + self.assertEqual(metadata.resolve_client(('1.2.3.3', None)), 'client3') + + metadata.session_cache[('1.2.3.3', None)] = (time.time() - 100, + 'client3') + mock_gethostbyaddr.return_value = ("client3", [], ['1.2.3.3']) + self.assertEqual(metadata.resolve_client(('1.2.3.3', None), + cleanup_cache=True), 'client3') + self.assertEqual(metadata.session_cache, dict()) + + mock_gethostbyaddr.return_value = ('client6', [], ['1.2.3.6']) + self.assertEqual(metadata.resolve_client(('1.2.3.6', None)), 'client6') + mock_gethostbyaddr.assert_called_with('1.2.3.6') + + mock_gethostbyaddr.reset_mock() + mock_gethostbyaddr.return_value = None + mock_gethostbyaddr.side_effect = socket.herror + self.assertRaises(MetadataConsistencyError, + metadata.resolve_client, + ('1.2.3.8', None)) + mock_gethostbyaddr.assert_called_with('1.2.3.8') + + def test_clients_xml_event(self): + pass + + +class TestDBMetadata_ClientsXML(TestDBMetadataBase): + """ test DBMetadata with a clients.xml. """ + __test__ = True + + def load_clients_data(self, metadata=None, xdata=None): + if metadata is None: + metadata = self.get_metadata_object() + metadata.core.fam = Mock() + metadata._handle_file("clients.xml") + metadata = TestMetadata.load_clients_data(self, metadata=metadata, + xdata=xdata) + return TestDBMetadataBase.load_clients_data(self, metadata=metadata, + xdata=xdata) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml") + @patch("Bcfg2.Server.Plugins.Metadata.Metadata._handle_clients_xml_event") + @patch("Bcfg2.Server.Plugins.DBMetadata.DBMetadata.list_clients") + def test_clients_xml_event(self, mock_list_clients, mock_handle_event, + mock_load_xml): + metadata = self.get_metadata_object() + metadata.profiles = ["group1", "group2"] + evt = Mock() + evt.filename = os.path.join(datastore, "DBMetadata", "clients.xml") + evt.code2str = Mock(return_value="changed") + metadata.HandleEvent(evt) + self.assertFalse(mock_handle_event.called) + self.assertFalse(mock_load_xml.called) + + mock_load_xml.reset_mock() + mock_handle_event.reset_mock() + mock_list_clients.reset_mock() + metadata._handle_file("clients.xml") + metadata.HandleEvent(evt) + mock_handle_event.assert_called_with(metadata, evt) + mock_list_clients.assert_any_call() + mock_load_xml.assert_any_call() diff --git a/testsuite/Testlib/TestServer/TestPlugins/TestMetadata.py b/testsuite/Testlib/TestServer/TestPlugins/TestMetadata.py index 8ea54a1e8..a0a3aaee1 100644 --- a/testsuite/Testlib/TestServer/TestPlugins/TestMetadata.py +++ b/testsuite/Testlib/TestServer/TestPlugins/TestMetadata.py @@ -31,10 +31,18 @@ clients_test_tree = lxml.etree.XML(''' <Client name="client8" profile="group1" auth="cert+password" address="1.2.3.5"/> <Client name="client9" profile="group2" secure="true" password="password3"/> + <Client name="client10" profile="group1" floating="true"/> </Clients>''').getroottree() groups_test_tree = lxml.etree.XML(''' <Groups xmlns:xi="http://www.w3.org/2001/XInclude"> + <Client name="client8"> + <Group name="group8"/> + </Client> + <Client name="client9"> + <Group name="group8"/> + </Client> + <Group name="group1" default="true" profile="true" public="true" category="category1"/> <Group name="group2" profile="true" public="true" category="category1"> @@ -54,6 +62,13 @@ groups_test_tree = lxml.etree.XML(''' </Group> <Group name="group8"> <Group name="group9"/> + <Client name="client9"> + <Group name="group11"/> + <Group name="group9" negate="true"/> + </Client> + <Group name="group1"> + <Group name="group10"/> + </Group> </Group> </Groups>''').getroottree() @@ -63,11 +78,13 @@ def get_metadata_object(core=None, watch_clients=False): if core is None: core = Mock() metadata = Metadata(core, datastore, watch_clients=watch_clients) - #metadata.logger = Mock() return metadata class TestXMLMetadataConfig(unittest.TestCase): + groups_test_tree = groups_test_tree + clients_test_tree = clients_test_tree + def get_config_object(self, basefile="clients.xml", core=None, watch_clients=False): self.metadata = get_metadata_object(core=core, @@ -79,8 +96,9 @@ class TestXMLMetadataConfig(unittest.TestCase): # we can't use assertRaises here because xdata is a property try: config.xdata + assert False except MetadataRuntimeError: - pass + assert True except: assert False config.data = "<test/>" @@ -91,8 +109,9 @@ class TestXMLMetadataConfig(unittest.TestCase): # we can't use assertRaises here because base_xdata is a property try: config.base_xdata + assert False except MetadataRuntimeError: - pass + assert True except: assert False config.basedata = "<test/>" @@ -103,16 +122,16 @@ class TestXMLMetadataConfig(unittest.TestCase): config = self.get_config_object(core=core) fname = "test.xml" - fpath = os.path.join(self.metadata.data, "test.xml") + fpath = os.path.join(self.metadata.data, fname) config.extras = [] config.add_monitor(fpath, fname) self.assertFalse(core.fam.AddMonitor.called) - self.assertEqual(config.extras, []) + self.assertEqual(config.extras, [fname]) config = self.get_config_object(core=core, watch_clients=True) config.add_monitor(fpath, fname) - core.fam.AddMonitor.assert_called_with(fpath, self.metadata) + core.fam.AddMonitor.assert_called_with(fpath, config.metadata) self.assertItemsEqual(config.extras, [fname]) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.add_monitor") @@ -174,7 +193,7 @@ class TestXMLMetadataConfig(unittest.TestCase): mock_islink.return_value = False - config.write_xml(fpath, clients_test_tree) + config.write_xml(fpath, self.clients_test_tree) mock_open.assert_called_with(tmpfile, "w") self.assertTrue(mock_open.return_value.write.called) mock_islink.assert_called_with(fpath) @@ -182,33 +201,33 @@ class TestXMLMetadataConfig(unittest.TestCase): mock_islink.return_value = True mock_readlink.return_value = linkdest - config.write_xml(fpath, clients_test_tree) + config.write_xml(fpath, self.clients_test_tree) mock_rename.assert_called_with(tmpfile, linkdest) mock_rename.side_effect = OSError self.assertRaises(MetadataRuntimeError, - config.write_xml, fpath, clients_test_tree) + config.write_xml, fpath, self.clients_test_tree) mock_open.return_value.write.side_effect = IOError self.assertRaises(MetadataRuntimeError, - config.write_xml, fpath, clients_test_tree) + config.write_xml, fpath, self.clients_test_tree) mock_unlink.assert_called_with(tmpfile) mock_open.side_effect = IOError self.assertRaises(MetadataRuntimeError, - config.write_xml, fpath, clients_test_tree) + config.write_xml, fpath, self.clients_test_tree) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) @patch('lxml.etree.parse') def test_find_xml_for_xpath(self, mock_parse): config = self.get_config_object("groups.xml") - config.basedata = groups_test_tree + config.basedata = self.groups_test_tree xpath = "//Group[@name='group1']" self.assertItemsEqual(config.find_xml_for_xpath(xpath), dict(filename=os.path.join(self.metadata.data, "groups.xml"), - xmltree=groups_test_tree, - xquery=groups_test_tree.xpath(xpath))) + xmltree=self.groups_test_tree, + xquery=self.groups_test_tree.xpath(xpath))) self.assertEqual(config.find_xml_for_xpath("//boguselement"), dict()) @@ -216,7 +235,7 @@ class TestXMLMetadataConfig(unittest.TestCase): def parse_side_effect(fname, parser=Bcfg2.Server.XMLParser): if fname == os.path.join(self.metadata.data, "clients.xml"): - return clients_test_tree + return self.clients_test_tree else: return lxml.etree.XML("<null/>").getroottree() @@ -225,8 +244,8 @@ class TestXMLMetadataConfig(unittest.TestCase): self.assertItemsEqual(config.find_xml_for_xpath(xpath), dict(filename=os.path.join(self.metadata.data, "clients.xml"), - xmltree=clients_test_tree, - xquery=clients_test_tree.xpath(xpath))) + xmltree=self.clients_test_tree, + xquery=self.clients_test_tree.xpath(xpath))) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml") def test_HandleEvent(self, mock_load_xml): @@ -247,18 +266,37 @@ class TestClientMetadata(unittest.TestCase): class TestMetadata(unittest.TestCase): - def test__init_no_fam(self): + groups_test_tree = groups_test_tree + clients_test_tree = clients_test_tree + + def get_metadata_object(self, core=None, watch_clients=False): + return get_metadata_object(core=core, watch_clients=watch_clients) + + def get_nonexistent_client(self, metadata, prefix="client"): + if metadata is None: + metadata = self.load_clients_data() + i = 0 + client_name = "%s%s" % (prefix, i) + while client_name in metadata.clients: + i += 1 + client_name = "%s%s" % (prefix, i) + return client_name + + def test__init(self): # test with watch_clients=False core = Mock() - metadata = get_metadata_object(core=core) - self.check_metadata_object(metadata) + metadata = self.get_metadata_object(core=core) + self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Plugin) + self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Metadata) + self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Statistics) + self.assertIsInstance(metadata.clients_xml, XMLMetadataConfig) + self.assertIsInstance(metadata.groups_xml, XMLMetadataConfig) + self.assertIsInstance(metadata.query, MetadataQuery) self.assertEqual(metadata.states, dict()) - def test__init_with_fam(self): # test with watch_clients=True - core = Mock() core.fam = Mock() - metadata = get_metadata_object(core=core, watch_clients=True) + metadata = self.get_metadata_object(core=core, watch_clients=True) self.assertEqual(len(metadata.states), 2) core.fam.AddMonitor.assert_any_call(os.path.join(metadata.data, "groups.xml"), @@ -270,64 +308,44 @@ class TestMetadata(unittest.TestCase): core.fam.reset_mock() core.fam.AddMonitor = Mock(side_effect=IOError) self.assertRaises(Bcfg2.Server.Plugin.PluginInitError, - get_metadata_object, + self.get_metadata_object, core=core, watch_clients=True) - def check_metadata_object(self, metadata): - self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Plugin) - self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Metadata) - self.assertIsInstance(metadata, Bcfg2.Server.Plugin.Statistics) - self.assertIsInstance(metadata.clients_xml, XMLMetadataConfig) - self.assertIsInstance(metadata.groups_xml, XMLMetadataConfig) - self.assertIsInstance(metadata.query, MetadataQuery) - @patch('os.makedirs', Mock()) @patch('__builtin__.open') def test_init_repo(self, mock_open): - groups = "groups %s" - os_selection = "os" - clients = "clients %s" - Metadata.init_repo(datastore, groups, os_selection, clients) + Metadata.init_repo(datastore, + groups_xml="groups", clients_xml="clients") mock_open.assert_any_call(os.path.join(datastore, "Metadata", "groups.xml"), "w") mock_open.assert_any_call(os.path.join(datastore, "Metadata", "clients.xml"), "w") - @patch('lxml.etree.parse') - def test_get_groups(self, mock_parse): - metadata = get_metadata_object() - mock_parse.return_value = groups_test_tree - groups = metadata.get_groups() - mock_parse.assert_called_with(os.path.join(datastore, "Metadata", - "groups.xml"), - parser=Bcfg2.Server.XMLParser) - self.assertIsInstance(groups, lxml.etree._Element) - - def test_search_xdata_name(self): + def test_search_xdata(self): # test finding a node with the proper name - metadata = get_metadata_object() - tree = groups_test_tree + metadata = self.get_metadata_object() + tree = self.groups_test_tree res = metadata._search_xdata("Group", "group1", tree) self.assertIsInstance(res, lxml.etree._Element) self.assertEqual(res.get("name"), "group1") - def test_search_xdata_alias(self): # test finding a node with the wrong name but correct alias - metadata = get_metadata_object() - tree = clients_test_tree + metadata = self.get_metadata_object() + tree = self.clients_test_tree res = metadata._search_xdata("Client", "alias3", tree, alias=True) self.assertIsInstance(res, lxml.etree._Element) self.assertNotEqual(res.get("name"), "alias3") - def test_search_xdata_not_found(self): # test failure finding a node - metadata = get_metadata_object() - tree = clients_test_tree - res = metadata._search_xdata("Client", "bogus_client", tree, alias=True) + metadata = self.get_metadata_object() + tree = self.clients_test_tree + res = metadata._search_xdata("Client", + self.get_nonexistent_client(metadata), + tree, alias=True) self.assertIsNone(res) def search_xdata(self, tag, name, tree, alias=False): - metadata = get_metadata_object() + metadata = self.get_metadata_object() res = metadata._search_xdata(tag, name, tree, alias=alias) self.assertIsInstance(res, lxml.etree._Element) if not alias: @@ -335,22 +353,22 @@ class TestMetadata(unittest.TestCase): def test_search_group(self): # test finding a group with the proper name - tree = groups_test_tree + tree = self.groups_test_tree self.search_xdata("Group", "group1", tree) def test_search_bundle(self): # test finding a bundle with the proper name - tree = groups_test_tree + tree = self.groups_test_tree self.search_xdata("Bundle", "bundle1", tree) def test_search_client(self): # test finding a client with the proper name - tree = clients_test_tree + tree = self.clients_test_tree self.search_xdata("Client", "client1", tree, alias=True) self.search_xdata("Client", "alias1", tree, alias=True) def test_add_group(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.groups_xml.write = Mock() metadata.groups_xml.data = lxml.etree.XML('<Groups/>').getroottree() metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) @@ -382,9 +400,9 @@ class TestMetadata(unittest.TestCase): self.assertFalse(metadata.groups_xml.write.called) def test_update_group(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.groups_xml.write_xml = Mock() - metadata.groups_xml.data = copy.deepcopy(groups_test_tree) + metadata.groups_xml.data = copy.deepcopy(self.groups_test_tree) metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) metadata.update_group("group1", dict(foo="bar")) @@ -399,9 +417,9 @@ class TestMetadata(unittest.TestCase): "bogus_group", dict()) def test_remove_group(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.groups_xml.write_xml = Mock() - metadata.groups_xml.data = copy.deepcopy(groups_test_tree) + metadata.groups_xml.data = copy.deepcopy(self.groups_test_tree) metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) metadata.remove_group("group5") @@ -414,7 +432,7 @@ class TestMetadata(unittest.TestCase): "bogus_group") def test_add_bundle(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.groups_xml.write = Mock() metadata.groups_xml.data = lxml.etree.XML('<Groups/>').getroottree() metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) @@ -437,9 +455,9 @@ class TestMetadata(unittest.TestCase): self.assertFalse(metadata.groups_xml.write.called) def test_remove_bundle(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.groups_xml.write_xml = Mock() - metadata.groups_xml.data = copy.deepcopy(groups_test_tree) + metadata.groups_xml.data = copy.deepcopy(self.groups_test_tree) metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) metadata.remove_bundle("bundle1") @@ -452,26 +470,29 @@ class TestMetadata(unittest.TestCase): "bogus_bundle") def test_add_client(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.clients_xml.write = Mock() metadata.clients_xml.data = lxml.etree.XML('<Clients/>').getroottree() metadata.clients_xml.basedata = copy.copy(metadata.clients_xml.data) - metadata.add_client("test1", dict()) + new1 = self.get_nonexistent_client(metadata) + metadata.add_client(new1, dict()) metadata.clients_xml.write.assert_any_call() - grp = metadata.search_client("test1", metadata.clients_xml.base_xdata) + grp = metadata.search_client(new1, metadata.clients_xml.base_xdata) self.assertIsNotNone(grp) - self.assertEqual(grp.attrib, dict(name='test1')) + self.assertEqual(grp.attrib, dict(name=new1)) # have to call this explicitly -- usually load_xml does this # on FAM events metadata.clients_xml.basedata = copy.copy(metadata.clients_xml.data) + metadata._handle_clients_xml_event(Mock()) - metadata.add_client("test2", dict(foo='bar')) + new2 = self.get_nonexistent_client(metadata) + metadata.add_client(new2, dict(foo='bar')) metadata.clients_xml.write.assert_any_call() - grp = metadata.search_client("test2", metadata.clients_xml.base_xdata) + grp = metadata.search_client(new2, metadata.clients_xml.base_xdata) self.assertIsNotNone(grp) - self.assertEqual(grp.attrib, dict(name='test2', foo='bar')) + self.assertEqual(grp.attrib, dict(name=new2, foo='bar')) # have to call this explicitly -- usually load_xml does this # on FAM events @@ -480,13 +501,13 @@ class TestMetadata(unittest.TestCase): metadata.clients_xml.write.reset_mock() self.assertRaises(MetadataConsistencyError, metadata.add_client, - "test1", dict()) + new1, dict()) self.assertFalse(metadata.clients_xml.write.called) def test_update_client(self): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.clients_xml.write_xml = Mock() - metadata.clients_xml.data = copy.deepcopy(clients_test_tree) + metadata.clients_xml.data = copy.deepcopy(self.clients_test_tree) metadata.clients_xml.basedata = copy.copy(metadata.clients_xml.data) metadata.update_client("client1", dict(foo="bar")) @@ -496,14 +517,15 @@ class TestMetadata(unittest.TestCase): self.assertEqual(grp.get("foo"), "bar") self.assertTrue(metadata.clients_xml.write_xml.called) + new = self.get_nonexistent_client(metadata) self.assertRaises(MetadataConsistencyError, metadata.update_client, - "bogus_client", dict()) + new, dict()) def load_clients_data(self, metadata=None, xdata=None): if metadata is None: - metadata = get_metadata_object() - metadata.clients_xml.data = xdata or copy.deepcopy(clients_test_tree) + metadata = self.get_metadata_object() + metadata.clients_xml.data = xdata or copy.deepcopy(self.clients_test_tree) metadata.clients_xml.basedata = copy.copy(metadata.clients_xml.data) evt = Mock() evt.filename = os.path.join(datastore, "Metadata", "clients.xml") @@ -513,39 +535,38 @@ class TestMetadata(unittest.TestCase): @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml") def test_clients_xml_event(self, mock_load_xml): - metadata = get_metadata_object() + metadata = self.get_metadata_object() metadata.profiles = ["group1", "group2"] self.load_clients_data(metadata=metadata) mock_load_xml.assert_any_call() self.assertItemsEqual(metadata.clients, dict([(c.get("name"), c.get("profile")) - for c in clients_test_tree.findall("//Client")])) + for c in self.clients_test_tree.findall("//Client")])) aliases = dict([(a.get("name"), a.getparent().get("name")) - for a in clients_test_tree.findall("//Alias")]) + for a in self.clients_test_tree.findall("//Alias")]) self.assertItemsEqual(metadata.aliases, aliases) raliases = dict([(c.get("name"), set()) - for c in clients_test_tree.findall("//Client")]) - for alias in clients_test_tree.findall("//Alias"): + for c in self.clients_test_tree.findall("//Client")]) + for alias in self.clients_test_tree.findall("//Alias"): raliases[alias.getparent().get("name")].add(alias.get("name")) self.assertItemsEqual(metadata.raliases, raliases) - self.assertEqual(metadata.bad_clients, dict()) self.assertEqual(metadata.secure, [c.get("name") - for c in clients_test_tree.findall("//Client[@secure='true']")]) - self.assertEqual(metadata.floating, ["client1"]) + for c in self.clients_test_tree.findall("//Client[@secure='true']")]) + self.assertEqual(metadata.floating, ["client1", "client10"]) addresses = dict([(c.get("address"), []) - for c in clients_test_tree.findall("//*[@address]")]) + for c in self.clients_test_tree.findall("//*[@address]")]) raddresses = dict() - for client in clients_test_tree.findall("//Client[@address]"): + for client in self.clients_test_tree.findall("//Client[@address]"): addresses[client.get("address")].append(client.get("name")) try: raddresses[client.get("name")].append(client.get("address")) except KeyError: raddresses[client.get("name")] = [client.get("address")] - for alias in clients_test_tree.findall("//Alias[@address]"): + for alias in self.clients_test_tree.findall("//Alias[@address]"): addresses[alias.get("address")].append(alias.getparent().get("name")) try: raddresses[alias.getparent().get("name")].append(alias.get("address")) @@ -556,25 +577,10 @@ class TestMetadata(unittest.TestCase): self.assertItemsEqual(metadata.raddresses, raddresses) self.assertTrue(metadata.states['clients.xml']) - @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) - def test_clients_xml_event_bad_clients(self): - metadata = get_metadata_object() - metadata.profiles = ["group2"] - self.load_clients_data(metadata=metadata) - clients = dict() - badclients = dict() - for client in clients_test_tree.findall("//Client"): - if client.get("profile") in metadata.profiles: - clients[client.get("name")] = client.get("profile") - else: - badclients[client.get("name")] = client.get("profile") - self.assertItemsEqual(metadata.clients, clients) - self.assertItemsEqual(metadata.bad_clients, badclients) - def load_groups_data(self, metadata=None, xdata=None): if metadata is None: - metadata = get_metadata_object() - metadata.groups_xml.data = xdata or copy.deepcopy(groups_test_tree) + metadata = self.get_metadata_object() + metadata.groups_xml.data = xdata or copy.deepcopy(self.groups_test_tree) metadata.groups_xml.basedata = copy.copy(metadata.groups_xml.data) evt = Mock() evt.filename = os.path.join(datastore, "Metadata", "groups.xml") @@ -584,34 +590,60 @@ class TestMetadata(unittest.TestCase): @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml") def test_groups_xml_event(self, mock_load_xml): - dup_data = copy.deepcopy(groups_test_tree) + dup_data = copy.deepcopy(self.groups_test_tree) lxml.etree.SubElement(dup_data.getroot(), "Group", name="group1") metadata = self.load_groups_data(xdata=dup_data) mock_load_xml.assert_any_call() - self.assertEqual(metadata.public, ["group1", "group2"]) - self.assertEqual(metadata.private, ["group3"]) - self.assertEqual(metadata.profiles, ["group1", "group2"]) + self.assertTrue(metadata.states['groups.xml']) + self.assertTrue(metadata.groups['group1'].is_public) + self.assertTrue(metadata.groups['group2'].is_public) + self.assertFalse(metadata.groups['group3'].is_public) + self.assertFalse(metadata.groups['group1'].is_private) + self.assertFalse(metadata.groups['group2'].is_private) + self.assertTrue(metadata.groups['group3'].is_private) + self.assertTrue(metadata.groups['group1'].is_profile) + self.assertTrue(metadata.groups['group2'].is_profile) + self.assertFalse(metadata.groups['group3'].is_profile) self.assertItemsEqual(metadata.groups.keys(), - [g.get("name") - for g in groups_test_tree.findall("/Group")]) - self.assertEqual(metadata.categories, - dict(group1="category1", - group2="category1", - group3="category2", - group4="category1")) + set(g.get("name") + for g in self.groups_test_tree.findall("//Group"))) + self.assertEqual(metadata.groups['group1'].category, 'category1') + self.assertEqual(metadata.groups['group2'].category, 'category1') + self.assertEqual(metadata.groups['group3'].category, 'category2') + self.assertEqual(metadata.groups['group4'].category, 'category1') self.assertEqual(metadata.default, "group1") - self.assertTrue(metadata.states['groups.xml']) + all_groups = [] + negated_groups = [] + for group in dup_data.xpath("//Groups/Client//*") + \ + dup_data.xpath("//Groups/Group//*"): + if group.tag == 'Group' and not group.getchildren(): + if group.get("negate", "false").lower() == 'true': + negated_groups.append(group.get("name")) + else: + all_groups.append(group.get("name")) + self.assertItemsEqual([g.name + for g in metadata.group_membership.values()], + all_groups) + self.assertItemsEqual([g.name + for g in metadata.negated_groups.values()], + negated_groups) + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) - @patch("Bcfg2.Server.Plugins.Metadata.Metadata.add_client") - @patch("Bcfg2.Server.Plugins.Metadata.Metadata.update_client") - def test_set_profile(self, mock_update_client, mock_add_client): - metadata = get_metadata_object() - metadata.states['clients.xml'] = False - self.assertRaises(MetadataRuntimeError, - metadata.set_profile, - None, None, None) + @patch("Bcfg2.Server.Plugins.Metadata.Metadata._set_profile") + def test_set_profile(self, mock_set_profile, inherited_set_profile=None): + if inherited_set_profile: + # allow a subclass of TestMetadata to patch a different + # _set_profile object and pass it in. this probably isn't + # the best way to accomplish that, but it seems to work. + mock_set_profile = inherited_set_profile + metadata = self.get_metadata_object() + if 'clients.xml' in metadata.states: + metadata.states['clients.xml'] = False + self.assertRaises(MetadataRuntimeError, + metadata.set_profile, + None, None, None) self.load_groups_data(metadata=metadata) self.load_clients_data(metadata=metadata) @@ -620,27 +652,47 @@ class TestMetadata(unittest.TestCase): metadata.set_profile, "client1", "group5", None) + self.assertRaises(MetadataConsistencyError, + metadata.set_profile, + "client1", "group3", None) + + metadata.set_profile("client1", "group5", None, force=True) + mock_set_profile.assert_called_with("client1", "group5", None) + + metadata.set_profile("client1", "group3", None, force=True) + mock_set_profile.assert_called_with("client1", "group3", None) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + @patch("Bcfg2.Server.Plugins.Metadata.Metadata.add_client") + @patch("Bcfg2.Server.Plugins.Metadata.Metadata.update_client") + def test__set_profile(self, mock_update_client, mock_add_client): + metadata = self.get_metadata_object() + self.load_groups_data(metadata=metadata) + self.load_clients_data(metadata=metadata) + metadata.clients_xml.write = Mock() - metadata.set_profile("client1", "group2", None) + metadata._set_profile("client1", "group2", None) mock_update_client.assert_called_with("client1", dict(profile="group2")) metadata.clients_xml.write.assert_any_call() - self.assertEqual(metadata.clients["client1"], "group2") + self.assertEqual(metadata.clientgroups["client1"], ["group2"]) metadata.clients_xml.write.reset_mock() - metadata.set_profile("client_new", "group1", None) - mock_add_client.assert_called_with("client_new", dict(profile="group1")) + new1 = self.get_nonexistent_client(metadata) + metadata._set_profile(new1, "group1", None) + mock_add_client.assert_called_with(new1, dict(profile="group1")) metadata.clients_xml.write.assert_any_call() - self.assertEqual(metadata.clients["client_new"], "group1") + self.assertEqual(metadata.clientgroups[new1], ["group1"]) - metadata.session_cache[('1.2.3.6', None)] = (None, 'client_new2') metadata.clients_xml.write.reset_mock() - metadata.set_profile("uuid_new", "group1", ('1.2.3.6', None)) - mock_add_client.assert_called_with("client_new2", + new2 = self.get_nonexistent_client(metadata) + metadata.session_cache[('1.2.3.6', None)] = (None, new2) + metadata._set_profile("uuid_new", "group1", ('1.2.3.6', None)) + mock_add_client.assert_called_with(new2, dict(uuid='uuid_new', profile="group1", address='1.2.3.6')) metadata.clients_xml.write.assert_any_call() - self.assertEqual(metadata.clients["uuid_new"], "group1") + self.assertEqual(metadata.clientgroups["uuid_new"], ["group1"]) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) @patch("socket.gethostbyaddr") @@ -682,63 +734,102 @@ class TestMetadata(unittest.TestCase): @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.write_xml", Mock()) @patch("Bcfg2.Server.Plugins.Metadata.ClientMetadata") def test_get_initial_metadata(self, mock_clientmetadata): - metadata = get_metadata_object() - metadata.states['clients.xml'] = False - self.assertRaises(MetadataRuntimeError, - metadata.get_initial_metadata, None) + metadata = self.get_metadata_object() + if 'clients.xml' in metadata.states: + metadata.states['clients.xml'] = False + self.assertRaises(MetadataRuntimeError, + metadata.get_initial_metadata, None) self.load_groups_data(metadata=metadata) self.load_clients_data(metadata=metadata) + # test address, password metadata.get_initial_metadata("client1") self.assertEqual(mock_clientmetadata.call_args[0][:9], ("client1", "group1", set(["group1"]), set(), set(), - set(["1.2.3.1"]), dict(), None, 'password2')) + set(["1.2.3.1"]), dict(category1='group1'), None, + 'password2')) + # test address, bundles, category suppression metadata.get_initial_metadata("client2") self.assertEqual(mock_clientmetadata.call_args[0][:9], - ("client2", "group2", set(["group1", "group2"]), + ("client2", "group2", set(["group2"]), set(["bundle1", "bundle2"]), set(), - set(["1.2.3.2"]), dict(category1="group1"), + set(["1.2.3.2"]), dict(category1="group2"), None, None)) + # test aliases, address, uuid, password imd = metadata.get_initial_metadata("alias1") self.assertEqual(mock_clientmetadata.call_args[0][:9], ("client3", "group1", set(["group1"]), set(), - set(['alias1']), set(["1.2.3.3"]), dict(), 'uuid1', - 'password2')) + set(['alias1']), set(["1.2.3.3"]), + dict(category1="group1"), 'uuid1', 'password2')) - imd = metadata.get_initial_metadata("client_new") + # test new client creation + new1 = self.get_nonexistent_client(metadata) + imd = metadata.get_initial_metadata(new1) self.assertEqual(mock_clientmetadata.call_args[0][:9], - ("client_new", "group1", set(["group1"]), set(), - set(), set(), dict(), None, None)) + (new1, "group1", set(["group1"]), set(), + set(), set(), dict(category1="group1"), None, None)) + # test nested groups, address, per-client groups + imd = metadata.get_initial_metadata("client8") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client8", "group1", + set(["group1", "group8", "group9", "group10"]), set(), + set(), set(["1.2.3.5"]), dict(category1="group1"), + None, None)) + + # test setting per-client groups, group negation, nested groups + imd = metadata.get_initial_metadata("client9") + self.assertEqual(mock_clientmetadata.call_args[0][:9], + ("client9", "group2", + set(["group2", "group8", "group11"]), + set(["bundle1", "bundle2"]), set(), set(), + dict(category1="group2"), None, "password3")) + + # test new client with no default profile metadata.default = None + new2 = self.get_nonexistent_client(metadata) self.assertRaises(MetadataConsistencyError, - metadata.get_initial_metadata, - "client_new2") + metadata.get_initial_metadata, new2) + + @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) + def test_merge_groups(self): + metadata = self.get_metadata_object() + self.load_groups_data(metadata=metadata) + self.load_clients_data(metadata=metadata) + + imd = metadata.get_initial_metadata("client1") + self.assertEqual(metadata._merge_groups(imd, imd.groups, + categories=imd.categories), + (imd.groups, imd.categories)) + imd = metadata.get_initial_metadata("client8") + self.assertEqual(metadata._merge_groups(imd, imd.groups, + categories=imd.categories), + (imd.groups.union(['group10']), imd.categories)) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) def test_get_all_group_names(self): metadata = self.load_groups_data() self.assertItemsEqual(metadata.get_all_group_names(), set([g.get("name") - for g in groups_test_tree.findall("//Group")])) + for g in self.groups_test_tree.findall("//Group")])) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) def test_get_all_groups_in_category(self): metadata = self.load_groups_data() self.assertItemsEqual(metadata.get_all_groups_in_category("category1"), set([g.get("name") - for g in groups_test_tree.findall("//Group[@category='category1']")])) + for g in self.groups_test_tree.findall("//Group[@category='category1']")])) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) def test_get_client_names_by_profiles(self): metadata = self.load_clients_data(metadata=self.load_groups_data()) - self.assertItemsEqual(metadata.get_client_names_by_profiles("group2"), + self.assertItemsEqual(metadata.get_client_names_by_profiles(["group2"]), [c.get("name") - for c in clients_test_tree.findall("//Client[@profile='group2']")]) + for c in self.clients_test_tree.findall("//Client[@profile='group2']")]) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) def test_get_client_names_by_groups(self): @@ -753,7 +844,7 @@ class TestMetadata(unittest.TestCase): lambda c: metadata.get_initial_metadata(c) self.assertItemsEqual(metadata.get_client_names_by_groups(["group2"]), [c.get("name") - for c in clients_test_tree.findall("//Client[@profile='group2']")]) + for c in self.clients_test_tree.findall("//Client[@profile='group2']")]) @patch("Bcfg2.Server.Plugins.Metadata.XMLMetadataConfig.load_xml", Mock()) def test_merge_additional_groups(self): |