From 553c693618321fad2a88030b16d42d3253befaec Mon Sep 17 00:00:00 2001 From: Fabian Affolter Date: Mon, 8 Nov 2010 23:52:30 +0100 Subject: Directories moved --- doc/getting_help/index.txt | 33 +++++ doc/getting_help/irc.txt | 45 ++++++ doc/getting_help/mailinglist.txt | 24 ++++ doc/help/index.txt | 33 ----- doc/help/irc.txt | 45 ------ doc/help/mailinglist.txt | 24 ---- doc/reports/dynamic.txt | 270 +++++++++++++++++++++++++++++++++++ doc/reports/index.txt | 29 ++++ doc/reports/install.txt | 183 ++++++++++++++++++++++++ doc/reports/item_detail.jpg | Bin 0 -> 98249 bytes doc/reports/node_dropdown.jpg | Bin 0 -> 148486 bytes doc/reports/static.txt | 100 +++++++++++++ doc/reports/summary_cal.jpg | Bin 0 -> 175328 bytes doc/server/reports/dynamic.txt | 270 ----------------------------------- doc/server/reports/index.txt | 29 ---- doc/server/reports/install.txt | 183 ------------------------ doc/server/reports/item_detail.jpg | Bin 98249 -> 0 bytes doc/server/reports/node_dropdown.jpg | Bin 148486 -> 0 bytes doc/server/reports/static.txt | 100 ------------- doc/server/reports/summary_cal.jpg | Bin 175328 -> 0 bytes 20 files changed, 684 insertions(+), 684 deletions(-) create mode 100644 doc/getting_help/index.txt create mode 100644 doc/getting_help/irc.txt create mode 100644 doc/getting_help/mailinglist.txt delete mode 100644 doc/help/index.txt delete mode 100644 doc/help/irc.txt delete mode 100644 doc/help/mailinglist.txt create mode 100644 doc/reports/dynamic.txt create mode 100644 doc/reports/index.txt create mode 100644 doc/reports/install.txt create mode 100644 doc/reports/item_detail.jpg create mode 100644 doc/reports/node_dropdown.jpg create mode 100644 doc/reports/static.txt create mode 100644 doc/reports/summary_cal.jpg delete mode 100644 doc/server/reports/dynamic.txt delete mode 100644 doc/server/reports/index.txt delete mode 100644 doc/server/reports/install.txt delete mode 100644 doc/server/reports/item_detail.jpg delete mode 100644 doc/server/reports/node_dropdown.jpg delete mode 100644 doc/server/reports/static.txt delete mode 100644 doc/server/reports/summary_cal.jpg diff --git a/doc/getting_help/index.txt b/doc/getting_help/index.txt new file mode 100644 index 000000000..bc713bcbe --- /dev/null +++ b/doc/getting_help/index.txt @@ -0,0 +1,33 @@ +.. -*- mode: rst -*- + +.. _help-index: + +======================= +Getting Help with Bcfg2 +======================= + +Having trouble? We'd like to help! + +* Try the :ref:`FAQ ` -- it's got answers to many common questions. +* Looking for specific information? Try the :ref:`genindex`, :ref:`modindex` or the :ref:`detailed table of contents `. +* Search for information in the :ref:`help-mailinglist`. +* Visit our :ref:`help-irc` channel. + +.. _report-a-bug: + +Report A Bug +============ + +Report bugs with Bcfg2 on the `Trac ticket tracker`_. + +.. _Bcfg2 mailing list archives: http://trac.mcs.anl.gov/projects/bcfg2/wiki/MailingList +.. _Trac ticket tracker: http://trac.mcs.anl.gov/projects/bcfg2/wiki + +Other ways to get in touch +========================== + +.. toctree:: + :maxdepth: 2 + + irc + mailinglist diff --git a/doc/getting_help/irc.txt b/doc/getting_help/irc.txt new file mode 100644 index 000000000..37d2099d3 --- /dev/null +++ b/doc/getting_help/irc.txt @@ -0,0 +1,45 @@ +.. -*- mode: rst -*- + +.. _help-irc: + +=== +IRC +=== + +The Bcfg2 IRC channel is `#bcfg2 on chat.freenode.net`_. It is home +to both support and development discussions. If you have a question, +suggestion, or just want to know about Bcfg2, please drop in and say hi. + +.. _#bcfg2 on chat.freenode.net: irc://chat.freenode.net/bcfg2 + +Archives are available at: http://colabti.org/irclogger/irclogger_logs/bcfg2. + +.. raw:: html + +
+ + +
+ in the timespan + + (yyyymmdd-yyyymmdd) +

Options:
+
+
+

+ +Administrative Note +=================== + +If the IRC logging stops working for a while, coordinate on #bcfg2 and +then bug **feb** on #irclogger (freenode), and stick around on that +channel until you get an answer (**feb** is great, but busy and in a +non-US time zone). Actually as long as **ilogger2** is logged in you +should be okay (**feb** looks at the logs). + +If you have private logs for the period of time **ilogger2** was off the +channel, you can convert them to the `format shown here`_, and **feb** +will incorporate them into the online logs. Be sure to strip out any +private messages in your logs first :-) + +.. _format shown here: http://colabti.org/irclogger/irclogger_log/bcfg2?date=2008-03-21,Fri;raw=on diff --git a/doc/getting_help/mailinglist.txt b/doc/getting_help/mailinglist.txt new file mode 100644 index 000000000..c77caa0c4 --- /dev/null +++ b/doc/getting_help/mailinglist.txt @@ -0,0 +1,24 @@ +.. -*- mode: rst -*- + +.. _help-mailinglist: + +============ +Mailing List +============ + +To subscribe to the mailing list for Bcfg2 please visit the `bcfg-dev +mailman page`_ + +`Searchable archives`_ are available from Gmane. You can also read the +mailing list from any NNTP client via Gmane. + +.. _bcfg-dev mailman page: https://lists.mcs.anl.gov/mailman/listinfo/bcfg-dev +.. _Searchable archives: http://dir.gmane.org/gmane.comp.sysutils.bcfg2.devel + +.. raw:: html + +
+ + + +
diff --git a/doc/help/index.txt b/doc/help/index.txt deleted file mode 100644 index bc713bcbe..000000000 --- a/doc/help/index.txt +++ /dev/null @@ -1,33 +0,0 @@ -.. -*- mode: rst -*- - -.. _help-index: - -======================= -Getting Help with Bcfg2 -======================= - -Having trouble? We'd like to help! - -* Try the :ref:`FAQ ` -- it's got answers to many common questions. -* Looking for specific information? Try the :ref:`genindex`, :ref:`modindex` or the :ref:`detailed table of contents `. -* Search for information in the :ref:`help-mailinglist`. -* Visit our :ref:`help-irc` channel. - -.. _report-a-bug: - -Report A Bug -============ - -Report bugs with Bcfg2 on the `Trac ticket tracker`_. - -.. _Bcfg2 mailing list archives: http://trac.mcs.anl.gov/projects/bcfg2/wiki/MailingList -.. _Trac ticket tracker: http://trac.mcs.anl.gov/projects/bcfg2/wiki - -Other ways to get in touch -========================== - -.. toctree:: - :maxdepth: 2 - - irc - mailinglist diff --git a/doc/help/irc.txt b/doc/help/irc.txt deleted file mode 100644 index 37d2099d3..000000000 --- a/doc/help/irc.txt +++ /dev/null @@ -1,45 +0,0 @@ -.. -*- mode: rst -*- - -.. _help-irc: - -=== -IRC -=== - -The Bcfg2 IRC channel is `#bcfg2 on chat.freenode.net`_. It is home -to both support and development discussions. If you have a question, -suggestion, or just want to know about Bcfg2, please drop in and say hi. - -.. _#bcfg2 on chat.freenode.net: irc://chat.freenode.net/bcfg2 - -Archives are available at: http://colabti.org/irclogger/irclogger_logs/bcfg2. - -.. raw:: html - -
- - -
- in the timespan - - (yyyymmdd-yyyymmdd) -

Options:
-
-
-

- -Administrative Note -=================== - -If the IRC logging stops working for a while, coordinate on #bcfg2 and -then bug **feb** on #irclogger (freenode), and stick around on that -channel until you get an answer (**feb** is great, but busy and in a -non-US time zone). Actually as long as **ilogger2** is logged in you -should be okay (**feb** looks at the logs). - -If you have private logs for the period of time **ilogger2** was off the -channel, you can convert them to the `format shown here`_, and **feb** -will incorporate them into the online logs. Be sure to strip out any -private messages in your logs first :-) - -.. _format shown here: http://colabti.org/irclogger/irclogger_log/bcfg2?date=2008-03-21,Fri;raw=on diff --git a/doc/help/mailinglist.txt b/doc/help/mailinglist.txt deleted file mode 100644 index c77caa0c4..000000000 --- a/doc/help/mailinglist.txt +++ /dev/null @@ -1,24 +0,0 @@ -.. -*- mode: rst -*- - -.. _help-mailinglist: - -============ -Mailing List -============ - -To subscribe to the mailing list for Bcfg2 please visit the `bcfg-dev -mailman page`_ - -`Searchable archives`_ are available from Gmane. You can also read the -mailing list from any NNTP client via Gmane. - -.. _bcfg-dev mailman page: https://lists.mcs.anl.gov/mailman/listinfo/bcfg-dev -.. _Searchable archives: http://dir.gmane.org/gmane.comp.sysutils.bcfg2.devel - -.. raw:: html - -
- - - -
diff --git a/doc/reports/dynamic.txt b/doc/reports/dynamic.txt new file mode 100644 index 000000000..1ec41af64 --- /dev/null +++ b/doc/reports/dynamic.txt @@ -0,0 +1,270 @@ +.. -*- mode: rst -*- + +.. _server-reports-dynamic: + +============================== +Bcfg2 Dynamic Reporting System +============================== + +.. versionadded:: 0.8.2 + +Installation +============ + +Prerequisites +------------- + +* sqlite3 +* pysqlite2 +* `Django `_ +* mod-python + +Install +------- + +Be sure to include the specified fields included in the example +``bcfg2.conf`` file. These can be specified in either ``/etc/bcfg2.conf``, +if it is readable by the webserver user, or ``/etc/bcfg2-web.conf``. Any +database supported by Django can be used. If you are not using sqlite +(the default choice), please see the :ref:`alternative-databases` +section below. + +.. note:: + + Distributed environments can share a single remote database for + reporting. + +The recommended statistics plugin is DBStats. To use it, add it to the +**plugins** line in your ``bcfg2.conf``. Alternatively, the Statistics +plugin can be used in conjunction with a crontab entry to run +``/usr/sbin/bcfg2-admin reports load_stats``. + +Detailed installation instructions can be found :ref:`here `. + +.. _dynamic-http-install: + +Apache configuration for web-based reports +------------------------------------------ + +.. note:: + + Reports no longer needs to be installed at the root URL for a given + host. Therefore, reports no longer require their own virtual host. + + In order to make this work, you will need to specify your web prefix + by adding a **web_prefix** setting in the [statistics] section of + your ``bcfg2.conf``. + +An example site config is included below for the vhost "reports.mcs.anl.gov":: + + + ServerAdmin webmaster@mcs.anl.gov + ServerName reports.mcs.anl.gov + DocumentRoot /var/www/reports + + Order deny,allow + Deny from all + Allow from localhost #you may want to change this + AllowOverride None + + + # Possible values include: debug, info, notice, warn, error, crit, + # alert, emerg. + LogLevel warn + + ServerSignature Off + + # Stop TRACE/TRACK vulnerability + + RewriteEngine on + RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) + RewriteRule .* - [F] + + + SetHandler python-program + PythonHandler django.core.handlers.modpython + SetEnv DJANGO_SETTINGS_MODULE Bcfg2.Server.Reports.settings + PythonDebug On + + + SetHandler None + + + +The first three lines of this configuration must be customized per site. + +The ``bcfg2-tarball/reports/site_media/`` directory needs to be copied +to ``/var/www/reports/site_media/`` It could live anywhere; as long as +the contents are accessible on the virtual host at ``/site_media/``. + +At this point you should be able to point your web browser to the +virtualhost you created and see the new reports + +Example WSGI configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +entry.wsgi:: + + import os, sys + os.environ['DJANGO_SETTINGS_MODULE'] = 'Bcfg2.Server.Reports.settings' + import django.core.handlers.wsgi + application = django.core.handlers.wsgi.WSGIHandler() + +Apache conf:: + + Alias /bcfg2reports/site_media "/path/to/site_media" + + Order deny,allow + Allow from all + AllowOverride None + + # If Python is installed in a non-standard prefix: + #WSGIPythonHome /python/prefix + #WSGIPythonPath /python/prefix/lib/python2.6/site-packages + WSGIScriptAlias /bcfg2reports "/another/path/to/entry.wsgi" + +.. _alternative-databases: + +Notes on Alternative Databases +------------------------------ + +If you choose to use a different database, you'll need to edit +``/etc/bcfg2.conf``. These fields should be updated in the [statistics] +section: + +* database_engine + + * ex: database_engine = mysql + * ex: database_engine = postgresql_psycopg2 + +* database_name +* database_user +* database_password +* database_host +* database_port (optional) + +Summary and Features +==================== + +The new reporting system was implemented to address a number of +deficiencies in the previous system. By storing statistics data in a +relational database, we are now able to view and analyze more information +about the state of the configuration, including information about previous +configuration. Specific features in the new system include: + +* The ability to look at a :ref:`reports-calendar-summary` with past + statistics information. +* More recent data concerning hosts. +* Additional information display in reports. Primarily, reasons for + :ref:`configuration item verification failure ` + are now accessible. +* Instead of static pages, pages are generated on the fly, allowing + users to drill down to find out about a :ref:`specific host + `, rather than only having one huge page with + too much information. + +Planned improvements include +============================ + +* Accounts, customized displays for each admin. And privacy of config data. +* Config browsing capabilities; to look at your config in an interesting way. +* Fixing all the known bugs from below. + +Unfortunately with all the improvements, there are a few less exciting +elements about the new reporting system. The new reporting system +moves away from static pages and towards a real web application, which +causes mainly problems with dependencies and makes installation more +difficult. This should become less of a problem over time as we develop +a better installation process for a web application. + +Usage +===== + +bcfg2-admin reports (command line script) +----------------------------------------- + +The bcfg2-admin tool provides management and maintenance capabilities for +the reporting database. A few useful `Django `_ +commands are provided as well. + +* init: Initialize a new database +* load_stats: Load statistics data from the Statistics plugin into the + database. This was importscript.py. +* scrub: Scrub the database for duplicate reasons. +* update: Apply any updates to the reporting database. Unlike the syncdb + command, this will modify existing tables. + +Django commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* syncdb: Create the tables for any models not installed. Django will + not modify any existing tables. +* sqlall: Print the sql statements used to create the database. Note: + This does not show the fixture data. +* validate: Validate the database against the current models. + +bcfg2-reports (command line script) +----------------------------------- + +bcfg2-reports allows you to retrieve data from the database about clients, +and the states of their current interactions. It also allows you to +change the expired/unexpired states. + +The utility runs as a standalone application. It does, however, use the +models from ``/src/lib/Server/Reports/reports/models.py``. + +A number of different options can be used to change what bcfg2-reports +displays:: + + Usage: python bcfg2-reports [option] ... + + Options and arguments (and corresponding environment variables): + -a : shows all hosts, including expired hosts + -b NAME : single-host mode - shows bad entries from the + current interaction of NAME + -c : shows only clean hosts + -d : shows only dirty hosts + -e NAME : single-host mode - shows extra entries from the + current interaction of NAME + -h : shows help and usage info about bcfg2-reports + -s NAME : single-host mode - shows bad and extra entries from + the current interaction of NAME + -x NAME : toggles expired/unexpired state of NAME + --badentry=KIND,NAME : shows only hosts whose current interaction has bad + entries in of KIND kind and NAME name; if a single + argument ARG1 is given, then KIND,NAME pairs will be + read from a file of name ARG1 + --extraentry=KIND,NAME : shows only hosts whose current interaction has extra + entries in of KIND kind and NAME name; if a single + argument ARG1 is given, then KIND,NAME pairs will be + read from a file of name ARG1 + --fields=ARG1,ARG2,... : only displays the fields ARG1,ARG2,... + (name,time,state) + --sort=ARG1,ARG2,... : sorts output on ARG1,ARG2,... (name,time,state) + --stale : shows hosts which haven't run in the last 24 hours + +Screenshots +=========== + +.. _reports-calendar-summary: + +Calendar Summary +---------------- + +.. image:: summary_cal.jpg + :alt: Calendar summary + +.. _reports-item-detail: + +Item detail +----------- + +.. image:: item_detail.jpg + :alt: Item detail + +Node dropdown +------------- + +.. _reports-node-dropdown: + +.. image:: node_dropdown.jpg + :alt: Node dropdown diff --git a/doc/reports/index.txt b/doc/reports/index.txt new file mode 100644 index 000000000..bef686305 --- /dev/null +++ b/doc/reports/index.txt @@ -0,0 +1,29 @@ +.. -*- mode: rst -*- + +.. _server-reports-index: + +The Bcfg2 Reporting System +========================== + +Bcfg2's reporting system is its killer feature. It allows administrators +to gain a broad understanding of the configuration state of their entire +environment. It summarizes + +* Configuration changes and when they were made +* Discrepancies between the specification and current client states + + * Clients can be grouped by misconfiguration type + +* Configuration entries that are not specified +* Overall client summaries according to these types + +There are two systems, the old system, which builds static reports based +on a series of XSLT stylesheets and a new dynamic reporting system that +uses django and a database backend. + +.. toctree:: + :maxdepth: 2 + + install + static + dynamic diff --git a/doc/reports/install.txt b/doc/reports/install.txt new file mode 100644 index 000000000..80f9342ae --- /dev/null +++ b/doc/reports/install.txt @@ -0,0 +1,183 @@ +.. -*- mode: rst -*- + +.. _EPEL: http://fedoraproject.org/wiki/EPEL + +.. This is combination of the Ubuntu guide and the Centos guide for + installing the web reports. + +.. _server-reports-install: + +===================== +Dynamic (web) Reports +===================== + +The first step is to install the needed software components like the +Django framework and the database (SQlite2). All packages for Fedora +are in the Fedora Package Collection or in EPEL_ for CentOS/RHEL:: + + [root@system01 ~]# yum -y install Django python-simplejson python-sqlite2 + +Of course is a web server needed as well:: + + [root@system01 ~]# yum -y install httpd mod_python + +The same packages are needed for Ubuntu systems:: + + [root@system01 ~]# aptitude install python-django apache2 libapache2-mod-python + +Now we need to create the sqlite database. Use the following command on +Fedora, CentOS, or RHEL.:: + + [root@system01 ~]# python /usr/lib/python2.4/site-packages/Bcfg2/Server/Reports/manage.py syncdb + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + + You just installed Django's auth system, which means you don't have any superusers defined. + Would you like to create one now? (yes/no): no + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + +.. note:: There are different versions of Python available. If you are + unsure about your installed version use the following line instead of + the line above.:: + + [root@system01 ~]# PYVER=`python -c 'import sys;print(sys.version[0:3])'`; python /usr/lib/python$PYVER/site-packages/Bcfg2/site-packages/Bcfg2/Server/Reports/manage.py syncdb + +The path on Ubuntu systems is different. Please use the same path as shown +in the following command to execute the script on an Ubuntu machine in +the next steps:: + + [root@system01 ~]# python /usr/share/pyshared/Bcfg2/Server/Reports/manage.py syncdb + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + + You just installed Django's auth system, which means you don't have any superusers defined. + Would you like to create one now? (yes/no): no + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + +The server should be tested to make sure that there are no mistakes:: + + [root@system01 ~]# python /usr/lib/python2.6/site-packages/Bcfg2/Server/Reports/manage.py testserver + Creating test database... + Creating table auth_permission + Creating table auth_group + Creating table auth_user + Creating table auth_message + Creating table django_content_type + Creating table django_session + Creating table django_site + Creating table django_admin_log + Creating table reports_client + Creating table reports_ping + Creating table reports_interaction + Creating table reports_reason + Creating table reports_entries + Creating table reports_entries_interactions + Creating table reports_performance + Creating table reports_internaldatabaseversion + Installing index for auth.Permission model + Installing index for auth.Message model + Installing index for admin.LogEntry model + Installing index for reports.Client model + Installing index for reports.Ping model + Installing index for reports.Interaction model + Installing index for reports.Entries model + Installing index for reports.Entries_interactions model + Validating models... + 0 errors found + + Django version 1.1.1, using settings 'Reports.settings' + Development server is running at http://127.0.0.1:8000/ + Quit the server with CONTROL-C. + +Add DBStats to the plugins line of ``bcfg2.conf``. The resulting +**[server]** section should look something like this:: + + [server] + repository = /var/lib/bcfg2 + plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase + +Start/restart the Bcfg2 server:: + + [root@system01 ~]# /etc/init.d/bcfg2-server restart + +Run the Bcfg2 client in order to populate the statistics database +(this run should take a bit longer since you are uploading the client +statistics to the database). + +Download the static reports content:: + + [root@system01 ~]# cd /var/www/ + [root@system01 ~]# svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2/reports + +Configure Apache using :ref:`dynamic-http-install` as a guide + +Copy server/statistics sections of ``bcfg2.conf`` to +``/etc/bcfg2-web.conf`` (make sure it is world-readable). You should +then have something like this:: + + [server] + repository = /var/lib/bcfg2 + plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase + + [statistics] + sendmailpath = /usr/lib/sendmail + database_engine = sqlite3 + # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'. + database_name = + # Or path to database file if using sqlite3. + #/etc/brpt.sqlite is default path if left empty + database_user = + # Not used with sqlite3. + database_password = + # Not used with sqlite3. + database_host = + # Not used with sqlite3. + database_port = + # Set to empty string for default. Not used with sqlite3. + web_debug = True + +Restart apache and point a browser to your Bcfg2 server. + +If using sqlite be sure the sql database file and directory containing the database are writable to apache. diff --git a/doc/reports/item_detail.jpg b/doc/reports/item_detail.jpg new file mode 100644 index 000000000..198880599 Binary files /dev/null and b/doc/reports/item_detail.jpg differ diff --git a/doc/reports/node_dropdown.jpg b/doc/reports/node_dropdown.jpg new file mode 100644 index 000000000..d3f7d4f14 Binary files /dev/null and b/doc/reports/node_dropdown.jpg differ diff --git a/doc/reports/static.txt b/doc/reports/static.txt new file mode 100644 index 000000000..d5d96fe12 --- /dev/null +++ b/doc/reports/static.txt @@ -0,0 +1,100 @@ +.. -*- mode: rst -*- + +.. _server-reports-static: + +============================= +Bcfg2 Static Reporting System +============================= + +The Bcfg2 reporting system collects and displays information about the +operation of the Bcfg2 client, and the configuration states of target +machines. + +Goals +===== + +The reporting system provides an interface to administrators describing +a few important tasks + +* Client configuration state, particularly aspects that do not match the configuration specification. + Information about bad and extra configuration elements is included. +* Client execution results (a list of configuration elements that were modified) +* Client execution performance data (including operation retry counts, and timings for several critical execution regions) + +This data can be used to understand the current configuration state +of the entire network, the operations performed by the client, how the +configuration changes propagate, and any reconfiguration operations that +have failed. + +Retention Model +=============== + +The current reporting system stores statistics in an XML data store, by +default to {{{/etc/statistics.xml}}}. It retains either one or two +statistic sets per host. If the client has a clean configuration state, +the most recent (clean) record is retained. If the client has a dirty +configuration state, two records are retained. One record is the last +clean record. The other record is the most recent record collected, +detailing the incorrect state. + +This retention model, while non-optimal, does manage to persistently +record most of the data that users would like. + +Setup +===== + +In order to configure your Bcfg2 server for receiving reports, you +will need to list the Statistics plugin in the plugins line of your +``bcfg2.conf``. You will also need a [statistics] section +in your ``bcfg2.conf``. You can find out more about what goes there in the +``bcfg2.conf`` manpage. + +Output +====== + +Several output reports can be generated from the statistics store with +the command line tool {{{bcfg2-build-reports}}}. + +* Nodes Digest +* Nodes Individual +* Overview Statistics +* Performance + +The data generated by these reports can be delivered by several +mechanisms: + +* HTML +* Email +* RSS + +Shortcomings and Planned Enhancements +===================================== + +When designing the current reporting system, we were overly concerned with +the potential explosion in data size over time. In order to address this, +we opted to use the retention scheme described above. This approach has +several shortcomings: + +* A comprehensive list of reconfiguration operations (with associated + timestamps) isn't retained +* Client results for any given day (except the last one) aren't uniformly + retained. This means that inter-client analysis is difficult, if + not impossible + +We plan to move to a database backend to address the dataset size +problem and start retaining all information. The move to a SQL backend +will allow many more types of queries to be efficiently processed. It +will also make on-demand reports simpler. + +Other sorts of information would also be useful to track. We plan to +add the ability to tag a particular configuration element as security +related, and include this in reports. This will aid in the effective +prioritization of manual and failed reconfiguration tasks. + +Capability Goals (posed as questions) +------------------------------------- + +* What machines have not yet applied critical updates? +* How long did critical updates take to be applied? +* What configuration did machine X have on a particular date? +* When did machine X perform configuration update Y? diff --git a/doc/reports/summary_cal.jpg b/doc/reports/summary_cal.jpg new file mode 100644 index 000000000..fab010762 Binary files /dev/null and b/doc/reports/summary_cal.jpg differ diff --git a/doc/server/reports/dynamic.txt b/doc/server/reports/dynamic.txt deleted file mode 100644 index 1ec41af64..000000000 --- a/doc/server/reports/dynamic.txt +++ /dev/null @@ -1,270 +0,0 @@ -.. -*- mode: rst -*- - -.. _server-reports-dynamic: - -============================== -Bcfg2 Dynamic Reporting System -============================== - -.. versionadded:: 0.8.2 - -Installation -============ - -Prerequisites -------------- - -* sqlite3 -* pysqlite2 -* `Django `_ -* mod-python - -Install -------- - -Be sure to include the specified fields included in the example -``bcfg2.conf`` file. These can be specified in either ``/etc/bcfg2.conf``, -if it is readable by the webserver user, or ``/etc/bcfg2-web.conf``. Any -database supported by Django can be used. If you are not using sqlite -(the default choice), please see the :ref:`alternative-databases` -section below. - -.. note:: - - Distributed environments can share a single remote database for - reporting. - -The recommended statistics plugin is DBStats. To use it, add it to the -**plugins** line in your ``bcfg2.conf``. Alternatively, the Statistics -plugin can be used in conjunction with a crontab entry to run -``/usr/sbin/bcfg2-admin reports load_stats``. - -Detailed installation instructions can be found :ref:`here `. - -.. _dynamic-http-install: - -Apache configuration for web-based reports ------------------------------------------- - -.. note:: - - Reports no longer needs to be installed at the root URL for a given - host. Therefore, reports no longer require their own virtual host. - - In order to make this work, you will need to specify your web prefix - by adding a **web_prefix** setting in the [statistics] section of - your ``bcfg2.conf``. - -An example site config is included below for the vhost "reports.mcs.anl.gov":: - - - ServerAdmin webmaster@mcs.anl.gov - ServerName reports.mcs.anl.gov - DocumentRoot /var/www/reports - - Order deny,allow - Deny from all - Allow from localhost #you may want to change this - AllowOverride None - - - # Possible values include: debug, info, notice, warn, error, crit, - # alert, emerg. - LogLevel warn - - ServerSignature Off - - # Stop TRACE/TRACK vulnerability - - RewriteEngine on - RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK) - RewriteRule .* - [F] - - - SetHandler python-program - PythonHandler django.core.handlers.modpython - SetEnv DJANGO_SETTINGS_MODULE Bcfg2.Server.Reports.settings - PythonDebug On - - - SetHandler None - - - -The first three lines of this configuration must be customized per site. - -The ``bcfg2-tarball/reports/site_media/`` directory needs to be copied -to ``/var/www/reports/site_media/`` It could live anywhere; as long as -the contents are accessible on the virtual host at ``/site_media/``. - -At this point you should be able to point your web browser to the -virtualhost you created and see the new reports - -Example WSGI configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -entry.wsgi:: - - import os, sys - os.environ['DJANGO_SETTINGS_MODULE'] = 'Bcfg2.Server.Reports.settings' - import django.core.handlers.wsgi - application = django.core.handlers.wsgi.WSGIHandler() - -Apache conf:: - - Alias /bcfg2reports/site_media "/path/to/site_media" - - Order deny,allow - Allow from all - AllowOverride None - - # If Python is installed in a non-standard prefix: - #WSGIPythonHome /python/prefix - #WSGIPythonPath /python/prefix/lib/python2.6/site-packages - WSGIScriptAlias /bcfg2reports "/another/path/to/entry.wsgi" - -.. _alternative-databases: - -Notes on Alternative Databases ------------------------------- - -If you choose to use a different database, you'll need to edit -``/etc/bcfg2.conf``. These fields should be updated in the [statistics] -section: - -* database_engine - - * ex: database_engine = mysql - * ex: database_engine = postgresql_psycopg2 - -* database_name -* database_user -* database_password -* database_host -* database_port (optional) - -Summary and Features -==================== - -The new reporting system was implemented to address a number of -deficiencies in the previous system. By storing statistics data in a -relational database, we are now able to view and analyze more information -about the state of the configuration, including information about previous -configuration. Specific features in the new system include: - -* The ability to look at a :ref:`reports-calendar-summary` with past - statistics information. -* More recent data concerning hosts. -* Additional information display in reports. Primarily, reasons for - :ref:`configuration item verification failure ` - are now accessible. -* Instead of static pages, pages are generated on the fly, allowing - users to drill down to find out about a :ref:`specific host - `, rather than only having one huge page with - too much information. - -Planned improvements include -============================ - -* Accounts, customized displays for each admin. And privacy of config data. -* Config browsing capabilities; to look at your config in an interesting way. -* Fixing all the known bugs from below. - -Unfortunately with all the improvements, there are a few less exciting -elements about the new reporting system. The new reporting system -moves away from static pages and towards a real web application, which -causes mainly problems with dependencies and makes installation more -difficult. This should become less of a problem over time as we develop -a better installation process for a web application. - -Usage -===== - -bcfg2-admin reports (command line script) ------------------------------------------ - -The bcfg2-admin tool provides management and maintenance capabilities for -the reporting database. A few useful `Django `_ -commands are provided as well. - -* init: Initialize a new database -* load_stats: Load statistics data from the Statistics plugin into the - database. This was importscript.py. -* scrub: Scrub the database for duplicate reasons. -* update: Apply any updates to the reporting database. Unlike the syncdb - command, this will modify existing tables. - -Django commands -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* syncdb: Create the tables for any models not installed. Django will - not modify any existing tables. -* sqlall: Print the sql statements used to create the database. Note: - This does not show the fixture data. -* validate: Validate the database against the current models. - -bcfg2-reports (command line script) ------------------------------------ - -bcfg2-reports allows you to retrieve data from the database about clients, -and the states of their current interactions. It also allows you to -change the expired/unexpired states. - -The utility runs as a standalone application. It does, however, use the -models from ``/src/lib/Server/Reports/reports/models.py``. - -A number of different options can be used to change what bcfg2-reports -displays:: - - Usage: python bcfg2-reports [option] ... - - Options and arguments (and corresponding environment variables): - -a : shows all hosts, including expired hosts - -b NAME : single-host mode - shows bad entries from the - current interaction of NAME - -c : shows only clean hosts - -d : shows only dirty hosts - -e NAME : single-host mode - shows extra entries from the - current interaction of NAME - -h : shows help and usage info about bcfg2-reports - -s NAME : single-host mode - shows bad and extra entries from - the current interaction of NAME - -x NAME : toggles expired/unexpired state of NAME - --badentry=KIND,NAME : shows only hosts whose current interaction has bad - entries in of KIND kind and NAME name; if a single - argument ARG1 is given, then KIND,NAME pairs will be - read from a file of name ARG1 - --extraentry=KIND,NAME : shows only hosts whose current interaction has extra - entries in of KIND kind and NAME name; if a single - argument ARG1 is given, then KIND,NAME pairs will be - read from a file of name ARG1 - --fields=ARG1,ARG2,... : only displays the fields ARG1,ARG2,... - (name,time,state) - --sort=ARG1,ARG2,... : sorts output on ARG1,ARG2,... (name,time,state) - --stale : shows hosts which haven't run in the last 24 hours - -Screenshots -=========== - -.. _reports-calendar-summary: - -Calendar Summary ----------------- - -.. image:: summary_cal.jpg - :alt: Calendar summary - -.. _reports-item-detail: - -Item detail ------------ - -.. image:: item_detail.jpg - :alt: Item detail - -Node dropdown -------------- - -.. _reports-node-dropdown: - -.. image:: node_dropdown.jpg - :alt: Node dropdown diff --git a/doc/server/reports/index.txt b/doc/server/reports/index.txt deleted file mode 100644 index bef686305..000000000 --- a/doc/server/reports/index.txt +++ /dev/null @@ -1,29 +0,0 @@ -.. -*- mode: rst -*- - -.. _server-reports-index: - -The Bcfg2 Reporting System -========================== - -Bcfg2's reporting system is its killer feature. It allows administrators -to gain a broad understanding of the configuration state of their entire -environment. It summarizes - -* Configuration changes and when they were made -* Discrepancies between the specification and current client states - - * Clients can be grouped by misconfiguration type - -* Configuration entries that are not specified -* Overall client summaries according to these types - -There are two systems, the old system, which builds static reports based -on a series of XSLT stylesheets and a new dynamic reporting system that -uses django and a database backend. - -.. toctree:: - :maxdepth: 2 - - install - static - dynamic diff --git a/doc/server/reports/install.txt b/doc/server/reports/install.txt deleted file mode 100644 index 80f9342ae..000000000 --- a/doc/server/reports/install.txt +++ /dev/null @@ -1,183 +0,0 @@ -.. -*- mode: rst -*- - -.. _EPEL: http://fedoraproject.org/wiki/EPEL - -.. This is combination of the Ubuntu guide and the Centos guide for - installing the web reports. - -.. _server-reports-install: - -===================== -Dynamic (web) Reports -===================== - -The first step is to install the needed software components like the -Django framework and the database (SQlite2). All packages for Fedora -are in the Fedora Package Collection or in EPEL_ for CentOS/RHEL:: - - [root@system01 ~]# yum -y install Django python-simplejson python-sqlite2 - -Of course is a web server needed as well:: - - [root@system01 ~]# yum -y install httpd mod_python - -The same packages are needed for Ubuntu systems:: - - [root@system01 ~]# aptitude install python-django apache2 libapache2-mod-python - -Now we need to create the sqlite database. Use the following command on -Fedora, CentOS, or RHEL.:: - - [root@system01 ~]# python /usr/lib/python2.4/site-packages/Bcfg2/Server/Reports/manage.py syncdb - Creating table auth_permission - Creating table auth_group - Creating table auth_user - Creating table auth_message - Creating table django_content_type - Creating table django_session - Creating table django_site - Creating table django_admin_log - Creating table reports_client - Creating table reports_ping - Creating table reports_interaction - Creating table reports_reason - Creating table reports_entries - Creating table reports_entries_interactions - Creating table reports_performance - Creating table reports_internaldatabaseversion - - You just installed Django's auth system, which means you don't have any superusers defined. - Would you like to create one now? (yes/no): no - Installing index for auth.Permission model - Installing index for auth.Message model - Installing index for admin.LogEntry model - Installing index for reports.Client model - Installing index for reports.Ping model - Installing index for reports.Interaction model - Installing index for reports.Entries model - Installing index for reports.Entries_interactions model - -.. note:: There are different versions of Python available. If you are - unsure about your installed version use the following line instead of - the line above.:: - - [root@system01 ~]# PYVER=`python -c 'import sys;print(sys.version[0:3])'`; python /usr/lib/python$PYVER/site-packages/Bcfg2/site-packages/Bcfg2/Server/Reports/manage.py syncdb - -The path on Ubuntu systems is different. Please use the same path as shown -in the following command to execute the script on an Ubuntu machine in -the next steps:: - - [root@system01 ~]# python /usr/share/pyshared/Bcfg2/Server/Reports/manage.py syncdb - Creating table auth_permission - Creating table auth_group - Creating table auth_user - Creating table auth_message - Creating table django_content_type - Creating table django_session - Creating table django_site - Creating table django_admin_log - Creating table reports_client - Creating table reports_ping - Creating table reports_interaction - Creating table reports_reason - Creating table reports_entries - Creating table reports_entries_interactions - Creating table reports_performance - Creating table reports_internaldatabaseversion - - You just installed Django's auth system, which means you don't have any superusers defined. - Would you like to create one now? (yes/no): no - Installing index for auth.Permission model - Installing index for auth.Message model - Installing index for admin.LogEntry model - Installing index for reports.Client model - Installing index for reports.Ping model - Installing index for reports.Interaction model - Installing index for reports.Entries model - Installing index for reports.Entries_interactions model - -The server should be tested to make sure that there are no mistakes:: - - [root@system01 ~]# python /usr/lib/python2.6/site-packages/Bcfg2/Server/Reports/manage.py testserver - Creating test database... - Creating table auth_permission - Creating table auth_group - Creating table auth_user - Creating table auth_message - Creating table django_content_type - Creating table django_session - Creating table django_site - Creating table django_admin_log - Creating table reports_client - Creating table reports_ping - Creating table reports_interaction - Creating table reports_reason - Creating table reports_entries - Creating table reports_entries_interactions - Creating table reports_performance - Creating table reports_internaldatabaseversion - Installing index for auth.Permission model - Installing index for auth.Message model - Installing index for admin.LogEntry model - Installing index for reports.Client model - Installing index for reports.Ping model - Installing index for reports.Interaction model - Installing index for reports.Entries model - Installing index for reports.Entries_interactions model - Validating models... - 0 errors found - - Django version 1.1.1, using settings 'Reports.settings' - Development server is running at http://127.0.0.1:8000/ - Quit the server with CONTROL-C. - -Add DBStats to the plugins line of ``bcfg2.conf``. The resulting -**[server]** section should look something like this:: - - [server] - repository = /var/lib/bcfg2 - plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase - -Start/restart the Bcfg2 server:: - - [root@system01 ~]# /etc/init.d/bcfg2-server restart - -Run the Bcfg2 client in order to populate the statistics database -(this run should take a bit longer since you are uploading the client -statistics to the database). - -Download the static reports content:: - - [root@system01 ~]# cd /var/www/ - [root@system01 ~]# svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2/reports - -Configure Apache using :ref:`dynamic-http-install` as a guide - -Copy server/statistics sections of ``bcfg2.conf`` to -``/etc/bcfg2-web.conf`` (make sure it is world-readable). You should -then have something like this:: - - [server] - repository = /var/lib/bcfg2 - plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase - - [statistics] - sendmailpath = /usr/lib/sendmail - database_engine = sqlite3 - # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'. - database_name = - # Or path to database file if using sqlite3. - #/etc/brpt.sqlite is default path if left empty - database_user = - # Not used with sqlite3. - database_password = - # Not used with sqlite3. - database_host = - # Not used with sqlite3. - database_port = - # Set to empty string for default. Not used with sqlite3. - web_debug = True - -Restart apache and point a browser to your Bcfg2 server. - -If using sqlite be sure the sql database file and directory containing the database are writable to apache. diff --git a/doc/server/reports/item_detail.jpg b/doc/server/reports/item_detail.jpg deleted file mode 100644 index 198880599..000000000 Binary files a/doc/server/reports/item_detail.jpg and /dev/null differ diff --git a/doc/server/reports/node_dropdown.jpg b/doc/server/reports/node_dropdown.jpg deleted file mode 100644 index d3f7d4f14..000000000 Binary files a/doc/server/reports/node_dropdown.jpg and /dev/null differ diff --git a/doc/server/reports/static.txt b/doc/server/reports/static.txt deleted file mode 100644 index d5d96fe12..000000000 --- a/doc/server/reports/static.txt +++ /dev/null @@ -1,100 +0,0 @@ -.. -*- mode: rst -*- - -.. _server-reports-static: - -============================= -Bcfg2 Static Reporting System -============================= - -The Bcfg2 reporting system collects and displays information about the -operation of the Bcfg2 client, and the configuration states of target -machines. - -Goals -===== - -The reporting system provides an interface to administrators describing -a few important tasks - -* Client configuration state, particularly aspects that do not match the configuration specification. - Information about bad and extra configuration elements is included. -* Client execution results (a list of configuration elements that were modified) -* Client execution performance data (including operation retry counts, and timings for several critical execution regions) - -This data can be used to understand the current configuration state -of the entire network, the operations performed by the client, how the -configuration changes propagate, and any reconfiguration operations that -have failed. - -Retention Model -=============== - -The current reporting system stores statistics in an XML data store, by -default to {{{/etc/statistics.xml}}}. It retains either one or two -statistic sets per host. If the client has a clean configuration state, -the most recent (clean) record is retained. If the client has a dirty -configuration state, two records are retained. One record is the last -clean record. The other record is the most recent record collected, -detailing the incorrect state. - -This retention model, while non-optimal, does manage to persistently -record most of the data that users would like. - -Setup -===== - -In order to configure your Bcfg2 server for receiving reports, you -will need to list the Statistics plugin in the plugins line of your -``bcfg2.conf``. You will also need a [statistics] section -in your ``bcfg2.conf``. You can find out more about what goes there in the -``bcfg2.conf`` manpage. - -Output -====== - -Several output reports can be generated from the statistics store with -the command line tool {{{bcfg2-build-reports}}}. - -* Nodes Digest -* Nodes Individual -* Overview Statistics -* Performance - -The data generated by these reports can be delivered by several -mechanisms: - -* HTML -* Email -* RSS - -Shortcomings and Planned Enhancements -===================================== - -When designing the current reporting system, we were overly concerned with -the potential explosion in data size over time. In order to address this, -we opted to use the retention scheme described above. This approach has -several shortcomings: - -* A comprehensive list of reconfiguration operations (with associated - timestamps) isn't retained -* Client results for any given day (except the last one) aren't uniformly - retained. This means that inter-client analysis is difficult, if - not impossible - -We plan to move to a database backend to address the dataset size -problem and start retaining all information. The move to a SQL backend -will allow many more types of queries to be efficiently processed. It -will also make on-demand reports simpler. - -Other sorts of information would also be useful to track. We plan to -add the ability to tag a particular configuration element as security -related, and include this in reports. This will aid in the effective -prioritization of manual and failed reconfiguration tasks. - -Capability Goals (posed as questions) -------------------------------------- - -* What machines have not yet applied critical updates? -* How long did critical updates take to be applied? -* What configuration did machine X have on a particular date? -* When did machine X perform configuration update Y? diff --git a/doc/server/reports/summary_cal.jpg b/doc/server/reports/summary_cal.jpg deleted file mode 100644 index fab010762..000000000 Binary files a/doc/server/reports/summary_cal.jpg and /dev/null differ -- cgit v1.2.3-1-g7c22