Directories moved

7 files changed, 582 insertions, 0 deletions

diff --git a/doc/reports/dynamic.txt b/doc/reports/dynamic.txt
new file mode 100644
index 000000000..1ec41af64
--- /dev/null
+++ b/doc/reports/dynamic.txt
@@ -0,0 +1,270 @@
+.. -*- mode: rst -*-
+
+.. _server-reports-dynamic:
+
+==============================
+Bcfg2 Dynamic Reporting System
+==============================
+
+.. versionadded:: 0.8.2
+
+Installation
+============
+
+Prerequisites
+-------------
+
+* sqlite3
+* pysqlite2
+* `Django <http://www.djangoproject.com>`_
+* mod-python
+
+Install
+-------
+
+Be sure to include the specified fields included in the example
+``bcfg2.conf`` file. These can be specified in either ``/etc/bcfg2.conf``,
+if it is readable by the webserver user, or ``/etc/bcfg2-web.conf``. Any
+database supported by Django can be used. If you are not using sqlite
+(the default choice), please see the :ref:`alternative-databases`
+section below.
+
+.. note::
+
+    Distributed environments can share a single remote database for
+    reporting.
+
+The recommended statistics plugin is DBStats. To use it, add it to the
+**plugins** line in your ``bcfg2.conf``.  Alternatively, the Statistics
+plugin can be used in conjunction with a crontab entry to run
+``/usr/sbin/bcfg2-admin reports load_stats``.
+
+Detailed installation instructions can be found :ref:`here <server-reports-install>`.
+
+.. _dynamic-http-install:
+
+Apache configuration for web-based reports
+------------------------------------------
+
+.. note::
+
+    Reports no longer needs to be installed at the root URL for a given
+    host. Therefore, reports no longer require their own virtual host.
+
+    In order to make this work, you will need to specify your web prefix
+    by adding a **web_prefix** setting in the [statistics] section of
+    your ``bcfg2.conf``.
+
+An example site config is included below for the vhost "reports.mcs.anl.gov"::
+
+    <VirtualHost reports.mcs.anl.gov>
+            ServerAdmin webmaster@mcs.anl.gov
+            ServerName reports.mcs.anl.gov
+            DocumentRoot /var/www/reports
+            <Directory /var/www/reports>
+                    Order deny,allow
+                    Deny from all
+                    Allow from localhost #you may want to change this
+                    AllowOverride None
+            </Directory>
+
+            # Possible values include: debug, info, notice, warn, error, crit,
+            # alert, emerg.
+            LogLevel warn
+
+            ServerSignature Off
+
+            # Stop TRACE/TRACK vulnerability
+            <IfModule mod_rewrite.c>
+                    RewriteEngine on
+                    RewriteCond %{REQUEST_METHOD} ^(TRACE|TRACK)
+                    RewriteRule .* - [F]
+            </IfModule>
+    <Location "/">
+            SetHandler python-program
+            PythonHandler django.core.handlers.modpython
+            SetEnv DJANGO_SETTINGS_MODULE Bcfg2.Server.Reports.settings
+            PythonDebug On
+    </Location>
+    <Location "/site_media/">
+            SetHandler None
+    </Location>
+    </VirtualHost>
+
+The first three lines of this configuration must be customized per site.
+
+The ``bcfg2-tarball/reports/site_media/`` directory needs to be copied
+to ``/var/www/reports/site_media/`` It could live anywhere; as long as
+the contents are accessible on the virtual host at ``/site_media/``.
+
+At this point you should be able to point your web browser to the
+virtualhost you created and see the new reports
+
+Example WSGI configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+entry.wsgi::
+
+    import os, sys
+    os.environ['DJANGO_SETTINGS_MODULE'] = 'Bcfg2.Server.Reports.settings'
+    import django.core.handlers.wsgi
+    application = django.core.handlers.wsgi.WSGIHandler()
+
+Apache conf::
+
+    Alias /bcfg2reports/site_media "/path/to/site_media"
+    <Directory /path/to>
+        Order deny,allow
+        Allow from all
+        AllowOverride None
+    </Directory>
+    # If Python is installed in a non-standard prefix:
+    #WSGIPythonHome /python/prefix
+    #WSGIPythonPath /python/prefix/lib/python2.6/site-packages
+    WSGIScriptAlias /bcfg2reports "/another/path/to/entry.wsgi"
+
+.. _alternative-databases:
+
+Notes on Alternative Databases
+------------------------------
+
+If you choose to use a different database, you'll need to edit
+``/etc/bcfg2.conf``. These fields should be updated in the [statistics]
+section:
+
+* database_engine
+
+  * ex: database_engine = mysql
+  * ex: database_engine = postgresql_psycopg2
+
+* database_name
+* database_user
+* database_password
+* database_host
+* database_port (optional)
+
+Summary and Features
+====================
+
+The new reporting system was implemented to address a number of
+deficiencies in the previous system. By storing statistics data in a
+relational database, we are now able to view and analyze more information
+about the state of the configuration, including information about previous
+configuration. Specific features in the new system include:
+
+* The ability to look at a :ref:`reports-calendar-summary` with past
+  statistics information.
+* More recent data concerning hosts.
+* Additional information display in reports. Primarily, reasons for
+  :ref:`configuration item verification failure <reports-item-detail>`
+  are now accessible.
+* Instead of static pages, pages are generated on the fly, allowing
+  users to drill down to find out about a :ref:`specific host
+  <reports-node-dropdown>`, rather than only having one huge page with
+  too much information.
+
+Planned improvements include
+============================
+
+* Accounts, customized displays for each admin. And privacy of config data.
+* Config browsing capabilities; to look at your config in an interesting way.
+* Fixing all the known bugs from below.
+
+Unfortunately with all the improvements, there are a few less exciting
+elements about the new reporting system. The new reporting system
+moves away from static pages and towards a real web application, which
+causes mainly problems with dependencies and makes installation more
+difficult. This should become less of a problem over time as we develop
+a better installation process for a web application.
+
+Usage
+=====
+
+bcfg2-admin reports (command line script)
+-----------------------------------------
+
+The bcfg2-admin tool provides management and maintenance capabilities for
+the reporting database.  A few useful `Django <http://www.djangoproject.com>`_
+commands are provided as well.
+
+* init: Initialize a new database
+* load_stats: Load statistics data from the Statistics plugin into the
+  database. This was importscript.py.
+* scrub: Scrub the database for duplicate reasons.
+* update: Apply any updates to the reporting database. Unlike the syncdb
+  command, this will modify existing tables.
+
+Django commands
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* syncdb:  Create the tables for any models not installed.  Django will
+  not modify any existing tables.
+* sqlall:  Print the sql statements used to create the database.  Note:
+  This does not show the fixture data.
+* validate:  Validate the database against the current models.
+
+bcfg2-reports (command line script)
+-----------------------------------
+
+bcfg2-reports allows you to retrieve data from the database about clients,
+and the states of their current interactions. It also allows you to
+change the expired/unexpired states.
+
+The utility runs as a standalone application. It does, however, use the
+models from ``/src/lib/Server/Reports/reports/models.py``.
+
+A number of different options can be used to change what bcfg2-reports
+displays::
+
+    Usage: python bcfg2-reports [option] ...
+
+    Options and arguments (and corresponding environment variables):
+    -a                     : shows all hosts, including expired hosts
+    -b NAME                : single-host mode - shows bad entries from the
+                             current interaction of NAME
+    -c                     : shows only clean hosts
+    -d                     : shows only dirty hosts
+    -e NAME                : single-host mode - shows extra entries from the
+                             current interaction of NAME
+    -h                     : shows help and usage info about bcfg2-reports
+    -s NAME                : single-host mode - shows bad and extra entries from
+                             the current interaction of NAME
+    -x NAME                : toggles expired/unexpired state of NAME
+    --badentry=KIND,NAME   : shows only hosts whose current interaction has bad
+                             entries in of KIND kind and NAME name; if a single
+                             argument ARG1 is given, then KIND,NAME pairs will be
+                             read from a file of name ARG1
+    --extraentry=KIND,NAME : shows only hosts whose current interaction has extra
+                             entries in of KIND kind and NAME name; if a single
+                             argument ARG1 is given, then KIND,NAME pairs will be
+                             read from a file of name ARG1
+    --fields=ARG1,ARG2,... : only displays the fields ARG1,ARG2,...
+                             (name,time,state)
+    --sort=ARG1,ARG2,...   : sorts output on ARG1,ARG2,... (name,time,state)
+    --stale                : shows hosts which haven't run in the last 24 hours
+
+Screenshots
+===========
+
+.. _reports-calendar-summary:
+
+Calendar Summary
+----------------
+
+.. image:: summary_cal.jpg
+   :alt: Calendar summary
+
+.. _reports-item-detail:
+
+Item detail
+-----------
+
+.. image:: item_detail.jpg
+   :alt: Item detail
+
+Node dropdown
+-------------
+
+.. _reports-node-dropdown:
+
+.. image:: node_dropdown.jpg
+   :alt: Node dropdown
diff --git a/doc/reports/index.txt b/doc/reports/index.txt
new file mode 100644
index 000000000..bef686305
--- /dev/null
+++ b/doc/reports/index.txt
@@ -0,0 +1,29 @@
+.. -*- mode: rst -*-
+
+.. _server-reports-index:
+
+The Bcfg2 Reporting System
+==========================
+
+Bcfg2's reporting system is its killer feature. It allows administrators
+to gain a broad understanding of the configuration state of their entire
+environment. It summarizes
+
+* Configuration changes and when they were made
+* Discrepancies between the specification and current client states
+
+  * Clients can be grouped by misconfiguration type
+
+* Configuration entries that are not specified
+* Overall client summaries according to these types
+
+There are two systems, the old system, which builds static reports based
+on a series of XSLT stylesheets and a new dynamic reporting system that
+uses django and a database backend.
+
+.. toctree::
+   :maxdepth: 2
+
+   install
+   static
+   dynamic
diff --git a/doc/reports/install.txt b/doc/reports/install.txt
new file mode 100644
index 000000000..80f9342ae
--- /dev/null
+++ b/doc/reports/install.txt
@@ -0,0 +1,183 @@
+.. -*- mode: rst -*-
+
+.. _EPEL: http://fedoraproject.org/wiki/EPEL
+
+.. This is combination of the Ubuntu guide and the Centos guide for
+   installing the web reports.
+
+.. _server-reports-install:
+
+=====================
+Dynamic (web) Reports
+=====================
+
+The first step is to install the needed software components like the
+Django framework and the database (SQlite2). All packages for Fedora
+are in the Fedora Package Collection or in EPEL_ for CentOS/RHEL::
+
+    [root@system01 ~]# yum -y install Django python-simplejson python-sqlite2
+
+Of course is a web server needed as well::
+
+    [root@system01 ~]# yum -y install httpd mod_python
+
+The same packages are needed for Ubuntu systems::
+
+    [root@system01 ~]# aptitude install python-django apache2 libapache2-mod-python
+
+Now we need to create the sqlite database. Use the following command on
+Fedora, CentOS, or RHEL.::
+
+    [root@system01 ~]# python /usr/lib/python2.4/site-packages/Bcfg2/Server/Reports/manage.py syncdb
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+
+    You just installed Django's auth system, which means you don't have any superusers defined.
+    Would you like to create one now? (yes/no): no
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+
+.. note:: There are different versions of Python available. If you are
+          unsure about your installed version use the following line instead of
+          the line above.::
+
+              [root@system01 ~]# PYVER=`python -c 'import sys;print(sys.version[0:3])'`; python /usr/lib/python$PYVER/site-packages/Bcfg2/site-packages/Bcfg2/Server/Reports/manage.py syncdb
+
+The path on Ubuntu systems is different. Please use the same path as shown
+in the following command to execute the script on an Ubuntu machine in
+the next steps::
+
+    [root@system01 ~]# python /usr/share/pyshared/Bcfg2/Server/Reports/manage.py syncdb
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+
+    You just installed Django's auth system, which means you don't have any superusers defined.
+    Would you like to create one now? (yes/no): no
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+
+The server should be tested to make sure that there are no mistakes::
+
+    [root@system01 ~]# python /usr/lib/python2.6/site-packages/Bcfg2/Server/Reports/manage.py testserver
+    Creating test database...
+    Creating table auth_permission
+    Creating table auth_group
+    Creating table auth_user
+    Creating table auth_message
+    Creating table django_content_type
+    Creating table django_session
+    Creating table django_site
+    Creating table django_admin_log
+    Creating table reports_client
+    Creating table reports_ping
+    Creating table reports_interaction
+    Creating table reports_reason
+    Creating table reports_entries
+    Creating table reports_entries_interactions
+    Creating table reports_performance
+    Creating table reports_internaldatabaseversion
+    Installing index for auth.Permission model
+    Installing index for auth.Message model
+    Installing index for admin.LogEntry model
+    Installing index for reports.Client model
+    Installing index for reports.Ping model
+    Installing index for reports.Interaction model
+    Installing index for reports.Entries model
+    Installing index for reports.Entries_interactions model
+    Validating models...
+    0 errors found
+
+    Django version 1.1.1, using settings 'Reports.settings'
+    Development server is running at http://127.0.0.1:8000/
+    Quit the server with CONTROL-C.
+
+Add DBStats to the plugins line of ``bcfg2.conf``. The resulting
+**[server]** section should look something like this::
+
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase
+
+Start/restart the Bcfg2 server::
+
+    [root@system01 ~]# /etc/init.d/bcfg2-server restart
+
+Run the Bcfg2 client in order to populate the statistics database
+(this run should take a bit longer since you are uploading the client
+statistics to the database).
+
+Download the static reports content::
+
+    [root@system01 ~]# cd /var/www/
+    [root@system01 ~]# svn co https://svn.mcs.anl.gov/repos/bcfg/trunk/bcfg2/reports
+
+Configure Apache using :ref:`dynamic-http-install` as a guide
+
+Copy server/statistics sections of ``bcfg2.conf`` to
+``/etc/bcfg2-web.conf`` (make sure it is world-readable). You should
+then have something like this::
+
+    [server]
+    repository = /var/lib/bcfg2
+    plugins = Base,Bundler,Cfg,DBStats,Metadata,Packages,Probes,Rules,SSHbase
+
+    [statistics]
+    sendmailpath = /usr/lib/sendmail
+    database_engine = sqlite3
+    # 'postgresql', 'mysql', 'mysql_old', 'sqlite3' or 'ado_mssql'.
+    database_name =
+    # Or path to database file if using sqlite3.
+    #<repository>/etc/brpt.sqlite is default path if left empty
+    database_user =
+    # Not used with sqlite3.
+    database_password =
+    # Not used with sqlite3.
+    database_host =
+    # Not used with sqlite3.
+    database_port =
+    # Set to empty string for default. Not used with sqlite3.
+    web_debug = True
+
+Restart apache and point a browser to your Bcfg2 server.
+
+If using sqlite be sure the sql database file and directory containing the database are writable to apache.
diff --git a/doc/reports/item_detail.jpg b/doc/reports/item_detail.jpg
new file mode 100644
index 000000000..198880599
--- /dev/null
+++ b/doc/reports/item_detail.jpg
diff --git a/doc/reports/node_dropdown.jpg b/doc/reports/node_dropdown.jpg
new file mode 100644
index 000000000..d3f7d4f14
--- /dev/null
+++ b/doc/reports/node_dropdown.jpg
diff --git a/doc/reports/static.txt b/doc/reports/static.txt
new file mode 100644
index 000000000..d5d96fe12
--- /dev/null
+++ b/doc/reports/static.txt
@@ -0,0 +1,100 @@
+.. -*- mode: rst -*-
+
+.. _server-reports-static:
+
+=============================
+Bcfg2 Static Reporting System
+=============================
+
+The Bcfg2 reporting system collects and displays information about the
+operation of the Bcfg2 client, and the configuration states of target
+machines.
+
+Goals
+=====
+
+The reporting system provides an interface to administrators describing
+a few important tasks
+
+* Client configuration state, particularly aspects that do not match the configuration specification.
+  Information about bad and extra configuration elements is included.
+* Client execution results (a list of configuration elements that were modified)
+* Client execution performance data (including operation retry counts, and timings for several critical execution regions)
+
+This data can be used to understand the current configuration state
+of the entire network, the operations performed by the client, how the
+configuration changes propagate, and any reconfiguration operations that
+have failed.
+
+Retention Model
+===============
+
+The current reporting system stores statistics in an XML data store, by
+default to {{{<repo>/etc/statistics.xml}}}. It retains either one or two
+statistic sets per host. If the client has a clean configuration state,
+the most recent (clean) record is retained.  If the client has a dirty
+configuration state, two records are retained. One record is the last
+clean record. The other record is the most recent record collected,
+detailing the incorrect state.
+
+This retention model, while non-optimal, does manage to persistently
+record most of the data that users would like.
+
+Setup
+=====
+
+In order to configure your Bcfg2 server for receiving reports, you
+will need to list the Statistics plugin in the plugins line of your
+``bcfg2.conf``. You will also need a [statistics] section
+in your ``bcfg2.conf``. You can find out more about what goes there in the
+``bcfg2.conf`` manpage.
+
+Output
+======
+
+Several output reports can be generated from the statistics store with
+the command line tool {{{bcfg2-build-reports}}}.
+
+* Nodes Digest
+* Nodes Individual
+* Overview Statistics
+* Performance
+
+The data generated by these reports can be delivered by several
+mechanisms:
+
+* HTML
+* Email
+* RSS
+
+Shortcomings and Planned Enhancements
+=====================================
+
+When designing the current reporting system, we were overly concerned with
+the potential explosion in data size over time. In order to address this,
+we opted to use the retention scheme described above. This approach has
+several shortcomings:
+
+* A comprehensive list of reconfiguration operations (with associated
+  timestamps) isn't retained
+* Client results for any given day (except the last one) aren't uniformly
+  retained. This means that inter-client analysis is difficult, if
+  not impossible
+
+We plan to move to a database backend to address the dataset size
+problem and start retaining all information. The move to a SQL backend
+will allow many more types of queries to be efficiently processed. It
+will also make on-demand reports simpler.
+
+Other sorts of information would also be useful to track. We plan to
+add the ability to tag a particular configuration element as security
+related, and include this in reports. This will aid in the effective
+prioritization of manual and failed reconfiguration tasks.
+
+Capability Goals (posed as questions)
+-------------------------------------
+
+* What machines have not yet applied critical updates?
+* How long did critical updates take to be applied?
+* What configuration did machine X have on a particular date?
+* When did machine X perform configuration update Y?
diff --git a/doc/reports/summary_cal.jpg b/doc/reports/summary_cal.jpg
new file mode 100644
index 000000000..fab010762
--- /dev/null
+++ b/doc/reports/summary_cal.jpg