path: root/buildbot/docs/buildbot.info-1

This is buildbot.info, produced by makeinfo version 4.11 from
buildbot.texinfo.

This is the BuildBot manual.

   Copyright (C) 2005,2006 Brian Warner

   Copying and distribution of this file, with or without
modification, are permitted in any medium without royalty provided
the copyright notice and this notice are preserved.


File: buildbot.info,  Node: Top,  Next: Introduction,  Prev: (dir),  Up: (dir)

BuildBot
********

This is the BuildBot manual.

   Copyright (C) 2005,2006 Brian Warner

   Copying and distribution of this file, with or without
modification, are permitted in any medium without royalty provided
the copyright notice and this notice are preserved.

* Menu:

* Introduction::                What the BuildBot does.
* Installation::                Creating a buildmaster and buildslaves,
                                running them.
* Concepts::                    What goes on in the buildbot's little mind.
* Configuration::               Controlling the buildbot.
* Getting Source Code Changes::  Discovering when to run a build.
* Build Process::               Controlling how each build is run.
* Status Delivery::             Telling the world about the build's results.
* Command-line tool::
* Resources::                   Getting help.
* Developer's Appendix::
* Index of Useful Classes::
* Index of master.cfg keys::
* Index::                       Complete index.

 --- The Detailed Node Listing ---

Introduction

* History and Philosophy::
* System Architecture::
* Control Flow::

System Architecture

* BuildSlave Connections::
* Buildmaster Architecture::
* Status Delivery Architecture::

Installation

* Requirements::
* Installing the code::
* Creating a buildmaster::
* Upgrading an Existing Buildmaster::
* Creating a buildslave::
* Launching the daemons::
* Logfiles::
* Shutdown::
* Maintenance::
* Troubleshooting::

Creating a buildslave

* Buildslave Options::

Troubleshooting

* Starting the buildslave::
* Connecting to the buildmaster::
* Forcing Builds::

Concepts

* Version Control Systems::
* Schedulers::
* BuildSet::
* BuildRequest::
* Builder::
* Users::
* Build Properties::

Version Control Systems

* Generalizing VC Systems::
* Source Tree Specifications::
* How Different VC Systems Specify Sources::
* Attributes of Changes::

Users

* Doing Things With Users::
* Email Addresses::
* IRC Nicknames::
* Live Status Clients::

Configuration

* Config File Format::
* Loading the Config File::
* Testing the Config File::
* Defining the Project::
* Change Sources and Schedulers::
* Setting the slaveport::
* Buildslave Specifiers::
* On-Demand ("Latent") Buildslaves::
* Defining Global Properties::
* Defining Builders::
* Defining Status Targets::
* Debug options::

Change Sources and Schedulers

* Scheduler Scheduler::
* AnyBranchScheduler::
* Dependent Scheduler::
* Periodic Scheduler::
* Nightly Scheduler::
* Try Schedulers::
* Triggerable Scheduler::

Buildslave Specifiers
* When Buildslaves Go Missing::

On-Demand ("Latent") Buildslaves
* Amazon Web Services Elastic Compute Cloud ("AWS EC2")::
* Dangers with Latent Buildslaves::
* Writing New Latent Buildslaves::

Getting Source Code Changes

* Change Sources::
* Choosing ChangeSources::
* CVSToys - PBService::
* Mail-parsing ChangeSources::
* PBChangeSource::
* P4Source::
* BonsaiPoller::
* SVNPoller::
* MercurialHook::
* Bzr Hook::
* Bzr Poller::

Mail-parsing ChangeSources

* Subscribing the Buildmaster::
* Using Maildirs::
* Parsing Email Change Messages::

Parsing Email Change Messages

* FCMaildirSource::
* SyncmailMaildirSource::
* BonsaiMaildirSource::
* SVNCommitEmailMaildirSource::

Build Process

* Build Steps::
* Interlocks::
* Build Factories::

Build Steps

* Common Parameters::
* Using Build Properties::
* Source Checkout::
* ShellCommand::
* Simple ShellCommand Subclasses::
* Python BuildSteps::
* Transferring Files::
* Steps That Run on the Master::
* Triggering Schedulers::
* Writing New BuildSteps::

Source Checkout

* CVS::
* SVN::
* Darcs::
* Mercurial::
* Arch::
* Bazaar::
* Bzr::
* P4::
* Git::

Simple ShellCommand Subclasses

* Configure::
* Compile::
* Test::
* TreeSize::
* PerlModuleTest::
* SetProperty::

Python BuildSteps

* BuildEPYDoc::
* PyFlakes::
* PyLint::

Writing New BuildSteps

* BuildStep LogFiles::
* Reading Logfiles::
* Adding LogObservers::
* BuildStep URLs::

Build Factories

* BuildStep Objects::
* BuildFactory::
* Process-Specific build factories::

BuildStep Objects

* BuildFactory Attributes::
* Quick builds::

BuildFactory

* BuildFactory Attributes::
* Quick builds::

Process-Specific build factories

* GNUAutoconf::
* CPAN::
* Python distutils::
* Python/Twisted/trial projects::

Status Delivery

* WebStatus::
* MailNotifier::
* IRC Bot::
* PBListener::
* Writing New Status Plugins::

WebStatus

* WebStatus Configuration Parameters::
* Buildbot Web Resources::
* XMLRPC server::
* HTML Waterfall::

Command-line tool

* Administrator Tools::
* Developer Tools::
* Other Tools::
* .buildbot config directory::

Developer Tools

* statuslog::
* statusgui::
* try::

waiting for results

* try --diff::

Other Tools

* sendchange::
* debugclient::


File: buildbot.info,  Node: Introduction,  Next: Installation,  Prev: Top,  Up: Top

1 Introduction
**************

The BuildBot is a system to automate the compile/test cycle required
by most software projects to validate code changes. By automatically
rebuilding and testing the tree each time something has changed,
build problems are pinpointed quickly, before other developers are
inconvenienced by the failure. The guilty developer can be identified
and harassed without human intervention. By running the builds on a
variety of platforms, developers who do not have the facilities to
test their changes everywhere before checkin will at least know
shortly afterwards whether they have broken the build or not. Warning
counts, lint checks, image size, compile time, and other build
parameters can be tracked over time, are more visible, and are
therefore easier to improve.

   The overall goal is to reduce tree breakage and provide a platform
to run tests or code-quality checks that are too annoying or pedantic
for any human to waste their time with. Developers get immediate (and
potentially public) feedback about their changes, encouraging them to
be more careful about testing before checkin.

   Features:

   * run builds on a variety of slave platforms

   * arbitrary build process: handles projects using C, Python,
     whatever

   * minimal host requirements: python and Twisted

   * slaves can be behind a firewall if they can still do checkout

   * status delivery through web page, email, IRC, other protocols

   * track builds in progress, provide estimated completion time

   * flexible configuration by subclassing generic build process
     classes

   * debug tools to force a new build, submit fake Changes, query
     slave status

   * released under the GPL

* Menu:

* History and Philosophy::
* System Architecture::
* Control Flow::


File: buildbot.info,  Node: History and Philosophy,  Next: System Architecture,  Prev: Introduction,  Up: Introduction

1.1 History and Philosophy
==========================

The Buildbot was inspired by a similar project built for a development
team writing a cross-platform embedded system. The various components
of the project were supposed to compile and run on several flavors of
unix (linux, solaris, BSD), but individual developers had their own
preferences and tended to stick to a single platform. From time to
time, incompatibilities would sneak in (some unix platforms want to
use `string.h', some prefer `strings.h'), and then the tree would
compile for some developers but not others. The buildbot was written
to automate the human process of walking into the office, updating a
tree, compiling (and discovering the breakage), finding the developer
at fault, and complaining to them about the problem they had
introduced. With multiple platforms it was difficult for developers to
do the right thing (compile their potential change on all platforms);
the buildbot offered a way to help.

   Another problem was when programmers would change the behavior of a
library without warning its users, or change internal aspects that
other code was (unfortunately) depending upon. Adding unit tests to
the codebase helps here: if an application's unit tests pass despite
changes in the libraries it uses, you can have more confidence that
the library changes haven't broken anything. Many developers
complained that the unit tests were inconvenient or took too long to
run: having the buildbot run them reduces the developer's workload to
a minimum.

   In general, having more visibility into the project is always good,
and automation makes it easier for developers to do the right thing.
When everyone can see the status of the project, developers are
encouraged to keep the tree in good working order. Unit tests that
aren't run on a regular basis tend to suffer from bitrot just like
code does: exercising them on a regular basis helps to keep them
functioning and useful.

   The current version of the Buildbot is additionally targeted at
distributed free-software projects, where resources and platforms are
only available when provided by interested volunteers. The buildslaves
are designed to require an absolute minimum of configuration, reducing
the effort a potential volunteer needs to expend to be able to
contribute a new test environment to the project. The goal is for
anyone who wishes that a given project would run on their favorite
platform should be able to offer that project a buildslave, running on
that platform, where they can verify that their portability code
works, and keeps working.


File: buildbot.info,  Node: System Architecture,  Next: Control Flow,  Prev: History and Philosophy,  Up: Introduction

1.2 System Architecture
=======================

The Buildbot consists of a single `buildmaster' and one or more
`buildslaves', connected in a star topology. The buildmaster makes
all decisions about what, when, and how to build. It sends commands
to be run on the build slaves, which simply execute the commands and
return the results. (certain steps involve more local decision
making, where the overhead of sending a lot of commands back and
forth would be inappropriate, but in general the buildmaster is
responsible for everything).

   The buildmaster is usually fed `Changes' by some sort of version
control system (*note Change Sources::), which may cause builds to be
run. As the builds are performed, various status messages are
produced, which are then sent to any registered Status Targets (*note
Status Delivery::).


                  +------------------+           +-----------+
                  |                  |---------->|  Browser  |
                  |   BuildMaster    |           +-----------+
        Changes   |                  |--------------->+--------+
     +----------->|                  | Build Status   | email  |
     |            |                  |------------+   +--------+
     |            |                  |-------+    |     +---------------+
     |            +------------------+       |    +---->| Status Client |
+----------+         | ^      | ^            |          +---------------+
| Change   |         | |     C| |            |             +-----+
|  Sources |         | |     o| |            +------------>| IRC |
|          |         | |     m| |R                         +-----+
| CVS      |         v |     m| |e
| SVN      |    +---------+  a| |s
| Darcs    |    |  Build  |  n| |u
| .. etc   |    |  Slave  |  d| |l
|          |    +---------+  s| |t
|          |                  v |s
+----------+                +---------+
                            |  Build  |
                            |  Slave  |
                            +---------+

   The buildmaster is configured and maintained by the "buildmaster
admin", who is generally the project team member responsible for
build process issues. Each buildslave is maintained by a "buildslave
admin", who do not need to be quite as involved. Generally slaves are
run by anyone who has an interest in seeing the project work well on
their favorite platform.

* Menu:

* BuildSlave Connections::
* Buildmaster Architecture::
* Status Delivery Architecture::


File: buildbot.info,  Node: BuildSlave Connections,  Next: Buildmaster Architecture,  Prev: System Architecture,  Up: System Architecture

1.2.1 BuildSlave Connections
----------------------------

The buildslaves are typically run on a variety of separate machines,
at least one per platform of interest. These machines connect to the
buildmaster over a TCP connection to a publically-visible port. As a
result, the buildslaves can live behind a NAT box or similar
firewalls, as long as they can get to buildmaster. The TCP connections
are initiated by the buildslave and accepted by the buildmaster, but
commands and results travel both ways within this connection. The
buildmaster is always in charge, so all commands travel exclusively
from the buildmaster to the buildslave.

   To perform builds, the buildslaves must typically obtain source
code from a CVS/SVN/etc repository. Therefore they must also be able
to reach the repository. The buildmaster provides instructions for
performing builds, but does not provide the source code itself.



Repository|  |       BuildMaster   |      |
 (CVS/SVN)|  |                    ^|^^^   |
          |  |                   / c   \  |
----------+  +------------------/--o----\-+
        ^                      /   m  ^  \
        |                     /    m  |   \
 checkout/update              --+  a  | +--
        |                    TCP|  n  | |TCP
        |                       |  d  | |
        |                       |  s  | |
        |                       |  |  | |
        |                       |  |  r |
        |                       |  |  e |
 -N-A-T-|- - - - -N-A-T- - - - -|- |- s-|- - - - -N-A-T- - -
        |                       |  |  u |
        |                       |  |  l |
        |    +------------------|--|--t-|-+
        |    |                  |  |  s | |
        +----|                     v  |   |
             |                        |   |
             |                        |   |
             |                            |
             |       BuildSlave           |
             +----------------------------+


File: buildbot.info,  Node: Buildmaster Architecture,  Next: Status Delivery Architecture,  Prev: BuildSlave Connections,  Up: System Architecture

1.2.2 Buildmaster Architecture
------------------------------

The Buildmaster consists of several pieces:



 +---------------+
 | Change Source |----->----+
 +---------------+          |
                         Changes
                            |
 +---------------+          v
 | Change Source |----->----+
 +---------------+          v
                      +-----+-------+
                      |             |
                      v             v
              +-----------+    +-----------+
              | Scheduler |    | Scheduler |
              +-----------+    +-----------+
                 |                |  |
          +------+---------+  +---+  +-----+
          |                |  |            |
          v                |  |          Build
      :      :           : v  v :        Request
      :      :           :      :          |
      : ---- :           :      :          |
      : ---- :           : ---- :          |
      +======+           +======+      :   v  :
         |                  |          :      :
         v                  v          :      :
   +---------+        +---------+      :queue :
   | Builder |        | Builder |      +======+
   +---------+        +---------+         |
                                          v
                                    +---------+
                                    | Builder |
                                    +---------+

   * Change Sources, which create a Change object each time something
     is modified in the VC repository. Most ChangeSources listen for
     messages from a hook script of some sort. Some sources actively
     poll the repository on a regular basis. All Changes are fed to
     the Schedulers.

   * Schedulers, which decide when builds should be performed. They
     collect Changes into BuildRequests, which are then queued for
     delivery to Builders until a buildslave is available.

   * Builders, which control exactly _how_ each build is performed
     (with a series of BuildSteps, configured in a BuildFactory). Each
     Build is run on a single buildslave.

   * Status plugins, which deliver information about the build results
     through protocols like HTTP, mail, and IRC.




                       +-----------------+
                       |  BuildSlave     |
                       |                 |
                       |                 |
 +-------+             | +------------+  |
 |Builder|----Build----->|SlaveBuilder|  |
 +-------+             | +------------+  |
                       |                 |
                       | +------------+  |
             +-Build---->|SlaveBuilder|  |
             |         | +------------+  |
 +-------+   |         |                 |
 |Builder|---+         +-----------------+
 +-------+   |
             |
             |      +-----------------+
           Build    |  BuildSlave     |
             |      |                 |
             |      |                 |
             |      | +------------+  |
             +------->|SlaveBuilder|  |
                    | +------------+  |
 +-------+          |                 |
 |Builder|--+       | +------------+  |
 +-------+  +-------->|SlaveBuilder|  |
                    | +------------+  |
                    |                 |
                    +-----------------+

   Each Builder is configured with a list of BuildSlaves that it will
use for its builds. These buildslaves are expected to behave
identically: the only reason to use multiple BuildSlaves for a single
Builder is to provide a measure of load-balancing.

   Within a single BuildSlave, each Builder creates its own
SlaveBuilder instance. These SlaveBuilders operate independently from
each other.  Each gets its own base directory to work in. It is quite
common to have many Builders sharing the same buildslave. For
example, there might be two buildslaves: one for i386, and a second
for PowerPC.  There may then be a pair of Builders that do a full
compile/test run, one for each architecture, and a lone Builder that
creates snapshot source tarballs if the full builders complete
successfully. The full builders would each run on a single
buildslave, whereas the tarball creation step might run on either
buildslave (since the platform doesn't matter when creating source
tarballs). In this case, the mapping would look like:

     Builder(full-i386)  ->  BuildSlaves(slave-i386)
     Builder(full-ppc)   ->  BuildSlaves(slave-ppc)
     Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc)

   and each BuildSlave would have two SlaveBuilders inside it, one
for a full builder, and a second for the source-tarball builder.

   Once a SlaveBuilder is available, the Builder pulls one or more
BuildRequests off its incoming queue. (It may pull more than one if it
determines that it can merge the requests together; for example, there
may be multiple requests to build the current HEAD revision). These
requests are merged into a single Build instance, which includes the
SourceStamp that describes what exact version of the source code
should be used for the build. The Build is then randomly assigned to a
free SlaveBuilder and the build begins.

   The behaviour when BuildRequests are merged can be customized,
*note Merging BuildRequests::.


File: buildbot.info,  Node: Status Delivery Architecture,  Prev: Buildmaster Architecture,  Up: System Architecture

1.2.3 Status Delivery Architecture
----------------------------------

The buildmaster maintains a central Status object, to which various
status plugins are connected. Through this Status object, a full
hierarchy of build status objects can be obtained.



  Status Objects            Status Plugins       User Clients

 +------+                   +---------+        +-----------+
 |Status|<--------------+-->|Waterfall|<-------|Web Browser|
 +------+               |   +---------+        +-----------+
    |  +-----+          |
    v        v          |
+-------+  +-------+    |     +---+            +----------+
|Builder|  |Builder|    +---->|IRC|<----------->IRC Server|
|Status |  |Status |    |     +---+            +----------+
+-------+  +-------+    |
    |  +----+           |
    v       v           |   +------------+     +----+
+------+  +------+      +-->|MailNotifier|---->|SMTP|
|Build |  |Build |          +------------+     +----+
|Status|  |Status|
+------+  +------+
    | +-----+
    v       v
+------+  +------+
|Step  |  |Step  |
|Status|  |Status|
+------+  +------+
   | +---+
   v     v
+----+ +----+
|Log | |Log |
|File| |File|
+----+ +----+

   The configuration file controls which status plugins are active.
Each status plugin gets a reference to the top-level Status object.
From there they can request information on each Builder, Build, Step,
and LogFile. This query-on-demand interface is used by the
html.Waterfall plugin to create the main status page each time a web
browser hits the main URL.

   The status plugins can also subscribe to hear about new Builds as
they occur: this is used by the MailNotifier to create new email
messages for each recently-completed Build.

   The Status object records the status of old builds on disk in the
buildmaster's base directory. This allows it to return information
about historical builds.

   There are also status objects that correspond to Schedulers and
BuildSlaves. These allow status plugins to report information about
upcoming builds, and the online/offline status of each buildslave.


File: buildbot.info,  Node: Control Flow,  Prev: System Architecture,  Up: Introduction

1.3 Control Flow
================

A day in the life of the buildbot:

   * A developer commits some source code changes to the repository.
     A hook script or commit trigger of some sort sends information
     about this change to the buildmaster through one of its
     configured Change Sources. This notification might arrive via
     email, or over a network connection (either initiated by the
     buildmaster as it "subscribes" to changes, or by the commit
     trigger as it pushes Changes towards the buildmaster). The
     Change contains information about who made the change, what
     files were modified, which revision contains the change, and any
     checkin comments.

   * The buildmaster distributes this change to all of its configured
     Schedulers. Any "important" changes cause the "tree-stable-timer"
     to be started, and the Change is added to a list of those that
     will go into a new Build. When the timer expires, a Build is
     started on each of a set of configured Builders, all
     compiling/testing the same source code. Unless configured
     otherwise, all Builds run in parallel on the various buildslaves.

   * The Build consists of a series of Steps. Each Step causes some
     number of commands to be invoked on the remote buildslave
     associated with that Builder. The first step is almost always to
     perform a checkout of the appropriate revision from the same VC
     system that produced the Change. The rest generally perform a
     compile and run unit tests. As each Step runs, the buildslave
     reports back command output and return status to the buildmaster.

   * As the Build runs, status messages like "Build Started", "Step
     Started", "Build Finished", etc, are published to a collection of
     Status Targets. One of these targets is usually the HTML
     "Waterfall" display, which shows a chronological list of events,
     and summarizes the results of the most recent build at the top
     of each column.  Developers can periodically check this page to
     see how their changes have fared. If they see red, they know
     that they've made a mistake and need to fix it. If they see
     green, they know that they've done their duty and don't need to
     worry about their change breaking anything.

   * If a MailNotifier status target is active, the completion of a
     build will cause email to be sent to any developers whose
     Changes were incorporated into this Build. The MailNotifier can
     be configured to only send mail upon failing builds, or for
     builds which have just transitioned from passing to failing.
     Other status targets can provide similar real-time notification
     via different communication channels, like IRC.



File: buildbot.info,  Node: Installation,  Next: Concepts,  Prev: Introduction,  Up: Top

2 Installation
**************

* Menu:

* Requirements::
* Installing the code::
* Creating a buildmaster::
* Upgrading an Existing Buildmaster::
* Creating a buildslave::
* Launching the daemons::
* Logfiles::
* Shutdown::
* Maintenance::
* Troubleshooting::


File: buildbot.info,  Node: Requirements,  Next: Installing the code,  Prev: Installation,  Up: Installation

2.1 Requirements
================

At a bare minimum, you'll need the following (for both the buildmaster
and a buildslave):

   * Python: http://www.python.org

     Buildbot requires python-2.3 or later, and is primarily developed
     against python-2.4. It is also tested against python-2.5 .

   * Twisted: http://twistedmatrix.com

     Both the buildmaster and the buildslaves require Twisted-2.0.x or
     later. It has been tested against all releases of Twisted up to
     Twisted-2.5.0 (the most recent as of this writing). As always,
     the most recent version is recommended.

     Twisted is delivered as a collection of subpackages. You'll need
     at least "Twisted" (the core package), and you'll also want
     TwistedMail, TwistedWeb, and TwistedWords (for sending email,
     serving a web status page, and delivering build status via IRC,
     respectively). You might also want TwistedConch (for the
     encrypted Manhole debug port). Note that Twisted requires
     ZopeInterface to be installed as well.


   Certain other packages may be useful on the system running the
buildmaster:

   * CVSToys: http://purl.net/net/CVSToys

     If your buildmaster uses FreshCVSSource to receive change
     notification from a cvstoys daemon, it will require CVSToys be
     installed (tested with CVSToys-1.0.10). If the it doesn't use
     that source (i.e. if you only use a mail-parsing change source,
     or the SVN notification script), you will not need CVSToys.


   And of course, your project's build process will impose additional
requirements on the buildslaves. These hosts must have all the tools
necessary to compile and test your project's source code.


File: buildbot.info,  Node: Installing the code,  Next: Creating a buildmaster,  Prev: Requirements,  Up: Installation

2.2 Installing the code
=======================

The Buildbot is installed using the standard python `distutils'
module. After unpacking the tarball, the process is:

     python setup.py build
     python setup.py install

   where the install step may need to be done as root. This will put
the bulk of the code in somewhere like
/usr/lib/python2.3/site-packages/buildbot . It will also install the
`buildbot' command-line tool in /usr/bin/buildbot.

   To test this, shift to a different directory (like /tmp), and run:

     buildbot --version

   If it shows you the versions of Buildbot and Twisted, the install
went ok. If it says `no such command' or it gets an `ImportError'
when it tries to load the libaries, then something went wrong.
`pydoc buildbot' is another useful diagnostic tool.

   Windows users will find these files in other places. You will need
to make sure that python can find the libraries, and will probably
find it convenient to have `buildbot' on your PATH.

   If you wish, you can run the buildbot unit test suite like this:

     PYTHONPATH=. trial buildbot.test

   This should run up to 192 tests, depending upon what VC tools you
have installed. On my desktop machine it takes about five minutes to
complete. Nothing should fail, a few might be skipped. If any of the
tests fail, you should stop and investigate the cause before
continuing the installation process, as it will probably be easier to
track down the bug early.

   If you cannot or do not wish to install the buildbot into a
site-wide location like `/usr' or `/usr/local', you can also install
it into the account's home directory. Do the install command like
this:

     python setup.py install --home=~

   That will populate `~/lib/python' and create `~/bin/buildbot'.
Make sure this lib directory is on your `PYTHONPATH'.


File: buildbot.info,  Node: Creating a buildmaster,  Next: Upgrading an Existing Buildmaster,  Prev: Installing the code,  Up: Installation

2.3 Creating a buildmaster
==========================

As you learned earlier (*note System Architecture::), the buildmaster
runs on a central host (usually one that is publically visible, so
everybody can check on the status of the project), and controls all
aspects of the buildbot system. Let us call this host
`buildbot.example.org'.

   You may wish to create a separate user account for the buildmaster,
perhaps named `buildmaster'. This can help keep your personal
configuration distinct from that of the buildmaster and is useful if
you have to use a mail-based notification system (*note Change
Sources::). However, the Buildbot will work just fine with your
regular user account.

   You need to choose a directory for the buildmaster, called the
`basedir'. This directory will be owned by the buildmaster, which
will use configuration files therein, and create status files as it
runs. `~/Buildbot' is a likely value. If you run multiple
buildmasters in the same account, or if you run both masters and
slaves, you may want a more distinctive name like
`~/Buildbot/master/gnomovision' or `~/Buildmasters/fooproject'. If
you are using a separate user account, this might just be
`~buildmaster/masters/fooproject'.

   Once you've picked a directory, use the `buildbot create-master'
command to create the directory and populate it with startup files:

     buildbot create-master BASEDIR

   You will need to create a configuration file (*note
Configuration::) before starting the buildmaster. Most of the rest of
this manual is dedicated to explaining how to do this. A sample
configuration file is placed in the working directory, named
`master.cfg.sample', which can be copied to `master.cfg' and edited
to suit your purposes.

   (Internal details: This command creates a file named
`buildbot.tac' that contains all the state necessary to create the
buildmaster. Twisted has a tool called `twistd' which can use this
.tac file to create and launch a buildmaster instance. twistd takes
care of logging and daemonization (running the program in the
background). `/usr/bin/buildbot' is a front end which runs twistd for
you.)

   In addition to `buildbot.tac', a small `Makefile.sample' is
installed. This can be used as the basis for customized daemon
startup, *Note Launching the daemons::.


File: buildbot.info,  Node: Upgrading an Existing Buildmaster,  Next: Creating a buildslave,  Prev: Creating a buildmaster,  Up: Installation

2.4 Upgrading an Existing Buildmaster
=====================================

If you have just installed a new version of the Buildbot code, and you
have buildmasters that were created using an older version, you'll
need to upgrade these buildmasters before you can use them. The
upgrade process adds and modifies files in the buildmaster's base
directory to make it compatible with the new code.

     buildbot upgrade-master BASEDIR

   This command will also scan your `master.cfg' file for
incompatbilities (by loading it and printing any errors or deprecation
warnings that occur). Each buildbot release tries to be compatible
with configurations that worked cleanly (i.e. without deprecation
warnings) on the previous release: any functions or classes that are
to be removed will first be deprecated in a release, to give users a
chance to start using their replacement.

   The 0.7.6 release introduced the `public_html/' directory, which
contains `index.html' and other files served by the `WebStatus' and
`Waterfall' status displays. The `upgrade-master' command will create
these files if they do not already exist. It will not modify existing
copies, but it will write a new copy in e.g. `index.html.new' if the
new version differs from the version that already exists.

   The `upgrade-master' command is idempotent. It is safe to run it
multiple times. After each upgrade of the buildbot code, you should
use `upgrade-master' on all your buildmasters.


File: buildbot.info,  Node: Creating a buildslave,  Next: Launching the daemons,  Prev: Upgrading an Existing Buildmaster,  Up: Installation

2.5 Creating a buildslave
=========================

Typically, you will be adding a buildslave to an existing buildmaster,
to provide additional architecture coverage. The buildbot
administrator will give you several pieces of information necessary to
connect to the buildmaster. You should also be somewhat familiar with
the project being tested, so you can troubleshoot build problems
locally.

   The buildbot exists to make sure that the project's stated "how to
build it" process actually works. To this end, the buildslave should
run in an environment just like that of your regular developers.
Typically the project build process is documented somewhere
(`README', `INSTALL', etc), in a document that should mention all
library dependencies and contain a basic set of build instructions.
This document will be useful as you configure the host and account in
which the buildslave runs.

   Here's a good checklist for setting up a buildslave:

  1. Set up the account

     It is recommended (although not mandatory) to set up a separate
     user account for the buildslave. This account is frequently named
     `buildbot' or `buildslave'. This serves to isolate your personal
     working environment from that of the slave's, and helps to
     minimize the security threat posed by letting possibly-unknown
     contributors run arbitrary code on your system. The account
     should have a minimum of fancy init scripts.

  2. Install the buildbot code

     Follow the instructions given earlier (*note Installing the
     code::).  If you use a separate buildslave account, and you
     didn't install the buildbot code to a shared location, then you
     will need to install it with `--home=~' for each account that
     needs it.

  3. Set up the host

     Make sure the host can actually reach the buildmaster. Usually
     the buildmaster is running a status webserver on the same
     machine, so simply point your web browser at it and see if you
     can get there.  Install whatever additional packages or
     libraries the project's INSTALL document advises. (or not: if
     your buildslave is supposed to make sure that building without
     optional libraries still works, then don't install those
     libraries).

     Again, these libraries don't necessarily have to be installed to
     a site-wide shared location, but they must be available to your
     build process. Accomplishing this is usually very specific to
     the build process, so installing them to `/usr' or `/usr/local'
     is usually the best approach.

  4. Test the build process

     Follow the instructions in the INSTALL document, in the
     buildslave's account. Perform a full CVS (or whatever) checkout,
     configure, make, run tests, etc. Confirm that the build works
     without manual fussing.  If it doesn't work when you do it by
     hand, it will be unlikely to work when the buildbot attempts to
     do it in an automated fashion.

  5. Choose a base directory

     This should be somewhere in the buildslave's account, typically
     named after the project which is being tested. The buildslave
     will not touch any file outside of this directory. Something
     like `~/Buildbot' or `~/Buildslaves/fooproject' is appropriate.

  6. Get the buildmaster host/port, botname, and password

     When the buildbot admin configures the buildmaster to accept and
     use your buildslave, they will provide you with the following
     pieces of information:

        * your buildslave's name

        * the password assigned to your buildslave

        * the hostname and port number of the buildmaster, i.e.
          buildbot.example.org:8007

  7. Create the buildslave

     Now run the 'buildbot' command as follows:

          buildbot create-slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD

     This will create the base directory and a collection of files
     inside, including the `buildbot.tac' file that contains all the
     information you passed to the `buildbot' command.

  8. Fill in the hostinfo files

     When it first connects, the buildslave will send a few files up
     to the buildmaster which describe the host that it is running
     on. These files are presented on the web status display so that
     developers have more information to reproduce any test failures
     that are witnessed by the buildbot. There are sample files in
     the `info' subdirectory of the buildbot's base directory. You
     should edit these to correctly describe you and your host.

     `BASEDIR/info/admin' should contain your name and email address.
     This is the "buildslave admin address", and will be visible from
     the build status page (so you may wish to munge it a bit if
     address-harvesting spambots are a concern).

     `BASEDIR/info/host' should be filled with a brief description of
     the host: OS, version, memory size, CPU speed, versions of
     relevant libraries installed, and finally the version of the
     buildbot code which is running the buildslave.

     If you run many buildslaves, you may want to create a single
     `~buildslave/info' file and share it among all the buildslaves
     with symlinks.


* Menu:

* Buildslave Options::


File: buildbot.info,  Node: Buildslave Options,  Prev: Creating a buildslave,  Up: Creating a buildslave

2.5.1 Buildslave Options
------------------------

There are a handful of options you might want to use when creating the
buildslave with the `buildbot create-slave <options> DIR <params>'
command. You can type `buildbot create-slave --help' for a summary.
To use these, just include them on the `buildbot create-slave'
command line, like this:

     buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd

`--usepty'
     This is a boolean flag that tells the buildslave whether to
     launch child processes in a PTY or with regular pipes (the
     default) when the master does not specify.  This option is
     deprecated, as this particular parameter is better specified on
     the master.

`--umask'
     This is a string (generally an octal representation of an
     integer) which will cause the buildslave process' "umask" value
     to be set shortly after initialization. The "twistd"
     daemonization utility forces the umask to 077 at startup (which
     means that all files created by the buildslave or its child
     processes will be unreadable by any user other than the
     buildslave account). If you want build products to be readable
     by other accounts, you can add `--umask=022' to tell the
     buildslave to fix the umask after twistd clobbers it. If you want
     build products to be _writable_ by other accounts too, use
     `--umask=000', but this is likely to be a security problem.

`--keepalive'
     This is a number that indicates how frequently "keepalive"
     messages should be sent from the buildslave to the buildmaster,
     expressed in seconds. The default (600) causes a message to be
     sent to the buildmaster at least once every 10 minutes. To set
     this to a lower value, use e.g. `--keepalive=120'.

     If the buildslave is behind a NAT box or stateful firewall, these
     messages may help to keep the connection alive: some NAT boxes
     tend to forget about a connection if it has not been used in a
     while. When this happens, the buildmaster will think that the
     buildslave has disappeared, and builds will time out. Meanwhile
     the buildslave will not realize than anything is wrong.

`--maxdelay'
     This is a number that indicates the maximum amount of time the
     buildslave will wait between connection attempts, expressed in
     seconds. The default (300) causes the buildslave to wait at most
     5 minutes before trying to connect to the buildmaster again.

`--log-size'
     This is the size in bytes when to rotate the Twisted log files.

`--log-count'
     This is the number of log rotations to keep around. You can
     either specify a number or `None' (the default) to keep all
     `twistd.log' files around.



File: buildbot.info,  Node: Launching the daemons,  Next: Logfiles,  Prev: Creating a buildslave,  Up: Installation

2.6 Launching the daemons
=========================

Both the buildmaster and the buildslave run as daemon programs. To
launch them, pass the working directory to the `buildbot' command:

     buildbot start BASEDIR

   This command will start the daemon and then return, so normally it
will not produce any output. To verify that the programs are indeed
running, look for a pair of files named `twistd.log' and `twistd.pid'
that should be created in the working directory.  `twistd.pid'
contains the process ID of the newly-spawned daemon.

   When the buildslave connects to the buildmaster, new directories
will start appearing in its base directory. The buildmaster tells the
slave to create a directory for each Builder which will be using that
slave.  All build operations are performed within these directories:
CVS checkouts, compiles, and tests.

   Once you get everything running, you will want to arrange for the
buildbot daemons to be started at boot time. One way is to use
`cron', by putting them in a @reboot crontab entry(1):

     @reboot buildbot start BASEDIR

   When you run `crontab' to set this up, remember to do it as the
buildmaster or buildslave account! If you add this to your crontab
when running as your regular account (or worse yet, root), then the
daemon will run as the wrong user, quite possibly as one with more
authority than you intended to provide.

   It is important to remember that the environment provided to cron
jobs and init scripts can be quite different that your normal runtime.
There may be fewer environment variables specified, and the PATH may
be shorter than usual. It is a good idea to test out this method of
launching the buildslave by using a cron job with a time in the near
future, with the same command, and then check `twistd.log' to make
sure the slave actually started correctly. Common problems here are
for `/usr/local' or `~/bin' to not be on your `PATH', or for
`PYTHONPATH' to not be set correctly.  Sometimes `HOME' is messed up
too.

   To modify the way the daemons are started (perhaps you want to set
some environment variables first, or perform some cleanup each time),
you can create a file named `Makefile.buildbot' in the base
directory. When the `buildbot' front-end tool is told to `start' the
daemon, and it sees this file (and `/usr/bin/make' exists), it will
do `make -f Makefile.buildbot start' instead of its usual action
(which involves running `twistd'). When the buildmaster or buildslave
is installed, a `Makefile.sample' is created which implements the
same behavior as the the `buildbot' tool uses, so if you want to
customize the process, just copy `Makefile.sample' to
`Makefile.buildbot' and edit it as necessary.

   Some distributions may include conveniences to make starting
buildbot at boot time easy.  For instance, with the default buildbot
package in Debian-based distributions, you may only need to modify
`/etc/default/buildbot' (see also `/etc/init.d/buildbot', which reads
the configuration in `/etc/default/buildbot').

   ---------- Footnotes ----------

   (1) this @reboot syntax is understood by Vixie cron, which is the
flavor usually provided with linux systems. Other unices may have a
cron that doesn't understand @reboot


File: buildbot.info,  Node: Logfiles,  Next: Shutdown,  Prev: Launching the daemons,  Up: Installation

2.7 Logfiles
============

While a buildbot daemon runs, it emits text to a logfile, named
`twistd.log'. A command like `tail -f twistd.log' is useful to watch
the command output as it runs.

   The buildmaster will announce any errors with its configuration
file in the logfile, so it is a good idea to look at the log at
startup time to check for any problems. Most buildmaster activities
will cause lines to be added to the log.


File: buildbot.info,  Node: Shutdown,  Next: Maintenance,  Prev: Logfiles,  Up: Installation

2.8 Shutdown
============

To stop a buildmaster or buildslave manually, use:

     buildbot stop BASEDIR

   This simply looks for the `twistd.pid' file and kills whatever
process is identified within.

   At system shutdown, all processes are sent a `SIGKILL'. The
buildmaster and buildslave will respond to this by shutting down
normally.

   The buildmaster will respond to a `SIGHUP' by re-reading its
config file. Of course, this only works on unix-like systems with
signal support, and won't work on Windows. The following shortcut is
available:

     buildbot reconfig BASEDIR

   When you update the Buildbot code to a new release, you will need
to restart the buildmaster and/or buildslave before it can take
advantage of the new code. You can do a `buildbot stop BASEDIR' and
`buildbot start BASEDIR' in quick succession, or you can use the
`restart' shortcut, which does both steps for you:

     buildbot restart BASEDIR

   There are certain configuration changes that are not handled
cleanly by `buildbot reconfig'. If this occurs, `buildbot restart' is
a more robust tool to fully switch over to the new configuration.

   `buildbot restart' may also be used to start a stopped Buildbot
instance. This behaviour is useful when writing scripts that stop,
start and restart Buildbot.

   A buildslave may also be gracefully shutdown from the *note
WebStatus:: status plugin. This is useful to shutdown a buildslave
without interrupting any current builds. The buildmaster will wait
until the buildslave is finished all its current builds, and will
then tell the buildslave to shutdown.


File: buildbot.info,  Node: Maintenance,  Next: Troubleshooting,  Prev: Shutdown,  Up: Installation

2.9 Maintenance
===============

It is a good idea to check the buildmaster's status page every once in
a while, to see if your buildslave is still online. Eventually the
buildbot will probably be enhanced to send you email (via the
`info/admin' email address) when the slave has been offline for more
than a few hours.

   If you find you can no longer provide a buildslave to the project,
please let the project admins know, so they can put out a call for a
replacement.

   The Buildbot records status and logs output continually, each time
a build is performed. The status tends to be small, but the build logs
can become quite large. Each build and log are recorded in a separate
file, arranged hierarchically under the buildmaster's base directory.
To prevent these files from growing without bound, you should
periodically delete old build logs. A simple cron job to delete
anything older than, say, two weeks should do the job. The only trick
is to leave the `buildbot.tac' and other support files alone, for
which find's `-mindepth' argument helps skip everything in the top
directory. You can use something like the following:

     @weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm {} \;
     @weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \;


File: buildbot.info,  Node: Troubleshooting,  Prev: Maintenance,  Up: Installation

2.10 Troubleshooting
====================

Here are a few hints on diagnosing common problems.

* Menu:

* Starting the buildslave::
* Connecting to the buildmaster::
* Forcing Builds::


File: buildbot.info,  Node: Starting the buildslave,  Next: Connecting to the buildmaster,  Prev: Troubleshooting,  Up: Troubleshooting

2.10.1 Starting the buildslave
------------------------------

Cron jobs are typically run with a minimal shell (`/bin/sh', not
`/bin/bash'), and tilde expansion is not always performed in such
commands. You may want to use explicit paths, because the `PATH' is
usually quite short and doesn't include anything set by your shell's
startup scripts (`.profile', `.bashrc', etc). If you've installed
buildbot (or other python libraries) to an unusual location, you may
need to add a `PYTHONPATH' specification (note that python will do
tilde-expansion on `PYTHONPATH' elements by itself). Sometimes it is
safer to fully-specify everything:

     @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir

   Take the time to get the @reboot job set up. Otherwise, things
will work fine for a while, but the first power outage or system
reboot you have will stop the buildslave with nothing but the cries
of sorrowful developers to remind you that it has gone away.


File: buildbot.info,  Node: Connecting to the buildmaster,  Next: Forcing Builds,  Prev: Starting the buildslave,  Up: Troubleshooting

2.10.2 Connecting to the buildmaster
------------------------------------

If the buildslave cannot connect to the buildmaster, the reason should
be described in the `twistd.log' logfile. Some common problems are an
incorrect master hostname or port number, or a mistyped bot name or
password. If the buildslave loses the connection to the master, it is
supposed to attempt to reconnect with an exponentially-increasing
backoff. Each attempt (and the time of the next attempt) will be
logged. If you get impatient, just manually stop and re-start the
buildslave.

   When the buildmaster is restarted, all slaves will be disconnected,
and will attempt to reconnect as usual. The reconnect time will depend
upon how long the buildmaster is offline (i.e. how far up the
exponential backoff curve the slaves have travelled). Again,
`buildbot stop BASEDIR; buildbot start BASEDIR' will speed up the
process.


File: buildbot.info,  Node: Forcing Builds,  Prev: Connecting to the buildmaster,  Up: Troubleshooting

2.10.3 Forcing Builds
---------------------

From the buildmaster's main status web page, you can force a build to
be run on your build slave. Figure out which column is for a builder
that runs on your slave, click on that builder's name, and the page
that comes up will have a "Force Build" button. Fill in the form, hit
the button, and a moment later you should see your slave's
`twistd.log' filling with commands being run. Using `pstree' or `top'
should also reveal the cvs/make/gcc/etc processes being run by the
buildslave. Note that the same web page should also show the `admin'
and `host' information files that you configured earlier.


File: buildbot.info,  Node: Concepts,  Next: Configuration,  Prev: Installation,  Up: Top

3 Concepts
**********

This chapter defines some of the basic concepts that the Buildbot
uses. You'll need to understand how the Buildbot sees the world to
configure it properly.

* Menu:

* Version Control Systems::
* Schedulers::
* BuildSet::
* BuildRequest::
* Builder::
* Users::
* Build Properties::


File: buildbot.info,  Node: Version Control Systems,  Next: Schedulers,  Prev: Concepts,  Up: Concepts

3.1 Version Control Systems
===========================

These source trees come from a Version Control System of some kind.
CVS and Subversion are two popular ones, but the Buildbot supports
others. All VC systems have some notion of an upstream `repository'
which acts as a server(1), from which clients can obtain source trees
according to various parameters. The VC repository provides source
trees of various projects, for different branches, and from various
points in time. The first thing we have to do is to specify which
source tree we want to get.

* Menu:

* Generalizing VC Systems::
* Source Tree Specifications::
* How Different VC Systems Specify Sources::
* Attributes of Changes::

   ---------- Footnotes ----------

   (1) except Darcs, but since the Buildbot never modifies its local
source tree we can ignore the fact that Darcs uses a less centralized
model


File: buildbot.info,  Node: Generalizing VC Systems,  Next: Source Tree Specifications,  Prev: Version Control Systems,  Up: Version Control Systems

3.1.1 Generalizing VC Systems
-----------------------------

For the purposes of the Buildbot, we will try to generalize all VC
systems as having repositories that each provide sources for a variety
of projects. Each project is defined as a directory tree with source
files. The individual files may each have revisions, but we ignore
that and treat the project as a whole as having a set of revisions
(CVS is really the only VC system still in widespread use that has
per-file revisions.. everything modern has moved to atomic tree-wide
changesets). Each time someone commits a change to the project, a new
revision becomes available. These revisions can be described by a
tuple with two items: the first is a branch tag, and the second is
some kind of revision stamp or timestamp. Complex projects may have
multiple branch tags, but there is always a default branch. The
timestamp may be an actual timestamp (such as the -D option to CVS),
or it may be a monotonically-increasing transaction number (such as
the change number used by SVN and P4, or the revision number used by
Arch/Baz/Bazaar, or a labeled tag used in CVS)(1). The SHA1 revision
ID used by Monotone, Mercurial, and Git is also a kind of revision
stamp, in that it specifies a unique copy of the source tree, as does
a Darcs "context" file.

   When we aren't intending to make any changes to the sources we
check out (at least not any that need to be committed back upstream),
there are two basic ways to use a VC system:

   * Retrieve a specific set of source revisions: some tag or key is
     used to index this set, which is fixed and cannot be changed by
     subsequent developers committing new changes to the tree.
     Releases are built from tagged revisions like this, so that they
     can be rebuilt again later (probably with controlled
     modifications).

   * Retrieve the latest sources along a specific branch: some tag is
     used to indicate which branch is to be used, but within that
     constraint we want to get the latest revisions.

   Build personnel or CM staff typically use the first approach: the
build that results is (ideally) completely specified by the two
parameters given to the VC system: repository and revision tag. This
gives QA and end-users something concrete to point at when reporting
bugs. Release engineers are also reportedly fond of shipping code that
can be traced back to a concise revision tag of some sort.

   Developers are more likely to use the second approach: each morning
the developer does an update to pull in the changes committed by the
team over the last day. These builds are not easy to fully specify: it
depends upon exactly when you did a checkout, and upon what local
changes the developer has in their tree. Developers do not normally
tag each build they produce, because there is usually significant
overhead involved in creating these tags. Recreating the trees used by
one of these builds can be a challenge. Some VC systems may provide
implicit tags (like a revision number), while others may allow the use
of timestamps to mean "the state of the tree at time X" as opposed to
a tree-state that has been explicitly marked.

   The Buildbot is designed to help developers, so it usually works in
terms of _the latest_ sources as opposed to specific tagged
revisions. However, it would really prefer to build from reproducible
source trees, so implicit revisions are used whenever possible.

   ---------- Footnotes ----------

   (1) many VC systems provide more complexity than this: in
particular the local views that P4 and ClearCase can assemble out of
various source directories are more complex than we're prepared to
take advantage of here


File: buildbot.info,  Node: Source Tree Specifications,  Next: How Different VC Systems Specify Sources,  Prev: Generalizing VC Systems,  Up: Version Control Systems

3.1.2 Source Tree Specifications
--------------------------------

So for the Buildbot's purposes we treat each VC system as a server
which can take a list of specifications as input and produce a source
tree as output. Some of these specifications are static: they are
attributes of the builder and do not change over time. Others are more
variable: each build will have a different value. The repository is
changed over time by a sequence of Changes, each of which represents a
single developer making changes to some set of files. These Changes
are cumulative(1).

   For normal builds, the Buildbot wants to get well-defined source
trees that contain specific Changes, and exclude other Changes that
may have occurred after the desired ones. We assume that the Changes
arrive at the buildbot (through one of the mechanisms described in
*note Change Sources::) in the same order in which they are committed
to the repository. The Buildbot waits for the tree to become "stable"
before initiating a build, for two reasons. The first is that
developers frequently make multiple related commits in quick
succession, even when the VC system provides ways to make atomic
transactions involving multiple files at the same time. Running a
build in the middle of these sets of changes would use an inconsistent
set of source files, and is likely to fail (and is certain to be less
useful than a build which uses the full set of changes). The
tree-stable-timer is intended to avoid these useless builds that
include some of the developer's changes but not all. The second reason
is that some VC systems (i.e. CVS) do not provide repository-wide
transaction numbers, so that timestamps are the only way to refer to
a specific repository state. These timestamps may be somewhat
ambiguous, due to processing and notification delays. By waiting until
the tree has been stable for, say, 10 minutes, we can choose a
timestamp from the middle of that period to use for our source
checkout, and then be reasonably sure that any clock-skew errors will
not cause the build to be performed on an inconsistent set of source
files.

   The Schedulers always use the tree-stable-timer, with a timeout
that is configured to reflect a reasonable tradeoff between build
latency and change frequency. When the VC system provides coherent
repository-wide revision markers (such as Subversion's revision
numbers, or in fact anything other than CVS's timestamps), the
resulting Build is simply performed against a source tree defined by
that revision marker. When the VC system does not provide this, a
timestamp from the middle of the tree-stable period is used to
generate the source tree(2).

   ---------- Footnotes ----------

   (1) Monotone's _multiple heads_ feature violates this assumption
of cumulative Changes, but in most situations the changes don't occur
frequently enough for this to be a significant problem

   (2) this `checkoutDelay' defaults to half the tree-stable timer,
but it can be overridden with an argument to the Source Step


File: buildbot.info,  Node: How Different VC Systems Specify Sources,  Next: Attributes of Changes,  Prev: Source Tree Specifications,  Up: Version Control Systems

3.1.3 How Different VC Systems Specify Sources
----------------------------------------------

For CVS, the static specifications are `repository' and `module'. In
addition to those, each build uses a timestamp (or omits the
timestamp to mean `the latest') and `branch tag' (which defaults to
HEAD). These parameters collectively specify a set of sources from
which a build may be performed.

   Subversion (http://subversion.tigris.org) combines the repository,
module, and branch into a single `Subversion URL' parameter. Within
that scope, source checkouts can be specified by a numeric `revision
number' (a repository-wide monotonically-increasing marker, such that
each transaction that changes the repository is indexed by a
different revision number), or a revision timestamp. When branches
are used, the repository and module form a static `baseURL', while
each build has a `revision number' and a `branch' (which defaults to a
statically-specified `defaultBranch'). The `baseURL' and `branch' are
simply concatenated together to derive the `svnurl' to use for the
checkout.

   Perforce (http://www.perforce.com/) is similar. The server is
specified through a `P4PORT' parameter. Module and branch are
specified in a single depot path, and revisions are depot-wide. When
branches are used, the `p4base' and `defaultBranch' are concatenated
together to produce the depot path.

   Arch (http://wiki.gnuarch.org/) and Bazaar
(http://bazaar.canonical.com/) specify a repository by URL, as well
as a `version' which is kind of like a branch name.  Arch uses the
word `archive' to represent the repository. Arch lets you push
changes from one archive to another, removing the strict
centralization required by CVS and SVN. It retains the distinction
between repository and working directory that most other VC systems
use. For complex multi-module directory structures, Arch has a
built-in `build config' layer with which the checkout process has two
steps. First, an initial bootstrap checkout is performed to retrieve
a set of build-config files. Second, one of these files is used to
figure out which archives/modules should be used to populate
subdirectories of the initial checkout.

   Builders which use Arch and Bazaar therefore have a static archive
`url', and a default "branch" (which is a string that specifies a
complete category-branch-version triple). Each build can have its own
branch (the category-branch-version string) to override the default,
as well as a revision number (which is turned into a -patch-NN suffix
when performing the checkout).

   Bzr (http://bazaar-vcs.org) (which is a descendant of Arch/Bazaar,
and is frequently referred to as "Bazaar") has the same sort of
repository-vs-workspace model as Arch, but the repository data can
either be stored inside the working directory or kept elsewhere
(either on the same machine or on an entirely different machine). For
the purposes of Buildbot (which never commits changes), the repository
is specified with a URL and a revision number.

   The most common way to obtain read-only access to a bzr tree is via
HTTP, simply by making the repository visible through a web server
like Apache. Bzr can also use FTP and SFTP servers, if the buildslave
process has sufficient privileges to access them. Higher performance
can be obtained by running a special Bazaar-specific server. None of
these matter to the buildbot: the repository URL just has to match the
kind of server being used. The `repoURL' argument provides the
location of the repository.

   Branches are expressed as subdirectories of the main central
repository, which means that if branches are being used, the BZR step
is given a `baseURL' and `defaultBranch' instead of getting the
`repoURL' argument.

   Darcs (http://darcs.net/) doesn't really have the notion of a
single master repository. Nor does it really have branches. In Darcs,
each working directory is also a repository, and there are operations
to push and pull patches from one of these `repositories' to another.
For the Buildbot's purposes, all you need to do is specify the URL of
a repository that you want to build from. The build slave will then
pull the latest patches from that repository and build them. Multiple
branches are implemented by using multiple repositories (possibly
living on the same server).

   Builders which use Darcs therefore have a static `repourl' which
specifies the location of the repository. If branches are being used,
the source Step is instead configured with a `baseURL' and a
`defaultBranch', and the two strings are simply concatenated together
to obtain the repository's URL. Each build then has a specific branch
which replaces `defaultBranch', or just uses the default one. Instead
of a revision number, each build can have a "context", which is a
string that records all the patches that are present in a given tree
(this is the output of `darcs changes --context', and is considerably
less concise than, e.g. Subversion's revision number, but the
patch-reordering flexibility of Darcs makes it impossible to provide
a shorter useful specification).

   Mercurial (http://selenic.com/mercurial) is like Darcs, in that
each branch is stored in a separate repository. The `repourl',
`baseURL', and `defaultBranch' arguments are all handled the same way
as with Darcs. The "revision", however, is the hash identifier
returned by `hg identify'.

   Git (http://git.or.cz/) also follows a decentralized model, and
each repository can have several branches and tags. The source Step is
configured with a static `repourl' which specifies the location of
the repository. In addition, an optional `branch' parameter can be
specified to check out code from a specific branch instead of the
default "master" branch. The "revision" is specified as a SHA1 hash
as returned by e.g. `git rev-parse'. No attempt is made to ensure
that the specified revision is actually a subset of the specified
branch.


File: buildbot.info,  Node: Attributes of Changes,  Prev: How Different VC Systems Specify Sources,  Up: Version Control Systems

3.1.4 Attributes of Changes
---------------------------

Who
===

Each Change has a `who' attribute, which specifies which developer is
responsible for the change. This is a string which comes from a
namespace controlled by the VC repository. Frequently this means it
is a username on the host which runs the repository, but not all VC
systems require this (Arch, for example, uses a fully-qualified `Arch
ID', which looks like an email address, as does Darcs).  Each
StatusNotifier will map the `who' attribute into something
appropriate for their particular means of communication: an email
address, an IRC handle, etc.

Files
=====

It also has a list of `files', which are just the tree-relative
filenames of any files that were added, deleted, or modified for this
Change. These filenames are used by the `fileIsImportant' function
(in the Scheduler) to decide whether it is worth triggering a new
build or not, e.g. the function could use the following function to
only run a build if a C file were checked in:

     def has_C_files(change):
         for name in change.files:
             if name.endswith(".c"):
                 return True
         return False

   Certain BuildSteps can also use the list of changed files to run a
more targeted series of tests, e.g. the `python_twisted.Trial' step
can run just the unit tests that provide coverage for the modified
.py files instead of running the full test suite.

Comments
========

The Change also has a `comments' attribute, which is a string
containing any checkin comments.

Revision
========

Each Change can have a `revision' attribute, which describes how to
get a tree with a specific state: a tree which includes this Change
(and all that came before it) but none that come after it. If this
information is unavailable, the `.revision' attribute will be `None'.
These revisions are provided by the ChangeSource, and consumed by the
`computeSourceRevision' method in the appropriate `step.Source' class.

`CVS'
     `revision' is an int, seconds since the epoch

`SVN'
     `revision' is an int, the changeset number (r%d)

`Darcs'
     `revision' is a large string, the output of `darcs changes
     --context'

`Mercurial'
     `revision' is a short string (a hash ID), the output of `hg
     identify'

`Arch/Bazaar'
     `revision' is the full revision ID (ending in -patch-%d)

`P4'
     `revision' is an int, the transaction number

`Git'
     `revision' is a short string (a SHA1 hash), the output of e.g.
     `git rev-parse'

Branches
========

The Change might also have a `branch' attribute. This indicates that
all of the Change's files are in the same named branch. The
Schedulers get to decide whether the branch should be built or not.

   For VC systems like CVS, Arch, Monotone, and Git, the `branch'
name is unrelated to the filename. (that is, the branch name and the
filename inhabit unrelated namespaces). For SVN, branches are
expressed as subdirectories of the repository, so the file's "svnurl"
is a combination of some base URL, the branch name, and the filename
within the branch. (In a sense, the branch name and the filename
inhabit the same namespace). Darcs branches are subdirectories of a
base URL just like SVN. Mercurial branches are the same as Darcs.

`CVS'
     branch='warner-newfeature', files=['src/foo.c']

`SVN'
     branch='branches/warner-newfeature', files=['src/foo.c']

`Darcs'
     branch='warner-newfeature', files=['src/foo.c']

`Mercurial'
     branch='warner-newfeature', files=['src/foo.c']

`Arch/Bazaar'
     branch='buildbot-usebranches-0', files=['buildbot/master.py']

`Git'
     branch='warner-newfeature', files=['src/foo.c']

Links
=====

Finally, the Change might have a `links' list, which is intended to
provide a list of URLs to a _viewcvs_-style web page that provides
more detail for this Change, perhaps including the full file diffs.


File: buildbot.info,  Node: Schedulers,  Next: BuildSet,  Prev: Version Control Systems,  Up: Concepts

3.2 Schedulers
==============

Each Buildmaster has a set of `Scheduler' objects, each of which gets
a copy of every incoming Change. The Schedulers are responsible for
deciding when Builds should be run. Some Buildbot installations might
have a single Scheduler, while others may have several, each for a
different purpose.

   For example, a "quick" scheduler might exist to give immediate
feedback to developers, hoping to catch obvious problems in the code
that can be detected quickly. These typically do not run the full test
suite, nor do they run on a wide variety of platforms. They also
usually do a VC update rather than performing a brand-new checkout
each time. You could have a "quick" scheduler which used a 30 second
timeout, and feeds a single "quick" Builder that uses a VC
`mode='update'' setting.

   A separate "full" scheduler would run more comprehensive tests a
little while later, to catch more subtle problems. This scheduler
would have a longer tree-stable-timer, maybe 30 minutes, and would
feed multiple Builders (with a `mode=' of `'copy'', `'clobber'', or
`'export'').

   The `tree-stable-timer' and `fileIsImportant' decisions are made
by the Scheduler. Dependencies are also implemented here.  Periodic
builds (those which are run every N seconds rather than after new
Changes arrive) are triggered by a special `Periodic' Scheduler
subclass. The default Scheduler class can also be told to watch for
specific branches, ignoring Changes on other branches. This may be
useful if you have a trunk and a few release branches which should be
tracked, but when you don't want to have the Buildbot pay attention
to several dozen private user branches.

   When the setup has multiple sources of Changes the `category' can
be used for `Scheduler' objects to filter out a subset of the
Changes.  Note that not all change sources can attach a category.

   Some Schedulers may trigger builds for other reasons, other than
recent Changes. For example, a Scheduler subclass could connect to a
remote buildmaster and watch for builds of a library to succeed before
triggering a local build that uses that library.

   Each Scheduler creates and submits `BuildSet' objects to the
`BuildMaster', which is then responsible for making sure the
individual `BuildRequests' are delivered to the target `Builders'.

   `Scheduler' instances are activated by placing them in the
`c['schedulers']' list in the buildmaster config file. Each Scheduler
has a unique name.


File: buildbot.info,  Node: BuildSet,  Next: BuildRequest,  Prev: Schedulers,  Up: Concepts

3.3 BuildSet
============

A `BuildSet' is the name given to a set of Builds that all
compile/test the same version of the tree on multiple Builders. In
general, all these component Builds will perform the same sequence of
Steps, using the same source code, but on different platforms or
against a different set of libraries.

   The `BuildSet' is tracked as a single unit, which fails if any of
the component Builds have failed, and therefore can succeed only if
_all_ of the component Builds have succeeded. There are two kinds of
status notification messages that can be emitted for a BuildSet: the
`firstFailure' type (which fires as soon as we know the BuildSet will
fail), and the `Finished' type (which fires once the BuildSet has
completely finished, regardless of whether the overall set passed or
failed).

   A `BuildSet' is created with a _source stamp_ tuple of (branch,
revision, changes, patch), some of which may be None, and a list of
Builders on which it is to be run. They are then given to the
BuildMaster, which is responsible for creating a separate
`BuildRequest' for each Builder.

   There are a couple of different likely values for the
`SourceStamp':

`(revision=None, changes=[CHANGES], patch=None)'
     This is a `SourceStamp' used when a series of Changes have
     triggered a build. The VC step will attempt to check out a tree
     that contains CHANGES (and any changes that occurred before
     CHANGES, but not any that occurred after them).

`(revision=None, changes=None, patch=None)'
     This builds the most recent code on the default branch. This is
     the sort of `SourceStamp' that would be used on a Build that was
     triggered by a user request, or a Periodic scheduler. It is also
     possible to configure the VC Source Step to always check out the
     latest sources rather than paying attention to the Changes in the
     SourceStamp, which will result in same behavior as this.

`(branch=BRANCH, revision=None, changes=None, patch=None)'
     This builds the most recent code on the given BRANCH. Again,
     this is generally triggered by a user request or Periodic build.

`(revision=REV, changes=None, patch=(LEVEL, DIFF))'
     This checks out the tree at the given revision REV, then applies
     a patch (using `patch -pLEVEL <DIFF'). The *note try:: feature
     uses this kind of `SourceStamp'. If `patch' is None, the patching
     step is bypassed.


   The buildmaster is responsible for turning the `BuildSet' into a
set of `BuildRequest' objects and queueing them on the appropriate
Builders.


File: buildbot.info,  Node: BuildRequest,  Next: Builder,  Prev: BuildSet,  Up: Concepts

3.4 BuildRequest
================

A `BuildRequest' is a request to build a specific set of sources on a
single specific `Builder'. Each `Builder' runs the `BuildRequest' as
soon as it can (i.e. when an associated buildslave becomes free).
`BuildRequest's are prioritized from oldest to newest, so when a
buildslave becomes free, the `Builder' with the oldest `BuildRequest'
is run.

   The `BuildRequest' contains the `SourceStamp' specification.  The
actual process of running the build (the series of Steps that will be
executed) is implemented by the `Build' object. In this future this
might be changed, to have the `Build' define _what_ gets built, and a
separate `BuildProcess' (provided by the Builder) to define _how_ it
gets built.

   `BuildRequest' is created with optional `Properties'.  One of
these, `owner', is collected by the resultant `Build' and added to
the set of _interested users_ to which status notifications will be
sent, depending on the configuration for each status object.

   The `BuildRequest' may be mergeable with other compatible
`BuildRequest's. Builds that are triggered by incoming Changes will
generally be mergeable. Builds that are triggered by user requests
are generally not, unless they are multiple requests to build the
_latest sources_ of the same branch.


File: buildbot.info,  Node: Builder,  Next: Users,  Prev: BuildRequest,  Up: Concepts

3.5 Builder
===========

The `Builder' is a long-lived object which controls all Builds of a
given type. Each one is created when the config file is first parsed,
and lives forever (or rather until it is removed from the config
file). It mediates the connections to the buildslaves that do all the
work, and is responsible for creating the `Build' objects that decide
_how_ a build is performed (i.e., which steps are executed in what
order).

   Each `Builder' gets a unique name, and the path name of a
directory where it gets to do all its work (there is a
buildmaster-side directory for keeping status information, as well as
a buildslave-side directory where the actual checkout/compile/test
commands are executed). It also gets a `BuildFactory', which is
responsible for creating new `Build' instances: because the `Build'
instance is what actually performs each build, choosing the
`BuildFactory' is the way to specify what happens each time a build
is done.

   Each `Builder' is associated with one of more `BuildSlaves'.  A
`Builder' which is used to perform OS-X builds (as opposed to Linux
or Solaris builds) should naturally be associated with an OS-X-based
buildslave.

   A `Builder' may be given a set of environment variables to be used
in its *note ShellCommand::s. These variables will override anything
in the buildslave's environment. Variables passed directly to a
ShellCommand will override variables of the same name passed to the
Builder.

   For example, if you a pool of identical slaves it is often easier
to manage variables like PATH from Buildbot rather than manually
editing it inside of the slaves' environment.

     f = factory.BuildFactory
     f.addStep(ShellCommand(
                   command=['bash', './configure']))
     f.addStep(Compile())

     c['builders'] = [
       {'name': 'test', 'slavenames': ['slave1', 'slave2', 'slave3', 'slave4',
                                        'slave5', 'slave6'],
         'builddir': 'test', 'factory': f',
         'env': {'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'}}


File: buildbot.info,  Node: Users,  Next: Build Properties,  Prev: Builder,  Up: Concepts

3.6 Users
=========

Buildbot has a somewhat limited awareness of _users_. It assumes the
world consists of a set of developers, each of whom can be described
by a couple of simple attributes. These developers make changes to
the source code, causing builds which may succeed or fail.

   Each developer is primarily known through the source control
system. Each Change object that arrives is tagged with a `who' field
that typically gives the account name (on the repository machine) of
the user responsible for that change. This string is the primary key
by which the User is known, and is displayed on the HTML status pages
and in each Build's "blamelist".

   To do more with the User than just refer to them, this username
needs to be mapped into an address of some sort. The responsibility
for this mapping is left up to the status module which needs the
address. The core code knows nothing about email addresses or IRC
nicknames, just user names.

* Menu:

* Doing Things With Users::
* Email Addresses::
* IRC Nicknames::
* Live Status Clients::


File: buildbot.info,  Node: Doing Things With Users,  Next: Email Addresses,  Prev: Users,  Up: Users

3.6.1 Doing Things With Users
-----------------------------

Each Change has a single User who is responsible for that Change. Most
Builds have a set of Changes: the Build represents the first time
these Changes have been built and tested by the Buildbot. The build
has a "blamelist" that consists of a simple union of the Users
responsible for all the Build's Changes.

   The Build provides (through the IBuildStatus interface) a list of
Users who are "involved" in the build. For now this is equal to the
blamelist, but in the future it will be expanded to include a "build
sheriff" (a person who is "on duty" at that time and responsible for
watching over all builds that occur during their shift), as well as
per-module owners who simply want to keep watch over their domain
(chosen by subdirectory or a regexp matched against the filenames
pulled out of the Changes). The Involved Users are those who probably
have an interest in the results of any given build.

   In the future, Buildbot will acquire the concept of "Problems",
which last longer than builds and have beginnings and ends. For
example, a test case which passed in one build and then failed in the
next is a Problem. The Problem lasts until the test case starts
passing again, at which point the Problem is said to be "resolved".

   If there appears to be a code change that went into the tree at the
same time as the test started failing, that Change is marked as being
resposible for the Problem, and the user who made the change is added
to the Problem's "Guilty" list. In addition to this user, there may
be others who share responsibility for the Problem (module owners,
sponsoring developers). In addition to the Responsible Users, there
may be a set of Interested Users, who take an interest in the fate of
the Problem.

   Problems therefore have sets of Users who may want to be kept
aware of the condition of the problem as it changes over time. If
configured, the Buildbot can pester everyone on the Responsible list
with increasing harshness until the problem is resolved, with the
most harshness reserved for the Guilty parties themselves. The
Interested Users may merely be told when the problem starts and
stops, as they are not actually responsible for fixing anything.


File: buildbot.info,  Node: Email Addresses,  Next: IRC Nicknames,  Prev: Doing Things With Users,  Up: Users

3.6.2 Email Addresses
---------------------

The `buildbot.status.mail.MailNotifier' class (*note MailNotifier::)
provides a status target which can send email about the results of
each build. It accepts a static list of email addresses to which each
message should be delivered, but it can also be configured to send
mail to the Build's Interested Users. To do this, it needs a way to
convert User names into email addresses.

   For many VC systems, the User Name is actually an account name on
the system which hosts the repository. As such, turning the name into
an email address is a simple matter of appending
"@repositoryhost.com". Some projects use other kinds of mappings (for
example the preferred email address may be at "project.org" despite
the repository host being named "cvs.project.org"), and some VC
systems have full separation between the concept of a user and that
of an account on the repository host (like Perforce). Some systems
(like Arch) put a full contact email address in every change.

   To convert these names to addresses, the MailNotifier uses an
EmailLookup object. This provides a .getAddress method which accepts
a name and (eventually) returns an address. The default `MailNotifier'
module provides an EmailLookup which simply appends a static string,
configurable when the notifier is created. To create more complex
behaviors (perhaps using an LDAP lookup, or using "finger" on a
central host to determine a preferred address for the developer),
provide a different object as the `lookup' argument.

   In the future, when the Problem mechanism has been set up, the
Buildbot will need to send mail to arbitrary Users. It will do this
by locating a MailNotifier-like object among all the buildmaster's
status targets, and asking it to send messages to various Users. This
means the User-to-address mapping only has to be set up once, in your
MailNotifier, and every email message the buildbot emits will take
advantage of it.


File: buildbot.info,  Node: IRC Nicknames,  Next: Live Status Clients,  Prev: Email Addresses,  Up: Users

3.6.3 IRC Nicknames
-------------------

Like MailNotifier, the `buildbot.status.words.IRC' class provides a
status target which can announce the results of each build. It also
provides an interactive interface by responding to online queries
posted in the channel or sent as private messages.

   In the future, the buildbot can be configured map User names to IRC
nicknames, to watch for the recent presence of these nicknames, and to
deliver build status messages to the interested parties. Like
`MailNotifier' does for email addresses, the `IRC' object will have
an `IRCLookup' which is responsible for nicknames. The mapping can be
set up statically, or it can be updated by online users themselves
(by claiming a username with some kind of "buildbot: i am user
warner" commands).

   Once the mapping is established, the rest of the buildbot can ask
the `IRC' object to send messages to various users. It can report on
the likelihood that the user saw the given message (based upon how
long the user has been inactive on the channel), which might prompt
the Problem Hassler logic to send them an email message instead.


File: buildbot.info,  Node: Live Status Clients,  Prev: IRC Nicknames,  Up: Users

3.6.4 Live Status Clients
-------------------------

The Buildbot also offers a PB-based status client interface which can
display real-time build status in a GUI panel on the developer's
desktop.  This interface is normally anonymous, but it could be
configured to let the buildmaster know _which_ developer is using the
status client. The status client could then be used as a
message-delivery service, providing an alternative way to deliver
low-latency high-interruption messages to the developer (like "hey,
you broke the build").


File: buildbot.info,  Node: Build Properties,  Prev: Users,  Up: Concepts

3.7 Build Properties
====================

Each build has a set of "Build Properties", which can be used by its
BuildStep to modify their actions.  These properties, in the form of
key-value pairs, provide a general framework for dynamically altering
the behavior of a build based on its circumstances.

   Properties come from a number of places:
   * global configuration - These properties apply to all builds.

   * schedulers - A scheduler can specify properties available to all
     the builds it starts.

   * buildslaves - A buildslave can pass properties on to the builds
     it performs.

   * builds - A build automatically sets a number of properties on
     itself.

   * steps - Steps of a build can set properties that are available
     to subsequent steps.  In particular, source steps set a number
     of properties.

   Properties are very flexible, and can be used to implement all
manner of functionality.  Here are some examples:

   Most Source steps record the revision that they checked out in the
`got_revision' property.  A later step could use this property to
specify the name of a fully-built tarball, dropped in an
easily-acessible directory for later testing.

   Some projects want to perform nightly builds as well as in response
to committed changes.  Such a project would run two schedulers, both
pointing to the same set of builders, but could provide an
`is_nightly' property so that steps can distinguish the nightly
builds, perhaps to run more resource-intensive tests.

   Some projects have different build processes on different systems.
Rather than create a build factory for each slave, the steps can use
buildslave properties to identify the unique aspects of each slave
and adapt the build process dynamically.


File: buildbot.info,  Node: Configuration,  Next: Getting Source Code Changes,  Prev: Concepts,  Up: Top

4 Configuration
***************

The buildbot's behavior is defined by the "config file", which
normally lives in the `master.cfg' file in the buildmaster's base
directory (but this can be changed with an option to the `buildbot
create-master' command). This file completely specifies which
Builders are to be run, which slaves they should use, how Changes
should be tracked, and where the status information is to be sent.
The buildmaster's `buildbot.tac' file names the base directory;
everything else comes from the config file.

   A sample config file was installed for you when you created the
buildmaster, but you will need to edit it before your buildbot will do
anything useful.

   This chapter gives an overview of the format of this file and the
various sections in it. You will need to read the later chapters to
understand how to fill in each section properly.

* Menu:

* Config File Format::
* Loading the Config File::
* Testing the Config File::
* Defining the Project::
* Change Sources and Schedulers::
* Merging BuildRequests::
* Setting the slaveport::
* Buildslave Specifiers::
* On-Demand ("Latent") Buildslaves::
* Defining Global Properties::
* Defining Builders::
* Defining Status Targets::
* Debug options::


File: buildbot.info,  Node: Config File Format,  Next: Loading the Config File,  Prev: Configuration,  Up: Configuration

4.1 Config File Format
======================

The config file is, fundamentally, just a piece of Python code which
defines a dictionary named `BuildmasterConfig', with a number of keys
that are treated specially. You don't need to know Python to do basic
configuration, though, you can just copy the syntax of the sample
file. If you _are_ comfortable writing Python code, however, you can
use all the power of a full programming language to achieve more
complicated configurations.

   The `BuildmasterConfig' name is the only one which matters: all
other names defined during the execution of the file are discarded.
When parsing the config file, the Buildmaster generally compares the
old configuration with the new one and performs the minimum set of
actions necessary to bring the buildbot up to date: Builders which are
not changed are left untouched, and Builders which are modified get to
keep their old event history.

   Basic Python syntax: comments start with a hash character ("#"),
tuples are defined with `(parenthesis, pairs)', arrays are defined
with `[square, brackets]', tuples and arrays are mostly
interchangeable. Dictionaries (data structures which map "keys" to
"values") are defined with curly braces: `{'key1': 'value1', 'key2':
'value2'} '. Function calls (and object instantiation) can use named
parameters, like `w = html.Waterfall(http_port=8010)'.

   The config file starts with a series of `import' statements, which
make various kinds of Steps and Status targets available for later
use. The main `BuildmasterConfig' dictionary is created, then it is
populated with a variety of keys. These keys are broken roughly into
the following sections, each of which is documented in the rest of
this chapter:

   * Project Definitions

   * Change Sources / Schedulers

   * Slaveport

   * Buildslave Configuration

   * Builders / Interlocks

   * Status Targets

   * Debug options

   The config file can use a few names which are placed into its
namespace:

`basedir'
     the base directory for the buildmaster. This string has not been
     expanded, so it may start with a tilde. It needs to be expanded
     before use. The config file is located in
     `os.path.expanduser(os.path.join(basedir, 'master.cfg'))'



File: buildbot.info,  Node: Loading the Config File,  Next: Testing the Config File,  Prev: Config File Format,  Up: Configuration

4.2 Loading the Config File
===========================

The config file is only read at specific points in time. It is first
read when the buildmaster is launched. Once it is running, there are
various ways to ask it to reload the config file. If you are on the
system hosting the buildmaster, you can send a `SIGHUP' signal to it:
the `buildbot' tool has a shortcut for this:

     buildbot reconfig BASEDIR

   This command will show you all of the lines from `twistd.log' that
relate to the reconfiguration. If there are any problems during the
config-file reload, they will be displayed in these lines.

   The debug tool (`buildbot debugclient --master HOST:PORT') has a
"Reload .cfg" button which will also trigger a reload. In the future,
there will be other ways to accomplish this step (probably a
password-protected button on the web page, as well as a privileged IRC
command).

   When reloading the config file, the buildmaster will endeavor to
change as little as possible about the running system. For example,
although old status targets may be shut down and new ones started up,
any status targets that were not changed since the last time the
config file was read will be left running and untouched. Likewise any
Builders which have not been changed will be left running. If a
Builder is modified (say, the build process is changed) while a Build
is currently running, that Build will keep running with the old
process until it completes. Any previously queued Builds (or Builds
which get queued after the reconfig) will use the new process.


File: buildbot.info,  Node: Testing the Config File,  Next: Defining the Project,  Prev: Loading the Config File,  Up: Configuration

4.3 Testing the Config File
===========================

To verify that the config file is well-formed and contains no
deprecated or invalid elements, use the "checkconfig" command:

     % buildbot checkconfig master.cfg
     Config file is good!

   If the config file has deprecated features (perhaps because you've
upgraded the buildmaster and need to update the config file to match),
they will be announced by checkconfig. In this case, the config file
will work, but you should really remove the deprecated items and use
the recommended replacements instead:

     % buildbot checkconfig master.cfg
     /usr/lib/python2.4/site-packages/buildbot/master.py:559: DeprecationWarning: c['sources'] is
     deprecated as of 0.7.6 and will be removed by 0.8.0 . Please use c['change_source'] instead.
       warnings.warn(m, DeprecationWarning)
     Config file is good!

   If the config file is simply broken, that will be caught too:

     % buildbot checkconfig master.cfg
     Traceback (most recent call last):
       File "/usr/lib/python2.4/site-packages/buildbot/scripts/runner.py", line 834, in doCheckConfig
         ConfigLoader(configFile)
       File "/usr/lib/python2.4/site-packages/buildbot/scripts/checkconfig.py", line 31, in __init__
         self.loadConfig(configFile)
       File "/usr/lib/python2.4/site-packages/buildbot/master.py", line 480, in loadConfig
         exec f in localDict
       File "/home/warner/BuildBot/master/foolscap/master.cfg", line 90, in ?
         c[bogus] = "stuff"
     NameError: name 'bogus' is not defined


File: buildbot.info,  Node: Defining the Project,  Next: Change Sources and Schedulers,  Prev: Testing the Config File,  Up: Configuration

4.4 Defining the Project
========================

There are a couple of basic settings that you use to tell the buildbot
what project it is working on. This information is used by status
reporters to let users find out more about the codebase being
exercised by this particular Buildbot installation.

     c['projectName'] = "Buildbot"
     c['projectURL'] = "http://buildbot.sourceforge.net/"
     c['buildbotURL'] = "http://localhost:8010/"

   `projectName' is a short string will be used to describe the
project that this buildbot is working on. For example, it is used as
the title of the waterfall HTML page.

   `projectURL' is a string that gives a URL for the project as a
whole. HTML status displays will show `projectName' as a link to
`projectURL', to provide a link from buildbot HTML pages to your
project's home page.

   The `buildbotURL' string should point to the location where the
buildbot's internal web server (usually the `html.Waterfall' page) is
visible. This typically uses the port number set when you create the
`Waterfall' object: the buildbot needs your help to figure out a
suitable externally-visible host name.

   When status notices are sent to users (either by email or over
IRC), `buildbotURL' will be used to create a URL to the specific build
or problem that they are being notified about. It will also be made
available to queriers (over IRC) who want to find out where to get
more information about this buildbot.

   The `logCompressionLimit' enables bz2-compression of build logs on
disk for logs that are bigger than the given size, or disables that
completely if given `False'. The default value is 4k, which should be
a reasonable default on most file systems. This setting has no impact
on status plugins, and merely affects the required disk space on the
master for build logs.


File: buildbot.info,  Node: Change Sources and Schedulers,  Next: Merging BuildRequests,  Prev: Defining the Project,  Up: Configuration

4.5 Change Sources and Schedulers
=================================

The `c['change_source']' key is the ChangeSource instance(1) that
defines how the buildmaster learns about source code changes. More
information about what goes here is available in *Note Getting Source
Code Changes::.

     from buildbot.changes.pb import PBChangeSource
     c['change_source'] = PBChangeSource()
   
   (note: in buildbot-0.7.5 and earlier, this key was named
`c['sources']', and required a list. `c['sources']' is deprecated as
of buildbot-0.7.6 and is scheduled to be removed in a future release).

   `c['schedulers']' is a list of Scheduler instances, each of which
causes builds to be started on a particular set of Builders. The two
basic Scheduler classes you are likely to start with are `Scheduler'
and `Periodic', but you can write a customized subclass to implement
more complicated build scheduling.

   Scheduler arguments should always be specified by name (as keyword
arguments), to allow for future expansion:

     sched = Scheduler(name="quick", builderNames=['lin', 'win'])

   All schedulers have several arguments in common:

`name'
     Each Scheduler must have a unique name. This is used in status
     displays, and is also available in the build property
     `scheduler'.

`builderNames'
     This is the set of builders which this scheduler should trigger,
     specified as a list of names (strings).

`properties'
     This is a dictionary specifying properties that will be
     transmitted to all builds started by this scheduler.


   Here is a brief catalog of the available Scheduler types. All these
Schedulers are classes in `buildbot.scheduler', and the docstrings
there are the best source of documentation on the arguments taken by
each one.

* Menu:

* Scheduler Scheduler::
* AnyBranchScheduler::
* Dependent Scheduler::
* Periodic Scheduler::
* Nightly Scheduler::
* Try Schedulers::
* Triggerable Scheduler::

   ---------- Footnotes ----------

   (1) To be precise, it is an object or a list of objects which all
implement the `buildbot.interfaces.IChangeSource' Interface. It is
unusual to have multiple ChangeSources, so this key accepts either a
single ChangeSource or a sequence of them.


File: buildbot.info,  Node: Scheduler Scheduler,  Next: AnyBranchScheduler,  Prev: Change Sources and Schedulers,  Up: Change Sources and Schedulers

4.5.1 Scheduler Scheduler
-------------------------

This is the original and still most popular Scheduler class. It
follows exactly one branch, and starts a configurable
tree-stable-timer after each change on that branch. When the timer
expires, it starts a build on some set of Builders. The Scheduler
accepts a `fileIsImportant' function which can be used to ignore some
Changes if they do not affect any "important" files.

   The arguments to this scheduler are:

`name'

`builderNames'

`properties'

`branch'
     This Scheduler will pay attention to a single branch, ignoring
     Changes that occur on other branches. Setting `branch' equal to
     the special value of `None' means it should only pay attention to
     the default branch. Note that `None' is a keyword, not a string,
     so you want to use `None' and not `"None"'.

`treeStableTimer'
     The Scheduler will wait for this many seconds before starting the
     build. If new changes are made during this interval, the timer
     will be restarted, so really the build will be started after a
     change and then after this many seconds of inactivity.

`fileIsImportant'
     A callable which takes one argument, a Change instance, and
     returns `True' if the change is worth building, and `False' if
     it is not.  Unimportant Changes are accumulated until the build
     is triggered by an important change.  The default value of None
     means that all Changes are important.

`categories'
     A list of categories of changes that this scheduler will respond
     to.  If this is specified, then any non-matching changes are
     ignored.


   Example:

     from buildbot import scheduler
     quick = scheduler.Scheduler(name="quick",
                         branch=None,
                         treeStableTimer=60,
                         builderNames=["quick-linux", "quick-netbsd"])
     full = scheduler.Scheduler(name="full",
                         branch=None,
                         treeStableTimer=5*60,
                         builderNames=["full-linux", "full-netbsd", "full-OSX"])
     c['schedulers'] = [quick, full]

   In this example, the two "quick" builders are triggered 60 seconds
after the tree has been changed. The "full" builds do not run quite
so quickly (they wait 5 minutes), so hopefully if the quick builds
fail due to a missing file or really simple typo, the developer can
discover and fix the problem before the full builds are started. Both
Schedulers only pay attention to the default branch: any changes on
other branches are ignored by these Schedulers. Each Scheduler
triggers a different set of Builders, referenced by name.


File: buildbot.info,  Node: AnyBranchScheduler,  Next: Dependent Scheduler,  Prev: Scheduler Scheduler,  Up: Change Sources and Schedulers

4.5.2 AnyBranchScheduler
------------------------

This scheduler uses a tree-stable-timer like the default one, but
follows multiple branches at once. Each branch gets a separate timer.

   The arguments to this scheduler are:

`name'

`builderNames'

`properties'

`branches'
     This Scheduler will pay attention to any number of branches,
     ignoring Changes that occur on other branches. Branches are
     specified just as for the `Scheduler' class.

`treeStableTimer'
     The Scheduler will wait for this many seconds before starting the
     build. If new changes are made during this interval, the timer
     will be restarted, so really the build will be started after a
     change and then after this many seconds of inactivity.

`fileIsImportant'
     A callable which takes one argument, a Change instance, and
     returns `True' if the change is worth building, and `False' if
     it is not.  Unimportant Changes are accumulated until the build
     is triggered by an important change.  The default value of None
     means that all Changes are important.


File: buildbot.info,  Node: Dependent Scheduler,  Next: Periodic Scheduler,  Prev: AnyBranchScheduler,  Up: Change Sources and Schedulers

4.5.3 Dependent Scheduler
-------------------------

It is common to wind up with one kind of build which should only be
performed if the same source code was successfully handled by some
other kind of build first. An example might be a packaging step: you
might only want to produce .deb or RPM packages from a tree that was
known to compile successfully and pass all unit tests. You could put
the packaging step in the same Build as the compile and testing steps,
but there might be other reasons to not do this (in particular you
might have several Builders worth of compiles/tests, but only wish to
do the packaging once). Another example is if you want to skip the
"full" builds after a failing "quick" build of the same source code.
Or, if one Build creates a product (like a compiled library) that is
used by some other Builder, you'd want to make sure the consuming
Build is run _after_ the producing one.

   You can use "Dependencies" to express this relationship to the
Buildbot. There is a special kind of Scheduler named
`scheduler.Dependent' that will watch an "upstream" Scheduler for
builds to complete successfully (on all of its Builders). Each time
that happens, the same source code (i.e. the same `SourceStamp') will
be used to start a new set of builds, on a different set of Builders.
This "downstream" scheduler doesn't pay attention to Changes at all.
It only pays attention to the upstream scheduler.

   If the build fails on any of the Builders in the upstream set, the
downstream builds will not fire.  Note that, for SourceStamps
generated by a ChangeSource, the `revision' is None, meaning HEAD.
If any changes are committed between the time the upstream scheduler
begins its build and the time the dependent scheduler begins its
build, then those changes will be included in the downstream build.
See the *note Triggerable Scheduler:: for a more flexible dependency
mechanism that can avoid this problem.

   The arguments to this scheduler are:

`name'

`builderNames'

`properties'

`upstream'
     The upstream scheduler to watch.  Note that this is an
     "instance", not the name of the scheduler.

   Example:

     from buildbot import scheduler
     tests = scheduler.Scheduler("just-tests", None, 5*60,
                                 ["full-linux", "full-netbsd", "full-OSX"])
     package = scheduler.Dependent("build-package",
                                   tests, # upstream scheduler -- no quotes!
                                   ["make-tarball", "make-deb", "make-rpm"])
     c['schedulers'] = [tests, package]


File: buildbot.info,  Node: Periodic Scheduler,  Next: Nightly Scheduler,  Prev: Dependent Scheduler,  Up: Change Sources and Schedulers

4.5.4 Periodic Scheduler
------------------------

This simple scheduler just triggers a build every N seconds.

   The arguments to this scheduler are:

`name'

`builderNames'

`properties'

`periodicBuildTimer'
     The time, in seconds, after which to start a build.

   Example:

     from buildbot import scheduler
     nightly = scheduler.Periodic(name="nightly",
                     builderNames=["full-solaris"],
                     periodicBuildTimer=24*60*60)
     c['schedulers'] = [nightly]

   The Scheduler in this example just runs the full solaris build once
per day. Note that this Scheduler only lets you control the time
between builds, not the absolute time-of-day of each Build, so this
could easily wind up a "daily" or "every afternoon" scheduler
depending upon when it was first activated.


File: buildbot.info,  Node: Nightly Scheduler,  Next: Try Schedulers,  Prev: Periodic Scheduler,  Up: Change Sources and Schedulers

4.5.5 Nightly Scheduler
-----------------------

This is highly configurable periodic build scheduler, which triggers
a build at particular times of day, week, month, or year. The
configuration syntax is very similar to the well-known `crontab'
format, in which you provide values for minute, hour, day, and month
(some of which can be wildcards), and a build is triggered whenever
the current time matches the given constraints. This can run a build
every night, every morning, every weekend, alternate Thursdays, on
your boss's birthday, etc.

   Pass some subset of `minute', `hour', `dayOfMonth', `month', and
`dayOfWeek'; each may be a single number or a list of valid values.
The builds will be triggered whenever the current time matches these
values. Wildcards are represented by a '*' string. All fields default
to a wildcard except 'minute', so with no fields this defaults to a
build every hour, on the hour.  The full list of parameters is:

`name'

`builderNames'

`properties'

`branch'
     The branch to build, just as for `Scheduler'.

`minute'
     The minute of the hour on which to start the build.  This
     defaults to 0, meaning an hourly build.

`hour'
     The hour of the day on which to start the build, in 24-hour
     notation.  This defaults to *, meaning every hour.

`month'
     The month in which to start the build, with January = 1.  This
     defaults to *, meaning every month.

`dayOfWeek'
     The day of the week to start a build, with Monday = 0.  This
     defauls to *, meaning every day of the week.

`onlyIfChanged'
     If this is true, then builds will not be scheduled at the
     designated time unless the source has changed since the previous
     build.

   For example, the following master.cfg clause will cause a build to
be started every night at 3:00am:

     s = scheduler.Nightly(name='nightly',
             builderNames=['builder1', 'builder2'],
             hour=3,
             minute=0)

   This scheduler will perform a build each monday morning at 6:23am
and again at 8:23am, but only if someone has committed code in the
interim:

     s = scheduler.Nightly(name='BeforeWork',
              builderNames=['builder1'],
              dayOfWeek=0,
              hour=[6,8],
              minute=23,
              onlyIfChanged=True)

   The following runs a build every two hours, using Python's `range'
function:

     s = Nightly(name='every2hours',
             builderNames=['builder1'],
             hour=range(0, 24, 2))

   Finally, this example will run only on December 24th:

     s = Nightly(name='SleighPreflightCheck',
             builderNames=['flying_circuits', 'radar'],
             month=12,
             dayOfMonth=24,
             hour=12,
             minute=0)


File: buildbot.info,  Node: Try Schedulers,  Next: Triggerable Scheduler,  Prev: Nightly Scheduler,  Up: Change Sources and Schedulers

4.5.6 Try Schedulers
--------------------

This scheduler allows developers to use the `buildbot try' command to
trigger builds of code they have not yet committed. See *note try::
for complete details.

   Two implementations are available: `Try_Jobdir' and
`Try_Userpass'.  The former monitors a job directory, specified by
the `jobdir' parameter, while the latter listens for PB connections
on a specific `port', and authenticates against `userport'.


File: buildbot.info,  Node: Triggerable Scheduler,  Prev: Try Schedulers,  Up: Change Sources and Schedulers

4.5.7 Triggerable Scheduler
---------------------------

The `Triggerable' scheduler waits to be triggered by a Trigger step
(see *note Triggering Schedulers::) in another build. That step can
optionally wait for the scheduler's builds to complete. This provides
two advantages over Dependent schedulers. First, the same scheduler
can be triggered from multiple builds. Second, the ability to wait
for a Triggerable's builds to complete provides a form of "subroutine
call", where one or more builds can "call" a scheduler to perform
some work for them, perhaps on other buildslaves.

   The parameters are just the basics:

`name'

`builderNames'

`properties'

   This class is only useful in conjunction with the `Trigger' step.
Here is a fully-worked example:

     from buildbot import scheduler
     from buildbot.process import factory
     from buildbot.steps import trigger

     checkin = scheduler.Scheduler(name="checkin",
                 branch=None,
                 treeStableTimer=5*60,
                 builderNames=["checkin"])
     nightly = scheduler.Nightly(name='nightly',
                 builderNames=['nightly'],
                 hour=3,
                 minute=0)

     mktarball = scheduler.Triggerable(name="mktarball",
                     builderNames=["mktarball"])
     build = scheduler.Triggerable(name="build-all-platforms",
                     builderNames=["build-all-platforms"])
     test = scheduler.Triggerable(name="distributed-test",
                     builderNames=["distributed-test"])
     package = scheduler.Triggerable(name="package-all-platforms",
                     builderNames=["package-all-platforms"])

     c['schedulers'] = [checkin, nightly, build, test, package]

     # on checkin, make a tarball, build it, and test it
     checkin_factory = factory.BuildFactory()
     checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'],
                                            waitForFinish=True))
     checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'],
                                        waitForFinish=True))
     checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'],
                                       waitForFinish=True))

     # and every night, make a tarball, build it, and package it
     nightly_factory = factory.BuildFactory()
     nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'],
                                            waitForFinish=True))
     nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'],
                                        waitForFinish=True))
     nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'],
                                          waitForFinish=True))


File: buildbot.info,  Node: Merging BuildRequests,  Next: Setting the slaveport,  Prev: Change Sources and Schedulers,  Up: Configuration

4.6 Merging BuildRequests
=========================

By default, buildbot merges BuildRequests that have the compatible
SourceStamps. This behaviour can be customized with the
`c['mergeRequests']' configuration key.  This key specifies a function
which is caleld with three arguments: a `Builder' and two
`BuildRequest' objects.  It should return true if the requests can be
merged.  For example:

     def mergeRequests(builder, req1, req2):
         """Don't merge buildrequest at all"""
         return False
     c['mergeRequests'] = mergeRequests

   In many cases, the details of the SourceStamps and BuildRequests
are important.  In this example, only BuildRequests with the same
"reason" are merged; thus developers forcing builds for different
reasons will see distinct builds.

     def mergeRequests(builder, req1, req2):
         if req1.source.canBeMergedWith(req2.source) and  req1.reason == req2.reason:
            return True
         return False
     c['mergeRequests'] = mergeRequests


File: buildbot.info,  Node: Setting the slaveport,  Next: Buildslave Specifiers,  Prev: Merging BuildRequests,  Up: Configuration

4.7 Setting the slaveport
=========================

The buildmaster will listen on a TCP port of your choosing for
connections from buildslaves. It can also use this port for
connections from remote Change Sources, status clients, and debug
tools. This port should be visible to the outside world, and you'll
need to tell your buildslave admins about your choice.

   It does not matter which port you pick, as long it is externally
visible, however you should probably use something larger than 1024,
since most operating systems don't allow non-root processes to bind to
low-numbered ports. If your buildmaster is behind a firewall or a NAT
box of some sort, you may have to configure your firewall to permit
inbound connections to this port.

     c['slavePortnum'] = 10000

   `c['slavePortnum']' is a _strports_ specification string, defined
in the `twisted.application.strports' module (try `pydoc
twisted.application.strports' to get documentation on the format).
This means that you can have the buildmaster listen on a
localhost-only port by doing:

     c['slavePortnum'] = "tcp:10000:interface=127.0.0.1"

   This might be useful if you only run buildslaves on the same
machine, and they are all configured to contact the buildmaster at
`localhost:10000'.


File: buildbot.info,  Node: Buildslave Specifiers,  Next: On-Demand ("Latent") Buildslaves,  Prev: Setting the slaveport,  Up: Configuration

4.8 Buildslave Specifiers
=========================

The `c['slaves']' key is a list of known buildslaves. In the common
case, each buildslave is defined by an instance of the BuildSlave
class.  It represents a standard, manually started machine that will
try to connect to the buildbot master as a slave.  Contrast these
with the "on-demand" latent buildslaves, such as the Amazon Web
Service Elastic Compute Cloud latent buildslave discussed below.

   The BuildSlave class is instantiated with two values: (slavename,
slavepassword). These are the same two values that need to be
provided to the buildslave administrator when they create the
buildslave.

   The slavenames must be unique, of course. The password exists to
prevent evildoers from interfering with the buildbot by inserting
their own (broken) buildslaves into the system and thus displacing the
real ones.

   Buildslaves with an unrecognized slavename or a non-matching
password will be rejected when they attempt to connect, and a message
describing the problem will be put in the log file (see *note
Logfiles::).

     from buildbot.buildslave import BuildSlave
     c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd')
                    BuildSlave('bot-bsd', 'bsdpasswd')
                   ]

   `BuildSlave' objects can also be created with an optional
`properties' argument, a dictionary specifying properties that will
be available to any builds performed on this slave.  For example:

     from buildbot.buildslave import BuildSlave
     c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd',
                         properties={'os':'solaris'}),
                   ]

   The `BuildSlave' constructor can also take an optional
`max_builds' parameter to limit the number of builds that it will
execute simultaneously:

     from buildbot.buildslave import BuildSlave
     c['slaves'] = [BuildSlave("bot-linux", "linuxpassword", max_builds=2)]

   Historical note: in buildbot-0.7.5 and earlier, the `c['bots']'
key was used instead, and it took a list of (name, password) tuples.
This key is accepted for backwards compatibility, but is deprecated as
of 0.7.6 and will go away in some future release.

* Menu:

* When Buildslaves Go Missing::


File: buildbot.info,  Node: When Buildslaves Go Missing,  Up: Buildslave Specifiers

4.8.1 When Buildslaves Go Missing
---------------------------------

Sometimes, the buildslaves go away. One very common reason for this is
when the buildslave process is started once (manually) and left
running, but then later the machine reboots and the process is not
automatically restarted.

   If you'd like to have the administrator of the buildslave (or other
people) be notified by email when the buildslave has been missing for
too long, just add the `notify_on_missing=' argument to the
`BuildSlave' definition:

     c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd',
                               notify_on_missing="bob@example.com"),
                   ]

   By default, this will send email when the buildslave has been
disconnected for more than one hour. Only one email per
connection-loss event will be sent. To change the timeout, use
`missing_timeout=' and give it a number of seconds (the default is
3600).

   You can have the buildmaster send email to multiple recipients:
just provide a list of addresses instead of a single one:

     c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd',
                               notify_on_missing=["bob@example.com",
                                                  "alice@example.org"],
                               missing_timeout=300, # notify after 5 minutes
                               ),
                   ]

   The email sent this way will use a MailNotifier (*note
MailNotifier::) status target, if one is configured. This provides a
way for you to control the "from" address of the email, as well as
the relayhost (aka "smarthost") to use as an SMTP server. If no
MailNotifier is configured on this buildmaster, the
buildslave-missing emails will be sent using a default configuration.

   Note that if you want to have a MailNotifier for buildslave-missing
emails but not for regular build emails, just create one with
builders=[], as follows:

     from buildbot.status import mail
     m = mail.MailNotifier(fromaddr="buildbot@localhost", builders=[],
                           relayhost="smtp.example.org")
     c['status'].append(m)
     c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd',
                               notify_on_missing="bob@example.com"),
                   ]


File: buildbot.info,  Node: On-Demand ("Latent") Buildslaves,  Next: Defining Global Properties,  Prev: Buildslave Specifiers,  Up: Configuration

4.9 On-Demand ("Latent") Buildslaves
====================================

The standard buildbot model has slaves started manually.  The
previous section described how to configure the master for this
approach.

   Another approach is to let the buildbot master start slaves when
builds are ready, on-demand.  Thanks to services such as Amazon Web
Services' Elastic Compute Cloud ("AWS EC2"), this is relatively easy
to set up, and can be very useful for some situations.

   The buildslaves that are started on-demand are called "latent"
buildslaves.  As of this writing, buildbot ships with an abstract
base class for building latent buildslaves, and a concrete
implementation for AWS EC2.

* Menu:

* Amazon Web Services Elastic Compute Cloud ("AWS EC2")::
* Dangers with Latent Buildslaves::
* Writing New Latent Buildslaves::


File: buildbot.info,  Node: Amazon Web Services Elastic Compute Cloud ("AWS EC2"),  Next: Dangers with Latent Buildslaves,  Up: On-Demand ("Latent") Buildslaves

4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2")
-----------------------------------------------------------

AWS EC2 is a web service that allows you to start virtual machines in
an Amazon data center. Please see their website for details, incuding
costs. Using the AWS EC2 latent buildslaves involves getting an EC2
account with AWS and setting up payment; customizing one or more EC2
machine images ("AMIs") on your desired operating system(s) and
publishing them (privately if needed); and configuring the buildbot
master to know how to start your customized images for
"substantiating" your latent slaves.

* Menu:

* Get an AWS EC2 Account::
* Create an AMI::
* Configure the Master with an EC2LatentBuildSlave::


File: buildbot.info,  Node: Get an AWS EC2 Account,  Next: Create an AMI,  Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2")

4.9.1.1 Get an AWS EC2 Account
..............................

To start off, to use the AWS EC2 latent buildslave, you need to get
an AWS developer account and sign up for EC2. These instructions may
help you get started:

   * Go to http://aws.amazon.com/ and click to "Sign Up Now" for an
     AWS account.

   * Once you are logged into your account, you need to sign up for
     EC2.  Instructions for how to do this have changed over time
     because Amazon changes their website, so the best advice is to
     hunt for it. After signing up for EC2, it may say it wants you
     to upload an x.509 cert. You will need this to create images
     (see below) but it is not technically necessary for the buildbot
     master configuration.

   * You must enter a valid credit card before you will be able to
     use EC2. Do that under 'Payment Method'.

   * Make sure you're signed up for EC2 by going to 'Your
     Account'->'Account Activity' and verifying EC2 is listed.


File: buildbot.info,  Node: Create an AMI,  Next: Configure the Master with an EC2LatentBuildSlave,  Prev: Get an AWS EC2 Account,  Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2")

4.9.1.2 Create an AMI
.....................

Now you need to create an AMI and configure the master.  You may need
to run through this cycle a few times to get it working, but these
instructions should get you started.

   Creating an AMI is out of the scope of this document.  The EC2
Getting Started Guide is a good resource for this task.  Here are a
few additional hints.

   * When an instance of the image starts, it needs to automatically
     start a buildbot slave that connects to your master (to create a
     buildbot slave, *note Creating a buildslave::; to make a daemon,
     *note Launching the daemons::).

   * You may want to make an instance of the buildbot slave,
     configure it as a standard buildslave in the master (i.e., not
     as a latent slave), and test and debug it that way before you
     turn it into an AMI and convert to a latent slave in the master.


File: buildbot.info,  Node: Configure the Master with an EC2LatentBuildSlave,  Prev: Create an AMI,  Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2")

4.9.1.3 Configure the Master with an EC2LatentBuildSlave
........................................................

Now let's assume you have an AMI that should work with the
EC2LatentBuildSlave.  It's now time to set up your buildbot master
configuration.

   You will need some information from your AWS account: the "Access
Key Id" and the "Secret Access Key".  If you've built the AMI
yourself, you probably already are familiar with these values.  If
you have not, and someone has given you access to an AMI, these hints
may help you find the necessary values:

   * While logged into your AWS account, find the "Access
     Identifiers" link (either on the left, or via "Your Account" ->
     "Access Identifiers".

   * On the page, you'll see alphanumeric values for "Your Access Key
     Id:" and "Your Secret Access Key:". Make a note of these. Later
     on, we'll call the first one your "identifier" and the second
     one your "secret_identifier."

   When creating an EC2LatentBuildSlave in the buildbot master
configuration, the first three arguments are required.  The name and
password are the first two arguments, and work the same as with
normal buildslaves.  The next argument specifies the type of the EC2
virtual machine (available options as of this writing include
"m1.small", "m1.large", 'm1.xlarge", "c1.medium", and "c1.xlarge";
see the EC2 documentation for descriptions of these machines).

   Here is the simplest example of configuring an EC2 latent
buildslave. It specifies all necessary remaining values explicitly in
the instantiation.

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
                                        ami='ami-12345',
                                        identifier='publickey',
                                        secret_identifier='privatekey'
                                        )]

   The "ami" argument specifies the AMI that the master should start.
The "identifier" argument specifies the AWS "Access Key Id," and the
"secret_identifier" specifies the AWS "Secret Access Key." Both the
AMI and the account information can be specified in alternate ways.

   Note that whoever has your identifier and secret_identifier values
can request AWS work charged to your account, so these values need to
be carefully protected. Another way to specify these access keys is
to put them in a separate file. You can then make the access
privileges stricter for this separate file, and potentially let more
people read your main configuration file.

   By default, you can make an .ec2 directory in the home folder of
the user running the buildbot master. In that directory, create a
file called aws_id.  The first line of that file should be your
access key id; the second line should be your secret access key id.
Then you can instantiate the build slave as follows.

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
                                        ami='ami-12345')]

   If you want to put the key information in another file, use the
"aws_id_file_path" initialization argument.

   Previous examples used a particular AMI.  If the Buildbot master
will be deployed in a process-controlled environment, it may be
convenient to specify the AMI more flexibly.  Rather than specifying
an individual AMI, specify one or two AMI filters.

   In all cases, the AMI that sorts last by its location (the S3
bucket and manifest name) will be preferred.

   One available filter is to specify the acceptable AMI owners, by
AWS account number (the 12 digit number, usually rendered in AWS with
hyphens like "1234-5678-9012", should be entered as in integer).

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
                                valid_ami_owners=[11111111111,
                                                  22222222222],
                                identifier='publickey',
                                secret_identifier='privatekey'
                                )

   The other available filter is to provide a regular expression
string that will be matched against each AMI's location (the S3
bucket and manifest name).

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     bot1 = EC2LatentBuildSlave(
         'bot1', 'sekrit', 'm1.large',
         valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml',
         identifier='publickey', secret_identifier='privatekey')

   The regular expression can specify a group, which will be
preferred for the sorting.  Only the first group is used; subsequent
groups are ignored.

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     bot1 = EC2LatentBuildSlave(
         'bot1', 'sekrit', 'm1.large',
         valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml',
         identifier='publickey', secret_identifier='privatekey')

   If the group can be cast to an integer, it will be.  This allows
10 to sort after 1, for instance.

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     bot1 = EC2LatentBuildSlave(
         'bot1', 'sekrit', 'm1.large',
         valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml',
         identifier='publickey', secret_identifier='privatekey')

   In addition to using the password as a handshake between the
master and the slave, you may want to use a firewall to assert that
only machines from a specific IP can connect as slaves.  This is
possible with AWS EC2 by using the Elastic IP feature.  To configure,
generate a Elastic IP in AWS, and then specify it in your
configuration using the "elastic_ip" argument.

     from buildbot.ec2buildslave import EC2LatentBuildSlave
     c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large',
                                        'ami-12345',
                                        identifier='publickey',
                                        secret_identifier='privatekey',
                                        elastic_ip='208.77.188.166'
                                        )]

   The EC2LatentBuildSlave supports all other configuration from the
standard BuildSlave.  The "missing_timeout" and "notify_on_missing"
specify how long to wait for an EC2 instance to attach before
considering the attempt to have failed, and email addresses to alert,
respectively.  "missing_timeout" defaults to 20 minutes.

   The "build_wait_timeout" allows you to specify how long an
EC2LatentBuildSlave should wait after a build for another build
before it shuts down the EC2 instance.  It defaults to 10 minutes.

   "keypair_name" and "security_name" allow you to specify different
names for these AWS EC2 values.  They both default to
"latent_buildbot_slave".


File: buildbot.info,  Node: Dangers with Latent Buildslaves,  Next: Writing New Latent Buildslaves,  Prev: Amazon Web Services Elastic Compute Cloud ("AWS EC2"),  Up: On-Demand ("Latent") Buildslaves

4.9.2 Dangers with Latent Buildslaves
-------------------------------------

Any latent build slave that interacts with a for-fee service, such as
the EC2LatentBuildSlave, brings significant risks. As already
identified, the configuraton will need access to account information
that, if obtained by a criminal, can be used to charge services to
your account. Also, bugs in the buildbot software may lead to
unnecessary charges. In particular, if the master neglects to shut
down an instance for some reason, a virtual machine may be running
unnecessarily, charging against your account. Manual and/or automatic
(e.g. nagios with a plugin using a library like boto) double-checking
may be appropriate.

   A comparitively trivial note is that currently if two instances
try to attach to the same latent buildslave, it is likely that the
system will become confused.  This should not occur, unless, for
instance, you configure a normal build slave to connect with the
authentication of a latent buildbot.  If the situation occurs, stop
all attached instances and restart the master.


File: buildbot.info,  Node: Writing New Latent Buildslaves,  Prev: Dangers with Latent Buildslaves,  Up: On-Demand ("Latent") Buildslaves

4.9.3 Writing New Latent Buildslaves
------------------------------------

Writing a new latent buildslave should only require subclassing
`buildbot.buildslave.AbstractLatentBuildSlave' and implementing
start_instance and stop_instance.

     def start_instance(self):
         # responsible for starting instance that will try to connect with this
         # master. Should return deferred. Problems should use an errback. The
         # callback value can be None, or can be an iterable of short strings to
         # include in the "substantiate success" status message, such as
         # identifying the instance that started.
         raise NotImplementedError

     def stop_instance(self, fast=False):
         # responsible for shutting down instance. Return a deferred. If `fast`,
         # we're trying to shut the master down, so callback as soon as is safe.
         # Callback value is ignored.
         raise NotImplementedError

   See `buildbot.ec2buildslave.EC2LatentBuildSlave' for an example,
or see the test example `buildbot.test_slaves.FakeLatentBuildSlave'.


File: buildbot.info,  Node: Defining Global Properties,  Next: Defining Builders,  Prev: On-Demand ("Latent") Buildslaves,  Up: Configuration

4.10 Defining Global Properties
===============================

The `'properties'' configuration key defines a dictionary of
properties that will be available to all builds started by the
buildmaster:

     c['properties'] = {
         'Widget-version' : '1.2',
         'release-stage' : 'alpha'
     }


File: buildbot.info,  Node: Defining Builders,  Next: Defining Status Targets,  Prev: Defining Global Properties,  Up: Configuration

4.11 Defining Builders
======================

The `c['builders']' key is a list of dictionaries which specify the
Builders. The Buildmaster runs a collection of Builders, each of
which handles a single type of build (e.g. full versus quick), on a
single build slave. A Buildbot which makes sure that the latest code
("HEAD") compiles correctly across four separate architecture will
have four Builders, each performing the same build but on different
slaves (one per platform).

   Each Builder gets a separate column in the waterfall display. In
general, each Builder runs independently (although various kinds of
interlocks can cause one Builder to have an effect on another).

   Each Builder specification dictionary has several required keys:

`name'
     This specifies the Builder's name, which is used in status
     reports.

`slavename'
     This specifies which buildslave will be used by this Builder.
     `slavename' must appear in the `c['slaves']' list. Each
     buildslave can accomodate multiple Builders.

`slavenames'
     If you provide `slavenames' instead of `slavename', you can give
     a list of buildslaves which are capable of running this Builder.
     If multiple buildslaves are available for any given Builder, you
     will have some measure of redundancy: in case one slave goes
     offline, the others can still keep the Builder working. In
     addition, multiple buildslaves will allow multiple simultaneous
     builds for the same Builder, which might be useful if you have a
     lot of forced or "try" builds taking place.

     If you use this feature, it is important to make sure that the
     buildslaves are all, in fact, capable of running the given
     build. The slave hosts should be configured similarly, otherwise
     you will spend a lot of time trying (unsuccessfully) to
     reproduce a failure that only occurs on some of the buildslaves
     and not the others. Different platforms, operating systems,
     versions of major programs or libraries, all these things mean
     you should use separate Builders.

`builddir'
     This specifies the name of a subdirectory (under the base
     directory) in which everything related to this builder will be
     placed. On the buildmaster, this holds build status information.
     On the buildslave, this is where checkouts, compiles, and tests
     are run.

`factory'
     This is a `buildbot.process.factory.BuildFactory' instance which
     controls how the build is performed. Full details appear in
     their own chapter, *Note Build Process::. Parameters like the
     location of the CVS repository and the compile-time options used
     for the build are generally provided as arguments to the
     factory's constructor.


   Other optional keys may be set on each Builder:

`category'
     If provided, this is a string that identifies a category for the
     builder to be a part of. Status clients can limit themselves to a
     subset of the available categories. A common use for this is to
     add new builders to your setup (for a new module, or for a new
     buildslave) that do not work correctly yet and allow you to
     integrate them with the active builders. You can put these new
     builders in a test category, make your main status clients
     ignore them, and have only private status clients pick them up.
     As soon as they work, you can move them over to the active
     category.



File: buildbot.info,  Node: Defining Status Targets,  Next: Debug options,  Prev: Defining Builders,  Up: Configuration

4.12 Defining Status Targets
============================

The Buildmaster has a variety of ways to present build status to
various users. Each such delivery method is a "Status Target" object
in the configuration's `status' list. To add status targets, you just
append more objects to this list:

     c['status'] = []

     from buildbot.status import html
     c['status'].append(html.Waterfall(http_port=8010))

     from buildbot.status import mail
     m = mail.MailNotifier(fromaddr="buildbot@localhost",
                           extraRecipients=["builds@lists.example.com"],
                           sendToInterestedUsers=False)
     c['status'].append(m)

     from buildbot.status import words
     c['status'].append(words.IRC(host="irc.example.com", nick="bb",
                                  channels=["#example"]))

   Status delivery has its own chapter, *Note Status Delivery::, in
which all the built-in status targets are documented.


File: buildbot.info,  Node: Debug options,  Prev: Defining Status Targets,  Up: Configuration

4.13 Debug options
==================

If you set `c['debugPassword']', then you can connect to the
buildmaster with the diagnostic tool launched by `buildbot
debugclient MASTER:PORT'. From this tool, you can reload the config
file, manually force builds, and inject changes, which may be useful
for testing your buildmaster without actually commiting changes to
your repository (or before you have the Change Sources set up). The
debug tool uses the same port number as the slaves do:
`c['slavePortnum']', and is authenticated with this password.

     c['debugPassword'] = "debugpassword"

   If you set `c['manhole']' to an instance of one of the classes in
`buildbot.manhole', you can telnet or ssh into the buildmaster and
get an interactive Python shell, which may be useful for debugging
buildbot internals. It is probably only useful for buildbot
developers. It exposes full access to the buildmaster's account
(including the ability to modify and delete files), so it should not
be enabled with a weak or easily guessable password.

   There are three separate `Manhole' classes. Two of them use SSH,
one uses unencrypted telnet. Two of them use a username+password
combination to grant access, one of them uses an SSH-style
`authorized_keys' file which contains a list of ssh public keys.

`manhole.AuthorizedKeysManhole'
     You construct this with the name of a file that contains one SSH
     public key per line, just like `~/.ssh/authorized_keys'. If you
     provide a non-absolute filename, it will be interpreted relative
     to the buildmaster's base directory.

`manhole.PasswordManhole'
     This one accepts SSH connections but asks for a username and
     password when authenticating. It accepts only one such pair.

`manhole.TelnetManhole'
     This accepts regular unencrypted telnet connections, and asks
     for a username/password pair before providing access. Because
     this username/password is transmitted in the clear, and because
     Manhole access to the buildmaster is equivalent to granting full
     shell privileges to both the buildmaster and all the buildslaves
     (and to all accounts which then run code produced by the
     buildslaves), it is highly recommended that you use one of the
     SSH manholes instead.


     # some examples:
     from buildbot import manhole
     c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys")
     c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword")
     c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please")

   The `Manhole' instance can be configured to listen on a specific
port. You may wish to have this listening port bind to the loopback
interface (sometimes known as "lo0", "localhost", or 127.0.0.1) to
restrict access to clients which are running on the same host.

     from buildbot.manhole import PasswordManhole
     c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd")

   To have the `Manhole' listen on all interfaces, use `"tcp:9999"'
or simply 9999. This port specification uses
`twisted.application.strports', so you can make it listen on SSL or
even UNIX-domain sockets if you want.

   Note that using any Manhole requires that the TwistedConch package
be installed, and that you be using Twisted version 2.0 or later.

   The buildmaster's SSH server will use a different host key than the
normal sshd running on a typical unix host. This will cause the ssh
client to complain about a "host key mismatch", because it does not
realize there are two separate servers running on the same host. To
avoid this, use a clause like the following in your `.ssh/config'
file:

     Host remotehost-buildbot
      HostName remotehost
      HostKeyAlias remotehost-buildbot
      Port 9999
      # use 'user' if you use PasswordManhole and your name is not 'admin'.
      # if you use AuthorizedKeysManhole, this probably doesn't matter.
      User admin


File: buildbot.info,  Node: Getting Source Code Changes,  Next: Build Process,  Prev: Configuration,  Up: Top

5 Getting Source Code Changes
*****************************

The most common way to use the Buildbot is centered around the idea of
`Source Trees': a directory tree filled with source code of some form
which can be compiled and/or tested. Some projects use languages that
don't involve any compilation step: nevertheless there may be a
`build' phase where files are copied or rearranged into a form that
is suitable for installation. Some projects do not have unit tests,
and the Buildbot is merely helping to make sure that the sources can
compile correctly. But in all of these cases, the thing-being-tested
is a single source tree.

   A Version Control System mantains a source tree, and tells the
buildmaster when it changes. The first step of each Build is typically
to acquire a copy of some version of this tree.

   This chapter describes how the Buildbot learns about what Changes
have occurred. For more information on VC systems and Changes, see
*note Version Control Systems::.

* Menu:

* Change Sources::
* Choosing ChangeSources::
* CVSToys - PBService::
* Mail-parsing ChangeSources::
* PBChangeSource::
* P4Source::
* BonsaiPoller::
* SVNPoller::
* MercurialHook::
* Bzr Hook::
* Bzr Poller::


File: buildbot.info,  Node: Change Sources,  Next: Choosing ChangeSources,  Prev: Getting Source Code Changes,  Up: Getting Source Code Changes

5.1 Change Sources
==================

Each Buildmaster watches a single source tree. Changes can be provided
by a variety of ChangeSource types, however any given project will
typically have only a single ChangeSource active. This section
provides a description of all available ChangeSource types and
explains how to set up each of them.

   There are a variety of ChangeSources available, some of which are
meant to be used in conjunction with other tools to deliver Change
events from the VC repository to the buildmaster.

   * CVSToys This ChangeSource opens a TCP connection from the
     buildmaster to a waiting FreshCVS daemon that lives on the
     repository machine, and subscribes to hear about Changes.

   * MaildirSource This one watches a local maildir-format inbox for
     email sent out by the repository when a change is made. When a
     message arrives, it is parsed to create the Change object. A
     variety of parsing functions are available to accomodate
     different email-sending tools.

   * PBChangeSource This ChangeSource listens on a local TCP socket
     for inbound connections from a separate tool. Usually, this tool
     would be run on the VC repository machine in a commit hook. It
     is expected to connect to the TCP socket and send a Change
     message over the network connection. The `buildbot sendchange'
     command is one example of a tool that knows how to send these
     messages, so you can write a commit script for your VC system
     that calls it to deliver the Change.  There are other tools in
     the contrib/ directory that use the same protocol.


   As a quick guide, here is a list of VC systems and the
ChangeSources that might be useful with them. All of these
ChangeSources are in the `buildbot.changes' module.

`CVS'
        * freshcvs.FreshCVSSource (connected via TCP to the freshcvs
          daemon)

        * mail.FCMaildirSource (watching for email sent by a freshcvs
          daemon)

        * mail.BonsaiMaildirSource (watching for email sent by Bonsai)

        * mail.SyncmailMaildirSource (watching for email sent by
          syncmail)

        * pb.PBChangeSource (listening for connections from `buildbot
          sendchange' run in a loginfo script)

        * pb.PBChangeSource (listening for connections from a
          long-running `contrib/viewcvspoll.py' polling process which
          examines the ViewCVS database directly

`SVN'
        * pb.PBChangeSource (listening for connections from
          `contrib/svn_buildbot.py' run in a postcommit script)

        * pb.PBChangeSource (listening for connections from a
          long-running `contrib/svn_watcher.py' or
          `contrib/svnpoller.py' polling process

        * mail.SVNCommitEmailMaildirSource (watching for email sent
          by commit-email.pl)

        * svnpoller.SVNPoller (polling the SVN repository)

`Darcs'
        * pb.PBChangeSource (listening for connections from
          `contrib/darcs_buildbot.py' in a commit script

`Mercurial'
        * pb.PBChangeSource (listening for connections from
          `contrib/hg_buildbot.py' run in an 'incoming' hook)

        * pb.PBChangeSource (listening for connections from
          `buildbot/changes/hgbuildbot.py' run as an in-process
          'changegroup' hook)

`Arch/Bazaar'
        * pb.PBChangeSource (listening for connections from
          `contrib/arch_buildbot.py' run in a commit hook)

`Bzr (the newer Bazaar)'
        * pb.PBChangeSource (listening for connections from
          `contrib/bzr_buildbot.py' run in a post-change-branch-tip
          or commit hook)

        * `contrib/bzr_buildbot.py''s BzrPoller (polling the Bzr
          repository)

`Git'
        * pb.PBChangeSource (listening for connections from
          `contrib/git_buildbot.py' run in the post-receive hook)


   All VC systems can be driven by a PBChangeSource and the `buildbot
sendchange' tool run from some form of commit script.  If you write
an email parsing function, they can also all be driven by a suitable
`MaildirSource'.


File: buildbot.info,  Node: Choosing ChangeSources,  Next: CVSToys - PBService,  Prev: Change Sources,  Up: Getting Source Code Changes

5.2 Choosing ChangeSources
==========================

The `master.cfg' configuration file has a dictionary key named
`BuildmasterConfig['change_source']', which holds the active
`IChangeSource' object. The config file will typically create an
object from one of the classes described below and stuff it into this
key.

   Each buildmaster typically has just a single ChangeSource, since
it is only watching a single source tree. But if, for some reason,
you need multiple sources, just set `c['change_source']' to a list of
ChangeSources.. it will accept that too.

     s = FreshCVSSourceNewcred(host="host", port=4519,
                               user="alice", passwd="secret",
                               prefix="Twisted")
     BuildmasterConfig['change_source'] = [s]

   Each source tree has a nominal `top'. Each Change has a list of
filenames, which are all relative to this top location. The
ChangeSource is responsible for doing whatever is necessary to
accomplish this. Most sources have a `prefix' argument: a partial
pathname which is stripped from the front of all filenames provided to
that `ChangeSource'. Files which are outside this sub-tree are
ignored by the changesource: it does not generate Changes for those
files.


File: buildbot.info,  Node: CVSToys - PBService,  Next: Mail-parsing ChangeSources,  Prev: Choosing ChangeSources,  Up: Getting Source Code Changes

5.3 CVSToys - PBService
=======================

The CVSToys (http://purl.net/net/CVSToys) package provides a server
which runs on the machine that hosts the CVS repository it watches.
It has a variety of ways to distribute commit notifications, and
offers a flexible regexp-based way to filter out uninteresting
changes. One of the notification options is named `PBService' and
works by listening on a TCP port for clients. These clients subscribe
to hear about commit notifications.

   The buildmaster has a CVSToys-compatible `PBService' client built
in. There are two versions of it, one for old versions of CVSToys
(1.0.9 and earlier) which used the `oldcred' authentication
framework, and one for newer versions (1.0.10 and later) which use
`newcred'. Both are classes in the `buildbot.changes.freshcvs'
package.

   `FreshCVSSourceNewcred' objects are created with the following
parameters:

``host' and `port''
     these specify where the CVSToys server can be reached

``user' and `passwd''
     these specify the login information for the CVSToys server
     (`freshcvs'). These must match the server's values, which are
     defined in the `freshCfg' configuration file (which lives in the
     CVSROOT directory of the repository).

``prefix''
     this is the prefix to be found and stripped from filenames
     delivered by the CVSToys server. Most projects live in
     sub-directories of the main repository, as siblings of the
     CVSROOT sub-directory, so typically this prefix is set to that
     top sub-directory name.


Example
=======

To set up the freshCVS server, add a statement like the following to
your `freshCfg' file:

     pb = ConfigurationSet([
         (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)),
         ])

   This will announce all changes to a client which connects to port
4519 using a username of 'foo' and a password of 'bar'.

   Then add a clause like this to your buildmaster's `master.cfg':

     BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519,
                                                         "foo", "bar",
                                                         prefix="glib/")

   where "cvs.example.com" is the host that is running the FreshCVS
daemon, and "glib" is the top-level directory (relative to the
repository's root) where all your source code lives. Most projects
keep one or more projects in the same repository (along with CVSROOT/
to hold admin files like loginfo and freshCfg); the prefix= argument
tells the buildmaster to ignore everything outside that directory,
and to strip that common prefix from all pathnames it handles.


File: buildbot.info,  Node: Mail-parsing ChangeSources,  Next: PBChangeSource,  Prev: CVSToys - PBService,  Up: Getting Source Code Changes

5.4 Mail-parsing ChangeSources
==============================

Many projects publish information about changes to their source tree
by sending an email message out to a mailing list, frequently named
PROJECT-commits or PROJECT-changes. Each message usually contains a
description of the change (who made the change, which files were
affected) and sometimes a copy of the diff. Humans can subscribe to
this list to stay informed about what's happening to the source tree.

   The Buildbot can also be subscribed to a -commits mailing list, and
can trigger builds in response to Changes that it hears about. The
buildmaster admin needs to arrange for these email messages to arrive
in a place where the buildmaster can find them, and configure the
buildmaster to parse the messages correctly. Once that is in place,
the email parser will create Change objects and deliver them to the
Schedulers (see *note Change Sources and Schedulers::) just like any
other ChangeSource.

   There are two components to setting up an email-based ChangeSource.
The first is to route the email messages to the buildmaster, which is
done by dropping them into a "maildir". The second is to actually
parse the messages, which is highly dependent upon the tool that was
used to create them. Each VC system has a collection of favorite
change-emailing tools, and each has a slightly different format, so
each has a different parsing function. There is a separate
ChangeSource variant for each parsing function.

   Once you've chosen a maildir location and a parsing function,
create the change source and put it in `c['change_source']':

     from buildbot.changes.mail import SyncmailMaildirSource
     c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot",
                                                prefix="/trunk/")

* Menu:

* Subscribing the Buildmaster::
* Using Maildirs::
* Parsing Email Change Messages::


File: buildbot.info,  Node: Subscribing the Buildmaster,  Next: Using Maildirs,  Prev: Mail-parsing ChangeSources,  Up: Mail-parsing ChangeSources

5.4.1 Subscribing the Buildmaster
---------------------------------

The recommended way to install the buildbot is to create a dedicated
account for the buildmaster. If you do this, the account will probably
have a distinct email address (perhaps <buildmaster@example.org>).
Then just arrange for this account's email to be delivered to a
suitable maildir (described in the next section).

   If the buildbot does not have its own account, "extension
addresses" can be used to distinguish between email intended for the
buildmaster and email intended for the rest of the account. In most
modern MTAs, the e.g. `foo@example.org' account has control over
every email address at example.org which begins with "foo", such that
email addressed to <account-foo@example.org> can be delivered to a
different destination than <account-bar@example.org>. qmail does this
by using separate .qmail files for the two destinations (`.qmail-foo'
and `.qmail-bar', with `.qmail' controlling the base address and
`.qmail-default' controlling all other extensions). Other MTAs have
similar mechanisms.

   Thus you can assign an extension address like
<foo-buildmaster@example.org> to the buildmaster, and retain
<foo@example.org> for your own use.


File: buildbot.info,  Node: Using Maildirs,  Next: Parsing Email Change Messages,  Prev: Subscribing the Buildmaster,  Up: Mail-parsing ChangeSources

5.4.2 Using Maildirs
--------------------

A "maildir" is a simple directory structure originally developed for
qmail that allows safe atomic update without locking. Create a base
directory with three subdirectories: "new", "tmp", and "cur".  When
messages arrive, they are put into a uniquely-named file (using pids,
timestamps, and random numbers) in "tmp". When the file is complete,
it is atomically renamed into "new". Eventually the buildmaster
notices the file in "new", reads and parses the contents, then moves
it into "cur". A cronjob can be used to delete files in "cur" at
leisure.

   Maildirs are frequently created with the `maildirmake' tool, but a
simple `mkdir -p ~/MAILDIR/{cur,new,tmp}' is pretty much equivalent.

   Many modern MTAs can deliver directly to maildirs. The usual
.forward or .procmailrc syntax is to name the base directory with a
trailing slash, so something like `~/MAILDIR/' . qmail and postfix are
maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail
Delivery Agent).

   For MTAs which cannot put files into maildirs directly, the
"safecat" tool can be executed from a .forward file to accomplish the
same thing.

   The Buildmaster uses the linux DNotify facility to receive
immediate notification when the maildir's "new" directory has
changed. When this facility is not available, it polls the directory
for new messages, every 10 seconds by default.


File: buildbot.info,  Node: Parsing Email Change Messages,  Prev: Using Maildirs,  Up: Mail-parsing ChangeSources

5.4.3 Parsing Email Change Messages
-----------------------------------

The second component to setting up an email-based ChangeSource is to
parse the actual notices. This is highly dependent upon the VC system
and commit script in use.

   A couple of common tools used to create these change emails are:

`CVS'

    `CVSToys MailNotifier'
          *note FCMaildirSource::

    `Bonsai notification'
          *note BonsaiMaildirSource::

    `syncmail'
          *note SyncmailMaildirSource::

`SVN'

    `svnmailer'
          http://opensource.perlig.de/en/svnmailer/

    `commit-email.pl'
          *note SVNCommitEmailMaildirSource::

`Mercurial'

    `NotifyExtension'
          http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension

`Git'

    `post-receive-email'
          http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD


   The following sections describe the parsers available for each of
these tools.

   Most of these parsers accept a `prefix=' argument, which is used
to limit the set of files that the buildmaster pays attention to. This
is most useful for systems like CVS and SVN which put multiple
projects in a single repository (or use repository names to indicate
branches). Each filename that appears in the email is tested against
the prefix: if the filename does not start with the prefix, the file
is ignored. If the filename _does_ start with the prefix, that prefix
is stripped from the filename before any further processing is done.
Thus the prefix usually ends with a slash.

* Menu:

* FCMaildirSource::
* SyncmailMaildirSource::
* BonsaiMaildirSource::
* SVNCommitEmailMaildirSource::


File: buildbot.info,  Node: FCMaildirSource,  Next: SyncmailMaildirSource,  Prev: Parsing Email Change Messages,  Up: Parsing Email Change Messages

5.4.3.1 FCMaildirSource
.......................

http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/

   This parser works with the CVSToys `MailNotification' action,
which will send email to a list of recipients for each commit. This
tends to work better than using `/bin/mail' from within the
CVSROOT/loginfo file directly, as CVSToys will batch together all
files changed during the same CVS invocation, and can provide more
information (like creating a ViewCVS URL for each file changed).

   The Buildbot's `FCMaildirSource' knows for to parse these CVSToys
messages and turn them into Change objects. It can be given two
parameters: the directory name of the maildir root, and the prefix to
strip.

     from buildbot.changes.mail import FCMaildirSource
     c['change_source'] = FCMaildirSource("~/maildir-buildbot")


File: buildbot.info,  Node: SyncmailMaildirSource,  Next: BonsaiMaildirSource,  Prev: FCMaildirSource,  Up: Parsing Email Change Messages

5.4.3.2 SyncmailMaildirSource
.............................

http://sourceforge.net/projects/cvs-syncmail

   `SyncmailMaildirSource' knows how to parse the message format used
by the CVS "syncmail" script.

     from buildbot.changes.mail import SyncmailMaildirSource
     c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot")


File: buildbot.info,  Node: BonsaiMaildirSource,  Next: SVNCommitEmailMaildirSource,  Prev: SyncmailMaildirSource,  Up: Parsing Email Change Messages

5.4.3.3 BonsaiMaildirSource
...........................

http://www.mozilla.org/bonsai.html

   `BonsaiMaildirSource' parses messages sent out by Bonsai, the CVS
tree-management system built by Mozilla.

     from buildbot.changes.mail import BonsaiMaildirSource
     c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot")


File: buildbot.info,  Node: SVNCommitEmailMaildirSource,  Prev: BonsaiMaildirSource,  Up: Parsing Email Change Messages

5.4.3.4 SVNCommitEmailMaildirSource
...................................

`SVNCommitEmailMaildirSource' parses message sent out by the
`commit-email.pl' script, which is included in the Subversion
distribution.

   It does not currently handle branches: all of the Change objects
that it creates will be associated with the default (i.e. trunk)
branch.

     from buildbot.changes.mail import SVNCommitEmailMaildirSource
     c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot")


File: buildbot.info,  Node: PBChangeSource,  Next: P4Source,  Prev: Mail-parsing ChangeSources,  Up: Getting Source Code Changes

5.5 PBChangeSource
==================

The last kind of ChangeSource actually listens on a TCP port for
clients to connect and push change notices _into_ the Buildmaster.
This is used by the built-in `buildbot sendchange' notification tool,
as well as the VC-specific `contrib/svn_buildbot.py',
`contrib/arch_buildbot.py', `contrib/hg_buildbot.py' tools, and the
`buildbot.changes.hgbuildbot' hook. These tools are run by the
repository (in a commit hook script), and connect to the buildmaster
directly each time a file is comitted. This is also useful for
creating new kinds of change sources that work on a `push' model
instead of some kind of subscription scheme, for example a script
which is run out of an email .forward file.

   This ChangeSource can be configured to listen on its own TCP port,
or it can share the port that the buildmaster is already using for the
buildslaves to connect. (This is possible because the
`PBChangeSource' uses the same protocol as the buildslaves, and they
can be distinguished by the `username' attribute used when the
initial connection is established). It might be useful to have it
listen on a different port if, for example, you wanted to establish
different firewall rules for that port. You could allow only the SVN
repository machine access to the `PBChangeSource' port, while
allowing only the buildslave machines access to the slave port. Or you
could just expose one port and run everything over it. _Note: this
feature is not yet implemented, the PBChangeSource will always share
the slave port and will always have a `user' name of `change', and a
passwd of `changepw'. These limitations will be removed in the
future._.

   The `PBChangeSource' is created with the following arguments. All
are optional.

``port''
     which port to listen on. If `None' (which is the default), it
     shares the port used for buildslave connections. _Not
     Implemented, always set to `None'_.

``user' and `passwd''
     The user/passwd account information that the client program must
     use to connect. Defaults to `change' and `changepw'. _Not
     Implemented, `user' is currently always set to `change',
     `passwd' is always set to `changepw'_.

``prefix''
     The prefix to be found and stripped from filenames delivered
     over the connection. Any filenames which do not start with this
     prefix will be removed. If all the filenames in a given Change
     are removed, the that whole Change will be dropped. This string
     should probably end with a directory separator.

     This is useful for changes coming from version control systems
     that represent branches as parent directories within the
     repository (like SVN and Perforce). Use a prefix of 'trunk/' or
     'project/branches/foobranch/' to only follow one branch and to
     get correct tree-relative filenames. Without a prefix, the
     PBChangeSource will probably deliver Changes with filenames like
     `trunk/foo.c' instead of just `foo.c'. Of course this also
     depends upon the tool sending the Changes in (like `buildbot
     sendchange') and what filenames it is delivering: that tool may
     be filtering and stripping prefixes at the sending end.



File: buildbot.info,  Node: P4Source,  Next: BonsaiPoller,  Prev: PBChangeSource,  Up: Getting Source Code Changes

5.6 P4Source
============

The `P4Source' periodically polls a Perforce
(http://www.perforce.com/) depot for changes. It accepts the
following arguments:

``p4base''
     The base depot path to watch, without the trailing '/...'.

``p4port''
     The Perforce server to connect to (as host:port).

``p4user''
     The Perforce user.

``p4passwd''
     The Perforce password.

``p4bin''
     An optional string parameter. Specify the location of the
     perforce command line binary (p4).  You only need to do this if
     the perforce binary is not in the path of the buildbot user.
     Defaults to "p4".

``split_file''
     A function that maps a pathname, without the leading `p4base',
     to a (branch, filename) tuple. The default just returns (None,
     branchfile), which effectively disables branch support. You
     should supply a function which understands your repository
     structure.

``pollinterval''
     How often to poll, in seconds. Defaults to 600 (10 minutes).

``histmax''
     The maximum number of changes to inspect at a time. If more than
     this number occur since the last poll, older changes will be
     silently ignored.

Example
=======

This configuration uses the `P4PORT', `P4USER', and `P4PASSWD'
specified in the buildmaster's environment. It watches a project in
which the branch name is simply the next path component, and the file
is all path components after.

     import buildbot.changes.p4poller
     s = p4poller.P4Source(p4base='//depot/project/',
                           split_file=lambda branchfile: branchfile.split('/',1),
                          )
     c['change_source'] = s


File: buildbot.info,  Node: BonsaiPoller,  Next: SVNPoller,  Prev: P4Source,  Up: Getting Source Code Changes

5.7 BonsaiPoller
================

The `BonsaiPoller' periodically polls a Bonsai server. This is a CGI
script accessed through a web server that provides information about
a CVS tree, for example the Mozilla bonsai server at
`http://bonsai.mozilla.org'. Bonsai servers are usable by both humans
and machines. In this case, the buildbot's change source forms a
query which asks about any files in the specified branch which have
changed since the last query.

   Please take a look at the BonsaiPoller docstring for details about
the arguments it accepts.


File: buildbot.info,  Node: SVNPoller,  Next: MercurialHook,  Prev: BonsaiPoller,  Up: Getting Source Code Changes

5.8 SVNPoller
=============

The `buildbot.changes.svnpoller.SVNPoller' is a ChangeSource which
periodically polls a Subversion (http://subversion.tigris.org/)
repository for new revisions, by running the `svn log' command in a
subshell. It can watch a single branch or multiple branches.

   `SVNPoller' accepts the following arguments:

`svnurl'
     The base URL path to watch, like
     `svn://svn.twistedmatrix.com/svn/Twisted/trunk', or
     `http://divmod.org/svn/Divmod/', or even
     `file:///home/svn/Repository/ProjectA/branches/1.5/'. This must
     include the access scheme, the location of the repository (both
     the hostname for remote ones, and any additional directory names
     necessary to get to the repository), and the sub-path within the
     repository's virtual filesystem for the project and branch of
     interest.

     The `SVNPoller' will only pay attention to files inside the
     subdirectory specified by the complete svnurl.

`split_file'
     A function to convert pathnames into (branch, relative_pathname)
     tuples. Use this to explain your repository's branch-naming
     policy to `SVNPoller'. This function must accept a single string
     and return a two-entry tuple. There are a few utility functions
     in `buildbot.changes.svnpoller' that can be used as a
     `split_file' function, see below for details.

     The default value always returns (None, path), which indicates
     that all files are on the trunk.

     Subclasses of `SVNPoller' can override the `split_file' method
     instead of using the `split_file=' argument.

`svnuser'
     An optional string parameter. If set, the `--user' argument will
     be added to all `svn' commands. Use this if you have to
     authenticate to the svn server before you can do `svn info' or
     `svn log' commands.

`svnpasswd'
     Like `svnuser', this will cause a `--password' argument to be
     passed to all svn commands.

`pollinterval'
     How often to poll, in seconds. Defaults to 600 (checking once
     every 10 minutes). Lower this if you want the buildbot to notice
     changes faster, raise it if you want to reduce the network and
     CPU load on your svn server. Please be considerate of public SVN
     repositories by using a large interval when polling them.

`histmax'
     The maximum number of changes to inspect at a time. Every
     POLLINTERVAL seconds, the `SVNPoller' asks for the last HISTMAX
     changes and looks through them for any ones it does not already
     know about. If more than HISTMAX revisions have been committed
     since the last poll, older changes will be silently ignored.
     Larger values of histmax will cause more time and memory to be
     consumed on each poll attempt.  `histmax' defaults to 100.

`svnbin'
     This controls the `svn' executable to use. If subversion is
     installed in a weird place on your system (outside of the
     buildmaster's `$PATH'), use this to tell `SVNPoller' where to
     find it. The default value of "svn" will almost always be
     sufficient.


Branches
========

Each source file that is tracked by a Subversion repository has a
fully-qualified SVN URL in the following form:
(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the
`SVNPoller', you give it a `svnurl' value that includes all of the
REPOURL and possibly some portion of the PROJECT-plus-BRANCH string.
The `SVNPoller' is responsible for producing Changes that contain a
branch name and a FILEPATH (which is relative to the top of a
checked-out tree). The details of how these strings are split up
depend upon how your repository names its branches.

PROJECT/BRANCHNAME/FILEPATH repositories
----------------------------------------

One common layout is to have all the various projects that share a
repository get a single top-level directory each. Then under a given
project's directory, you get two subdirectories, one named "trunk"
and another named "branches". Under "branches" you have a bunch of
other directories, one per branch, with names like "1.5.x" and
"testing". It is also common to see directories like "tags" and
"releases" next to "branches" and "trunk".

   For example, the Twisted project has a subversion server on
"svn.twistedmatrix.com" that hosts several sub-projects. The
repository is available through a SCHEME of "svn:". The primary
sub-project is Twisted, of course, with a repository root of
"svn://svn.twistedmatrix.com/svn/Twisted". Another sub-project is
Informant, with a root of
"svn://svn.twistedmatrix.com/svn/Informant", etc. Inside any
checked-out Twisted tree, there is a file named bin/trial (which is
used to run unit test suites).

   The trunk for Twisted is in
"svn://svn.twistedmatrix.com/svn/Twisted/trunk", and the
fully-qualified SVN URL for the trunk version of `trial' would be
"svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial". The same
SVNURL for that file on a branch named "1.5.x" would be
"svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial".

   To set up a `SVNPoller' that watches the Twisted trunk (and
nothing else), we would use the following:

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk")

   In this case, every Change that our `SVNPoller' produces will have
`.branch=None', to indicate that the Change is on the trunk.  No
other sub-projects or branches will be tracked.

   If we want our ChangeSource to follow multiple branches, we have
to do two things. First we have to change our `svnurl=' argument to
watch more than just ".../Twisted/trunk". We will set it to
".../Twisted" so that we'll see both the trunk and all the branches.
Second, we have to tell `SVNPoller' how to split the
(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository
out into (BRANCH) and (FILEPATH) pairs.

   We do the latter by providing a "split_file" function. This
function is responsible for splitting something like
"branches/1.5.x/bin/trial" into `branch'="branches/1.5.x" and
`filepath'="bin/trial". This function is always given a string that
names a file relative to the subdirectory pointed to by the
`SVNPoller''s `svnurl=' argument. It is expected to return a
(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the
branch indicated), or None to indicate that the file is outside any
project of interest.

   (note that we want to see "branches/1.5.x" rather than just
"1.5.x" because when we perform the SVN checkout, we will probably
append the branch name to the baseURL, which requires that we keep the
"branches" component in there. Other VC schemes use a different
approach towards branches and may not require this artifact.)

   If your repository uses this same PROJECT/BRANCH/FILEPATH naming
scheme, the following function will work:

     def split_file_branches(path):
         pieces = path.split('/')
         if pieces[0] == 'trunk':
             return (None, '/'.join(pieces[1:]))
         elif pieces[0] == 'branches':
             return ('/'.join(pieces[0:2]),
                     '/'.join(pieces[2:]))
         else:
             return None

   This function is provided as
`buildbot.changes.svnpoller.split_file_branches' for your
convenience. So to have our Twisted-watching `SVNPoller' follow
multiple branches, we would use this:

     from buildbot.changes.svnpoller import SVNPoller, split_file_branches
     c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted",
                                    split_file=split_file_branches)

   Changes for all sorts of branches (with names like
"branches/1.5.x", and None to indicate the trunk) will be delivered
to the Schedulers.  Each Scheduler is then free to use or ignore each
branch as it sees fit.

BRANCHNAME/PROJECT/FILEPATH repositories
----------------------------------------

Another common way to organize a Subversion repository is to put the
branch name at the top, and the projects underneath. This is
especially frequent when there are a number of related sub-projects
that all get released in a group.

   For example, Divmod.org hosts a project named "Nevow" as well as
one named "Quotient". In a checked-out Nevow tree there is a directory
named "formless" that contains a python source file named
"webform.py". This repository is accessible via webdav (and thus uses
an "http:" scheme) through the divmod.org hostname. There are many
branches in this repository, and they use a (BRANCHNAME)/(PROJECT)
naming policy.

   The fully-qualified SVN URL for the trunk version of webform.py is
`http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py'.  You
can do an `svn co' with that URL and get a copy of the latest
version. The 1.5.x branch version of this file would have a URL of
`http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py'.
The whole Nevow trunk would be checked out with
`http://divmod.org/svn/Divmod/trunk/Nevow', while the Quotient trunk
would be checked out using
`http://divmod.org/svn/Divmod/trunk/Quotient'.

   Now suppose we want to have an `SVNPoller' that only cares about
the Nevow trunk. This case looks just like the PROJECT/BRANCH layout
described earlier:

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow")

   But what happens when we want to track multiple Nevow branches? We
have to point our `svnurl=' high enough to see all those branches,
but we also don't want to include Quotient changes (since we're only
building Nevow). To accomplish this, we must rely upon the
`split_file' function to help us tell the difference between files
that belong to Nevow and those that belong to Quotient, as well as
figuring out which branch each one is on.

     from buildbot.changes.svnpoller import SVNPoller
     c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod",
                                    split_file=my_file_splitter)

   The `my_file_splitter' function will be called with
repository-relative pathnames like:

`trunk/Nevow/formless/webform.py'
     This is a Nevow file, on the trunk. We want the Change that
     includes this to see a filename of `formless/webform.py"', and a
     branch of None

`branches/1.5.x/Nevow/formless/webform.py'
     This is a Nevow file, on a branch. We want to get
     branch="branches/1.5.x" and filename="formless/webform.py".

`trunk/Quotient/setup.py'
     This is a Quotient file, so we want to ignore it by having
     `my_file_splitter' return None.

`branches/1.5.x/Quotient/setup.py'
     This is also a Quotient file, which should be ignored.

   The following definition for `my_file_splitter' will do the job:

     def my_file_splitter(path):
         pieces = path.split('/')
         if pieces[0] == 'trunk':
             branch = None
             pieces.pop(0) # remove 'trunk'
         elif pieces[0] == 'branches':
             pieces.pop(0) # remove 'branches'
             # grab branch name
             branch = 'branches/' + pieces.pop(0)
         else:
             return None # something weird
         projectname = pieces.pop(0)
         if projectname != 'Nevow':
             return None # wrong project
         return (branch, '/'.join(pieces))


File: buildbot.info,  Node: MercurialHook,  Next: Bzr Hook,  Prev: SVNPoller,  Up: Getting Source Code Changes

5.9 MercurialHook
=================

Since Mercurial is written in python, the hook script can invoke
Buildbot's `sendchange' function directly, rather than having to
spawn an external process. This function delivers the same sort of
changes as `buildbot sendchange' and the various hook scripts in
contrib/, so you'll need to add a `pb.PBChangeSource' to your
buildmaster to receive these changes.

   To set this up, first choose a Mercurial repository that represents
your central "official" source tree. This will be the same repository
that your buildslaves will eventually pull from. Install Buildbot on
the machine that hosts this repository, using the same version of
python as Mercurial is using (so that the Mercurial hook can import
code from buildbot). Then add the following to the `.hg/hgrc' file in
that repository, replacing the buildmaster hostname/portnumber as
appropriate for your buildbot:

     [hooks]
     changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook

     [hgbuildbot]
     master = buildmaster.example.org:9987

   (Note that Mercurial lets you define multiple `changegroup' hooks
by giving them distinct names, like `changegroup.foo' and
`changegroup.bar', which is why we use `changegroup.buildbot' in this
example. There is nothing magical about the "buildbot" suffix in the
hook name. The `[hgbuildbot]' section _is_ special, however, as it is
the only section that the buildbot hook pays attention to.)

   Also note that this runs as a `changegroup' hook, rather than as
an `incoming' hook. The `changegroup' hook is run with multiple
revisions at a time (say, if multiple revisions are being pushed to
this repository in a single `hg push' command), whereas the
`incoming' hook is run with just one revision at a time. The
`hgbuildbot.hook' function will only work with the `changegroup' hook.

   The `[hgbuildbot]' section has two other parameters that you might
specify, both of which control the name of the branch that is
attached to the changes coming from this hook.

   One common branch naming policy for Mercurial repositories is to
use it just like Darcs: each branch goes into a separate repository,
and all the branches for a single project share a common parent
directory.  For example, you might have `/var/repos/PROJECT/trunk/'
and `/var/repos/PROJECT/release'. To use this style, use the
`branchtype = dirname' setting, which simply uses the last component
of the repository's enclosing directory as the branch name:

     [hgbuildbot]
     master = buildmaster.example.org:9987
     branchtype = dirname

   Another approach is to use Mercurial's built-in branches (the kind
created with `hg branch' and listed with `hg branches'). This feature
associates persistent names with particular lines of descent within a
single repository. (note that the buildbot `source.Mercurial'
checkout step does not yet support this kind of branch). To have the
commit hook deliver this sort of branch name with the Change object,
use `branchtype = inrepo':

     [hgbuildbot]
     master = buildmaster.example.org:9987
     branchtype = inrepo

   Finally, if you want to simply specify the branchname directly, for
all changes, use `branch = BRANCHNAME'. This overrides `branchtype':

     [hgbuildbot]
     master = buildmaster.example.org:9987
     branch = trunk

   If you use `branch=' like this, you'll need to put a separate
.hgrc in each repository. If you use `branchtype=', you may be able
to use the same .hgrc for all your repositories, stored in `~/.hgrc'
or `/etc/mercurial/hgrc'.


File: buildbot.info,  Node: Bzr Hook,  Next: Bzr Poller,  Prev: MercurialHook,  Up: Getting Source Code Changes

5.10 Bzr Hook
=============

Bzr is also written in Python, and the Bzr hook depends on Twisted to
send the changes.

   To install, put `contrib/bzr_buildbot.py' in one of your plugins
locations a bzr plugins directory (e.g., `~/.bazaar/plugins'). Then,
in one of your bazaar conf files (e.g., `~/.bazaar/locations.conf'),
set the location you want to connect with buildbot with these keys:

`buildbot_on'
     one of 'commit', 'push, or 'change'. Turns the plugin on to
     report changes via commit, changes via push, or any changes to
     the trunk. 'change' is recommended.

`buildbot_server'
     (required to send to a buildbot master) the URL of the buildbot
     master to which you will connect (as of this writing, the same
     server and port to which slaves connect).

`buildbot_port'
     (optional, defaults to 9989) the port of the buildbot master to
     which you will connect (as of this writing, the same server and
     port to which slaves connect)

`buildbot_pqm'
     (optional, defaults to not pqm) Normally, the user that commits
     the revision is the user that is responsible for the change.
     When run in a pqm (Patch Queue Manager, see
     https://launchpad.net/pqm) environment, the user that commits is
     the Patch Queue Manager, and the user that committed the
     *parent* revision is responsible for the change. To turn on the
     pqm mode, set this value to any of (case-insensitive) "Yes",
     "Y", "True", or "T".

`buildbot_dry_run'
     (optional, defaults to not a dry run) Normally, the post-commit
     hook will attempt to communicate with the configured buildbot
     server and port. If this parameter is included and any of
     (case-insensitive) "Yes", "Y", "True", or "T", then the hook
     will simply print what it would have sent, but not attempt to
     contact the buildbot master.

`buildbot_send_branch_name'
     (optional, defaults to not sending the branch name) If your
     buildbot's bzr source build step uses a repourl, do *not* turn
     this on. If your buildbot's bzr build step uses a baseURL, then
     you may set this value to any of (case-insensitive) "Yes", "Y",
     "True", or "T" to have the buildbot master append the branch
     name to the baseURL.


   When buildbot no longer has a hardcoded password, it will be a
configuration option here as well.

   Here's a simple example that you might have in your
`~/.bazaar/locations.conf'.

     [chroot-*:///var/local/myrepo/mybranch]
     buildbot_on = change
     buildbot_server = localhost


File: buildbot.info,  Node: Bzr Poller,  Prev: Bzr Hook,  Up: Getting Source Code Changes

5.11 Bzr Poller
===============

If you cannot insert a Bzr hook in the server, you can use the Bzr
Poller. To use, put `contrib/bzr_buildbot.py' somewhere that your
buildbot configuration can import it. Even putting it in the same
directory as the master.cfg should work. Install the poller in the
buildbot configuration as with any other change source. Minimally,
provide a URL that you want to poll (bzr://, bzr+ssh://, or lp:),
though make sure the buildbot user has necessary privileges. You may
also want to specify these optional values.

`poll_interval'
     The number of seconds to wait between polls.  Defaults to 10
     minutes.

`branch_name'
     Any value to be used as the branch name. Defaults to None, or
     specify a string, or specify the constants from
     `bzr_buildbot.py' SHORT or FULL to get the short branch name or
     full branch address.

`blame_merge_author'
     normally, the user that commits the revision is the user that is
     responsible for the change. When run in a pqm (Patch Queue
     Manager, see https://launchpad.net/pqm) environment, the user
     that commits is the Patch Queue Manager, and the user that
     committed the merged, *parent* revision is responsible for the
     change. set this value to True if this is pointed against a
     PQM-managed branch.


File: buildbot.info,  Node: Build Process,  Next: Status Delivery,  Prev: Getting Source Code Changes,  Up: Top

6 Build Process
***************

A `Build' object is responsible for actually performing a build.  It
gets access to a remote `SlaveBuilder' where it may run commands, and
a `BuildStatus' object where it must emit status events. The `Build'
is created by the Builder's `BuildFactory'.

   The default `Build' class is made up of a fixed sequence of
`BuildSteps', executed one after another until all are complete (or
one of them indicates that the build should be halted early). The
default `BuildFactory' creates instances of this `Build' class with a
list of `BuildSteps', so the basic way to configure the build is to
provide a list of `BuildSteps' to your `BuildFactory'.

   More complicated `Build' subclasses can make other decisions:
execute some steps only if certain files were changed, or if certain
previous steps passed or failed. The base class has been written to
allow users to express basic control flow without writing code, but
you can always subclass and customize to achieve more specialized
behavior.

* Menu:

* Build Steps::
* Interlocks::
* Build Factories::


File: buildbot.info,  Node: Build Steps,  Next: Interlocks,  Prev: Build Process,  Up: Build Process

6.1 Build Steps
===============

`BuildStep's are usually specified in the buildmaster's configuration
file, in a list that goes into the `BuildFactory'.  The `BuildStep'
instances in this list are used as templates to construct new
independent copies for each build (so that state can be kept on the
`BuildStep' in one build without affecting a later build). Each
`BuildFactory' can be created with a list of steps, or the factory
can be created empty and then steps added to it using the `addStep'
method:

     from buildbot.steps import source, shell
     from buildbot.process import factory

     f = factory.BuildFactory()
     f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/"))
     f.addStep(shell.ShellCommand(command=["make", "all"]))
     f.addStep(shell.ShellCommand(command=["make", "test"]))

   In earlier versions (0.7.5 and older), these steps were specified
with a tuple of (step_class, keyword_arguments). Steps can still be
specified this way, but the preferred form is to pass actual
`BuildStep' instances to `addStep', because that gives the
`BuildStep' class a chance to do some validation on the arguments.

   If you have a common set of steps which are used in several
factories, the `addSteps' method may be handy.  It takes an iterable
of `BuildStep' instances.

     setup_steps = [
         source.SVN(svnurl="http://svn.example.org/Trunk/")
         shell.ShellCommand(command="./setup")
     ]
     quick = factory.BuildFactory()
     quick.addSteps(setup_steps)
     quick.addStep(shell.shellCommand(command="make quick"))

   The rest of this section lists all the standard BuildStep objects
available for use in a Build, and the parameters which can be used to
control each.

* Menu:

* Common Parameters::
* Using Build Properties::
* Source Checkout::
* ShellCommand::
* Simple ShellCommand Subclasses::
* Python BuildSteps::
* Transferring Files::
* Steps That Run on the Master::
* Triggering Schedulers::
* Writing New BuildSteps::


File: buildbot.info,  Node: Common Parameters,  Next: Using Build Properties,  Prev: Build Steps,  Up: Build Steps

6.1.1 Common Parameters
-----------------------

The standard `Build' runs a series of `BuildStep's in order, only
stopping when it runs out of steps or if one of them requests that
the build be halted. It collects status information from each one to
create an overall build status (of SUCCESS, WARNINGS, or FAILURE).

   All BuildSteps accept some common parameters. Some of these control
how their individual status affects the overall build. Others are used
to specify which `Locks' (see *note Interlocks::) should be acquired
before allowing the step to run.

   Arguments common to all `BuildStep' subclasses:

`name'
     the name used to describe the step on the status display. It is
     also used to give a name to any LogFiles created by this step.

`haltOnFailure'
     if True, a FAILURE of this build step will cause the build to
     halt immediately. Steps with `alwaysRun=True' are still run.
     Generally speaking, haltOnFailure implies flunkOnFailure (the
     default for most BuildSteps). In some cases, particularly series
     of tests, it makes sense to haltOnFailure if something fails
     early on but not flunkOnFailure.  This can be achieved with
     haltOnFailure=True, flunkOnFailure=False.

`flunkOnWarnings'
     when True, a WARNINGS or FAILURE of this build step will mark the
     overall build as FAILURE. The remaining steps will still be
     executed.

`flunkOnFailure'
     when True, a FAILURE of this build step will mark the overall
     build as a FAILURE. The remaining steps will still be executed.

`warnOnWarnings'
     when True, a WARNINGS or FAILURE of this build step will mark the
     overall build as having WARNINGS. The remaining steps will still
     be executed.

`warnOnFailure'
     when True, a FAILURE of this build step will mark the overall
     build as having WARNINGS. The remaining steps will still be
     executed.

`alwaysRun'
     if True, this build step will always be run, even if a previous
     buildstep with `haltOnFailure=True' has failed.

`locks'
     a list of Locks (instances of `buildbot.locks.SlaveLock' or
     `buildbot.locks.MasterLock') that should be acquired before
     starting this Step. The Locks will be released when the step is
     complete. Note that this is a list of actual Lock instances, not
     names. Also note that all Locks must have unique names.



File: buildbot.info,  Node: Using Build Properties,  Next: Source Checkout,  Prev: Common Parameters,  Up: Build Steps

6.1.2 Using Build Properties
----------------------------

Build properties are a generalized way to provide configuration
information to build steps; see *note Build Properties::.

   Some build properties are inherited from external sources - global
properties, schedulers, or buildslaves.  Some build properties are
set when the build starts, such as the SourceStamp information. Other
properties can be set by BuildSteps as they run, for example the
various Source steps will set the `got_revision' property to the
source revision that was actually checked out (which can be useful
when the SourceStamp in use merely requested the "latest revision":
`got_revision' will tell you what was actually built).

   In custom BuildSteps, you can get and set the build properties with
the `getProperty'/`setProperty' methods. Each takes a string for the
name of the property, and returns or accepts an arbitrary(1) object.
For example:

     class MakeTarball(ShellCommand):
         def start(self):
             if self.getProperty("os") == "win":
                 self.setCommand([ ... ]) # windows-only command
             else:
                 self.setCommand([ ... ]) # equivalent for other systems
             ShellCommand.start(self)

WithProperties
==============

You can use build properties in ShellCommands by using the
`WithProperties' wrapper when setting the arguments of the
ShellCommand. This interpolates the named build properties into the
generated shell command.  Most step parameters accept
`WithProperties'.  Please file bugs for any parameters which do not.

     from buildbot.steps.shell import ShellCommand
     from buildbot.process.properties import WithProperties

     f.addStep(ShellCommand(
               command=["tar", "czf",
                        WithProperties("build-%s.tar.gz", "revision"),
                        "source"]))

   If this BuildStep were used in a tree obtained from Subversion, it
would create a tarball with a name like `build-1234.tar.gz'.

   The `WithProperties' function does `printf'-style string
interpolation, using strings obtained by calling
`build.getProperty(propname)'. Note that for every `%s' (or `%d',
etc), you must have exactly one additional argument to indicate which
build property you want to insert.

   You can also use python dictionary-style string interpolation by
using the `%(propname)s' syntax. In this form, the property name goes
in the parentheses, and WithProperties takes _no_ additional
arguments:

     f.addStep(ShellCommand(
               command=["tar", "czf",
                        WithProperties("build-%(revision)s.tar.gz"),
                        "source"]))

   Don't forget the extra "s" after the closing parenthesis! This is
the cause of many confusing errors.

   The dictionary-style interpolation supports a number of more
advanced syntaxes, too.

`propname:-replacement'
     If `propname' exists, substitute its value; otherwise,
     substitute `replacement'.  `replacement' may be empty
     (`%(propname:-)s')

`propname:+replacement'
     If `propname' exists, substitute `replacement'; otherwise,
     substitute an empty string.


   Although these are similar to shell substitutions, no other
substitutions are currently supported, and `replacement' in the above
cannot contain more substitutions.

   Note: like python, you can either do positional-argument
interpolation _or_ keyword-argument interpolation, not both. Thus you
cannot use a string like `WithProperties("foo-%(revision)s-%s",
"branch")'.

Common Build Properties
=======================

The following build properties are set when the build is started, and
are available to all steps.

`branch'
     This comes from the build's SourceStamp, and describes which
     branch is being checked out. This will be `None' (which
     interpolates into `WithProperties' as an empty string) if the
     build is on the default branch, which is generally the trunk.
     Otherwise it will be a string like "branches/beta1.4". The exact
     syntax depends upon the VC system being used.

`revision'
     This also comes from the SourceStamp, and is the revision of the
     source code tree that was requested from the VC system. When a
     build is requested of a specific revision (as is generally the
     case when the build is triggered by Changes), this will contain
     the revision specification. This is always a string, although
     the syntax depends upon the VC system in use: for SVN it is an
     integer, for Mercurial it is a short string, for Darcs it is a
     rather large string, etc.

     If the "force build" button was pressed, the revision will be
     `None', which means to use the most recent revision available.
     This is a "trunk build". This will be interpolated as an empty
     string.

`got_revision'
     This is set when a Source step checks out the source tree, and
     provides the revision that was actually obtained from the VC
     system.  In general this should be the same as `revision',
     except for trunk builds, where `got_revision' indicates what
     revision was current when the checkout was performed. This can
     be used to rebuild the same source code later.

     Note that for some VC systems (Darcs in particular), the
     revision is a large string containing newlines, and is not
     suitable for interpolation into a filename.

`buildername'
     This is a string that indicates which Builder the build was a
     part of.  The combination of buildername and buildnumber
     uniquely identify a build.

`buildnumber'
     Each build gets a number, scoped to the Builder (so the first
     build performed on any given Builder will have a build number of
     0). This integer property contains the build's number.

`slavename'
     This is a string which identifies which buildslave the build is
     running on.

`scheduler'
     If the build was started from a scheduler, then this property
     will contain the name of that scheduler.


   ---------- Footnotes ----------

   (1) Build properties are serialized along with the build results,
so they must be serializable. For this reason, the value of any build
property should be simple inert data: strings, numbers, lists,
tuples, and dictionaries. They should not contain class instances.


File: buildbot.info,  Node: Source Checkout,  Next: ShellCommand,  Prev: Using Build Properties,  Up: Build Steps

6.1.3 Source Checkout
---------------------

The first step of any build is typically to acquire the source code
from which the build will be performed. There are several classes to
handle this, one for each of the different source control system that
Buildbot knows about. For a description of how Buildbot treats source
control in general, see *note Version Control Systems::.

   All source checkout steps accept some common parameters to control
how they get the sources and where they should be placed. The
remaining per-VC-system parameters are mostly to specify where
exactly the sources are coming from.

`mode'
     a string describing the kind of VC operation that is desired.
     Defaults to `update'.

    `update'
          specifies that the CVS checkout/update should be performed
          directly into the workdir. Each build is performed in the
          same directory, allowing for incremental builds. This
          minimizes disk space, bandwidth, and CPU time. However, it
          may encounter problems if the build process does not handle
          dependencies properly (sometimes you must do a "clean
          build" to make sure everything gets compiled), or if source
          files are deleted but generated files can influence test
          behavior (e.g. python's .pyc files), or when source
          directories are deleted but generated files prevent CVS
          from removing them. Builds ought to be correct regardless
          of whether they are done "from scratch" or incrementally,
          but it is useful to test both kinds: this mode exercises the
          incremental-build style.

    `copy'
          specifies that the CVS workspace should be maintained in a
          separate directory (called the 'copydir'), using checkout
          or update as necessary. For each build, a new workdir is
          created with a copy of the source tree (rm -rf workdir; cp
          -r copydir workdir). This doubles the disk space required,
          but keeps the bandwidth low (update instead of a full
          checkout). A full 'clean' build is performed each time. This
          avoids any generated-file build problems, but is still
          occasionally vulnerable to CVS problems such as a
          repository being manually rearranged, causing CVS errors on
          update which are not an issue with a full checkout.

    `clobber'
          specifes that the working directory should be deleted each
          time, necessitating a full checkout for each build. This
          insures a clean build off a complete checkout, avoiding any
          of the problems described above. This mode exercises the
          "from-scratch" build style.

    `export'
          this is like `clobber', except that the 'cvs export'
          command is used to create the working directory. This
          command removes all CVS metadata files (the CVS/
          directories) from the tree, which is sometimes useful for
          creating source tarballs (to avoid including the metadata
          in the tar file).

`workdir'
     like all Steps, this indicates the directory where the build
     will take place. Source Steps are special in that they perform
     some operations outside of the workdir (like creating the
     workdir itself).

`alwaysUseLatest'
     if True, bypass the usual "update to the last Change" behavior,
     and always update to the latest changes instead.

`retry'
     If set, this specifies a tuple of `(delay, repeats)' which means
     that when a full VC checkout fails, it should be retried up to
     REPEATS times, waiting DELAY seconds between attempts. If you
     don't provide this, it defaults to `None', which means VC
     operations should not be retried. This is provided to make life
     easier for buildslaves which are stuck behind poor network
     connections.


   My habit as a developer is to do a `cvs update' and `make' each
morning. Problems can occur, either because of bad code being checked
in, or by incomplete dependencies causing a partial rebuild to fail
where a complete from-scratch build might succeed. A quick Builder
which emulates this incremental-build behavior would use the
`mode='update'' setting.

   On the other hand, other kinds of dependency problems can cause a
clean build to fail where a partial build might succeed. This
frequently results from a link step that depends upon an object file
that was removed from a later version of the tree: in the partial
tree, the object file is still around (even though the Makefiles no
longer know how to create it).

   "official" builds (traceable builds performed from a known set of
source revisions) are always done as clean builds, to make sure it is
not influenced by any uncontrolled factors (like leftover files from a
previous build). A "full" Builder which behaves this way would want
to use the `mode='clobber'' setting.

   Each VC system has a corresponding source checkout class: their
arguments are described on the following pages.

* Menu:

* CVS::
* SVN::
* Darcs::
* Mercurial::
* Arch::
* Bazaar::
* Bzr::
* P4::
* Git::


File: buildbot.info,  Node: CVS,  Next: SVN,  Prev: Source Checkout,  Up: Source Checkout

6.1.3.1 CVS
...........

The `CVS' build step performs a CVS (http://www.nongnu.org/cvs/)
checkout or update. It takes the following arguments:

`cvsroot'
     (required): specify the CVSROOT value, which points to a CVS
     repository, probably on a remote machine. For example, the
     cvsroot value you would use to get a copy of the Buildbot source
     code is
     `:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot'

`cvsmodule'
     (required): specify the cvs `module', which is generally a
     subdirectory of the CVSROOT. The cvsmodule for the Buildbot
     source code is `buildbot'.

`branch'
     a string which will be used in a `-r' argument. This is most
     useful for specifying a branch to work on. Defaults to `HEAD'.

`global_options'
     a list of flags to be put before the verb in the CVS command.

`checkoutDelay'
     if set, the number of seconds to put between the timestamp of
     the last known Change and the value used for the `-D' option.
     Defaults to half of the parent Build's treeStableTimer.



File: buildbot.info,  Node: SVN,  Next: Darcs,  Prev: CVS,  Up: Source Checkout

6.1.3.2 SVN
...........

The `SVN' build step performs a Subversion
(http://subversion.tigris.org) checkout or update.  There are two
basic ways of setting up the checkout step, depending upon whether
you are using multiple branches or not.

   If all of your builds use the same branch, then you should create
the `SVN' step with the `svnurl' argument:

`svnurl'
     (required): this specifies the `URL' argument that will be given
     to the `svn checkout' command. It dictates both where the
     repository is located and which sub-tree should be extracted. In
     this respect, it is like a combination of the CVS `cvsroot' and
     `cvsmodule' arguments. For example, if you are using a remote
     Subversion repository which is accessible through HTTP at a URL
     of `http://svn.example.com/repos', and you wanted to check out
     the `trunk/calc' sub-tree, you would use
     `svnurl="http://svn.example.com/repos/trunk/calc"' as an argument
     to your `SVN' step.

   If, on the other hand, you are building from multiple branches,
then you should create the `SVN' step with the `baseURL' and
`defaultBranch' arguments instead:

`baseURL'
     (required): this specifies the base repository URL, to which a
     branch name will be appended. It should probably end in a slash.

`defaultBranch'
     this specifies the name of the branch to use when a Build does
     not provide one of its own. This will be appended to `baseURL' to
     create the string that will be passed to the `svn checkout'
     command.

`username'
     if specified, this will be passed to the `svn' binary with a
     `--username' option.

`password'
     if specified, this will be passed to the `svn' binary with a
     `--password' option.  The password itself will be suitably
     obfuscated in the logs.


   If you are using branches, you must also make sure your
`ChangeSource' will report the correct branch names.

branch example
==============

Let's suppose that the "MyProject" repository uses branches for the
trunk, for various users' individual development efforts, and for
several new features that will require some amount of work (involving
multiple developers) before they are ready to merge onto the trunk.
Such a repository might be organized as follows:

     svn://svn.example.org/MyProject/trunk
     svn://svn.example.org/MyProject/branches/User1/foo
     svn://svn.example.org/MyProject/branches/User1/bar
     svn://svn.example.org/MyProject/branches/User2/baz
     svn://svn.example.org/MyProject/features/newthing
     svn://svn.example.org/MyProject/features/otherthing

   Further assume that we want the Buildbot to run tests against the
trunk and against all the feature branches (i.e., do a
checkout/compile/build of branch X when a file has been changed on
branch X, when X is in the set [trunk, features/newthing,
features/otherthing]). We do not want the Buildbot to automatically
build any of the user branches, but it should be willing to build a
user branch when explicitly requested (most likely by the user who
owns that branch).

   There are three things that need to be set up to accomodate this
system. The first is a ChangeSource that is capable of identifying the
branch which owns any given file. This depends upon a user-supplied
function, in an external program that runs in the SVN commit hook and
connects to the buildmaster's `PBChangeSource' over a TCP connection.
(you can use the "`buildbot sendchange'" utility for this purpose,
but you will still need an external program to decide what value
should be passed to the `--branch=' argument).  For example, a change
to a file with the SVN url of
"svn://svn.example.org/MyProject/features/newthing/src/foo.c" should
be broken down into a Change instance with
`branch='features/newthing'' and `file='src/foo.c''.

   The second piece is an `AnyBranchScheduler' which will pay
attention to the desired branches. It will not pay attention to the
user branches, so it will not automatically start builds in response
to changes there. The AnyBranchScheduler class requires you to
explicitly list all the branches you want it to use, but it would not
be difficult to write a subclass which used
`branch.startswith('features/'' to remove the need for this explicit
list. Or, if you want to build user branches too, you can use
AnyBranchScheduler with `branches=None' to indicate that you want it
to pay attention to all branches.

   The third piece is an `SVN' checkout step that is configured to
handle the branches correctly, with a `baseURL' value that matches
the way the ChangeSource splits each file's URL into base, branch,
and file.

     from buildbot.changes.pb import PBChangeSource
     from buildbot.scheduler import AnyBranchScheduler
     from buildbot.process import source, factory
     from buildbot.steps import source, shell

     c['change_source'] = PBChangeSource()
     s1 = AnyBranchScheduler('main',
                             ['trunk', 'features/newthing', 'features/otherthing'],
                             10*60, ['test-i386', 'test-ppc'])
     c['schedulers'] = [s1]

     f = factory.BuildFactory()
     f.addStep(source.SVN(mode='update',
                          baseURL='svn://svn.example.org/MyProject/',
                          defaultBranch='trunk'))
     f.addStep(shell.Compile(command="make all"))
     f.addStep(shell.Test(command="make test"))

     c['builders'] = [
       {'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386',
                            'factory':f },
       {'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc',
                           'factory':f },
      ]

   In this example, when a change arrives with a `branch' attribute
of "trunk", the resulting build will have an SVN step that
concatenates "svn://svn.example.org/MyProject/" (the baseURL) with
"trunk" (the branch name) to get the correct svn command. If the
"newthing" branch has a change to "src/foo.c", then the SVN step will
concatenate "svn://svn.example.org/MyProject/" with
"features/newthing" to get the svnurl for checkout.


File: buildbot.info,  Node: Darcs,  Next: Mercurial,  Prev: SVN,  Up: Source Checkout

6.1.3.3 Darcs
.............

The `Darcs' build step performs a Darcs (http://darcs.net/) checkout
or update.

   Like *Note SVN::, this step can either be configured to always
check out a specific tree, or set up to pull from a particular branch
that gets specified separately for each build. Also like SVN, the
repository URL given to Darcs is created by concatenating a `baseURL'
with the branch name, and if no particular branch is requested, it
uses a `defaultBranch'. The only difference in usage is that each
potential Darcs repository URL must point to a fully-fledged
repository, whereas SVN URLs usually point to sub-trees of the main
Subversion repository. In other words, doing an SVN checkout of
`baseURL' is legal, but silly, since you'd probably wind up with a
copy of every single branch in the whole repository.  Doing a Darcs
checkout of `baseURL' is just plain wrong, since the parent directory
of a collection of Darcs repositories is not itself a valid
repository.

   The Darcs step takes the following arguments:

`repourl'
     (required unless `baseURL' is provided): the URL at which the
     Darcs source repository is available.

`baseURL'
     (required unless `repourl' is provided): the base repository URL,
     to which a branch name will be appended. It should probably end
     in a slash.

`defaultBranch'
     (allowed if and only if `baseURL' is provided): this specifies
     the name of the branch to use when a Build does not provide one
     of its own. This will be appended to `baseURL' to create the
     string that will be passed to the `darcs get' command.


File: buildbot.info,  Node: Mercurial,  Next: Arch,  Prev: Darcs,  Up: Source Checkout

6.1.3.4 Mercurial
.................

The `Mercurial' build step performs a Mercurial
(http://selenic.com/mercurial) (aka "hg") checkout or update.

   Branches are handled just like *Note Darcs::.

   The Mercurial step takes the following arguments:

`repourl'
     (required unless `baseURL' is provided): the URL at which the
     Mercurial source repository is available.

`baseURL'
     (required unless `repourl' is provided): the base repository URL,
     to which a branch name will be appended. It should probably end
     in a slash.

`defaultBranch'
     (allowed if and only if `baseURL' is provided): this specifies
     the name of the branch to use when a Build does not provide one
     of its own. This will be appended to `baseURL' to create the
     string that will be passed to the `hg clone' command.


File: buildbot.info,  Node: Arch,  Next: Bazaar,  Prev: Mercurial,  Up: Source Checkout

6.1.3.5 Arch
............

The `Arch' build step performs an Arch (http://gnuarch.org/) checkout
or update using the `tla' client. It takes the following arguments:

`url'
     (required): this specifies the URL at which the Arch source
     archive is available.

`version'
     (required): this specifies which "development line" (like a
     branch) should be used. This provides the default branch name,
     but individual builds may specify a different one.

`archive'
     (optional): Each repository knows its own archive name. If this
     parameter is provided, it must match the repository's archive
     name.  The parameter is accepted for compatibility with the
     `Bazaar' step, below.



File: buildbot.info,  Node: Bazaar,  Next: Bzr,  Prev: Arch,  Up: Source Checkout

6.1.3.6 Bazaar
..............

`Bazaar' is an alternate implementation of the Arch VC system, which
uses a client named `baz'. The checkout semantics are just different
enough from `tla' that there is a separate BuildStep for it.

   It takes exactly the same arguments as `Arch', except that the
`archive=' parameter is required. (baz does not emit the archive name
when you do `baz register-archive', so we must provide it ourselves).


File: buildbot.info,  Node: Bzr,  Next: P4,  Prev: Bazaar,  Up: Source Checkout

6.1.3.7 Bzr
...........

`bzr' is a descendant of Arch/Baz, and is frequently referred to as
simply "Bazaar". The repository-vs-workspace model is similar to
Darcs, but it uses a strictly linear sequence of revisions (one
history per branch) like Arch. Branches are put in subdirectories.
This makes it look very much like Mercurial, so it takes the same
arguments:

`repourl'
     (required unless `baseURL' is provided): the URL at which the
     Bzr source repository is available.

`baseURL'
     (required unless `repourl' is provided): the base repository URL,
     to which a branch name will be appended. It should probably end
     in a slash.

`defaultBranch'
     (allowed if and only if `baseURL' is provided): this specifies
     the name of the branch to use when a Build does not provide one
     of its own. This will be appended to `baseURL' to create the
     string that will be passed to the `bzr checkout' command.


File: buildbot.info,  Node: P4,  Next: Git,  Prev: Bzr,  Up: Source Checkout

6.1.3.8 P4
..........

The `P4' build step creates a Perforce (http://www.perforce.com/)
client specification and performs an update.

`p4base'
     A view into the Perforce depot without branch name or trailing
     "...".  Typically "//depot/proj/".

`defaultBranch'
     A branch name to append on build requests if none is specified.
     Typically "trunk".

`p4port'
     (optional): the host:port string describing how to get to the P4
     Depot (repository), used as the -p argument for all p4 commands.

`p4user'
     (optional): the Perforce user, used as the -u argument to all p4
     commands.

`p4passwd'
     (optional): the Perforce password, used as the -p argument to
     all p4 commands.

`p4extra_views'
     (optional): a list of (depotpath, clientpath) tuples containing
     extra views to be mapped into the client specification. Both
     will have "/..." appended automatically. The client name and
     source directory will be prepended to the client path.

`p4client'
     (optional): The name of the client to use. In mode='copy' and
     mode='update', it's particularly important that a unique name is
     used for each checkout directory to avoid incorrect
     synchronization. For this reason, Python percent substitution
     will be performed on this value to replace %(slave)s with the
     slave name and %(builder)s with the builder name. The default is
     "buildbot_%(slave)s_%(build)s".


File: buildbot.info,  Node: Git,  Prev: P4,  Up: Source Checkout

6.1.3.9 Git
...........

The `Git' build step clones or updates a Git (http://git.or.cz/)
repository and checks out the specified branch or revision. Note that
the buildbot supports Git version 1.2.0 and later: earlier versions
(such as the one shipped in Ubuntu 'Dapper') do not support the `git
init' command that the buildbot uses.

   The Git step takes the following arguments:

`repourl'
     (required): the URL of the upstream Git repository.

`branch'
     (optional): this specifies the name of the branch to use when a
     Build does not provide one of its own. If this this parameter is
     not specified, and the Build does not provide a branch, the
     "master" branch will be used.


File: buildbot.info,  Node: ShellCommand,  Next: Simple ShellCommand Subclasses,  Prev: Source Checkout,  Up: Build Steps

6.1.4 ShellCommand
------------------

This is a useful base class for just about everything you might want
to do during a build (except for the initial source checkout). It runs
a single command in a child shell on the buildslave. All stdout/stderr
is recorded into a LogFile. The step finishes with a status of FAILURE
if the command's exit code is non-zero, otherwise it has a status of
SUCCESS.

   The preferred way to specify the command is with a list of argv
strings, since this allows for spaces in filenames and avoids doing
any fragile shell-escaping. You can also specify the command with a
single string, in which case the string is given to '/bin/sh -c
COMMAND' for parsing.

   On Windows, commands are run via `cmd.exe /c' which works well.
However, if you're running a batch file, the error level does not get
propagated correctly unless you add 'call' before your batch file's
name: `cmd=['call', 'myfile.bat', ...]'.

   All ShellCommands are run by default in the "workdir", which
defaults to the "`build'" subdirectory of the slave builder's base
directory. The absolute path of the workdir will thus be the slave's
basedir (set as an option to `buildbot create-slave', *note Creating
a buildslave::) plus the builder's basedir (set in the builder's
`c['builddir']' key in master.cfg) plus the workdir itself (a
class-level attribute of the BuildFactory, defaults to "`build'").

   `ShellCommand' arguments:

`command'
     a list of strings (preferred) or single string (discouraged)
     which specifies the command to be run. A list of strings is
     preferred because it can be used directly as an argv array.
     Using a single string (with embedded spaces) requires the
     buildslave to pass the string to /bin/sh for interpretation,
     which raises all sorts of difficult questions about how to
     escape or interpret shell metacharacters.

`env'
     a dictionary of environment strings which will be added to the
     child command's environment. For example, to run tests with a
     different i18n language setting, you might use

          f.addStep(ShellCommand(command=["make", "test"],
                                 env={'LANG': 'fr_FR'}))

     These variable settings will override any existing ones in the
     buildslave's environment or the environment specified in the
     Builder. The exception is PYTHONPATH, which is merged with
     (actually prepended to) any existing $PYTHONPATH setting. The
     value is treated as a list of directories to prepend, and a
     single string is treated like a one-item list. For example, to
     prepend both `/usr/local/lib/python2.3' and
     `/home/buildbot/lib/python' to any existing $PYTHONPATH setting,
     you would do something like the following:

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        env={'PYTHONPATH': ["/usr/local/lib/python2.3",
                                             "/home/buildbot/lib/python"] }))

`want_stdout'
     if False, stdout from the child process is discarded rather than
     being sent to the buildmaster for inclusion in the step's
     LogFile.

`want_stderr'
     like `want_stdout' but for stderr. Note that commands run through
     a PTY do not have separate stdout/stderr streams: both are
     merged into stdout.

`usePTY'
     Should this command be run in a `pty'?  The default is to
     observe the configuration of the client (*note Buildslave
     Options::), but specifying `True' or `False' here will override
     the default.

     The advantage of using a PTY is that "grandchild" processes are
     more likely to be cleaned up if the build is interrupted or
     times out (since it enables the use of a "process group" in
     which all child processes will be placed). The disadvantages:
     some forms of Unix have problems with PTYs, some of your unit
     tests may behave differently when run under a PTY (generally
     those which check to see if they are being run interactively),
     and PTYs will merge the stdout and stderr streams into a single
     output stream (which means the red-vs-black coloring in the
     logfiles will be lost).

`logfiles'
     Sometimes commands will log interesting data to a local file,
     rather than emitting everything to stdout or stderr. For
     example, Twisted's "trial" command (which runs unit tests) only
     presents summary information to stdout, and puts the rest into a
     file named `_trial_temp/test.log'. It is often useful to watch
     these files as the command runs, rather than using `/bin/cat' to
     dump their contents afterwards.

     The `logfiles=' argument allows you to collect data from these
     secondary logfiles in near-real-time, as the step is running. It
     accepts a dictionary which maps from a local Log name (which is
     how the log data is presented in the build results) to a remote
     filename (interpreted relative to the build's working
     directory). Each named file will be polled on a regular basis
     (every couple of seconds) as the build runs, and any new text
     will be sent over to the buildmaster.

          f.addStep(ShellCommand(
                        command=["make", "test"],
                        logfiles={"triallog": "_trial_temp/test.log"}))

`timeout'
     if the command fails to produce any output for this many
     seconds, it is assumed to be locked up and will be killed.

`description'
     This will be used to describe the command (on the Waterfall
     display) while the command is still running. It should be a
     single imperfect-tense verb, like "compiling" or "testing". The
     preferred form is a list of short strings, which allows the HTML
     Waterfall display to create narrower columns by emitting a <br>
     tag between each word. You may also provide a single string.

`descriptionDone'
     This will be used to describe the command once it has finished. A
     simple noun like "compile" or "tests" should be used. Like
     `description', this may either be a list of short strings or a
     single string.

     If neither `description' nor `descriptionDone' are set, the
     actual command arguments will be used to construct the
     description.  This may be a bit too wide to fit comfortably on
     the Waterfall display.

          f.addStep(ShellCommand(command=["make", "test"],
                                 description=["testing"],
                                 descriptionDone=["tests"]))

`logEnviron'
     If this option is true (the default), then the step's logfile
     will describe the environment variables on the slave.  In
     situations where the environment is not relevant and is long, it
     may be easier to set `logEnviron=False'.



File: buildbot.info,  Node: Simple ShellCommand Subclasses,  Next: Python BuildSteps,  Prev: ShellCommand,  Up: Build Steps

6.1.5 Simple ShellCommand Subclasses
------------------------------------

Several subclasses of ShellCommand are provided as starting points for
common build steps. These are all very simple: they just override a
few parameters so you don't have to specify them yourself, making the
master.cfg file less verbose.

* Menu:

* Configure::
* Compile::
* Test::
* TreeSize::
* PerlModuleTest::
* SetProperty::


File: buildbot.info,  Node: Configure,  Next: Compile,  Prev: Simple ShellCommand Subclasses,  Up: Simple ShellCommand Subclasses

6.1.5.1 Configure
.................

This is intended to handle the `./configure' step from autoconf-style
projects, or the `perl Makefile.PL' step from perl MakeMaker.pm-style
modules. The default command is `./configure' but you can change this
by providing a `command=' parameter.


File: buildbot.info,  Node: Compile,  Next: Test,  Prev: Configure,  Up: Simple ShellCommand Subclasses

6.1.5.2 Compile
...............

This is meant to handle compiling or building a project written in C.
The default command is `make all'. When the compile is finished, the
log file is scanned for GCC warning messages, a summary log is
created with any problems that were seen, and the step is marked as
WARNINGS if any were discovered. The number of warnings is stored in a
Build Property named "warnings-count", which is accumulated over all
Compile steps (so if two warnings are found in one step, and three are
found in another step, the overall build will have a "warnings-count"
property of 5.

   The default regular expression used to detect a warning is
`'.*warning[: ].*'' , which is fairly liberal and may cause
false-positives. To use a different regexp, provide a
`warningPattern=' argument, or use a subclass which sets the
`warningPattern' attribute:

     f.addStep(Compile(command=["make", "test"],
                       warningPattern="^Warning: "))

   The `warningPattern=' can also be a pre-compiled python regexp
object: this makes it possible to add flags like `re.I' (to use
case-insensitive matching).

   (TODO: this step needs to be extended to look for GCC error
messages as well, and collect them into a separate logfile, along
with the source code filenames involved).


File: buildbot.info,  Node: Test,  Next: TreeSize,  Prev: Compile,  Up: Simple ShellCommand Subclasses

6.1.5.3 Test
............

This is meant to handle unit tests. The default command is `make
test', and the `warnOnFailure' flag is set.


File: buildbot.info,  Node: TreeSize,  Next: PerlModuleTest,  Prev: Test,  Up: Simple ShellCommand Subclasses

6.1.5.4 TreeSize
................

This is a simple command that uses the 'du' tool to measure the size
of the code tree. It puts the size (as a count of 1024-byte blocks,
aka 'KiB' or 'kibibytes') on the step's status text, and sets a build
property named 'tree-size-KiB' with the same value.


File: buildbot.info,  Node: PerlModuleTest,  Next: SetProperty,  Prev: TreeSize,  Up: Simple ShellCommand Subclasses

6.1.5.5 PerlModuleTest
......................

This is a simple command that knows how to run tests of perl modules.
It parses the output to determine the number of tests passed and
failed and total number executed, saving the results for later query.


File: buildbot.info,  Node: SetProperty,  Prev: PerlModuleTest,  Up: Simple ShellCommand Subclasses

6.1.5.6 SetProperty
...................

This buildstep is similar to ShellCommand, except that it captures the
output of the command into a property.  It is usually used like this:

     f.addStep(SetProperty(command="uname -a", property="uname"))

   This runs `uname -a' and captures its stdout, stripped of leading
and trailing whitespace, in the property "uname".  To avoid stripping,
add `strip=False'.  The `property' argument can be specified as a
`WithProperties' object.

   The more advanced usage allows you to specify a function to extract
properties from the command output.  Here you can use regular
expressions, string interpolation, or whatever you would like.  The
function is called with three arguments: the exit status of the
command, its standard output as a string, and its standard error as a
string.  It should return a dictionary containing all new properties.

     def glob2list(rc, stdout, stderr):
         jpgs = [ l.strip() for l in stdout.split('\n') ]
         return { 'jpgs' : jpgs }
     f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list))

   Note that any ordering relationship of the contents of stdout and
stderr is lost.  For example, given

     f.addStep(SetProperty(
         command="echo output1; echo error >&2; echo output2",
         extract_fn=my_extract))

   Then `my_extract' will see `stdout="output1\noutput2\n"' and
`stderr="error\n"'.


File: buildbot.info,  Node: Python BuildSteps,  Next: Transferring Files,  Prev: Simple ShellCommand Subclasses,  Up: Build Steps

6.1.6 Python BuildSteps
-----------------------

Here are some BuildSteps that are specifcally useful for projects
implemented in Python.

* Menu:

* BuildEPYDoc::
* PyFlakes::
* PyLint::


File: buildbot.info,  Node: BuildEPYDoc,  Next: PyFlakes,  Up: Python BuildSteps

6.1.6.1 BuildEPYDoc
...................

epydoc (http://epydoc.sourceforge.net/) is a tool for generating API
documentation for Python modules from their docstrings. It reads all
the .py files from your source tree, processes the docstrings
therein, and creates a large tree of .html files (or a single .pdf
file).

   The `buildbot.steps.python.BuildEPYDoc' step will run `epydoc' to
produce this API documentation, and will count the errors and
warnings from its output.

   You must supply the command line to be used. The default is `make
epydocs', which assumes that your project has a Makefile with an
"epydocs" target. You might wish to use something like `epydoc -o
apiref source/PKGNAME' instead. You might also want to add `--pdf' to
generate a PDF file instead of a large tree of HTML files.

   The API docs are generated in-place in the build tree (under the
workdir, in the subdirectory controlled by the "-o" argument). To
make them useful, you will probably have to copy them to somewhere
they can be read. A command like `rsync -ad apiref/
dev.example.com:~public_html/current-apiref/' might be useful. You
might instead want to bundle them into a tarball and publish it in the
same place where the generated install tarball is placed.

     from buildbot.steps.python import BuildEPYDoc

     ...
     f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"]))


File: buildbot.info,  Node: PyFlakes,  Next: PyLint,  Prev: BuildEPYDoc,  Up: Python BuildSteps

6.1.6.2 PyFlakes
................

PyFlakes (http://divmod.org/trac/wiki/DivmodPyflakes) is a tool to
perform basic static analysis of Python code to look for simple
errors, like missing imports and references of undefined names. It is
like a fast and simple form of the C "lint" program. Other tools
(like pychecker) provide more detailed results but take longer to run.

   The `buildbot.steps.python.PyFlakes' step will run pyflakes and
count the various kinds of errors and warnings it detects.

   You must supply the command line to be used. The default is `make
pyflakes', which assumes you have a top-level Makefile with a
"pyflakes" target. You might want to use something like `pyflakes .'
or `pyflakes src'.

     from buildbot.steps.python import PyFlakes

     ...
     f.addStep(PyFlakes(command=["pyflakes", "src"]))


File: buildbot.info,  Node: PyLint,  Prev: PyFlakes,  Up: Python BuildSteps

6.1.6.3 PyLint
..............

Similarly, the `buildbot.steps.python.PyLint' step will run pylint and
analyze the results.

   You must supply the command line to be used. There is no default.

     from buildbot.steps.python import PyLint

     ...
     f.addStep(PyLint(command=["pylint", "src"]))


File: buildbot.info,  Node: Transferring Files,  Next: Steps That Run on the Master,  Prev: Python BuildSteps,  Up: Build Steps

6.1.7 Transferring Files
------------------------

Most of the work involved in a build will take place on the
buildslave. But occasionally it is useful to do some work on the
buildmaster side. The most basic way to involve the buildmaster is
simply to move a file from the slave to the master, or vice versa.
There are a pair of BuildSteps named `FileUpload' and `FileDownload'
to provide this functionality. `FileUpload' moves a file _up to_ the
master, while `FileDownload' moves a file _down from_ the master.

   As an example, let's assume that there is a step which produces an
HTML file within the source tree that contains some sort of generated
project documentation. We want to move this file to the buildmaster,
into a `~/public_html' directory, so it can be visible to developers.
This file will wind up in the slave-side working directory under the
name `docs/reference.html'. We want to put it into the master-side
`~/public_html/ref.html'.

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import FileUpload

     f.addStep(ShellCommand(command=["make", "docs"]))
     f.addStep(FileUpload(slavesrc="docs/reference.html",
                          masterdest="~/public_html/ref.html"))

   The `masterdest=' argument will be passed to os.path.expanduser,
so things like "~" will be expanded properly. Non-absolute paths will
be interpreted relative to the buildmaster's base directory.
Likewise, the `slavesrc=' argument will be expanded and interpreted
relative to the builder's working directory.

   To move a file from the master to the slave, use the
`FileDownload' command. For example, let's assume that some step
requires a configuration file that, for whatever reason, could not be
recorded in the source code repository or generated on the buildslave
side:

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import FileUpload

     f.addStep(FileDownload(mastersrc="~/todays_build_config.txt",
                            slavedest="build_config.txt"))
     f.addStep(ShellCommand(command=["make", "config"]))

   Like `FileUpload', the `mastersrc=' argument is interpreted
relative to the buildmaster's base directory, and the `slavedest='
argument is relative to the builder's working directory. If the
buildslave is running in `~buildslave', and the builder's "builddir"
is something like `tests-i386', then the workdir is going to be
`~buildslave/tests-i386/build', and a `slavedest=' of `foo/bar.html'
will get put in `~buildslave/tests-i386/build/foo/bar.html'. Both of
these commands will create any missing intervening directories.

Other Parameters
----------------

The `maxsize=' argument lets you set a maximum size for the file to
be transferred. This may help to avoid surprises: transferring a
100MB coredump when you were expecting to move a 10kB status file
might take an awfully long time. The `blocksize=' argument controls
how the file is sent over the network: larger blocksizes are slightly
more efficient but also consume more memory on each end, and there is
a hard-coded limit of about 640kB.

   The `mode=' argument allows you to control the access permissions
of the target file, traditionally expressed as an octal integer. The
most common value is probably 0755, which sets the "x" executable bit
on the file (useful for shell scripts and the like). The default
value for `mode=' is None, which means the permission bits will
default to whatever the umask of the writing process is. The default
umask tends to be fairly restrictive, but at least on the buildslave
you can make it less restrictive with a -umask command-line option at
creation time (*note Buildslave Options::).

Transfering Directories
-----------------------

To transfer complete directories from the buildslave to the master,
there is a BuildStep named `DirectoryUpload'. It works like
`FileUpload', just for directories. However it does not support the
`maxsize', `blocksize' and `mode' arguments. As an example, let's
assume an generated project documentation, which consists of many
files (like the output of doxygen or epydoc). We want to move the
entire documentation to the buildmaster, into a `~/public_html/docs'
directory. On the slave-side the directory can be found under `docs':

     from buildbot.steps.shell import ShellCommand
     from buildbot.steps.transfer import DirectoryUpload

     f.addStep(ShellCommand(command=["make", "docs"]))
     f.addStep(DirectoryUpload(slavesrc="docs",
     				masterdest="~/public_html/docs"))

   The DirectoryUpload step will create all necessary directories and
transfers empty directories, too.


File: buildbot.info,  Node: Steps That Run on the Master,  Next: Triggering Schedulers,  Prev: Transferring Files,  Up: Build Steps

6.1.8 Steps That Run on the Master
----------------------------------

Occasionally, it is useful to execute some task on the master, for
example to create a directory, deploy a build result, or trigger some
other centralized processing.  This is possible, in a limited
fashion, with the `MasterShellCommand' step.

   This step operates similarly to a regular `ShellCommand', but
executes on the master, instead of the slave.  To be clear, the
enclosing `Build' object must still have a slave object, just as for
any other step - only, in this step, the slave does not do anything.

   In this example, the step renames a tarball based on the day of
the week.

     from buildbot.steps.transfer import FileUpload
     from buildbot.steps.master import MasterShellCommand

     f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz",
                          masterdest="/var/buildoutputs/widgetsoft-new.tar.gz"))
     f.addStep(MasterShellCommand(command="""
         cd /var/buildoutputs;
         mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz"""))


File: buildbot.info,  Node: Triggering Schedulers,  Next: Writing New BuildSteps,  Prev: Steps That Run on the Master,  Up: Build Steps

6.1.9 Triggering Schedulers
---------------------------

The counterpart to the Triggerable described in section *note
Triggerable Scheduler:: is the Trigger BuildStep.

     from buildbot.steps.trigger import Trigger
     f.addStep(Trigger(schedulerNames=['build-prep'],
                       waitForFinish=True,
                       updateSourceStamp=True))

   The `schedulerNames=' argument lists the Triggerables that should
be triggered when this step is executed.  Note that it is possible,
but not advisable, to create a cycle where a build continually
triggers itself, because the schedulers are specified by name.

   If `waitForFinish' is True, then the step will not finish until
all of the builds from the triggered schedulers have finished. If this
argument is False (the default) or not given, then the buildstep
succeeds immediately after triggering the schedulers.

   If `updateSourceStamp' is True (the default), then step updates
the SourceStamp given to the Triggerables to include `got_revision'
(the revision actually used in this build) as `revision' (the
revision to use in the triggered builds). This is useful to ensure
that all of the builds use exactly the same SourceStamp, even if
other Changes have occurred while the build was running.


File: buildbot.info,  Node: Writing New BuildSteps,  Prev: Triggering Schedulers,  Up: Build Steps

6.1.10 Writing New BuildSteps
-----------------------------

While it is a good idea to keep your build process self-contained in
the source code tree, sometimes it is convenient to put more
intelligence into your Buildbot configuration. One way to do this is
to write a custom BuildStep. Once written, this Step can be used in
the `master.cfg' file.

   The best reason for writing a custom BuildStep is to better parse
the results of the command being run. For example, a BuildStep that
knows about JUnit could look at the logfiles to determine which tests
had been run, how many passed and how many failed, and then report
more detailed information than a simple `rc==0' -based "good/bad"
decision.

* Menu:

* Writing BuildStep Constructors::
* BuildStep LogFiles::
* Reading Logfiles::
* Adding LogObservers::
* BuildStep URLs::


File: buildbot.info,  Node: Writing BuildStep Constructors,  Next: BuildStep LogFiles,  Up: Writing New BuildSteps

6.1.10.1 Writing BuildStep Constructors
.......................................

BuildStep classes have some extra equipment, because they are their
own factories.  Consider the use of a BuildStep in `master.cfg':

     f.addStep(MyStep(someopt="stuff", anotheropt=1))

   This creates a single instance of class `MyStep'.  However,
Buildbot needs a new object each time the step is executed.  this is
accomplished by storing the information required to instantiate a new
object in the `factory' attribute.  When the time comes to construct
a new Build, BuildFactory consults this attribute (via
`getStepFactory') and instantiates a new step object.

   When writing a new step class, then, keep in mind are that you
cannot do anything "interesting" in the constructor - limit yourself
to checking and storing arguments.  To ensure that these arguments
are provided to any new objects, call `self.addFactoryArguments' with
any keyword arguments your constructor needs.

   Keep a `**kwargs' argument on the end of your options, and pass
that up to the parent class's constructor.

   The whole thing looks like this:

     class Frobinfy(LoggingBuildStep):
         def __init__(self,
                 frob_what="frobee",
                 frob_how_many=None,
                 frob_how=None,
                 **kwargs)

             # check
             if frob_how_many is None:
                 raise TypeError("Frobinfy argument how_many is required")

             # call parent
             LoggingBuildStep.__init__(self, **kwargs)

             # and record arguments for later
             self.addFactoryArguments(
                 frob_what=frob_what,
                 frob_how_many=frob_how_many,
                 frob_how=frob_how)

     class FastFrobnify(Frobnify):
         def __init__(self,
                 speed=5,
                 **kwargs)
             Frobnify.__init__(self, **kwargs)
             self.addFactoryArguments(
                 speed=speed)


File: buildbot.info,  Node: BuildStep LogFiles,  Next: Reading Logfiles,  Prev: Writing BuildStep Constructors,  Up: Writing New BuildSteps

6.1.10.2 BuildStep LogFiles
...........................

Each BuildStep has a collection of "logfiles". Each one has a short
name, like "stdio" or "warnings". Each LogFile contains an arbitrary
amount of text, usually the contents of some output file generated
during a build or test step, or a record of everything that was
printed to stdout/stderr during the execution of some command.

   These LogFiles are stored to disk, so they can be retrieved later.

   Each can contain multiple "channels", generally limited to three
basic ones: stdout, stderr, and "headers". For example, when a
ShellCommand runs, it writes a few lines to the "headers" channel to
indicate the exact argv strings being run, which directory the command
is being executed in, and the contents of the current environment
variables. Then, as the command runs, it adds a lot of "stdout" and
"stderr" messages. When the command finishes, a final "header" line
is added with the exit code of the process.

   Status display plugins can format these different channels in
different ways. For example, the web page shows LogFiles as text/html,
with header lines in blue text, stdout in black, and stderr in red. A
different URL is available which provides a text/plain format, in
which stdout and stderr are collapsed together, and header lines are
stripped completely. This latter option makes it easy to save the
results to a file and run `grep' or whatever against the output.

   Each BuildStep contains a mapping (implemented in a python
dictionary) from LogFile name to the actual LogFile objects. Status
plugins can get a list of LogFiles to display, for example, a list of
HREF links that, when clicked, provide the full contents of the
LogFile.

Using LogFiles in custom BuildSteps
===================================

The most common way for a custom BuildStep to use a LogFile is to
summarize the results of a ShellCommand (after the command has
finished running). For example, a compile step with thousands of lines
of output might want to create a summary of just the warning messages.
If you were doing this from a shell, you would use something like:

     grep "warning:" output.log >warnings.log

   In a custom BuildStep, you could instead create a "warnings"
LogFile that contained the same text. To do this, you would add code
to your `createSummary' method that pulls lines from the main output
log and creates a new LogFile with the results:

         def createSummary(self, log):
             warnings = []
             for line in log.readlines():
                 if "warning:" in line:
                     warnings.append()
             self.addCompleteLog('warnings', "".join(warnings))

   This example uses the `addCompleteLog' method, which creates a new
LogFile, puts some text in it, and then "closes" it, meaning that no
further contents will be added. This LogFile will appear in the HTML
display under an HREF with the name "warnings", since that is the
name of the LogFile.

   You can also use `addHTMLLog' to create a complete (closed)
LogFile that contains HTML instead of plain text. The normal LogFile
will be HTML-escaped if presented through a web page, but the HTML
LogFile will not. At the moment this is only used to present a pretty
HTML representation of an otherwise ugly exception traceback when
something goes badly wrong during the BuildStep.

   In contrast, you might want to create a new LogFile at the
beginning of the step, and add text to it as the command runs. You
can create the LogFile and attach it to the build by calling
`addLog', which returns the LogFile object. You then add text to this
LogFile by calling methods like `addStdout' and `addHeader'. When you
are done, you must call the `finish' method so the LogFile can be
closed. It may be useful to create and populate a LogFile like this
from a LogObserver method *Note Adding LogObservers::.

   The `logfiles=' argument to `ShellCommand' (see *note
ShellCommand::) creates new LogFiles and fills them in realtime by
asking the buildslave to watch a actual file on disk. The buildslave
will look for additions in the target file and report them back to
the BuildStep. These additions will be added to the LogFile by
calling `addStdout'. These secondary LogFiles can be used as the
source of a LogObserver just like the normal "stdio" LogFile.


File: buildbot.info,  Node: Reading Logfiles,  Next: Adding LogObservers,  Prev: BuildStep LogFiles,  Up: Writing New BuildSteps

6.1.10.3 Reading Logfiles
.........................

Once a LogFile has been added to a BuildStep with `addLog()',
`addCompleteLog()', `addHTMLLog()', or `logfiles=', your BuildStep
can retrieve it by using `getLog()':

     class MyBuildStep(ShellCommand):
         logfiles = { "nodelog": "_test/node.log" }

         def evaluateCommand(self, cmd):
             nodelog = self.getLog("nodelog")
             if "STARTED" in nodelog.getText():
                 return SUCCESS
             else:
                 return FAILURE

   For a complete list of the methods you can call on a LogFile,
please see the docstrings on the `IStatusLog' class in
`buildbot/interfaces.py'.


File: buildbot.info,  Node: Adding LogObservers,  Next: BuildStep URLs,  Prev: Reading Logfiles,  Up: Writing New BuildSteps

6.1.10.4 Adding LogObservers
............................

Most shell commands emit messages to stdout or stderr as they operate,
especially if you ask them nicely with a `--verbose' flag of some
sort. They may also write text to a log file while they run. Your
BuildStep can watch this output as it arrives, to keep track of how
much progress the command has made. You can get a better measure of
progress by counting the number of source files compiled or test cases
run than by merely tracking the number of bytes that have been written
to stdout. This improves the accuracy and the smoothness of the ETA
display.

   To accomplish this, you will need to attach a `LogObserver' to one
of the log channels, most commonly to the "stdio" channel but perhaps
to another one which tracks a log file. This observer is given all
text as it is emitted from the command, and has the opportunity to
parse that output incrementally. Once the observer has decided that
some event has occurred (like a source file being compiled), it can
use the `setProgress' method to tell the BuildStep about the progress
that this event represents.

   There are a number of pre-built `LogObserver' classes that you can
choose from (defined in `buildbot.process.buildstep', and of course
you can subclass them to add further customization. The
`LogLineObserver' class handles the grunt work of buffering and
scanning for end-of-line delimiters, allowing your parser to operate
on complete stdout/stderr lines. (Lines longer than a set maximum
length are dropped; the maximum defaults to 16384 bytes, but you can
change it by calling `setMaxLineLength()' on your `LogLineObserver'
instance.  Use `sys.maxint' for effective infinity.)

   For example, let's take a look at the `TrialTestCaseCounter',
which is used by the Trial step to count test cases as they are run.
As Trial executes, it emits lines like the following:

     buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK]
     buildbot.test.test_config.ConfigTest.testEmpty ... [OK]
     buildbot.test.test_config.ConfigTest.testIRC ... [FAIL]
     buildbot.test.test_config.ConfigTest.testLocks ... [OK]

   When the tests are finished, trial emits a long line of "======"
and then some lines which summarize the tests that failed. We want to
avoid parsing these trailing lines, because their format is less
well-defined than the "[OK]" lines.

   The parser class looks like this:

     from buildbot.process.buildstep import LogLineObserver

     class TrialTestCaseCounter(LogLineObserver):
         _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$')
         numTests = 0
         finished = False

         def outLineReceived(self, line):
             if self.finished:
                 return
             if line.startswith("=" * 40):
                 self.finished = True
                 return

             m = self._line_re.search(line.strip())
             if m:
                 testname, result = m.groups()
                 self.numTests += 1
                 self.step.setProgress('tests', self.numTests)

   This parser only pays attention to stdout, since that's where trial
writes the progress lines. It has a mode flag named `finished' to
ignore everything after the "====" marker, and a scary-looking
regular expression to match each line while hopefully ignoring other
messages that might get displayed as the test runs.

   Each time it identifies a test has been completed, it increments
its counter and delivers the new progress value to the step with
`self.step.setProgress'. This class is specifically measuring
progress along the "tests" metric, in units of test cases (as opposed
to other kinds of progress like the "output" metric, which measures
in units of bytes). The Progress-tracking code uses each progress
metric separately to come up with an overall completion percentage
and an ETA value.

   To connect this parser into the `Trial' BuildStep,
`Trial.__init__' ends with the following clause:

             # this counter will feed Progress along the 'test cases' metric
             counter = TrialTestCaseCounter()
             self.addLogObserver('stdio', counter)
             self.progressMetrics += ('tests',)

   This creates a TrialTestCaseCounter and tells the step that the
counter wants to watch the "stdio" log. The observer is automatically
given a reference to the step in its `.step' attribute.

A Somewhat Whimsical Example
----------------------------

Let's say that we've got some snazzy new unit-test framework called
Framboozle. It's the hottest thing since sliced bread. It slices, it
dices, it runs unit tests like there's no tomorrow. Plus if your unit
tests fail, you can use its name for a Web 2.1 startup company, make
millions of dollars, and hire engineers to fix the bugs for you, while
you spend your afternoons lazily hang-gliding along a scenic pacific
beach, blissfully unconcerned about the state of your tests.(1)

   To run a Framboozle-enabled test suite, you just run the
'framboozler' command from the top of your source code tree. The
'framboozler' command emits a bunch of stuff to stdout, but the most
interesting bit is that it emits the line "FNURRRGH!" every time it
finishes running a test case(2). You'd like to have a test-case
counting LogObserver that watches for these lines and counts them,
because counting them will help the buildbot more accurately
calculate how long the build will take, and this will let you know
exactly how long you can sneak out of the office for your
hang-gliding lessons without anyone noticing that you're gone.

   This will involve writing a new BuildStep (probably named
"Framboozle") which inherits from ShellCommand. The BuildStep class
definition itself will look something like this:

     # START
     from buildbot.steps.shell import ShellCommand
     from buildbot.process.buildstep import LogLineObserver

     class FNURRRGHCounter(LogLineObserver):
         numTests = 0
         def outLineReceived(self, line):
             if "FNURRRGH!" in line:
                 self.numTests += 1
                 self.step.setProgress('tests', self.numTests)

     class Framboozle(ShellCommand):
         command = ["framboozler"]

         def __init__(self, **kwargs):
             ShellCommand.__init__(self, **kwargs)   # always upcall!
             counter = FNURRRGHCounter())
             self.addLogObserver('stdio', counter)
             self.progressMetrics += ('tests',)
     # FINISH

   So that's the code that we want to wind up using. How do we
actually deploy it?

   You have a couple of different options.

   Option 1: The simplest technique is to simply put this text
(everything from START to FINISH) in your master.cfg file, somewhere
before the BuildFactory definition where you actually use it in a
clause like:

     f = BuildFactory()
     f.addStep(SVN(svnurl="stuff"))
     f.addStep(Framboozle())

   Remember that master.cfg is secretly just a python program with one
job: populating the BuildmasterConfig dictionary. And python programs
are allowed to define as many classes as they like. So you can define
classes and use them in the same file, just as long as the class is
defined before some other code tries to use it.

   This is easy, and it keeps the point of definition very close to
the point of use, and whoever replaces you after that unfortunate
hang-gliding accident will appreciate being able to easily figure out
what the heck this stupid "Framboozle" step is doing anyways. The
downside is that every time you reload the config file, the Framboozle
class will get redefined, which means that the buildmaster will think
that you've reconfigured all the Builders that use it, even though
nothing changed. Bleh.

   Option 2: Instead, we can put this code in a separate file, and
import it into the master.cfg file just like we would the normal
buildsteps like ShellCommand and SVN.

   Create a directory named ~/lib/python, put everything from START to
FINISH in ~/lib/python/framboozle.py, and run your buildmaster using:

      PYTHONPATH=~/lib/python buildbot start MASTERDIR

   or use the `Makefile.buildbot' to control the way `buildbot start'
works. Or add something like this to something like your ~/.bashrc or
~/.bash_profile or ~/.cshrc:

      export PYTHONPATH=~/lib/python

   Once we've done this, our master.cfg can look like:

     from framboozle import Framboozle
     f = BuildFactory()
     f.addStep(SVN(svnurl="stuff"))
     f.addStep(Framboozle())

   or:

     import framboozle
     f = BuildFactory()
     f.addStep(SVN(svnurl="stuff"))
     f.addStep(framboozle.Framboozle())

   (check out the python docs for details about how "import" and
"from A import B" work).

   What we've done here is to tell python that every time it handles
an "import" statement for some named module, it should look in our
~/lib/python/ for that module before it looks anywhere else. After our
directories, it will try in a bunch of standard directories too
(including the one where buildbot is installed). By setting the
PYTHONPATH environment variable, you can add directories to the front
of this search list.

   Python knows that once it "import"s a file, it doesn't need to
re-import it again. This means that reconfiguring the buildmaster
(with "buildbot reconfig", for example) won't make it think the
Framboozle class has changed every time, so the Builders that use it
will not be spuriously restarted. On the other hand, you either have
to start your buildmaster in a slightly weird way, or you have to
modify your environment to set the PYTHONPATH variable.

   Option 3: Install this code into a standard python library
directory

   Find out what your python's standard include path is by asking it:

     80:warner@luther% python
     Python 2.4.4c0 (#2, Oct  2 2006, 00:57:46)
     [GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2
     Type "help", "copyright", "credits" or "license" for more information.
     >>> import sys
     >>> import pprint
     >>> pprint.pprint(sys.path)
     ['',
      '/usr/lib/python24.zip',
      '/usr/lib/python2.4',
      '/usr/lib/python2.4/plat-linux2',
      '/usr/lib/python2.4/lib-tk',
      '/usr/lib/python2.4/lib-dynload',
      '/usr/local/lib/python2.4/site-packages',
      '/usr/lib/python2.4/site-packages',
      '/usr/lib/python2.4/site-packages/Numeric',
      '/var/lib/python-support/python2.4',
      '/usr/lib/site-python']

   In this case, putting the code into
/usr/local/lib/python2.4/site-packages/framboozle.py would work just
fine. We can use the same master.cfg "import framboozle" statement as
in Option 2. By putting it in a standard include directory (instead of
the decidedly non-standard ~/lib/python), we don't even have to set
PYTHONPATH to anything special. The downside is that you probably have
to be root to write to one of those standard include directories.

   Option 4: Submit the code for inclusion in the Buildbot
distribution

   Make a fork of buildbot on http://github.com/djmitche/buildbot or
post a patch in a bug at http://buildbot.net.  In either case, post a
note about your patch to the mailing list, so others can provide
feedback and, eventually, commit it.

     from buildbot.steps import framboozle
     f = BuildFactory()
     f.addStep(SVN(svnurl="stuff"))
     f.addStep(framboozle.Framboozle())

   And then you don't even have to install framboozle.py anywhere on
your system, since it will ship with Buildbot. You don't have to be
root, you don't have to set PYTHONPATH. But you do have to make a
good case for Framboozle being worth going into the main
distribution, you'll probably have to provide docs and some unit test
cases, you'll need to figure out what kind of beer the author likes,
and then you'll have to wait until the next release. But in some
environments, all this is easier than getting root on your
buildmaster box, so the tradeoffs may actually be worth it.

   Putting the code in master.cfg (1) makes it available to that
buildmaster instance. Putting it in a file in a personal library
directory (2) makes it available for any buildmasters you might be
running. Putting it in a file in a system-wide shared library
directory (3) makes it available for any buildmasters that anyone on
that system might be running. Getting it into the buildbot's upstream
repository (4) makes it available for any buildmasters that anyone in
the world might be running. It's all a matter of how widely you want
to deploy that new class.

   ---------- Footnotes ----------

   (1) framboozle.com is still available. Remember, I get 10% :).

   (2) Framboozle gets very excited about running unit tests.


File: buildbot.info,  Node: BuildStep URLs,  Prev: Adding LogObservers,  Up: Writing New BuildSteps

6.1.10.5 BuildStep URLs
.......................

Each BuildStep has a collection of "links". Like its collection of
LogFiles, each link has a name and a target URL. The web status page
creates HREFs for each link in the same box as it does for LogFiles,
except that the target of the link is the external URL instead of an
internal link to a page that shows the contents of the LogFile.

   These external links can be used to point at build information
hosted on other servers. For example, the test process might produce
an intricate description of which tests passed and failed, or some
sort of code coverage data in HTML form, or a PNG or GIF image with a
graph of memory usage over time. The external link can provide an
easy way for users to navigate from the buildbot's status page to
these external web sites or file servers. Note that the step itself is
responsible for insuring that there will be a document available at
the given URL (perhaps by using `scp' to copy the HTML output to a
`~/public_html/' directory on a remote web server). Calling `addURL'
does not magically populate a web server.

   To set one of these links, the BuildStep should call the `addURL'
method with the name of the link and the target URL. Multiple URLs can
be set.

   In this example, we assume that the `make test' command causes a
collection of HTML files to be created and put somewhere on the
coverage.example.org web server, in a filename that incorporates the
build number.

     class TestWithCodeCoverage(BuildStep):
         command = ["make", "test",
                    WithProperties("buildnum=%s" % "buildnumber")]

         def createSummary(self, log):
             buildnumber = self.getProperty("buildnumber")
             url = "http://coverage.example.org/builds/%s.html" % buildnumber
             self.addURL("coverage", url)

   You might also want to extract the URL from some special message
output by the build process itself:

     class TestWithCodeCoverage(BuildStep):
         command = ["make", "test",
                    WithProperties("buildnum=%s" % "buildnumber")]

         def createSummary(self, log):
             output = StringIO(log.getText())
             for line in output.readlines():
                 if line.startswith("coverage-url:"):
                     url = line[len("coverage-url:"):].strip()
                     self.addURL("coverage", url)
                     return

   Note that a build process which emits both stdout and stderr might
cause this line to be split or interleaved between other lines. It
might be necessary to restrict the getText() call to only stdout with
something like this:

             output = StringIO("".join([c[1]
                                        for c in log.getChunks()
                                        if c[0] == LOG_CHANNEL_STDOUT]))

   Of course if the build is run under a PTY, then stdout and stderr
will be merged before the buildbot ever sees them, so such
interleaving will be unavoidable.


File: buildbot.info,  Node: Interlocks,  Next: Build Factories,  Prev: Build Steps,  Up: Build Process

6.2 Interlocks
==============

Until now, we assumed that a master can run builds at any slave
whenever needed or desired.  Some times, you want to enforce
additional constraints on builds. For reasons like limited network
bandwidth, old slave machines, or a self-willed data base server, you
may want to limit the number of builds (or build steps) that can
access a resource.

   The mechanism used by Buildbot is known as the read/write lock.(1)
It allows either many readers or a single writer but not a
combination of readers and writers. The general lock has been
modified and extended for use in Buildbot. Firstly, the general lock
allows an infinite number of readers. In Buildbot, we often want to
put an upper limit on the number of readers, for example allowing two
out of five possible builds at the same time. To do this, the lock
counts the number of active readers. Secondly, the terms _read mode_
and _write mode_ are confusing in Buildbot context. They have been
replaced by _counting mode_ (since the lock counts them) and
_exclusive mode_.  As a result of these changes, locks in Buildbot
allow a number of builds (upto some fixed number) in counting mode,
or they allow one build in exclusive mode.

   Often, not all slaves are equal. To allow for this situation,
Buildbot allows to have a separate upper limit on the count for each
slave. In this way, you can have at most 3 concurrent builds at a
fast slave, 2 at a slightly older slave, and 1 at all other slaves.

   The final thing you can specify when you introduce a new lock is
its scope.  Some constraints are global - they must be enforced over
all slaves. Other constraints are local to each slave.  A _master
lock_ is used for the global constraints. You can ensure for example
that at most one build (of all builds running at all slaves) accesses
the data base server. With a _slave lock_ you can add a limit local
to each slave. With such a lock, you can for example enforce an upper
limit to the number of active builds at a slave, like above.

   Time for a few examples. Below a master lock is defined to protect
a data base, and a slave lock is created to limit the number of
builds at each slave.

     from buildbot import locks

     db_lock = locks.MasterLock("database")
     build_lock = locks.SlaveLock("slave_builds",
                                  maxCount = 1,
                                  maxCountForSlave = { 'fast': 3, 'new': 2 })

   After importing locks from buildbot, `db_lock' is defined to be a
master lock. The `"database"' string is used for uniquely identifying
the lock.  At the next line, a slave lock called `build_lock' is
created. It is identified by the `"slave_builds"' string. Since the
requirements of the lock are a bit more complicated, two optional
arguments are also specified. The `maxCount' parameter sets the
default limit for builds in counting mode to `1'. For the slave
called `'fast'' however, we want to have at most three builds, and
for the slave called `'new'' the upper limit is two builds running at
the same time.

   The next step is using the locks in builds.  Buildbot allows a
lock to be used during an entire build (from beginning to end), or
only during a single build step. In the latter case, the lock is
claimed for use just before the step starts, and released again when
the step ends. To prevent deadlocks,(2) it is not possible to claim
or release locks at other times.

   To use locks, you should add them with a `locks' argument.  Each
use of a lock is either in counting mode (that is, possibly shared
with other builds) or in exclusive mode. A build or build step
proceeds only when it has acquired all locks. If a build or step
needs a lot of locks, it may be starved(3) by other builds that need
fewer locks.

   To illustrate use of locks, a few examples.

     from buildbot import locks
     from buildbot.steps import source, shell
     from buildbot.process import factory

     db_lock = locks.MasterLock("database")
     build_lock = locks.SlaveLock("slave_builds",
                                  maxCount = 1,
                                  maxCountForSlave = { 'fast': 3, 'new': 2 })

     f = factory.BuildFactory()
     f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk"))
     f.addStep(shell.ShellCommand(command="make all"))
     f.addStep(shell.ShellCommand(command="make test",
                                  locks=[db_lock.access('exclusive')]))

     b1 = {'name': 'full1', 'slavename': 'fast',  'builddir': 'f1', 'factory': f,
            'locks': [build_lock.access('counting')] }

     b2 = {'name': 'full2', 'slavename': 'new',   'builddir': 'f2', 'factory': f.
            'locks': [build_lock.access('counting')] }

     b3 = {'name': 'full3', 'slavename': 'old',   'builddir': 'f3', 'factory': f.
            'locks': [build_lock.access('counting')] }

     b4 = {'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f.
            'locks': [build_lock.access('counting')] }

     c['builders'] = [b1, b2, b3, b4]

   Here we have four slaves `b1', `b2', `b3', and `b4'. Each slave
performs the same checkout, make, and test build step sequence.  We
want to enforce that at most one test step is executed between all
slaves due to restrictions with the data base server. This is done by
adding the `locks=' parameter with the third step. It takes a list of
locks with their access mode. In this case only the `db_lock' is
needed. The exclusive access mode is used to ensure there is at most
one slave that executes the test step.

   In addition to exclusive accessing the data base, we also want
slaves to stay responsive even under the load of a large number of
builds being triggered.  For this purpose, the slave lock called
`build_lock' is defined. Since the restraint holds for entire builds,
the lock is specified in the builder with `'locks':
[build_lock.access('counting')]'.

   ---------- Footnotes ----------

   (1) See http://en.wikipedia.org/wiki/Read/write_lock_pattern for
more information.

   (2) Deadlock is the situation where two or more slaves each hold a
lock in exclusive mode, and in addition want to claim the lock held by
the other slave exclusively as well. Since locks allow at most one
exclusive user, both slaves will wait forever.

   (3) Starving is the situation that only a few locks are available,
and they are immediately grabbed by another build. As a result, it
may take a long time before all locks needed by the starved build are
free at the same time.


File: buildbot.info,  Node: Build Factories,  Prev: Interlocks,  Up: Build Process

6.3 Build Factories
===================

Each Builder is equipped with a "build factory", which is responsible
for producing the actual `Build' objects that perform each build.
This factory is created in the configuration file, and attached to a
Builder through the `factory' element of its dictionary.

   The standard `BuildFactory' object creates `Build' objects by
default. These Builds will each execute a collection of BuildSteps in
a fixed sequence. Each step can affect the results of the build, but
in general there is little intelligence to tie the different steps
together. You can create subclasses of `Build' to implement more
sophisticated build processes, and then use a subclass of
`BuildFactory' (or simply set the `buildClass' attribute) to create
instances of your new Build subclass.

* Menu:

* BuildStep Objects::
* BuildFactory::
* Process-Specific build factories::


File: buildbot.info,  Node: BuildStep Objects,  Next: BuildFactory,  Prev: Build Factories,  Up: Build Factories

6.3.1 BuildStep Objects
-----------------------

The steps used by these builds are all subclasses of `BuildStep'.
The standard ones provided with Buildbot are documented later, *Note
Build Steps::. You can also write your own subclasses to use in
builds.

   The basic behavior for a `BuildStep' is to:

   * run for a while, then stop

   * possibly invoke some RemoteCommands on the attached build slave

   * possibly produce a set of log files

   * finish with a status described by one of four values defined in
     buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED

   * provide a list of short strings to describe the step

   * define a color (generally green, orange, or red) with which the
     step should be displayed

   More sophisticated steps may produce additional information and
provide it to later build steps, or store it in the factory to provide
to later builds.

* Menu:

* BuildFactory Attributes::
* Quick builds::


File: buildbot.info,  Node: BuildFactory,  Next: Process-Specific build factories,  Prev: BuildStep Objects,  Up: Build Factories

6.3.2 BuildFactory
------------------

The default `BuildFactory', provided in the
`buildbot.process.factory' module, contains an internal list of
"BuildStep specifications": a list of `(step_class, kwargs)' tuples
for each. These specification tuples are constructed when the config
file is read, by asking the instances passed to `addStep' for their
subclass and arguments.

   When asked to create a Build, the `BuildFactory' puts a copy of
the list of step specifications into the new Build object. When the
Build is actually started, these step specifications are used to
create the actual set of BuildSteps, which are then executed one at a
time. This serves to give each Build an independent copy of each step.
For example, a build which consists of a CVS checkout followed by a
`make build' would be constructed as follows:

     from buildbot.steps import source, shell
     from buildbot.process import factory

     f = factory.BuildFactory()
     f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"))
     f.addStep(shell.Compile(command=["make", "build"]))

   (To support config files from buildbot-0.7.5 and earlier,
`addStep' also accepts the `f.addStep(shell.Compile,
command=["make","build"])' form, although its use is discouraged
because then the `Compile' step doesn't get to validate or complain
about its arguments until build time. The modern pass-by-instance
approach allows this validation to occur while the config file is
being loaded, where the admin has a better chance of noticing
problems).

   It is also possible to pass a list of steps into the
`BuildFactory' when it is created. Using `addStep' is usually
simpler, but there are cases where is is more convenient to create
the list of steps ahead of time.:

     from buildbot.steps import source, shell
     from buildbot.process import factory

     all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"),
                  shell.Compile(command=["make", "build"]),
                 ]
     f = factory.BuildFactory(all_steps)

   Each step can affect the build process in the following ways:

   * If the step's `haltOnFailure' attribute is True, then a failure
     in the step (i.e. if it completes with a result of FAILURE) will
     cause the whole build to be terminated immediately: no further
     steps will be executed, with the exception of steps with
     `alwaysRun' set to True. `haltOnFailure' is useful for setup
     steps upon which the rest of the build depends: if the CVS
     checkout or `./configure' process fails, there is no point in
     trying to compile or test the resulting tree.

   * If the step's `alwaysRun' attribute is True, then it will always
     be run, regardless of if previous steps have failed. This is
     useful for cleanup steps that should always be run to return the
     build directory or build slave into a good state.

   * If the `flunkOnFailure' or `flunkOnWarnings' flag is set, then a
     result of FAILURE or WARNINGS will mark the build as a whole as
     FAILED. However, the remaining steps will still be executed.
     This is appropriate for things like multiple testing steps: a
     failure in any one of them will indicate that the build has
     failed, however it is still useful to run them all to completion.

   * Similarly, if the `warnOnFailure' or `warnOnWarnings' flag is
     set, then a result of FAILURE or WARNINGS will mark the build as
     having WARNINGS, and the remaining steps will still be executed.
     This may be appropriate for certain kinds of optional build or
     test steps.  For example, a failure experienced while building
     documentation files should be made visible with a WARNINGS
     result but not be serious enough to warrant marking the whole
     build with a FAILURE.


   In addition, each Step produces its own results, may create
logfiles, etc. However only the flags described above have any effect
on the build as a whole.

   The pre-defined BuildSteps like `CVS' and `Compile' have
reasonably appropriate flags set on them already. For example, without
a source tree there is no point in continuing the build, so the `CVS'
class has the `haltOnFailure' flag set to True. Look in
`buildbot/steps/*.py' to see how the other Steps are marked.

   Each Step is created with an additional `workdir' argument that
indicates where its actions should take place. This is specified as a
subdirectory of the slave builder's base directory, with a default
value of `build'. This is only implemented as a step argument (as
opposed to simply being a part of the base directory) because the
CVS/SVN steps need to perform their checkouts from the parent
directory.

* Menu:

* BuildFactory Attributes::
* Quick builds::


File: buildbot.info,  Node: BuildFactory Attributes,  Next: Quick builds,  Prev: BuildFactory,  Up: BuildFactory

6.3.2.1 BuildFactory Attributes
...............................

Some attributes from the BuildFactory are copied into each Build.

`useProgress'
     (defaults to True): if True, the buildmaster keeps track of how
     long each step takes, so it can provide estimates of how long
     future builds will take. If builds are not expected to take a
     consistent amount of time (such as incremental builds in which a
     random set of files are recompiled or tested each time), this
     should be set to False to inhibit progress-tracking.



File: buildbot.info,  Node: Quick builds,  Prev: BuildFactory Attributes,  Up: BuildFactory

6.3.2.2 Quick builds
....................

The difference between a "full build" and a "quick build" is that
quick builds are generally done incrementally, starting with the tree
where the previous build was performed. That simply means that the
source-checkout step should be given a `mode='update'' flag, to do
the source update in-place.

   In addition to that, the `useProgress' flag should be set to
False. Incremental builds will (or at least the ought to) compile as
few files as necessary, so they will take an unpredictable amount of
time to run. Therefore it would be misleading to claim to predict how
long the build will take.


File: buildbot.info,  Node: Process-Specific build factories,  Prev: BuildFactory,  Up: Build Factories

6.3.3 Process-Specific build factories
--------------------------------------

Many projects use one of a few popular build frameworks to simplify
the creation and maintenance of Makefiles or other compilation
structures. Buildbot provides several pre-configured BuildFactory
subclasses which let you build these projects with a minimum of fuss.

* Menu:

* GNUAutoconf::
* CPAN::
* Python distutils::
* Python/Twisted/trial projects::


File: buildbot.info,  Node: GNUAutoconf,  Next: CPAN,  Prev: Process-Specific build factories,  Up: Process-Specific build factories

6.3.3.1 GNUAutoconf
...................

GNU Autoconf (http://www.gnu.org/software/autoconf/) is a software
portability tool, intended to make it possible to write programs in C
(and other languages) which will run on a variety of UNIX-like
systems. Most GNU software is built using autoconf. It is frequently
used in combination with GNU automake. These tools both encourage a
build process which usually looks like this:

     % CONFIG_ENV=foo ./configure --with-flags
     % make all
     % make check
     # make install

   (except of course the Buildbot always skips the `make install'
part).

   The Buildbot's `buildbot.process.factory.GNUAutoconf' factory is
designed to build projects which use GNU autoconf and/or automake. The
configuration environment variables, the configure flags, and command
lines used for the compile and test are all configurable, in general
the default values will be suitable.

   Example:

     # use the s() convenience function defined earlier
     f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"),
                             flags=["--disable-nls"])

   Required Arguments:

`source'
     This argument must be a step specification tuple that provides a
     BuildStep to generate the source tree.

   Optional Arguments:

`configure'
     The command used to configure the tree. Defaults to
     `./configure'. Accepts either a string or a list of shell argv
     elements.

`configureEnv'
     The environment used for the initial configuration step. This
     accepts a dictionary which will be merged into the buildslave's
     normal environment. This is commonly used to provide things like
     `CFLAGS="-O2 -g"' (to turn off debug symbols during the compile).
     Defaults to an empty dictionary.

`configureFlags'
     A list of flags to be appended to the argument list of the
     configure command. This is commonly used to enable or disable
     specific features of the autoconf-controlled package, like
     `["--without-x"]' to disable windowing support. Defaults to an
     empty list.

`compile'
     this is a shell command or list of argv values which is used to
     actually compile the tree. It defaults to `make all'. If set to
     None, the compile step is skipped.

`test'
     this is a shell command or list of argv values which is used to
     run the tree's self-tests. It defaults to `make check'. If set to
     None, the test step is skipped.



File: buildbot.info,  Node: CPAN,  Next: Python distutils,  Prev: GNUAutoconf,  Up: Process-Specific build factories

6.3.3.2 CPAN
............

Most Perl modules available from the CPAN (http://www.cpan.org/)
archive use the `MakeMaker' module to provide configuration, build,
and test services. The standard build routine for these modules looks
like:

     % perl Makefile.PL
     % make
     % make test
     # make install

   (except again Buildbot skips the install step)

   Buildbot provides a `CPAN' factory to compile and test these
projects.

   Arguments:
`source'
     (required): A step specification tuple, like that used by
     GNUAutoconf.

`perl'
     A string which specifies the `perl' executable to use. Defaults
     to just `perl'.



File: buildbot.info,  Node: Python distutils,  Next: Python/Twisted/trial projects,  Prev: CPAN,  Up: Process-Specific build factories

6.3.3.3 Python distutils
........................

Most Python modules use the `distutils' package to provide
configuration and build services. The standard build process looks
like:

     % python ./setup.py build
     % python ./setup.py install

   Unfortunately, although Python provides a standard unit-test
framework named `unittest', to the best of my knowledge `distutils'
does not provide a standardized target to run such unit tests. (Please
let me know if I'm wrong, and I will update this factory.)

   The `Distutils' factory provides support for running the build
part of this process. It accepts the same `source=' parameter as the
other build factories.

   Arguments:
`source'
     (required): A step specification tuple, like that used by
     GNUAutoconf.

`python'
     A string which specifies the `python' executable to use. Defaults
     to just `python'.

`test'
     Provides a shell command which runs unit tests. This accepts
     either a string or a list. The default value is None, which
     disables the test step (since there is no common default command
     to run unit tests in distutils modules).



File: buildbot.info,  Node: Python/Twisted/trial projects,  Prev: Python distutils,  Up: Process-Specific build factories

6.3.3.4 Python/Twisted/trial projects
.....................................

Twisted provides a unit test tool named `trial' which provides a few
improvements over Python's built-in `unittest' module. Many python
projects which use Twisted for their networking or application
services also use trial for their unit tests. These modules are
usually built and tested with something like the following:

     % python ./setup.py build
     % PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test
     % python ./setup.py install

   Unfortunately, the `build/lib' directory into which the
built/copied .py files are placed is actually architecture-dependent,
and I do not yet know of a simple way to calculate its value. For many
projects it is sufficient to import their libraries "in place" from
the tree's base directory (`PYTHONPATH=.').

   In addition, the PROJECTNAME value where the test files are
located is project-dependent: it is usually just the project's
top-level library directory, as common practice suggests the unit test
files are put in the `test' sub-module. This value cannot be guessed,
the `Trial' class must be told where to find the test files.

   The `Trial' class provides support for building and testing
projects which use distutils and trial. If the test module name is
specified, trial will be invoked. The library path used for testing
can also be set.

   One advantage of trial is that the Buildbot happens to know how to
parse trial output, letting it identify which tests passed and which
ones failed. The Buildbot can then provide fine-grained reports about
how many tests have failed, when individual tests fail when they had
been passing previously, etc.

   Another feature of trial is that you can give it a series of source
.py files, and it will search them for special `test-case-name' tags
that indicate which test cases provide coverage for that file.  Trial
can then run just the appropriate tests. This is useful for quick
builds, where you want to only run the test cases that cover the
changed functionality.

   Arguments:
`source'
     (required): A step specification tuple, like that used by
     GNUAutoconf.

`buildpython'
     A list (argv array) of strings which specifies the `python'
     executable to use when building the package. Defaults to just
     `['python']'. It may be useful to add flags here, to supress
     warnings during compilation of extension modules. This list is
     extended with `['./setup.py', 'build']' and then executed in a
     ShellCommand.

`testpath'
     Provides a directory to add to `PYTHONPATH' when running the unit
     tests, if tests are being run. Defaults to `.' to include the
     project files in-place. The generated build library is frequently
     architecture-dependent, but may simply be `build/lib' for
     pure-python modules.

`trialpython'
     Another list of strings used to build the command that actually
     runs trial. This is prepended to the contents of the `trial'
     argument below. It may be useful to add `-W' flags here to
     supress warnings that occur while tests are being run. Defaults
     to an empty list, meaning `trial' will be run without an explicit
     interpreter, which is generally what you want if you're using
     `/usr/bin/trial' instead of, say, the `./bin/trial' that lives
     in the Twisted source tree.

`trial'
     provides the name of the `trial' command. It is occasionally
     useful to use an alternate executable, such as `trial2.2' which
     might run the tests under an older version of Python. Defaults to
     `trial'.

`tests'
     Provides a module name or names which contain the unit tests for
     this project. Accepts a string, typically `PROJECTNAME.test', or
     a list of strings. Defaults to None, indicating that no tests
     should be run. You must either set this or `useTestCaseNames' to
     do anyting useful with the Trial factory.

`useTestCaseNames'
     Tells the Step to provide the names of all changed .py files to
     trial, so it can look for test-case-name tags and run just the
     matching test cases. Suitable for use in quick builds. Defaults
     to False.

`randomly'
     If `True', tells Trial (with the `--random=0' argument) to run
     the test cases in random order, which sometimes catches subtle
     inter-test dependency bugs. Defaults to `False'.

`recurse'
     If `True', tells Trial (with the `--recurse' argument) to look
     in all subdirectories for additional test cases. It isn't clear
     to me how this works, but it may be useful to deal with the
     unknown-PROJECTNAME problem described above, and is currently
     used in the Twisted buildbot to accomodate the fact that test
     cases are now distributed through multiple
     twisted.SUBPROJECT.test directories.


   Unless one of `trialModule' or `useTestCaseNames' are set, no
tests will be run.

   Some quick examples follow. Most of these examples assume that the
target python code (the "code under test") can be reached directly
from the root of the target tree, rather than being in a `lib/'
subdirectory.

     #  Trial(source, tests="toplevel.test") does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to toplevel.test

     #  Trial(source, tests=["toplevel.test", "other.test"]) does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to toplevel.test other.test

     #  Trial(source, useTestCaseNames=True) does:
     #   python ./setup.py build
     #   PYTHONPATH=. trial -to --testmodule=foo/bar.py..  (from Changes)

     #  Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"):
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. trial -to foo.tests

     #  Trial(source, trialpython="python2.3", trial="/usr/bin/trial",
     #        tests="foo.tests") does:
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests

     # For running trial out of the tree being tested (only useful when the
     # tree being built is Twisted itself):
     #  Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial",
     #        tests="foo.tests") does:
     #   python2.3 -Wall ./setup.py build
     #   PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests

   If the output directory of `./setup.py build' is known, you can
pull the python code from the built location instead of the source
directories. This should be able to handle variations in where the
source comes from, as well as accomodating binary extension modules:

     # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3')
     # does:
     #  python ./setup.py build
     #  PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test


File: buildbot.info,  Node: Status Delivery,  Next: Command-line tool,  Prev: Build Process,  Up: Top

7 Status Delivery
*****************

More details are available in the docstrings for each class, use a
command like `pydoc buildbot.status.html.WebStatus' to see them.
Most status delivery objects take a `categories=' argument, which can
contain a list of "category" names: in this case, it will only show
status for Builders that are in one of the named categories.

   (implementor's note: each of these objects should be a
service.MultiService which will be attached to the BuildMaster object
when the configuration is processed. They should use
`self.parent.getStatus()' to get access to the top-level IStatus
object, either inside `startService' or later. They may call
`status.subscribe()' in `startService' to receive notifications of
builder events, in which case they must define `builderAdded' and
related methods. See the docstrings in `buildbot/interfaces.py' for
full details.)

* Menu:

* WebStatus::
* MailNotifier::
* IRC Bot::
* PBListener::
* Writing New Status Plugins::


File: buildbot.info,  Node: WebStatus,  Next: MailNotifier,  Prev: Status Delivery,  Up: Status Delivery

7.1 WebStatus
=============

The `buildbot.status.html.WebStatus' status target runs a small web
server inside the buildmaster. You can point a browser at this web
server and retrieve information about every build the buildbot knows
about, as well as find out what the buildbot is currently working on.

   The first page you will see is the "Welcome Page", which contains
links to all the other useful pages. This page is simply served from
the `public_html/index.html' file in the buildmaster's base
directory, where it is created by the `buildbot create-master'
command along with the rest of the buildmaster.

   The most complex resource provided by `WebStatus' is the
"Waterfall Display", which shows a time-based chart of events. This
somewhat-busy display provides detailed information about all steps of
all recent builds, and provides hyperlinks to look at individual build
logs and source changes. By simply reloading this page on a regular
basis, you will see a complete description of everything the buildbot
is currently working on.

   There are also pages with more specialized information. For
example, there is a page which shows the last 20 builds performed by
the buildbot, one line each. Each line is a link to detailed
information about that build. By adding query arguments to the URL
used to reach this page, you can narrow the display to builds that
involved certain branches, or which ran on certain Builders. These
pages are described in great detail below.

   When the buildmaster is created, a subdirectory named
`public_html/' is created in its base directory. By default,
`WebStatus' will serve files from this directory: for example, when a
user points their browser at the buildbot's `WebStatus' URL, they
will see the contents of the `public_html/index.html' file. Likewise,
`public_html/robots.txt', `public_html/buildbot.css', and
`public_html/favicon.ico' are all useful things to have in there.
The first time a buildmaster is created, the `public_html' directory
is populated with some sample files, which you will probably want to
customize for your own project. These files are all static: the
buildbot does not modify them in any way as it serves them to HTTP
clients.

     from buildbot.status.html import WebStatus
     c['status'].append(WebStatus(8080))

   Note that the initial robots.txt file has Disallow lines for all of
the dynamically-generated buildbot pages, to discourage web spiders
and search engines from consuming a lot of CPU time as they crawl
through the entire history of your buildbot. If you are running the
buildbot behind a reverse proxy, you'll probably need to put the
robots.txt file somewhere else (at the top level of the parent web
server), and replace the URL prefixes in it with more suitable values.

   If you would like to use an alternative root directory, add the
`public_html=..' option to the `WebStatus' creation:

     c['status'].append(WebStatus(8080, public_html="/var/www/buildbot"))

   In addition, if you are familiar with twisted.web _Resource
Trees_, you can write code to add additional pages at places inside
this web space. Just use `webstatus.putChild' to place these
resources.

   The following section describes the special URLs and the status
views they provide.

* Menu:

* WebStatus Configuration Parameters::
* Buildbot Web Resources::
* XMLRPC server::
* HTML Waterfall::


File: buildbot.info,  Node: WebStatus Configuration Parameters,  Next: Buildbot Web Resources,  Prev: WebStatus,  Up: WebStatus

7.1.1 WebStatus Configuration Parameters
----------------------------------------

The most common way to run a `WebStatus' is on a regular TCP port. To
do this, just pass in the TCP port number when you create the
`WebStatus' instance; this is called the `http_port' argument:

     from buildbot.status.html import WebStatus
     c['status'].append(WebStatus(8080))

   The `http_port' argument is actually a "strports specification"
for the port that the web server should listen on. This can be a
simple port number, or a string like `tcp:8080:interface=127.0.0.1'
(to limit connections to the loopback interface, and therefore to
clients running on the same host)(1).

   If instead (or in addition) you provide the `distrib_port'
argument, a twisted.web distributed server will be started either on a
TCP port (if `distrib_port' is like `"tcp:12345"') or more likely on
a UNIX socket (if `distrib_port' is like `"unix:/path/to/socket"').

   The `distrib_port' option means that, on a host with a
suitably-configured twisted-web server, you do not need to consume a
separate TCP port for the buildmaster's status web page. When the web
server is constructed with `mktap web --user', URLs that point to
`http://host/~username/' are dispatched to a sub-server that is
listening on a UNIX socket at `~username/.twisted-web-pb'. On such a
system, it is convenient to create a dedicated `buildbot' user, then
set `distrib_port' to
`"unix:"+os.path.expanduser("~/.twistd-web-pb")'. This configuration
will make the HTML status page available at `http://host/~buildbot/'
. Suitable URL remapping can make it appear at
`http://host/buildbot/', and the right virtual host setup can even
place it at `http://buildbot.host/' .

   The other `WebStatus' argument is `allowForce'. If set to True,
then the web page will provide a "Force Build" button that allows
visitors to manually trigger builds. This is useful for developers to
re-run builds that have failed because of intermittent problems in
the test suite, or because of libraries that were not installed at
the time of the previous build. You may not wish to allow strangers
to cause a build to run: in that case, set this to False to remove
these buttons. The default value is False.

   ---------- Footnotes ----------

   (1) It may even be possible to provide SSL access by using a
specification like
`"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"', but this is
completely untested