diff options
Diffstat (limited to 'buildbot/docs/buildbot.info-1')
| -rw-r--r-- | buildbot/docs/buildbot.info-1 | 7278 |
1 files changed, 7278 insertions, 0 deletions
diff --git a/buildbot/docs/buildbot.info-1 b/buildbot/docs/buildbot.info-1 new file mode 100644 index 0000000..5dcf913 --- /dev/null +++ b/buildbot/docs/buildbot.info-1 @@ -0,0 +1,7278 @@ +This is buildbot.info, produced by makeinfo version 4.11 from +buildbot.texinfo. + +This is the BuildBot manual. + + Copyright (C) 2005,2006 Brian Warner + + Copying and distribution of this file, with or without +modification, are permitted in any medium without royalty provided +the copyright notice and this notice are preserved. + + +File: buildbot.info, Node: Top, Next: Introduction, Prev: (dir), Up: (dir) + +BuildBot +******** + +This is the BuildBot manual. + + Copyright (C) 2005,2006 Brian Warner + + Copying and distribution of this file, with or without +modification, are permitted in any medium without royalty provided +the copyright notice and this notice are preserved. + +* Menu: + +* Introduction:: What the BuildBot does. +* Installation:: Creating a buildmaster and buildslaves, + running them. +* Concepts:: What goes on in the buildbot's little mind. +* Configuration:: Controlling the buildbot. +* Getting Source Code Changes:: Discovering when to run a build. +* Build Process:: Controlling how each build is run. +* Status Delivery:: Telling the world about the build's results. +* Command-line tool:: +* Resources:: Getting help. +* Developer's Appendix:: +* Index of Useful Classes:: +* Index of master.cfg keys:: +* Index:: Complete index. + + --- The Detailed Node Listing --- + +Introduction + +* History and Philosophy:: +* System Architecture:: +* Control Flow:: + +System Architecture + +* BuildSlave Connections:: +* Buildmaster Architecture:: +* Status Delivery Architecture:: + +Installation + +* Requirements:: +* Installing the code:: +* Creating a buildmaster:: +* Upgrading an Existing Buildmaster:: +* Creating a buildslave:: +* Launching the daemons:: +* Logfiles:: +* Shutdown:: +* Maintenance:: +* Troubleshooting:: + +Creating a buildslave + +* Buildslave Options:: + +Troubleshooting + +* Starting the buildslave:: +* Connecting to the buildmaster:: +* Forcing Builds:: + +Concepts + +* Version Control Systems:: +* Schedulers:: +* BuildSet:: +* BuildRequest:: +* Builder:: +* Users:: +* Build Properties:: + +Version Control Systems + +* Generalizing VC Systems:: +* Source Tree Specifications:: +* How Different VC Systems Specify Sources:: +* Attributes of Changes:: + +Users + +* Doing Things With Users:: +* Email Addresses:: +* IRC Nicknames:: +* Live Status Clients:: + +Configuration + +* Config File Format:: +* Loading the Config File:: +* Testing the Config File:: +* Defining the Project:: +* Change Sources and Schedulers:: +* Setting the slaveport:: +* Buildslave Specifiers:: +* On-Demand ("Latent") Buildslaves:: +* Defining Global Properties:: +* Defining Builders:: +* Defining Status Targets:: +* Debug options:: + +Change Sources and Schedulers + +* Scheduler Scheduler:: +* AnyBranchScheduler:: +* Dependent Scheduler:: +* Periodic Scheduler:: +* Nightly Scheduler:: +* Try Schedulers:: +* Triggerable Scheduler:: + +Buildslave Specifiers +* When Buildslaves Go Missing:: + +On-Demand ("Latent") Buildslaves +* Amazon Web Services Elastic Compute Cloud ("AWS EC2"):: +* Dangers with Latent Buildslaves:: +* Writing New Latent Buildslaves:: + +Getting Source Code Changes + +* Change Sources:: +* Choosing ChangeSources:: +* CVSToys - PBService:: +* Mail-parsing ChangeSources:: +* PBChangeSource:: +* P4Source:: +* BonsaiPoller:: +* SVNPoller:: +* MercurialHook:: +* Bzr Hook:: +* Bzr Poller:: + +Mail-parsing ChangeSources + +* Subscribing the Buildmaster:: +* Using Maildirs:: +* Parsing Email Change Messages:: + +Parsing Email Change Messages + +* FCMaildirSource:: +* SyncmailMaildirSource:: +* BonsaiMaildirSource:: +* SVNCommitEmailMaildirSource:: + +Build Process + +* Build Steps:: +* Interlocks:: +* Build Factories:: + +Build Steps + +* Common Parameters:: +* Using Build Properties:: +* Source Checkout:: +* ShellCommand:: +* Simple ShellCommand Subclasses:: +* Python BuildSteps:: +* Transferring Files:: +* Steps That Run on the Master:: +* Triggering Schedulers:: +* Writing New BuildSteps:: + +Source Checkout + +* CVS:: +* SVN:: +* Darcs:: +* Mercurial:: +* Arch:: +* Bazaar:: +* Bzr:: +* P4:: +* Git:: + +Simple ShellCommand Subclasses + +* Configure:: +* Compile:: +* Test:: +* TreeSize:: +* PerlModuleTest:: +* SetProperty:: + +Python BuildSteps + +* BuildEPYDoc:: +* PyFlakes:: +* PyLint:: + +Writing New BuildSteps + +* BuildStep LogFiles:: +* Reading Logfiles:: +* Adding LogObservers:: +* BuildStep URLs:: + +Build Factories + +* BuildStep Objects:: +* BuildFactory:: +* Process-Specific build factories:: + +BuildStep Objects + +* BuildFactory Attributes:: +* Quick builds:: + +BuildFactory + +* BuildFactory Attributes:: +* Quick builds:: + +Process-Specific build factories + +* GNUAutoconf:: +* CPAN:: +* Python distutils:: +* Python/Twisted/trial projects:: + +Status Delivery + +* WebStatus:: +* MailNotifier:: +* IRC Bot:: +* PBListener:: +* Writing New Status Plugins:: + +WebStatus + +* WebStatus Configuration Parameters:: +* Buildbot Web Resources:: +* XMLRPC server:: +* HTML Waterfall:: + +Command-line tool + +* Administrator Tools:: +* Developer Tools:: +* Other Tools:: +* .buildbot config directory:: + +Developer Tools + +* statuslog:: +* statusgui:: +* try:: + +waiting for results + +* try --diff:: + +Other Tools + +* sendchange:: +* debugclient:: + + +File: buildbot.info, Node: Introduction, Next: Installation, Prev: Top, Up: Top + +1 Introduction +************** + +The BuildBot is a system to automate the compile/test cycle required +by most software projects to validate code changes. By automatically +rebuilding and testing the tree each time something has changed, +build problems are pinpointed quickly, before other developers are +inconvenienced by the failure. The guilty developer can be identified +and harassed without human intervention. By running the builds on a +variety of platforms, developers who do not have the facilities to +test their changes everywhere before checkin will at least know +shortly afterwards whether they have broken the build or not. Warning +counts, lint checks, image size, compile time, and other build +parameters can be tracked over time, are more visible, and are +therefore easier to improve. + + The overall goal is to reduce tree breakage and provide a platform +to run tests or code-quality checks that are too annoying or pedantic +for any human to waste their time with. Developers get immediate (and +potentially public) feedback about their changes, encouraging them to +be more careful about testing before checkin. + + Features: + + * run builds on a variety of slave platforms + + * arbitrary build process: handles projects using C, Python, + whatever + + * minimal host requirements: python and Twisted + + * slaves can be behind a firewall if they can still do checkout + + * status delivery through web page, email, IRC, other protocols + + * track builds in progress, provide estimated completion time + + * flexible configuration by subclassing generic build process + classes + + * debug tools to force a new build, submit fake Changes, query + slave status + + * released under the GPL + +* Menu: + +* History and Philosophy:: +* System Architecture:: +* Control Flow:: + + +File: buildbot.info, Node: History and Philosophy, Next: System Architecture, Prev: Introduction, Up: Introduction + +1.1 History and Philosophy +========================== + +The Buildbot was inspired by a similar project built for a development +team writing a cross-platform embedded system. The various components +of the project were supposed to compile and run on several flavors of +unix (linux, solaris, BSD), but individual developers had their own +preferences and tended to stick to a single platform. From time to +time, incompatibilities would sneak in (some unix platforms want to +use `string.h', some prefer `strings.h'), and then the tree would +compile for some developers but not others. The buildbot was written +to automate the human process of walking into the office, updating a +tree, compiling (and discovering the breakage), finding the developer +at fault, and complaining to them about the problem they had +introduced. With multiple platforms it was difficult for developers to +do the right thing (compile their potential change on all platforms); +the buildbot offered a way to help. + + Another problem was when programmers would change the behavior of a +library without warning its users, or change internal aspects that +other code was (unfortunately) depending upon. Adding unit tests to +the codebase helps here: if an application's unit tests pass despite +changes in the libraries it uses, you can have more confidence that +the library changes haven't broken anything. Many developers +complained that the unit tests were inconvenient or took too long to +run: having the buildbot run them reduces the developer's workload to +a minimum. + + In general, having more visibility into the project is always good, +and automation makes it easier for developers to do the right thing. +When everyone can see the status of the project, developers are +encouraged to keep the tree in good working order. Unit tests that +aren't run on a regular basis tend to suffer from bitrot just like +code does: exercising them on a regular basis helps to keep them +functioning and useful. + + The current version of the Buildbot is additionally targeted at +distributed free-software projects, where resources and platforms are +only available when provided by interested volunteers. The buildslaves +are designed to require an absolute minimum of configuration, reducing +the effort a potential volunteer needs to expend to be able to +contribute a new test environment to the project. The goal is for +anyone who wishes that a given project would run on their favorite +platform should be able to offer that project a buildslave, running on +that platform, where they can verify that their portability code +works, and keeps working. + + +File: buildbot.info, Node: System Architecture, Next: Control Flow, Prev: History and Philosophy, Up: Introduction + +1.2 System Architecture +======================= + +The Buildbot consists of a single `buildmaster' and one or more +`buildslaves', connected in a star topology. The buildmaster makes +all decisions about what, when, and how to build. It sends commands +to be run on the build slaves, which simply execute the commands and +return the results. (certain steps involve more local decision +making, where the overhead of sending a lot of commands back and +forth would be inappropriate, but in general the buildmaster is +responsible for everything). + + The buildmaster is usually fed `Changes' by some sort of version +control system (*note Change Sources::), which may cause builds to be +run. As the builds are performed, various status messages are +produced, which are then sent to any registered Status Targets (*note +Status Delivery::). + + + +------------------+ +-----------+ + | |---------->| Browser | + | BuildMaster | +-----------+ + Changes | |--------------->+--------+ + +----------->| | Build Status | email | + | | |------------+ +--------+ + | | |-------+ | +---------------+ + | +------------------+ | +---->| Status Client | ++----------+ | ^ | ^ | +---------------+ +| Change | | | C| | | +-----+ +| Sources | | | o| | +------------>| IRC | +| | | | m| |R +-----+ +| CVS | v | m| |e +| SVN | +---------+ a| |s +| Darcs | | Build | n| |u +| .. etc | | Slave | d| |l +| | +---------+ s| |t +| | v |s ++----------+ +---------+ + | Build | + | Slave | + +---------+ + + The buildmaster is configured and maintained by the "buildmaster +admin", who is generally the project team member responsible for +build process issues. Each buildslave is maintained by a "buildslave +admin", who do not need to be quite as involved. Generally slaves are +run by anyone who has an interest in seeing the project work well on +their favorite platform. + +* Menu: + +* BuildSlave Connections:: +* Buildmaster Architecture:: +* Status Delivery Architecture:: + + +File: buildbot.info, Node: BuildSlave Connections, Next: Buildmaster Architecture, Prev: System Architecture, Up: System Architecture + +1.2.1 BuildSlave Connections +---------------------------- + +The buildslaves are typically run on a variety of separate machines, +at least one per platform of interest. These machines connect to the +buildmaster over a TCP connection to a publically-visible port. As a +result, the buildslaves can live behind a NAT box or similar +firewalls, as long as they can get to buildmaster. The TCP connections +are initiated by the buildslave and accepted by the buildmaster, but +commands and results travel both ways within this connection. The +buildmaster is always in charge, so all commands travel exclusively +from the buildmaster to the buildslave. + + To perform builds, the buildslaves must typically obtain source +code from a CVS/SVN/etc repository. Therefore they must also be able +to reach the repository. The buildmaster provides instructions for +performing builds, but does not provide the source code itself. + + + +Repository| | BuildMaster | | + (CVS/SVN)| | ^|^^^ | + | | / c \ | +----------+ +------------------/--o----\-+ + ^ / m ^ \ + | / m | \ + checkout/update --+ a | +-- + | TCP| n | |TCP + | | d | | + | | s | | + | | | | | + | | | r | + | | | e | + -N-A-T-|- - - - -N-A-T- - - - -|- |- s-|- - - - -N-A-T- - - + | | | u | + | | | l | + | +------------------|--|--t-|-+ + | | | | s | | + +----| v | | + | | | + | | | + | | + | BuildSlave | + +----------------------------+ + + +File: buildbot.info, Node: Buildmaster Architecture, Next: Status Delivery Architecture, Prev: BuildSlave Connections, Up: System Architecture + +1.2.2 Buildmaster Architecture +------------------------------ + +The Buildmaster consists of several pieces: + + + + +---------------+ + | Change Source |----->----+ + +---------------+ | + Changes + | + +---------------+ v + | Change Source |----->----+ + +---------------+ v + +-----+-------+ + | | + v v + +-----------+ +-----------+ + | Scheduler | | Scheduler | + +-----------+ +-----------+ + | | | + +------+---------+ +---+ +-----+ + | | | | + v | | Build + : : : v v : Request + : : : : | + : ---- : : : | + : ---- : : ---- : | + +======+ +======+ : v : + | | : : + v v : : + +---------+ +---------+ :queue : + | Builder | | Builder | +======+ + +---------+ +---------+ | + v + +---------+ + | Builder | + +---------+ + + * Change Sources, which create a Change object each time something + is modified in the VC repository. Most ChangeSources listen for + messages from a hook script of some sort. Some sources actively + poll the repository on a regular basis. All Changes are fed to + the Schedulers. + + * Schedulers, which decide when builds should be performed. They + collect Changes into BuildRequests, which are then queued for + delivery to Builders until a buildslave is available. + + * Builders, which control exactly _how_ each build is performed + (with a series of BuildSteps, configured in a BuildFactory). Each + Build is run on a single buildslave. + + * Status plugins, which deliver information about the build results + through protocols like HTTP, mail, and IRC. + + + + + +-----------------+ + | BuildSlave | + | | + | | + +-------+ | +------------+ | + |Builder|----Build----->|SlaveBuilder| | + +-------+ | +------------+ | + | | + | +------------+ | + +-Build---->|SlaveBuilder| | + | | +------------+ | + +-------+ | | | + |Builder|---+ +-----------------+ + +-------+ | + | + | +-----------------+ + Build | BuildSlave | + | | | + | | | + | | +------------+ | + +------->|SlaveBuilder| | + | +------------+ | + +-------+ | | + |Builder|--+ | +------------+ | + +-------+ +-------->|SlaveBuilder| | + | +------------+ | + | | + +-----------------+ + + Each Builder is configured with a list of BuildSlaves that it will +use for its builds. These buildslaves are expected to behave +identically: the only reason to use multiple BuildSlaves for a single +Builder is to provide a measure of load-balancing. + + Within a single BuildSlave, each Builder creates its own +SlaveBuilder instance. These SlaveBuilders operate independently from +each other. Each gets its own base directory to work in. It is quite +common to have many Builders sharing the same buildslave. For +example, there might be two buildslaves: one for i386, and a second +for PowerPC. There may then be a pair of Builders that do a full +compile/test run, one for each architecture, and a lone Builder that +creates snapshot source tarballs if the full builders complete +successfully. The full builders would each run on a single +buildslave, whereas the tarball creation step might run on either +buildslave (since the platform doesn't matter when creating source +tarballs). In this case, the mapping would look like: + + Builder(full-i386) -> BuildSlaves(slave-i386) + Builder(full-ppc) -> BuildSlaves(slave-ppc) + Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc) + + and each BuildSlave would have two SlaveBuilders inside it, one +for a full builder, and a second for the source-tarball builder. + + Once a SlaveBuilder is available, the Builder pulls one or more +BuildRequests off its incoming queue. (It may pull more than one if it +determines that it can merge the requests together; for example, there +may be multiple requests to build the current HEAD revision). These +requests are merged into a single Build instance, which includes the +SourceStamp that describes what exact version of the source code +should be used for the build. The Build is then randomly assigned to a +free SlaveBuilder and the build begins. + + The behaviour when BuildRequests are merged can be customized, +*note Merging BuildRequests::. + + +File: buildbot.info, Node: Status Delivery Architecture, Prev: Buildmaster Architecture, Up: System Architecture + +1.2.3 Status Delivery Architecture +---------------------------------- + +The buildmaster maintains a central Status object, to which various +status plugins are connected. Through this Status object, a full +hierarchy of build status objects can be obtained. + + + + Status Objects Status Plugins User Clients + + +------+ +---------+ +-----------+ + |Status|<--------------+-->|Waterfall|<-------|Web Browser| + +------+ | +---------+ +-----------+ + | +-----+ | + v v | ++-------+ +-------+ | +---+ +----------+ +|Builder| |Builder| +---->|IRC|<----------->IRC Server| +|Status | |Status | | +---+ +----------+ ++-------+ +-------+ | + | +----+ | + v v | +------------+ +----+ ++------+ +------+ +-->|MailNotifier|---->|SMTP| +|Build | |Build | +------------+ +----+ +|Status| |Status| ++------+ +------+ + | +-----+ + v v ++------+ +------+ +|Step | |Step | +|Status| |Status| ++------+ +------+ + | +---+ + v v ++----+ +----+ +|Log | |Log | +|File| |File| ++----+ +----+ + + The configuration file controls which status plugins are active. +Each status plugin gets a reference to the top-level Status object. +From there they can request information on each Builder, Build, Step, +and LogFile. This query-on-demand interface is used by the +html.Waterfall plugin to create the main status page each time a web +browser hits the main URL. + + The status plugins can also subscribe to hear about new Builds as +they occur: this is used by the MailNotifier to create new email +messages for each recently-completed Build. + + The Status object records the status of old builds on disk in the +buildmaster's base directory. This allows it to return information +about historical builds. + + There are also status objects that correspond to Schedulers and +BuildSlaves. These allow status plugins to report information about +upcoming builds, and the online/offline status of each buildslave. + + +File: buildbot.info, Node: Control Flow, Prev: System Architecture, Up: Introduction + +1.3 Control Flow +================ + +A day in the life of the buildbot: + + * A developer commits some source code changes to the repository. + A hook script or commit trigger of some sort sends information + about this change to the buildmaster through one of its + configured Change Sources. This notification might arrive via + email, or over a network connection (either initiated by the + buildmaster as it "subscribes" to changes, or by the commit + trigger as it pushes Changes towards the buildmaster). The + Change contains information about who made the change, what + files were modified, which revision contains the change, and any + checkin comments. + + * The buildmaster distributes this change to all of its configured + Schedulers. Any "important" changes cause the "tree-stable-timer" + to be started, and the Change is added to a list of those that + will go into a new Build. When the timer expires, a Build is + started on each of a set of configured Builders, all + compiling/testing the same source code. Unless configured + otherwise, all Builds run in parallel on the various buildslaves. + + * The Build consists of a series of Steps. Each Step causes some + number of commands to be invoked on the remote buildslave + associated with that Builder. The first step is almost always to + perform a checkout of the appropriate revision from the same VC + system that produced the Change. The rest generally perform a + compile and run unit tests. As each Step runs, the buildslave + reports back command output and return status to the buildmaster. + + * As the Build runs, status messages like "Build Started", "Step + Started", "Build Finished", etc, are published to a collection of + Status Targets. One of these targets is usually the HTML + "Waterfall" display, which shows a chronological list of events, + and summarizes the results of the most recent build at the top + of each column. Developers can periodically check this page to + see how their changes have fared. If they see red, they know + that they've made a mistake and need to fix it. If they see + green, they know that they've done their duty and don't need to + worry about their change breaking anything. + + * If a MailNotifier status target is active, the completion of a + build will cause email to be sent to any developers whose + Changes were incorporated into this Build. The MailNotifier can + be configured to only send mail upon failing builds, or for + builds which have just transitioned from passing to failing. + Other status targets can provide similar real-time notification + via different communication channels, like IRC. + + + +File: buildbot.info, Node: Installation, Next: Concepts, Prev: Introduction, Up: Top + +2 Installation +************** + +* Menu: + +* Requirements:: +* Installing the code:: +* Creating a buildmaster:: +* Upgrading an Existing Buildmaster:: +* Creating a buildslave:: +* Launching the daemons:: +* Logfiles:: +* Shutdown:: +* Maintenance:: +* Troubleshooting:: + + +File: buildbot.info, Node: Requirements, Next: Installing the code, Prev: Installation, Up: Installation + +2.1 Requirements +================ + +At a bare minimum, you'll need the following (for both the buildmaster +and a buildslave): + + * Python: http://www.python.org + + Buildbot requires python-2.3 or later, and is primarily developed + against python-2.4. It is also tested against python-2.5 . + + * Twisted: http://twistedmatrix.com + + Both the buildmaster and the buildslaves require Twisted-2.0.x or + later. It has been tested against all releases of Twisted up to + Twisted-2.5.0 (the most recent as of this writing). As always, + the most recent version is recommended. + + Twisted is delivered as a collection of subpackages. You'll need + at least "Twisted" (the core package), and you'll also want + TwistedMail, TwistedWeb, and TwistedWords (for sending email, + serving a web status page, and delivering build status via IRC, + respectively). You might also want TwistedConch (for the + encrypted Manhole debug port). Note that Twisted requires + ZopeInterface to be installed as well. + + + Certain other packages may be useful on the system running the +buildmaster: + + * CVSToys: http://purl.net/net/CVSToys + + If your buildmaster uses FreshCVSSource to receive change + notification from a cvstoys daemon, it will require CVSToys be + installed (tested with CVSToys-1.0.10). If the it doesn't use + that source (i.e. if you only use a mail-parsing change source, + or the SVN notification script), you will not need CVSToys. + + + And of course, your project's build process will impose additional +requirements on the buildslaves. These hosts must have all the tools +necessary to compile and test your project's source code. + + +File: buildbot.info, Node: Installing the code, Next: Creating a buildmaster, Prev: Requirements, Up: Installation + +2.2 Installing the code +======================= + +The Buildbot is installed using the standard python `distutils' +module. After unpacking the tarball, the process is: + + python setup.py build + python setup.py install + + where the install step may need to be done as root. This will put +the bulk of the code in somewhere like +/usr/lib/python2.3/site-packages/buildbot . It will also install the +`buildbot' command-line tool in /usr/bin/buildbot. + + To test this, shift to a different directory (like /tmp), and run: + + buildbot --version + + If it shows you the versions of Buildbot and Twisted, the install +went ok. If it says `no such command' or it gets an `ImportError' +when it tries to load the libaries, then something went wrong. +`pydoc buildbot' is another useful diagnostic tool. + + Windows users will find these files in other places. You will need +to make sure that python can find the libraries, and will probably +find it convenient to have `buildbot' on your PATH. + + If you wish, you can run the buildbot unit test suite like this: + + PYTHONPATH=. trial buildbot.test + + This should run up to 192 tests, depending upon what VC tools you +have installed. On my desktop machine it takes about five minutes to +complete. Nothing should fail, a few might be skipped. If any of the +tests fail, you should stop and investigate the cause before +continuing the installation process, as it will probably be easier to +track down the bug early. + + If you cannot or do not wish to install the buildbot into a +site-wide location like `/usr' or `/usr/local', you can also install +it into the account's home directory. Do the install command like +this: + + python setup.py install --home=~ + + That will populate `~/lib/python' and create `~/bin/buildbot'. +Make sure this lib directory is on your `PYTHONPATH'. + + +File: buildbot.info, Node: Creating a buildmaster, Next: Upgrading an Existing Buildmaster, Prev: Installing the code, Up: Installation + +2.3 Creating a buildmaster +========================== + +As you learned earlier (*note System Architecture::), the buildmaster +runs on a central host (usually one that is publically visible, so +everybody can check on the status of the project), and controls all +aspects of the buildbot system. Let us call this host +`buildbot.example.org'. + + You may wish to create a separate user account for the buildmaster, +perhaps named `buildmaster'. This can help keep your personal +configuration distinct from that of the buildmaster and is useful if +you have to use a mail-based notification system (*note Change +Sources::). However, the Buildbot will work just fine with your +regular user account. + + You need to choose a directory for the buildmaster, called the +`basedir'. This directory will be owned by the buildmaster, which +will use configuration files therein, and create status files as it +runs. `~/Buildbot' is a likely value. If you run multiple +buildmasters in the same account, or if you run both masters and +slaves, you may want a more distinctive name like +`~/Buildbot/master/gnomovision' or `~/Buildmasters/fooproject'. If +you are using a separate user account, this might just be +`~buildmaster/masters/fooproject'. + + Once you've picked a directory, use the `buildbot create-master' +command to create the directory and populate it with startup files: + + buildbot create-master BASEDIR + + You will need to create a configuration file (*note +Configuration::) before starting the buildmaster. Most of the rest of +this manual is dedicated to explaining how to do this. A sample +configuration file is placed in the working directory, named +`master.cfg.sample', which can be copied to `master.cfg' and edited +to suit your purposes. + + (Internal details: This command creates a file named +`buildbot.tac' that contains all the state necessary to create the +buildmaster. Twisted has a tool called `twistd' which can use this +.tac file to create and launch a buildmaster instance. twistd takes +care of logging and daemonization (running the program in the +background). `/usr/bin/buildbot' is a front end which runs twistd for +you.) + + In addition to `buildbot.tac', a small `Makefile.sample' is +installed. This can be used as the basis for customized daemon +startup, *Note Launching the daemons::. + + +File: buildbot.info, Node: Upgrading an Existing Buildmaster, Next: Creating a buildslave, Prev: Creating a buildmaster, Up: Installation + +2.4 Upgrading an Existing Buildmaster +===================================== + +If you have just installed a new version of the Buildbot code, and you +have buildmasters that were created using an older version, you'll +need to upgrade these buildmasters before you can use them. The +upgrade process adds and modifies files in the buildmaster's base +directory to make it compatible with the new code. + + buildbot upgrade-master BASEDIR + + This command will also scan your `master.cfg' file for +incompatbilities (by loading it and printing any errors or deprecation +warnings that occur). Each buildbot release tries to be compatible +with configurations that worked cleanly (i.e. without deprecation +warnings) on the previous release: any functions or classes that are +to be removed will first be deprecated in a release, to give users a +chance to start using their replacement. + + The 0.7.6 release introduced the `public_html/' directory, which +contains `index.html' and other files served by the `WebStatus' and +`Waterfall' status displays. The `upgrade-master' command will create +these files if they do not already exist. It will not modify existing +copies, but it will write a new copy in e.g. `index.html.new' if the +new version differs from the version that already exists. + + The `upgrade-master' command is idempotent. It is safe to run it +multiple times. After each upgrade of the buildbot code, you should +use `upgrade-master' on all your buildmasters. + + +File: buildbot.info, Node: Creating a buildslave, Next: Launching the daemons, Prev: Upgrading an Existing Buildmaster, Up: Installation + +2.5 Creating a buildslave +========================= + +Typically, you will be adding a buildslave to an existing buildmaster, +to provide additional architecture coverage. The buildbot +administrator will give you several pieces of information necessary to +connect to the buildmaster. You should also be somewhat familiar with +the project being tested, so you can troubleshoot build problems +locally. + + The buildbot exists to make sure that the project's stated "how to +build it" process actually works. To this end, the buildslave should +run in an environment just like that of your regular developers. +Typically the project build process is documented somewhere +(`README', `INSTALL', etc), in a document that should mention all +library dependencies and contain a basic set of build instructions. +This document will be useful as you configure the host and account in +which the buildslave runs. + + Here's a good checklist for setting up a buildslave: + + 1. Set up the account + + It is recommended (although not mandatory) to set up a separate + user account for the buildslave. This account is frequently named + `buildbot' or `buildslave'. This serves to isolate your personal + working environment from that of the slave's, and helps to + minimize the security threat posed by letting possibly-unknown + contributors run arbitrary code on your system. The account + should have a minimum of fancy init scripts. + + 2. Install the buildbot code + + Follow the instructions given earlier (*note Installing the + code::). If you use a separate buildslave account, and you + didn't install the buildbot code to a shared location, then you + will need to install it with `--home=~' for each account that + needs it. + + 3. Set up the host + + Make sure the host can actually reach the buildmaster. Usually + the buildmaster is running a status webserver on the same + machine, so simply point your web browser at it and see if you + can get there. Install whatever additional packages or + libraries the project's INSTALL document advises. (or not: if + your buildslave is supposed to make sure that building without + optional libraries still works, then don't install those + libraries). + + Again, these libraries don't necessarily have to be installed to + a site-wide shared location, but they must be available to your + build process. Accomplishing this is usually very specific to + the build process, so installing them to `/usr' or `/usr/local' + is usually the best approach. + + 4. Test the build process + + Follow the instructions in the INSTALL document, in the + buildslave's account. Perform a full CVS (or whatever) checkout, + configure, make, run tests, etc. Confirm that the build works + without manual fussing. If it doesn't work when you do it by + hand, it will be unlikely to work when the buildbot attempts to + do it in an automated fashion. + + 5. Choose a base directory + + This should be somewhere in the buildslave's account, typically + named after the project which is being tested. The buildslave + will not touch any file outside of this directory. Something + like `~/Buildbot' or `~/Buildslaves/fooproject' is appropriate. + + 6. Get the buildmaster host/port, botname, and password + + When the buildbot admin configures the buildmaster to accept and + use your buildslave, they will provide you with the following + pieces of information: + + * your buildslave's name + + * the password assigned to your buildslave + + * the hostname and port number of the buildmaster, i.e. + buildbot.example.org:8007 + + 7. Create the buildslave + + Now run the 'buildbot' command as follows: + + buildbot create-slave BASEDIR MASTERHOST:PORT SLAVENAME PASSWORD + + This will create the base directory and a collection of files + inside, including the `buildbot.tac' file that contains all the + information you passed to the `buildbot' command. + + 8. Fill in the hostinfo files + + When it first connects, the buildslave will send a few files up + to the buildmaster which describe the host that it is running + on. These files are presented on the web status display so that + developers have more information to reproduce any test failures + that are witnessed by the buildbot. There are sample files in + the `info' subdirectory of the buildbot's base directory. You + should edit these to correctly describe you and your host. + + `BASEDIR/info/admin' should contain your name and email address. + This is the "buildslave admin address", and will be visible from + the build status page (so you may wish to munge it a bit if + address-harvesting spambots are a concern). + + `BASEDIR/info/host' should be filled with a brief description of + the host: OS, version, memory size, CPU speed, versions of + relevant libraries installed, and finally the version of the + buildbot code which is running the buildslave. + + If you run many buildslaves, you may want to create a single + `~buildslave/info' file and share it among all the buildslaves + with symlinks. + + +* Menu: + +* Buildslave Options:: + + +File: buildbot.info, Node: Buildslave Options, Prev: Creating a buildslave, Up: Creating a buildslave + +2.5.1 Buildslave Options +------------------------ + +There are a handful of options you might want to use when creating the +buildslave with the `buildbot create-slave <options> DIR <params>' +command. You can type `buildbot create-slave --help' for a summary. +To use these, just include them on the `buildbot create-slave' +command line, like this: + + buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd + +`--usepty' + This is a boolean flag that tells the buildslave whether to + launch child processes in a PTY or with regular pipes (the + default) when the master does not specify. This option is + deprecated, as this particular parameter is better specified on + the master. + +`--umask' + This is a string (generally an octal representation of an + integer) which will cause the buildslave process' "umask" value + to be set shortly after initialization. The "twistd" + daemonization utility forces the umask to 077 at startup (which + means that all files created by the buildslave or its child + processes will be unreadable by any user other than the + buildslave account). If you want build products to be readable + by other accounts, you can add `--umask=022' to tell the + buildslave to fix the umask after twistd clobbers it. If you want + build products to be _writable_ by other accounts too, use + `--umask=000', but this is likely to be a security problem. + +`--keepalive' + This is a number that indicates how frequently "keepalive" + messages should be sent from the buildslave to the buildmaster, + expressed in seconds. The default (600) causes a message to be + sent to the buildmaster at least once every 10 minutes. To set + this to a lower value, use e.g. `--keepalive=120'. + + If the buildslave is behind a NAT box or stateful firewall, these + messages may help to keep the connection alive: some NAT boxes + tend to forget about a connection if it has not been used in a + while. When this happens, the buildmaster will think that the + buildslave has disappeared, and builds will time out. Meanwhile + the buildslave will not realize than anything is wrong. + +`--maxdelay' + This is a number that indicates the maximum amount of time the + buildslave will wait between connection attempts, expressed in + seconds. The default (300) causes the buildslave to wait at most + 5 minutes before trying to connect to the buildmaster again. + +`--log-size' + This is the size in bytes when to rotate the Twisted log files. + +`--log-count' + This is the number of log rotations to keep around. You can + either specify a number or `None' (the default) to keep all + `twistd.log' files around. + + + +File: buildbot.info, Node: Launching the daemons, Next: Logfiles, Prev: Creating a buildslave, Up: Installation + +2.6 Launching the daemons +========================= + +Both the buildmaster and the buildslave run as daemon programs. To +launch them, pass the working directory to the `buildbot' command: + + buildbot start BASEDIR + + This command will start the daemon and then return, so normally it +will not produce any output. To verify that the programs are indeed +running, look for a pair of files named `twistd.log' and `twistd.pid' +that should be created in the working directory. `twistd.pid' +contains the process ID of the newly-spawned daemon. + + When the buildslave connects to the buildmaster, new directories +will start appearing in its base directory. The buildmaster tells the +slave to create a directory for each Builder which will be using that +slave. All build operations are performed within these directories: +CVS checkouts, compiles, and tests. + + Once you get everything running, you will want to arrange for the +buildbot daemons to be started at boot time. One way is to use +`cron', by putting them in a @reboot crontab entry(1): + + @reboot buildbot start BASEDIR + + When you run `crontab' to set this up, remember to do it as the +buildmaster or buildslave account! If you add this to your crontab +when running as your regular account (or worse yet, root), then the +daemon will run as the wrong user, quite possibly as one with more +authority than you intended to provide. + + It is important to remember that the environment provided to cron +jobs and init scripts can be quite different that your normal runtime. +There may be fewer environment variables specified, and the PATH may +be shorter than usual. It is a good idea to test out this method of +launching the buildslave by using a cron job with a time in the near +future, with the same command, and then check `twistd.log' to make +sure the slave actually started correctly. Common problems here are +for `/usr/local' or `~/bin' to not be on your `PATH', or for +`PYTHONPATH' to not be set correctly. Sometimes `HOME' is messed up +too. + + To modify the way the daemons are started (perhaps you want to set +some environment variables first, or perform some cleanup each time), +you can create a file named `Makefile.buildbot' in the base +directory. When the `buildbot' front-end tool is told to `start' the +daemon, and it sees this file (and `/usr/bin/make' exists), it will +do `make -f Makefile.buildbot start' instead of its usual action +(which involves running `twistd'). When the buildmaster or buildslave +is installed, a `Makefile.sample' is created which implements the +same behavior as the the `buildbot' tool uses, so if you want to +customize the process, just copy `Makefile.sample' to +`Makefile.buildbot' and edit it as necessary. + + Some distributions may include conveniences to make starting +buildbot at boot time easy. For instance, with the default buildbot +package in Debian-based distributions, you may only need to modify +`/etc/default/buildbot' (see also `/etc/init.d/buildbot', which reads +the configuration in `/etc/default/buildbot'). + + ---------- Footnotes ---------- + + (1) this @reboot syntax is understood by Vixie cron, which is the +flavor usually provided with linux systems. Other unices may have a +cron that doesn't understand @reboot + + +File: buildbot.info, Node: Logfiles, Next: Shutdown, Prev: Launching the daemons, Up: Installation + +2.7 Logfiles +============ + +While a buildbot daemon runs, it emits text to a logfile, named +`twistd.log'. A command like `tail -f twistd.log' is useful to watch +the command output as it runs. + + The buildmaster will announce any errors with its configuration +file in the logfile, so it is a good idea to look at the log at +startup time to check for any problems. Most buildmaster activities +will cause lines to be added to the log. + + +File: buildbot.info, Node: Shutdown, Next: Maintenance, Prev: Logfiles, Up: Installation + +2.8 Shutdown +============ + +To stop a buildmaster or buildslave manually, use: + + buildbot stop BASEDIR + + This simply looks for the `twistd.pid' file and kills whatever +process is identified within. + + At system shutdown, all processes are sent a `SIGKILL'. The +buildmaster and buildslave will respond to this by shutting down +normally. + + The buildmaster will respond to a `SIGHUP' by re-reading its +config file. Of course, this only works on unix-like systems with +signal support, and won't work on Windows. The following shortcut is +available: + + buildbot reconfig BASEDIR + + When you update the Buildbot code to a new release, you will need +to restart the buildmaster and/or buildslave before it can take +advantage of the new code. You can do a `buildbot stop BASEDIR' and +`buildbot start BASEDIR' in quick succession, or you can use the +`restart' shortcut, which does both steps for you: + + buildbot restart BASEDIR + + There are certain configuration changes that are not handled +cleanly by `buildbot reconfig'. If this occurs, `buildbot restart' is +a more robust tool to fully switch over to the new configuration. + + `buildbot restart' may also be used to start a stopped Buildbot +instance. This behaviour is useful when writing scripts that stop, +start and restart Buildbot. + + A buildslave may also be gracefully shutdown from the *note +WebStatus:: status plugin. This is useful to shutdown a buildslave +without interrupting any current builds. The buildmaster will wait +until the buildslave is finished all its current builds, and will +then tell the buildslave to shutdown. + + +File: buildbot.info, Node: Maintenance, Next: Troubleshooting, Prev: Shutdown, Up: Installation + +2.9 Maintenance +=============== + +It is a good idea to check the buildmaster's status page every once in +a while, to see if your buildslave is still online. Eventually the +buildbot will probably be enhanced to send you email (via the +`info/admin' email address) when the slave has been offline for more +than a few hours. + + If you find you can no longer provide a buildslave to the project, +please let the project admins know, so they can put out a call for a +replacement. + + The Buildbot records status and logs output continually, each time +a build is performed. The status tends to be small, but the build logs +can become quite large. Each build and log are recorded in a separate +file, arranged hierarchically under the buildmaster's base directory. +To prevent these files from growing without bound, you should +periodically delete old build logs. A simple cron job to delete +anything older than, say, two weeks should do the job. The only trick +is to leave the `buildbot.tac' and other support files alone, for +which find's `-mindepth' argument helps skip everything in the top +directory. You can use something like the following: + + @weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm {} \; + @weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \; + + +File: buildbot.info, Node: Troubleshooting, Prev: Maintenance, Up: Installation + +2.10 Troubleshooting +==================== + +Here are a few hints on diagnosing common problems. + +* Menu: + +* Starting the buildslave:: +* Connecting to the buildmaster:: +* Forcing Builds:: + + +File: buildbot.info, Node: Starting the buildslave, Next: Connecting to the buildmaster, Prev: Troubleshooting, Up: Troubleshooting + +2.10.1 Starting the buildslave +------------------------------ + +Cron jobs are typically run with a minimal shell (`/bin/sh', not +`/bin/bash'), and tilde expansion is not always performed in such +commands. You may want to use explicit paths, because the `PATH' is +usually quite short and doesn't include anything set by your shell's +startup scripts (`.profile', `.bashrc', etc). If you've installed +buildbot (or other python libraries) to an unusual location, you may +need to add a `PYTHONPATH' specification (note that python will do +tilde-expansion on `PYTHONPATH' elements by itself). Sometimes it is +safer to fully-specify everything: + + @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir + + Take the time to get the @reboot job set up. Otherwise, things +will work fine for a while, but the first power outage or system +reboot you have will stop the buildslave with nothing but the cries +of sorrowful developers to remind you that it has gone away. + + +File: buildbot.info, Node: Connecting to the buildmaster, Next: Forcing Builds, Prev: Starting the buildslave, Up: Troubleshooting + +2.10.2 Connecting to the buildmaster +------------------------------------ + +If the buildslave cannot connect to the buildmaster, the reason should +be described in the `twistd.log' logfile. Some common problems are an +incorrect master hostname or port number, or a mistyped bot name or +password. If the buildslave loses the connection to the master, it is +supposed to attempt to reconnect with an exponentially-increasing +backoff. Each attempt (and the time of the next attempt) will be +logged. If you get impatient, just manually stop and re-start the +buildslave. + + When the buildmaster is restarted, all slaves will be disconnected, +and will attempt to reconnect as usual. The reconnect time will depend +upon how long the buildmaster is offline (i.e. how far up the +exponential backoff curve the slaves have travelled). Again, +`buildbot stop BASEDIR; buildbot start BASEDIR' will speed up the +process. + + +File: buildbot.info, Node: Forcing Builds, Prev: Connecting to the buildmaster, Up: Troubleshooting + +2.10.3 Forcing Builds +--------------------- + +From the buildmaster's main status web page, you can force a build to +be run on your build slave. Figure out which column is for a builder +that runs on your slave, click on that builder's name, and the page +that comes up will have a "Force Build" button. Fill in the form, hit +the button, and a moment later you should see your slave's +`twistd.log' filling with commands being run. Using `pstree' or `top' +should also reveal the cvs/make/gcc/etc processes being run by the +buildslave. Note that the same web page should also show the `admin' +and `host' information files that you configured earlier. + + +File: buildbot.info, Node: Concepts, Next: Configuration, Prev: Installation, Up: Top + +3 Concepts +********** + +This chapter defines some of the basic concepts that the Buildbot +uses. You'll need to understand how the Buildbot sees the world to +configure it properly. + +* Menu: + +* Version Control Systems:: +* Schedulers:: +* BuildSet:: +* BuildRequest:: +* Builder:: +* Users:: +* Build Properties:: + + +File: buildbot.info, Node: Version Control Systems, Next: Schedulers, Prev: Concepts, Up: Concepts + +3.1 Version Control Systems +=========================== + +These source trees come from a Version Control System of some kind. +CVS and Subversion are two popular ones, but the Buildbot supports +others. All VC systems have some notion of an upstream `repository' +which acts as a server(1), from which clients can obtain source trees +according to various parameters. The VC repository provides source +trees of various projects, for different branches, and from various +points in time. The first thing we have to do is to specify which +source tree we want to get. + +* Menu: + +* Generalizing VC Systems:: +* Source Tree Specifications:: +* How Different VC Systems Specify Sources:: +* Attributes of Changes:: + + ---------- Footnotes ---------- + + (1) except Darcs, but since the Buildbot never modifies its local +source tree we can ignore the fact that Darcs uses a less centralized +model + + +File: buildbot.info, Node: Generalizing VC Systems, Next: Source Tree Specifications, Prev: Version Control Systems, Up: Version Control Systems + +3.1.1 Generalizing VC Systems +----------------------------- + +For the purposes of the Buildbot, we will try to generalize all VC +systems as having repositories that each provide sources for a variety +of projects. Each project is defined as a directory tree with source +files. The individual files may each have revisions, but we ignore +that and treat the project as a whole as having a set of revisions +(CVS is really the only VC system still in widespread use that has +per-file revisions.. everything modern has moved to atomic tree-wide +changesets). Each time someone commits a change to the project, a new +revision becomes available. These revisions can be described by a +tuple with two items: the first is a branch tag, and the second is +some kind of revision stamp or timestamp. Complex projects may have +multiple branch tags, but there is always a default branch. The +timestamp may be an actual timestamp (such as the -D option to CVS), +or it may be a monotonically-increasing transaction number (such as +the change number used by SVN and P4, or the revision number used by +Arch/Baz/Bazaar, or a labeled tag used in CVS)(1). The SHA1 revision +ID used by Monotone, Mercurial, and Git is also a kind of revision +stamp, in that it specifies a unique copy of the source tree, as does +a Darcs "context" file. + + When we aren't intending to make any changes to the sources we +check out (at least not any that need to be committed back upstream), +there are two basic ways to use a VC system: + + * Retrieve a specific set of source revisions: some tag or key is + used to index this set, which is fixed and cannot be changed by + subsequent developers committing new changes to the tree. + Releases are built from tagged revisions like this, so that they + can be rebuilt again later (probably with controlled + modifications). + + * Retrieve the latest sources along a specific branch: some tag is + used to indicate which branch is to be used, but within that + constraint we want to get the latest revisions. + + Build personnel or CM staff typically use the first approach: the +build that results is (ideally) completely specified by the two +parameters given to the VC system: repository and revision tag. This +gives QA and end-users something concrete to point at when reporting +bugs. Release engineers are also reportedly fond of shipping code that +can be traced back to a concise revision tag of some sort. + + Developers are more likely to use the second approach: each morning +the developer does an update to pull in the changes committed by the +team over the last day. These builds are not easy to fully specify: it +depends upon exactly when you did a checkout, and upon what local +changes the developer has in their tree. Developers do not normally +tag each build they produce, because there is usually significant +overhead involved in creating these tags. Recreating the trees used by +one of these builds can be a challenge. Some VC systems may provide +implicit tags (like a revision number), while others may allow the use +of timestamps to mean "the state of the tree at time X" as opposed to +a tree-state that has been explicitly marked. + + The Buildbot is designed to help developers, so it usually works in +terms of _the latest_ sources as opposed to specific tagged +revisions. However, it would really prefer to build from reproducible +source trees, so implicit revisions are used whenever possible. + + ---------- Footnotes ---------- + + (1) many VC systems provide more complexity than this: in +particular the local views that P4 and ClearCase can assemble out of +various source directories are more complex than we're prepared to +take advantage of here + + +File: buildbot.info, Node: Source Tree Specifications, Next: How Different VC Systems Specify Sources, Prev: Generalizing VC Systems, Up: Version Control Systems + +3.1.2 Source Tree Specifications +-------------------------------- + +So for the Buildbot's purposes we treat each VC system as a server +which can take a list of specifications as input and produce a source +tree as output. Some of these specifications are static: they are +attributes of the builder and do not change over time. Others are more +variable: each build will have a different value. The repository is +changed over time by a sequence of Changes, each of which represents a +single developer making changes to some set of files. These Changes +are cumulative(1). + + For normal builds, the Buildbot wants to get well-defined source +trees that contain specific Changes, and exclude other Changes that +may have occurred after the desired ones. We assume that the Changes +arrive at the buildbot (through one of the mechanisms described in +*note Change Sources::) in the same order in which they are committed +to the repository. The Buildbot waits for the tree to become "stable" +before initiating a build, for two reasons. The first is that +developers frequently make multiple related commits in quick +succession, even when the VC system provides ways to make atomic +transactions involving multiple files at the same time. Running a +build in the middle of these sets of changes would use an inconsistent +set of source files, and is likely to fail (and is certain to be less +useful than a build which uses the full set of changes). The +tree-stable-timer is intended to avoid these useless builds that +include some of the developer's changes but not all. The second reason +is that some VC systems (i.e. CVS) do not provide repository-wide +transaction numbers, so that timestamps are the only way to refer to +a specific repository state. These timestamps may be somewhat +ambiguous, due to processing and notification delays. By waiting until +the tree has been stable for, say, 10 minutes, we can choose a +timestamp from the middle of that period to use for our source +checkout, and then be reasonably sure that any clock-skew errors will +not cause the build to be performed on an inconsistent set of source +files. + + The Schedulers always use the tree-stable-timer, with a timeout +that is configured to reflect a reasonable tradeoff between build +latency and change frequency. When the VC system provides coherent +repository-wide revision markers (such as Subversion's revision +numbers, or in fact anything other than CVS's timestamps), the +resulting Build is simply performed against a source tree defined by +that revision marker. When the VC system does not provide this, a +timestamp from the middle of the tree-stable period is used to +generate the source tree(2). + + ---------- Footnotes ---------- + + (1) Monotone's _multiple heads_ feature violates this assumption +of cumulative Changes, but in most situations the changes don't occur +frequently enough for this to be a significant problem + + (2) this `checkoutDelay' defaults to half the tree-stable timer, +but it can be overridden with an argument to the Source Step + + +File: buildbot.info, Node: How Different VC Systems Specify Sources, Next: Attributes of Changes, Prev: Source Tree Specifications, Up: Version Control Systems + +3.1.3 How Different VC Systems Specify Sources +---------------------------------------------- + +For CVS, the static specifications are `repository' and `module'. In +addition to those, each build uses a timestamp (or omits the +timestamp to mean `the latest') and `branch tag' (which defaults to +HEAD). These parameters collectively specify a set of sources from +which a build may be performed. + + Subversion (http://subversion.tigris.org) combines the repository, +module, and branch into a single `Subversion URL' parameter. Within +that scope, source checkouts can be specified by a numeric `revision +number' (a repository-wide monotonically-increasing marker, such that +each transaction that changes the repository is indexed by a +different revision number), or a revision timestamp. When branches +are used, the repository and module form a static `baseURL', while +each build has a `revision number' and a `branch' (which defaults to a +statically-specified `defaultBranch'). The `baseURL' and `branch' are +simply concatenated together to derive the `svnurl' to use for the +checkout. + + Perforce (http://www.perforce.com/) is similar. The server is +specified through a `P4PORT' parameter. Module and branch are +specified in a single depot path, and revisions are depot-wide. When +branches are used, the `p4base' and `defaultBranch' are concatenated +together to produce the depot path. + + Arch (http://wiki.gnuarch.org/) and Bazaar +(http://bazaar.canonical.com/) specify a repository by URL, as well +as a `version' which is kind of like a branch name. Arch uses the +word `archive' to represent the repository. Arch lets you push +changes from one archive to another, removing the strict +centralization required by CVS and SVN. It retains the distinction +between repository and working directory that most other VC systems +use. For complex multi-module directory structures, Arch has a +built-in `build config' layer with which the checkout process has two +steps. First, an initial bootstrap checkout is performed to retrieve +a set of build-config files. Second, one of these files is used to +figure out which archives/modules should be used to populate +subdirectories of the initial checkout. + + Builders which use Arch and Bazaar therefore have a static archive +`url', and a default "branch" (which is a string that specifies a +complete category-branch-version triple). Each build can have its own +branch (the category-branch-version string) to override the default, +as well as a revision number (which is turned into a -patch-NN suffix +when performing the checkout). + + Bzr (http://bazaar-vcs.org) (which is a descendant of Arch/Bazaar, +and is frequently referred to as "Bazaar") has the same sort of +repository-vs-workspace model as Arch, but the repository data can +either be stored inside the working directory or kept elsewhere +(either on the same machine or on an entirely different machine). For +the purposes of Buildbot (which never commits changes), the repository +is specified with a URL and a revision number. + + The most common way to obtain read-only access to a bzr tree is via +HTTP, simply by making the repository visible through a web server +like Apache. Bzr can also use FTP and SFTP servers, if the buildslave +process has sufficient privileges to access them. Higher performance +can be obtained by running a special Bazaar-specific server. None of +these matter to the buildbot: the repository URL just has to match the +kind of server being used. The `repoURL' argument provides the +location of the repository. + + Branches are expressed as subdirectories of the main central +repository, which means that if branches are being used, the BZR step +is given a `baseURL' and `defaultBranch' instead of getting the +`repoURL' argument. + + Darcs (http://darcs.net/) doesn't really have the notion of a +single master repository. Nor does it really have branches. In Darcs, +each working directory is also a repository, and there are operations +to push and pull patches from one of these `repositories' to another. +For the Buildbot's purposes, all you need to do is specify the URL of +a repository that you want to build from. The build slave will then +pull the latest patches from that repository and build them. Multiple +branches are implemented by using multiple repositories (possibly +living on the same server). + + Builders which use Darcs therefore have a static `repourl' which +specifies the location of the repository. If branches are being used, +the source Step is instead configured with a `baseURL' and a +`defaultBranch', and the two strings are simply concatenated together +to obtain the repository's URL. Each build then has a specific branch +which replaces `defaultBranch', or just uses the default one. Instead +of a revision number, each build can have a "context", which is a +string that records all the patches that are present in a given tree +(this is the output of `darcs changes --context', and is considerably +less concise than, e.g. Subversion's revision number, but the +patch-reordering flexibility of Darcs makes it impossible to provide +a shorter useful specification). + + Mercurial (http://selenic.com/mercurial) is like Darcs, in that +each branch is stored in a separate repository. The `repourl', +`baseURL', and `defaultBranch' arguments are all handled the same way +as with Darcs. The "revision", however, is the hash identifier +returned by `hg identify'. + + Git (http://git.or.cz/) also follows a decentralized model, and +each repository can have several branches and tags. The source Step is +configured with a static `repourl' which specifies the location of +the repository. In addition, an optional `branch' parameter can be +specified to check out code from a specific branch instead of the +default "master" branch. The "revision" is specified as a SHA1 hash +as returned by e.g. `git rev-parse'. No attempt is made to ensure +that the specified revision is actually a subset of the specified +branch. + + +File: buildbot.info, Node: Attributes of Changes, Prev: How Different VC Systems Specify Sources, Up: Version Control Systems + +3.1.4 Attributes of Changes +--------------------------- + +Who +=== + +Each Change has a `who' attribute, which specifies which developer is +responsible for the change. This is a string which comes from a +namespace controlled by the VC repository. Frequently this means it +is a username on the host which runs the repository, but not all VC +systems require this (Arch, for example, uses a fully-qualified `Arch +ID', which looks like an email address, as does Darcs). Each +StatusNotifier will map the `who' attribute into something +appropriate for their particular means of communication: an email +address, an IRC handle, etc. + +Files +===== + +It also has a list of `files', which are just the tree-relative +filenames of any files that were added, deleted, or modified for this +Change. These filenames are used by the `fileIsImportant' function +(in the Scheduler) to decide whether it is worth triggering a new +build or not, e.g. the function could use the following function to +only run a build if a C file were checked in: + + def has_C_files(change): + for name in change.files: + if name.endswith(".c"): + return True + return False + + Certain BuildSteps can also use the list of changed files to run a +more targeted series of tests, e.g. the `python_twisted.Trial' step +can run just the unit tests that provide coverage for the modified +.py files instead of running the full test suite. + +Comments +======== + +The Change also has a `comments' attribute, which is a string +containing any checkin comments. + +Revision +======== + +Each Change can have a `revision' attribute, which describes how to +get a tree with a specific state: a tree which includes this Change +(and all that came before it) but none that come after it. If this +information is unavailable, the `.revision' attribute will be `None'. +These revisions are provided by the ChangeSource, and consumed by the +`computeSourceRevision' method in the appropriate `step.Source' class. + +`CVS' + `revision' is an int, seconds since the epoch + +`SVN' + `revision' is an int, the changeset number (r%d) + +`Darcs' + `revision' is a large string, the output of `darcs changes + --context' + +`Mercurial' + `revision' is a short string (a hash ID), the output of `hg + identify' + +`Arch/Bazaar' + `revision' is the full revision ID (ending in -patch-%d) + +`P4' + `revision' is an int, the transaction number + +`Git' + `revision' is a short string (a SHA1 hash), the output of e.g. + `git rev-parse' + +Branches +======== + +The Change might also have a `branch' attribute. This indicates that +all of the Change's files are in the same named branch. The +Schedulers get to decide whether the branch should be built or not. + + For VC systems like CVS, Arch, Monotone, and Git, the `branch' +name is unrelated to the filename. (that is, the branch name and the +filename inhabit unrelated namespaces). For SVN, branches are +expressed as subdirectories of the repository, so the file's "svnurl" +is a combination of some base URL, the branch name, and the filename +within the branch. (In a sense, the branch name and the filename +inhabit the same namespace). Darcs branches are subdirectories of a +base URL just like SVN. Mercurial branches are the same as Darcs. + +`CVS' + branch='warner-newfeature', files=['src/foo.c'] + +`SVN' + branch='branches/warner-newfeature', files=['src/foo.c'] + +`Darcs' + branch='warner-newfeature', files=['src/foo.c'] + +`Mercurial' + branch='warner-newfeature', files=['src/foo.c'] + +`Arch/Bazaar' + branch='buildbot-usebranches-0', files=['buildbot/master.py'] + +`Git' + branch='warner-newfeature', files=['src/foo.c'] + +Links +===== + +Finally, the Change might have a `links' list, which is intended to +provide a list of URLs to a _viewcvs_-style web page that provides +more detail for this Change, perhaps including the full file diffs. + + +File: buildbot.info, Node: Schedulers, Next: BuildSet, Prev: Version Control Systems, Up: Concepts + +3.2 Schedulers +============== + +Each Buildmaster has a set of `Scheduler' objects, each of which gets +a copy of every incoming Change. The Schedulers are responsible for +deciding when Builds should be run. Some Buildbot installations might +have a single Scheduler, while others may have several, each for a +different purpose. + + For example, a "quick" scheduler might exist to give immediate +feedback to developers, hoping to catch obvious problems in the code +that can be detected quickly. These typically do not run the full test +suite, nor do they run on a wide variety of platforms. They also +usually do a VC update rather than performing a brand-new checkout +each time. You could have a "quick" scheduler which used a 30 second +timeout, and feeds a single "quick" Builder that uses a VC +`mode='update'' setting. + + A separate "full" scheduler would run more comprehensive tests a +little while later, to catch more subtle problems. This scheduler +would have a longer tree-stable-timer, maybe 30 minutes, and would +feed multiple Builders (with a `mode=' of `'copy'', `'clobber'', or +`'export''). + + The `tree-stable-timer' and `fileIsImportant' decisions are made +by the Scheduler. Dependencies are also implemented here. Periodic +builds (those which are run every N seconds rather than after new +Changes arrive) are triggered by a special `Periodic' Scheduler +subclass. The default Scheduler class can also be told to watch for +specific branches, ignoring Changes on other branches. This may be +useful if you have a trunk and a few release branches which should be +tracked, but when you don't want to have the Buildbot pay attention +to several dozen private user branches. + + When the setup has multiple sources of Changes the `category' can +be used for `Scheduler' objects to filter out a subset of the +Changes. Note that not all change sources can attach a category. + + Some Schedulers may trigger builds for other reasons, other than +recent Changes. For example, a Scheduler subclass could connect to a +remote buildmaster and watch for builds of a library to succeed before +triggering a local build that uses that library. + + Each Scheduler creates and submits `BuildSet' objects to the +`BuildMaster', which is then responsible for making sure the +individual `BuildRequests' are delivered to the target `Builders'. + + `Scheduler' instances are activated by placing them in the +`c['schedulers']' list in the buildmaster config file. Each Scheduler +has a unique name. + + +File: buildbot.info, Node: BuildSet, Next: BuildRequest, Prev: Schedulers, Up: Concepts + +3.3 BuildSet +============ + +A `BuildSet' is the name given to a set of Builds that all +compile/test the same version of the tree on multiple Builders. In +general, all these component Builds will perform the same sequence of +Steps, using the same source code, but on different platforms or +against a different set of libraries. + + The `BuildSet' is tracked as a single unit, which fails if any of +the component Builds have failed, and therefore can succeed only if +_all_ of the component Builds have succeeded. There are two kinds of +status notification messages that can be emitted for a BuildSet: the +`firstFailure' type (which fires as soon as we know the BuildSet will +fail), and the `Finished' type (which fires once the BuildSet has +completely finished, regardless of whether the overall set passed or +failed). + + A `BuildSet' is created with a _source stamp_ tuple of (branch, +revision, changes, patch), some of which may be None, and a list of +Builders on which it is to be run. They are then given to the +BuildMaster, which is responsible for creating a separate +`BuildRequest' for each Builder. + + There are a couple of different likely values for the +`SourceStamp': + +`(revision=None, changes=[CHANGES], patch=None)' + This is a `SourceStamp' used when a series of Changes have + triggered a build. The VC step will attempt to check out a tree + that contains CHANGES (and any changes that occurred before + CHANGES, but not any that occurred after them). + +`(revision=None, changes=None, patch=None)' + This builds the most recent code on the default branch. This is + the sort of `SourceStamp' that would be used on a Build that was + triggered by a user request, or a Periodic scheduler. It is also + possible to configure the VC Source Step to always check out the + latest sources rather than paying attention to the Changes in the + SourceStamp, which will result in same behavior as this. + +`(branch=BRANCH, revision=None, changes=None, patch=None)' + This builds the most recent code on the given BRANCH. Again, + this is generally triggered by a user request or Periodic build. + +`(revision=REV, changes=None, patch=(LEVEL, DIFF))' + This checks out the tree at the given revision REV, then applies + a patch (using `patch -pLEVEL <DIFF'). The *note try:: feature + uses this kind of `SourceStamp'. If `patch' is None, the patching + step is bypassed. + + + The buildmaster is responsible for turning the `BuildSet' into a +set of `BuildRequest' objects and queueing them on the appropriate +Builders. + + +File: buildbot.info, Node: BuildRequest, Next: Builder, Prev: BuildSet, Up: Concepts + +3.4 BuildRequest +================ + +A `BuildRequest' is a request to build a specific set of sources on a +single specific `Builder'. Each `Builder' runs the `BuildRequest' as +soon as it can (i.e. when an associated buildslave becomes free). +`BuildRequest's are prioritized from oldest to newest, so when a +buildslave becomes free, the `Builder' with the oldest `BuildRequest' +is run. + + The `BuildRequest' contains the `SourceStamp' specification. The +actual process of running the build (the series of Steps that will be +executed) is implemented by the `Build' object. In this future this +might be changed, to have the `Build' define _what_ gets built, and a +separate `BuildProcess' (provided by the Builder) to define _how_ it +gets built. + + `BuildRequest' is created with optional `Properties'. One of +these, `owner', is collected by the resultant `Build' and added to +the set of _interested users_ to which status notifications will be +sent, depending on the configuration for each status object. + + The `BuildRequest' may be mergeable with other compatible +`BuildRequest's. Builds that are triggered by incoming Changes will +generally be mergeable. Builds that are triggered by user requests +are generally not, unless they are multiple requests to build the +_latest sources_ of the same branch. + + +File: buildbot.info, Node: Builder, Next: Users, Prev: BuildRequest, Up: Concepts + +3.5 Builder +=========== + +The `Builder' is a long-lived object which controls all Builds of a +given type. Each one is created when the config file is first parsed, +and lives forever (or rather until it is removed from the config +file). It mediates the connections to the buildslaves that do all the +work, and is responsible for creating the `Build' objects that decide +_how_ a build is performed (i.e., which steps are executed in what +order). + + Each `Builder' gets a unique name, and the path name of a +directory where it gets to do all its work (there is a +buildmaster-side directory for keeping status information, as well as +a buildslave-side directory where the actual checkout/compile/test +commands are executed). It also gets a `BuildFactory', which is +responsible for creating new `Build' instances: because the `Build' +instance is what actually performs each build, choosing the +`BuildFactory' is the way to specify what happens each time a build +is done. + + Each `Builder' is associated with one of more `BuildSlaves'. A +`Builder' which is used to perform OS-X builds (as opposed to Linux +or Solaris builds) should naturally be associated with an OS-X-based +buildslave. + + A `Builder' may be given a set of environment variables to be used +in its *note ShellCommand::s. These variables will override anything +in the buildslave's environment. Variables passed directly to a +ShellCommand will override variables of the same name passed to the +Builder. + + For example, if you a pool of identical slaves it is often easier +to manage variables like PATH from Buildbot rather than manually +editing it inside of the slaves' environment. + + f = factory.BuildFactory + f.addStep(ShellCommand( + command=['bash', './configure'])) + f.addStep(Compile()) + + c['builders'] = [ + {'name': 'test', 'slavenames': ['slave1', 'slave2', 'slave3', 'slave4', + 'slave5', 'slave6'], + 'builddir': 'test', 'factory': f', + 'env': {'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'}} + + +File: buildbot.info, Node: Users, Next: Build Properties, Prev: Builder, Up: Concepts + +3.6 Users +========= + +Buildbot has a somewhat limited awareness of _users_. It assumes the +world consists of a set of developers, each of whom can be described +by a couple of simple attributes. These developers make changes to +the source code, causing builds which may succeed or fail. + + Each developer is primarily known through the source control +system. Each Change object that arrives is tagged with a `who' field +that typically gives the account name (on the repository machine) of +the user responsible for that change. This string is the primary key +by which the User is known, and is displayed on the HTML status pages +and in each Build's "blamelist". + + To do more with the User than just refer to them, this username +needs to be mapped into an address of some sort. The responsibility +for this mapping is left up to the status module which needs the +address. The core code knows nothing about email addresses or IRC +nicknames, just user names. + +* Menu: + +* Doing Things With Users:: +* Email Addresses:: +* IRC Nicknames:: +* Live Status Clients:: + + +File: buildbot.info, Node: Doing Things With Users, Next: Email Addresses, Prev: Users, Up: Users + +3.6.1 Doing Things With Users +----------------------------- + +Each Change has a single User who is responsible for that Change. Most +Builds have a set of Changes: the Build represents the first time +these Changes have been built and tested by the Buildbot. The build +has a "blamelist" that consists of a simple union of the Users +responsible for all the Build's Changes. + + The Build provides (through the IBuildStatus interface) a list of +Users who are "involved" in the build. For now this is equal to the +blamelist, but in the future it will be expanded to include a "build +sheriff" (a person who is "on duty" at that time and responsible for +watching over all builds that occur during their shift), as well as +per-module owners who simply want to keep watch over their domain +(chosen by subdirectory or a regexp matched against the filenames +pulled out of the Changes). The Involved Users are those who probably +have an interest in the results of any given build. + + In the future, Buildbot will acquire the concept of "Problems", +which last longer than builds and have beginnings and ends. For +example, a test case which passed in one build and then failed in the +next is a Problem. The Problem lasts until the test case starts +passing again, at which point the Problem is said to be "resolved". + + If there appears to be a code change that went into the tree at the +same time as the test started failing, that Change is marked as being +resposible for the Problem, and the user who made the change is added +to the Problem's "Guilty" list. In addition to this user, there may +be others who share responsibility for the Problem (module owners, +sponsoring developers). In addition to the Responsible Users, there +may be a set of Interested Users, who take an interest in the fate of +the Problem. + + Problems therefore have sets of Users who may want to be kept +aware of the condition of the problem as it changes over time. If +configured, the Buildbot can pester everyone on the Responsible list +with increasing harshness until the problem is resolved, with the +most harshness reserved for the Guilty parties themselves. The +Interested Users may merely be told when the problem starts and +stops, as they are not actually responsible for fixing anything. + + +File: buildbot.info, Node: Email Addresses, Next: IRC Nicknames, Prev: Doing Things With Users, Up: Users + +3.6.2 Email Addresses +--------------------- + +The `buildbot.status.mail.MailNotifier' class (*note MailNotifier::) +provides a status target which can send email about the results of +each build. It accepts a static list of email addresses to which each +message should be delivered, but it can also be configured to send +mail to the Build's Interested Users. To do this, it needs a way to +convert User names into email addresses. + + For many VC systems, the User Name is actually an account name on +the system which hosts the repository. As such, turning the name into +an email address is a simple matter of appending +"@repositoryhost.com". Some projects use other kinds of mappings (for +example the preferred email address may be at "project.org" despite +the repository host being named "cvs.project.org"), and some VC +systems have full separation between the concept of a user and that +of an account on the repository host (like Perforce). Some systems +(like Arch) put a full contact email address in every change. + + To convert these names to addresses, the MailNotifier uses an +EmailLookup object. This provides a .getAddress method which accepts +a name and (eventually) returns an address. The default `MailNotifier' +module provides an EmailLookup which simply appends a static string, +configurable when the notifier is created. To create more complex +behaviors (perhaps using an LDAP lookup, or using "finger" on a +central host to determine a preferred address for the developer), +provide a different object as the `lookup' argument. + + In the future, when the Problem mechanism has been set up, the +Buildbot will need to send mail to arbitrary Users. It will do this +by locating a MailNotifier-like object among all the buildmaster's +status targets, and asking it to send messages to various Users. This +means the User-to-address mapping only has to be set up once, in your +MailNotifier, and every email message the buildbot emits will take +advantage of it. + + +File: buildbot.info, Node: IRC Nicknames, Next: Live Status Clients, Prev: Email Addresses, Up: Users + +3.6.3 IRC Nicknames +------------------- + +Like MailNotifier, the `buildbot.status.words.IRC' class provides a +status target which can announce the results of each build. It also +provides an interactive interface by responding to online queries +posted in the channel or sent as private messages. + + In the future, the buildbot can be configured map User names to IRC +nicknames, to watch for the recent presence of these nicknames, and to +deliver build status messages to the interested parties. Like +`MailNotifier' does for email addresses, the `IRC' object will have +an `IRCLookup' which is responsible for nicknames. The mapping can be +set up statically, or it can be updated by online users themselves +(by claiming a username with some kind of "buildbot: i am user +warner" commands). + + Once the mapping is established, the rest of the buildbot can ask +the `IRC' object to send messages to various users. It can report on +the likelihood that the user saw the given message (based upon how +long the user has been inactive on the channel), which might prompt +the Problem Hassler logic to send them an email message instead. + + +File: buildbot.info, Node: Live Status Clients, Prev: IRC Nicknames, Up: Users + +3.6.4 Live Status Clients +------------------------- + +The Buildbot also offers a PB-based status client interface which can +display real-time build status in a GUI panel on the developer's +desktop. This interface is normally anonymous, but it could be +configured to let the buildmaster know _which_ developer is using the +status client. The status client could then be used as a +message-delivery service, providing an alternative way to deliver +low-latency high-interruption messages to the developer (like "hey, +you broke the build"). + + +File: buildbot.info, Node: Build Properties, Prev: Users, Up: Concepts + +3.7 Build Properties +==================== + +Each build has a set of "Build Properties", which can be used by its +BuildStep to modify their actions. These properties, in the form of +key-value pairs, provide a general framework for dynamically altering +the behavior of a build based on its circumstances. + + Properties come from a number of places: + * global configuration - These properties apply to all builds. + + * schedulers - A scheduler can specify properties available to all + the builds it starts. + + * buildslaves - A buildslave can pass properties on to the builds + it performs. + + * builds - A build automatically sets a number of properties on + itself. + + * steps - Steps of a build can set properties that are available + to subsequent steps. In particular, source steps set a number + of properties. + + Properties are very flexible, and can be used to implement all +manner of functionality. Here are some examples: + + Most Source steps record the revision that they checked out in the +`got_revision' property. A later step could use this property to +specify the name of a fully-built tarball, dropped in an +easily-acessible directory for later testing. + + Some projects want to perform nightly builds as well as in response +to committed changes. Such a project would run two schedulers, both +pointing to the same set of builders, but could provide an +`is_nightly' property so that steps can distinguish the nightly +builds, perhaps to run more resource-intensive tests. + + Some projects have different build processes on different systems. +Rather than create a build factory for each slave, the steps can use +buildslave properties to identify the unique aspects of each slave +and adapt the build process dynamically. + + +File: buildbot.info, Node: Configuration, Next: Getting Source Code Changes, Prev: Concepts, Up: Top + +4 Configuration +*************** + +The buildbot's behavior is defined by the "config file", which +normally lives in the `master.cfg' file in the buildmaster's base +directory (but this can be changed with an option to the `buildbot +create-master' command). This file completely specifies which +Builders are to be run, which slaves they should use, how Changes +should be tracked, and where the status information is to be sent. +The buildmaster's `buildbot.tac' file names the base directory; +everything else comes from the config file. + + A sample config file was installed for you when you created the +buildmaster, but you will need to edit it before your buildbot will do +anything useful. + + This chapter gives an overview of the format of this file and the +various sections in it. You will need to read the later chapters to +understand how to fill in each section properly. + +* Menu: + +* Config File Format:: +* Loading the Config File:: +* Testing the Config File:: +* Defining the Project:: +* Change Sources and Schedulers:: +* Merging BuildRequests:: +* Setting the slaveport:: +* Buildslave Specifiers:: +* On-Demand ("Latent") Buildslaves:: +* Defining Global Properties:: +* Defining Builders:: +* Defining Status Targets:: +* Debug options:: + + +File: buildbot.info, Node: Config File Format, Next: Loading the Config File, Prev: Configuration, Up: Configuration + +4.1 Config File Format +====================== + +The config file is, fundamentally, just a piece of Python code which +defines a dictionary named `BuildmasterConfig', with a number of keys +that are treated specially. You don't need to know Python to do basic +configuration, though, you can just copy the syntax of the sample +file. If you _are_ comfortable writing Python code, however, you can +use all the power of a full programming language to achieve more +complicated configurations. + + The `BuildmasterConfig' name is the only one which matters: all +other names defined during the execution of the file are discarded. +When parsing the config file, the Buildmaster generally compares the +old configuration with the new one and performs the minimum set of +actions necessary to bring the buildbot up to date: Builders which are +not changed are left untouched, and Builders which are modified get to +keep their old event history. + + Basic Python syntax: comments start with a hash character ("#"), +tuples are defined with `(parenthesis, pairs)', arrays are defined +with `[square, brackets]', tuples and arrays are mostly +interchangeable. Dictionaries (data structures which map "keys" to +"values") are defined with curly braces: `{'key1': 'value1', 'key2': +'value2'} '. Function calls (and object instantiation) can use named +parameters, like `w = html.Waterfall(http_port=8010)'. + + The config file starts with a series of `import' statements, which +make various kinds of Steps and Status targets available for later +use. The main `BuildmasterConfig' dictionary is created, then it is +populated with a variety of keys. These keys are broken roughly into +the following sections, each of which is documented in the rest of +this chapter: + + * Project Definitions + + * Change Sources / Schedulers + + * Slaveport + + * Buildslave Configuration + + * Builders / Interlocks + + * Status Targets + + * Debug options + + The config file can use a few names which are placed into its +namespace: + +`basedir' + the base directory for the buildmaster. This string has not been + expanded, so it may start with a tilde. It needs to be expanded + before use. The config file is located in + `os.path.expanduser(os.path.join(basedir, 'master.cfg'))' + + + +File: buildbot.info, Node: Loading the Config File, Next: Testing the Config File, Prev: Config File Format, Up: Configuration + +4.2 Loading the Config File +=========================== + +The config file is only read at specific points in time. It is first +read when the buildmaster is launched. Once it is running, there are +various ways to ask it to reload the config file. If you are on the +system hosting the buildmaster, you can send a `SIGHUP' signal to it: +the `buildbot' tool has a shortcut for this: + + buildbot reconfig BASEDIR + + This command will show you all of the lines from `twistd.log' that +relate to the reconfiguration. If there are any problems during the +config-file reload, they will be displayed in these lines. + + The debug tool (`buildbot debugclient --master HOST:PORT') has a +"Reload .cfg" button which will also trigger a reload. In the future, +there will be other ways to accomplish this step (probably a +password-protected button on the web page, as well as a privileged IRC +command). + + When reloading the config file, the buildmaster will endeavor to +change as little as possible about the running system. For example, +although old status targets may be shut down and new ones started up, +any status targets that were not changed since the last time the +config file was read will be left running and untouched. Likewise any +Builders which have not been changed will be left running. If a +Builder is modified (say, the build process is changed) while a Build +is currently running, that Build will keep running with the old +process until it completes. Any previously queued Builds (or Builds +which get queued after the reconfig) will use the new process. + + +File: buildbot.info, Node: Testing the Config File, Next: Defining the Project, Prev: Loading the Config File, Up: Configuration + +4.3 Testing the Config File +=========================== + +To verify that the config file is well-formed and contains no +deprecated or invalid elements, use the "checkconfig" command: + + % buildbot checkconfig master.cfg + Config file is good! + + If the config file has deprecated features (perhaps because you've +upgraded the buildmaster and need to update the config file to match), +they will be announced by checkconfig. In this case, the config file +will work, but you should really remove the deprecated items and use +the recommended replacements instead: + + % buildbot checkconfig master.cfg + /usr/lib/python2.4/site-packages/buildbot/master.py:559: DeprecationWarning: c['sources'] is + deprecated as of 0.7.6 and will be removed by 0.8.0 . Please use c['change_source'] instead. + warnings.warn(m, DeprecationWarning) + Config file is good! + + If the config file is simply broken, that will be caught too: + + % buildbot checkconfig master.cfg + Traceback (most recent call last): + File "/usr/lib/python2.4/site-packages/buildbot/scripts/runner.py", line 834, in doCheckConfig + ConfigLoader(configFile) + File "/usr/lib/python2.4/site-packages/buildbot/scripts/checkconfig.py", line 31, in __init__ + self.loadConfig(configFile) + File "/usr/lib/python2.4/site-packages/buildbot/master.py", line 480, in loadConfig + exec f in localDict + File "/home/warner/BuildBot/master/foolscap/master.cfg", line 90, in ? + c[bogus] = "stuff" + NameError: name 'bogus' is not defined + + +File: buildbot.info, Node: Defining the Project, Next: Change Sources and Schedulers, Prev: Testing the Config File, Up: Configuration + +4.4 Defining the Project +======================== + +There are a couple of basic settings that you use to tell the buildbot +what project it is working on. This information is used by status +reporters to let users find out more about the codebase being +exercised by this particular Buildbot installation. + + c['projectName'] = "Buildbot" + c['projectURL'] = "http://buildbot.sourceforge.net/" + c['buildbotURL'] = "http://localhost:8010/" + + `projectName' is a short string will be used to describe the +project that this buildbot is working on. For example, it is used as +the title of the waterfall HTML page. + + `projectURL' is a string that gives a URL for the project as a +whole. HTML status displays will show `projectName' as a link to +`projectURL', to provide a link from buildbot HTML pages to your +project's home page. + + The `buildbotURL' string should point to the location where the +buildbot's internal web server (usually the `html.Waterfall' page) is +visible. This typically uses the port number set when you create the +`Waterfall' object: the buildbot needs your help to figure out a +suitable externally-visible host name. + + When status notices are sent to users (either by email or over +IRC), `buildbotURL' will be used to create a URL to the specific build +or problem that they are being notified about. It will also be made +available to queriers (over IRC) who want to find out where to get +more information about this buildbot. + + The `logCompressionLimit' enables bz2-compression of build logs on +disk for logs that are bigger than the given size, or disables that +completely if given `False'. The default value is 4k, which should be +a reasonable default on most file systems. This setting has no impact +on status plugins, and merely affects the required disk space on the +master for build logs. + + +File: buildbot.info, Node: Change Sources and Schedulers, Next: Merging BuildRequests, Prev: Defining the Project, Up: Configuration + +4.5 Change Sources and Schedulers +================================= + +The `c['change_source']' key is the ChangeSource instance(1) that +defines how the buildmaster learns about source code changes. More +information about what goes here is available in *Note Getting Source +Code Changes::. + + from buildbot.changes.pb import PBChangeSource + c['change_source'] = PBChangeSource() + + (note: in buildbot-0.7.5 and earlier, this key was named +`c['sources']', and required a list. `c['sources']' is deprecated as +of buildbot-0.7.6 and is scheduled to be removed in a future release). + + `c['schedulers']' is a list of Scheduler instances, each of which +causes builds to be started on a particular set of Builders. The two +basic Scheduler classes you are likely to start with are `Scheduler' +and `Periodic', but you can write a customized subclass to implement +more complicated build scheduling. + + Scheduler arguments should always be specified by name (as keyword +arguments), to allow for future expansion: + + sched = Scheduler(name="quick", builderNames=['lin', 'win']) + + All schedulers have several arguments in common: + +`name' + Each Scheduler must have a unique name. This is used in status + displays, and is also available in the build property + `scheduler'. + +`builderNames' + This is the set of builders which this scheduler should trigger, + specified as a list of names (strings). + +`properties' + This is a dictionary specifying properties that will be + transmitted to all builds started by this scheduler. + + + Here is a brief catalog of the available Scheduler types. All these +Schedulers are classes in `buildbot.scheduler', and the docstrings +there are the best source of documentation on the arguments taken by +each one. + +* Menu: + +* Scheduler Scheduler:: +* AnyBranchScheduler:: +* Dependent Scheduler:: +* Periodic Scheduler:: +* Nightly Scheduler:: +* Try Schedulers:: +* Triggerable Scheduler:: + + ---------- Footnotes ---------- + + (1) To be precise, it is an object or a list of objects which all +implement the `buildbot.interfaces.IChangeSource' Interface. It is +unusual to have multiple ChangeSources, so this key accepts either a +single ChangeSource or a sequence of them. + + +File: buildbot.info, Node: Scheduler Scheduler, Next: AnyBranchScheduler, Prev: Change Sources and Schedulers, Up: Change Sources and Schedulers + +4.5.1 Scheduler Scheduler +------------------------- + +This is the original and still most popular Scheduler class. It +follows exactly one branch, and starts a configurable +tree-stable-timer after each change on that branch. When the timer +expires, it starts a build on some set of Builders. The Scheduler +accepts a `fileIsImportant' function which can be used to ignore some +Changes if they do not affect any "important" files. + + The arguments to this scheduler are: + +`name' + +`builderNames' + +`properties' + +`branch' + This Scheduler will pay attention to a single branch, ignoring + Changes that occur on other branches. Setting `branch' equal to + the special value of `None' means it should only pay attention to + the default branch. Note that `None' is a keyword, not a string, + so you want to use `None' and not `"None"'. + +`treeStableTimer' + The Scheduler will wait for this many seconds before starting the + build. If new changes are made during this interval, the timer + will be restarted, so really the build will be started after a + change and then after this many seconds of inactivity. + +`fileIsImportant' + A callable which takes one argument, a Change instance, and + returns `True' if the change is worth building, and `False' if + it is not. Unimportant Changes are accumulated until the build + is triggered by an important change. The default value of None + means that all Changes are important. + +`categories' + A list of categories of changes that this scheduler will respond + to. If this is specified, then any non-matching changes are + ignored. + + + Example: + + from buildbot import scheduler + quick = scheduler.Scheduler(name="quick", + branch=None, + treeStableTimer=60, + builderNames=["quick-linux", "quick-netbsd"]) + full = scheduler.Scheduler(name="full", + branch=None, + treeStableTimer=5*60, + builderNames=["full-linux", "full-netbsd", "full-OSX"]) + c['schedulers'] = [quick, full] + + In this example, the two "quick" builders are triggered 60 seconds +after the tree has been changed. The "full" builds do not run quite +so quickly (they wait 5 minutes), so hopefully if the quick builds +fail due to a missing file or really simple typo, the developer can +discover and fix the problem before the full builds are started. Both +Schedulers only pay attention to the default branch: any changes on +other branches are ignored by these Schedulers. Each Scheduler +triggers a different set of Builders, referenced by name. + + +File: buildbot.info, Node: AnyBranchScheduler, Next: Dependent Scheduler, Prev: Scheduler Scheduler, Up: Change Sources and Schedulers + +4.5.2 AnyBranchScheduler +------------------------ + +This scheduler uses a tree-stable-timer like the default one, but +follows multiple branches at once. Each branch gets a separate timer. + + The arguments to this scheduler are: + +`name' + +`builderNames' + +`properties' + +`branches' + This Scheduler will pay attention to any number of branches, + ignoring Changes that occur on other branches. Branches are + specified just as for the `Scheduler' class. + +`treeStableTimer' + The Scheduler will wait for this many seconds before starting the + build. If new changes are made during this interval, the timer + will be restarted, so really the build will be started after a + change and then after this many seconds of inactivity. + +`fileIsImportant' + A callable which takes one argument, a Change instance, and + returns `True' if the change is worth building, and `False' if + it is not. Unimportant Changes are accumulated until the build + is triggered by an important change. The default value of None + means that all Changes are important. + + +File: buildbot.info, Node: Dependent Scheduler, Next: Periodic Scheduler, Prev: AnyBranchScheduler, Up: Change Sources and Schedulers + +4.5.3 Dependent Scheduler +------------------------- + +It is common to wind up with one kind of build which should only be +performed if the same source code was successfully handled by some +other kind of build first. An example might be a packaging step: you +might only want to produce .deb or RPM packages from a tree that was +known to compile successfully and pass all unit tests. You could put +the packaging step in the same Build as the compile and testing steps, +but there might be other reasons to not do this (in particular you +might have several Builders worth of compiles/tests, but only wish to +do the packaging once). Another example is if you want to skip the +"full" builds after a failing "quick" build of the same source code. +Or, if one Build creates a product (like a compiled library) that is +used by some other Builder, you'd want to make sure the consuming +Build is run _after_ the producing one. + + You can use "Dependencies" to express this relationship to the +Buildbot. There is a special kind of Scheduler named +`scheduler.Dependent' that will watch an "upstream" Scheduler for +builds to complete successfully (on all of its Builders). Each time +that happens, the same source code (i.e. the same `SourceStamp') will +be used to start a new set of builds, on a different set of Builders. +This "downstream" scheduler doesn't pay attention to Changes at all. +It only pays attention to the upstream scheduler. + + If the build fails on any of the Builders in the upstream set, the +downstream builds will not fire. Note that, for SourceStamps +generated by a ChangeSource, the `revision' is None, meaning HEAD. +If any changes are committed between the time the upstream scheduler +begins its build and the time the dependent scheduler begins its +build, then those changes will be included in the downstream build. +See the *note Triggerable Scheduler:: for a more flexible dependency +mechanism that can avoid this problem. + + The arguments to this scheduler are: + +`name' + +`builderNames' + +`properties' + +`upstream' + The upstream scheduler to watch. Note that this is an + "instance", not the name of the scheduler. + + Example: + + from buildbot import scheduler + tests = scheduler.Scheduler("just-tests", None, 5*60, + ["full-linux", "full-netbsd", "full-OSX"]) + package = scheduler.Dependent("build-package", + tests, # upstream scheduler -- no quotes! + ["make-tarball", "make-deb", "make-rpm"]) + c['schedulers'] = [tests, package] + + +File: buildbot.info, Node: Periodic Scheduler, Next: Nightly Scheduler, Prev: Dependent Scheduler, Up: Change Sources and Schedulers + +4.5.4 Periodic Scheduler +------------------------ + +This simple scheduler just triggers a build every N seconds. + + The arguments to this scheduler are: + +`name' + +`builderNames' + +`properties' + +`periodicBuildTimer' + The time, in seconds, after which to start a build. + + Example: + + from buildbot import scheduler + nightly = scheduler.Periodic(name="nightly", + builderNames=["full-solaris"], + periodicBuildTimer=24*60*60) + c['schedulers'] = [nightly] + + The Scheduler in this example just runs the full solaris build once +per day. Note that this Scheduler only lets you control the time +between builds, not the absolute time-of-day of each Build, so this +could easily wind up a "daily" or "every afternoon" scheduler +depending upon when it was first activated. + + +File: buildbot.info, Node: Nightly Scheduler, Next: Try Schedulers, Prev: Periodic Scheduler, Up: Change Sources and Schedulers + +4.5.5 Nightly Scheduler +----------------------- + +This is highly configurable periodic build scheduler, which triggers +a build at particular times of day, week, month, or year. The +configuration syntax is very similar to the well-known `crontab' +format, in which you provide values for minute, hour, day, and month +(some of which can be wildcards), and a build is triggered whenever +the current time matches the given constraints. This can run a build +every night, every morning, every weekend, alternate Thursdays, on +your boss's birthday, etc. + + Pass some subset of `minute', `hour', `dayOfMonth', `month', and +`dayOfWeek'; each may be a single number or a list of valid values. +The builds will be triggered whenever the current time matches these +values. Wildcards are represented by a '*' string. All fields default +to a wildcard except 'minute', so with no fields this defaults to a +build every hour, on the hour. The full list of parameters is: + +`name' + +`builderNames' + +`properties' + +`branch' + The branch to build, just as for `Scheduler'. + +`minute' + The minute of the hour on which to start the build. This + defaults to 0, meaning an hourly build. + +`hour' + The hour of the day on which to start the build, in 24-hour + notation. This defaults to *, meaning every hour. + +`month' + The month in which to start the build, with January = 1. This + defaults to *, meaning every month. + +`dayOfWeek' + The day of the week to start a build, with Monday = 0. This + defauls to *, meaning every day of the week. + +`onlyIfChanged' + If this is true, then builds will not be scheduled at the + designated time unless the source has changed since the previous + build. + + For example, the following master.cfg clause will cause a build to +be started every night at 3:00am: + + s = scheduler.Nightly(name='nightly', + builderNames=['builder1', 'builder2'], + hour=3, + minute=0) + + This scheduler will perform a build each monday morning at 6:23am +and again at 8:23am, but only if someone has committed code in the +interim: + + s = scheduler.Nightly(name='BeforeWork', + builderNames=['builder1'], + dayOfWeek=0, + hour=[6,8], + minute=23, + onlyIfChanged=True) + + The following runs a build every two hours, using Python's `range' +function: + + s = Nightly(name='every2hours', + builderNames=['builder1'], + hour=range(0, 24, 2)) + + Finally, this example will run only on December 24th: + + s = Nightly(name='SleighPreflightCheck', + builderNames=['flying_circuits', 'radar'], + month=12, + dayOfMonth=24, + hour=12, + minute=0) + + +File: buildbot.info, Node: Try Schedulers, Next: Triggerable Scheduler, Prev: Nightly Scheduler, Up: Change Sources and Schedulers + +4.5.6 Try Schedulers +-------------------- + +This scheduler allows developers to use the `buildbot try' command to +trigger builds of code they have not yet committed. See *note try:: +for complete details. + + Two implementations are available: `Try_Jobdir' and +`Try_Userpass'. The former monitors a job directory, specified by +the `jobdir' parameter, while the latter listens for PB connections +on a specific `port', and authenticates against `userport'. + + +File: buildbot.info, Node: Triggerable Scheduler, Prev: Try Schedulers, Up: Change Sources and Schedulers + +4.5.7 Triggerable Scheduler +--------------------------- + +The `Triggerable' scheduler waits to be triggered by a Trigger step +(see *note Triggering Schedulers::) in another build. That step can +optionally wait for the scheduler's builds to complete. This provides +two advantages over Dependent schedulers. First, the same scheduler +can be triggered from multiple builds. Second, the ability to wait +for a Triggerable's builds to complete provides a form of "subroutine +call", where one or more builds can "call" a scheduler to perform +some work for them, perhaps on other buildslaves. + + The parameters are just the basics: + +`name' + +`builderNames' + +`properties' + + This class is only useful in conjunction with the `Trigger' step. +Here is a fully-worked example: + + from buildbot import scheduler + from buildbot.process import factory + from buildbot.steps import trigger + + checkin = scheduler.Scheduler(name="checkin", + branch=None, + treeStableTimer=5*60, + builderNames=["checkin"]) + nightly = scheduler.Nightly(name='nightly', + builderNames=['nightly'], + hour=3, + minute=0) + + mktarball = scheduler.Triggerable(name="mktarball", + builderNames=["mktarball"]) + build = scheduler.Triggerable(name="build-all-platforms", + builderNames=["build-all-platforms"]) + test = scheduler.Triggerable(name="distributed-test", + builderNames=["distributed-test"]) + package = scheduler.Triggerable(name="package-all-platforms", + builderNames=["package-all-platforms"]) + + c['schedulers'] = [checkin, nightly, build, test, package] + + # on checkin, make a tarball, build it, and test it + checkin_factory = factory.BuildFactory() + checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) + checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) + checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'], + waitForFinish=True)) + + # and every night, make a tarball, build it, and package it + nightly_factory = factory.BuildFactory() + nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) + nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) + nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'], + waitForFinish=True)) + + +File: buildbot.info, Node: Merging BuildRequests, Next: Setting the slaveport, Prev: Change Sources and Schedulers, Up: Configuration + +4.6 Merging BuildRequests +========================= + +By default, buildbot merges BuildRequests that have the compatible +SourceStamps. This behaviour can be customized with the +`c['mergeRequests']' configuration key. This key specifies a function +which is caleld with three arguments: a `Builder' and two +`BuildRequest' objects. It should return true if the requests can be +merged. For example: + + def mergeRequests(builder, req1, req2): + """Don't merge buildrequest at all""" + return False + c['mergeRequests'] = mergeRequests + + In many cases, the details of the SourceStamps and BuildRequests +are important. In this example, only BuildRequests with the same +"reason" are merged; thus developers forcing builds for different +reasons will see distinct builds. + + def mergeRequests(builder, req1, req2): + if req1.source.canBeMergedWith(req2.source) and req1.reason == req2.reason: + return True + return False + c['mergeRequests'] = mergeRequests + + +File: buildbot.info, Node: Setting the slaveport, Next: Buildslave Specifiers, Prev: Merging BuildRequests, Up: Configuration + +4.7 Setting the slaveport +========================= + +The buildmaster will listen on a TCP port of your choosing for +connections from buildslaves. It can also use this port for +connections from remote Change Sources, status clients, and debug +tools. This port should be visible to the outside world, and you'll +need to tell your buildslave admins about your choice. + + It does not matter which port you pick, as long it is externally +visible, however you should probably use something larger than 1024, +since most operating systems don't allow non-root processes to bind to +low-numbered ports. If your buildmaster is behind a firewall or a NAT +box of some sort, you may have to configure your firewall to permit +inbound connections to this port. + + c['slavePortnum'] = 10000 + + `c['slavePortnum']' is a _strports_ specification string, defined +in the `twisted.application.strports' module (try `pydoc +twisted.application.strports' to get documentation on the format). +This means that you can have the buildmaster listen on a +localhost-only port by doing: + + c['slavePortnum'] = "tcp:10000:interface=127.0.0.1" + + This might be useful if you only run buildslaves on the same +machine, and they are all configured to contact the buildmaster at +`localhost:10000'. + + +File: buildbot.info, Node: Buildslave Specifiers, Next: On-Demand ("Latent") Buildslaves, Prev: Setting the slaveport, Up: Configuration + +4.8 Buildslave Specifiers +========================= + +The `c['slaves']' key is a list of known buildslaves. In the common +case, each buildslave is defined by an instance of the BuildSlave +class. It represents a standard, manually started machine that will +try to connect to the buildbot master as a slave. Contrast these +with the "on-demand" latent buildslaves, such as the Amazon Web +Service Elastic Compute Cloud latent buildslave discussed below. + + The BuildSlave class is instantiated with two values: (slavename, +slavepassword). These are the same two values that need to be +provided to the buildslave administrator when they create the +buildslave. + + The slavenames must be unique, of course. The password exists to +prevent evildoers from interfering with the buildbot by inserting +their own (broken) buildslaves into the system and thus displacing the +real ones. + + Buildslaves with an unrecognized slavename or a non-matching +password will be rejected when they attempt to connect, and a message +describing the problem will be put in the log file (see *note +Logfiles::). + + from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd') + BuildSlave('bot-bsd', 'bsdpasswd') + ] + + `BuildSlave' objects can also be created with an optional +`properties' argument, a dictionary specifying properties that will +be available to any builds performed on this slave. For example: + + from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + properties={'os':'solaris'}), + ] + + The `BuildSlave' constructor can also take an optional +`max_builds' parameter to limit the number of builds that it will +execute simultaneously: + + from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave("bot-linux", "linuxpassword", max_builds=2)] + + Historical note: in buildbot-0.7.5 and earlier, the `c['bots']' +key was used instead, and it took a list of (name, password) tuples. +This key is accepted for backwards compatibility, but is deprecated as +of 0.7.6 and will go away in some future release. + +* Menu: + +* When Buildslaves Go Missing:: + + +File: buildbot.info, Node: When Buildslaves Go Missing, Up: Buildslave Specifiers + +4.8.1 When Buildslaves Go Missing +--------------------------------- + +Sometimes, the buildslaves go away. One very common reason for this is +when the buildslave process is started once (manually) and left +running, but then later the machine reboots and the process is not +automatically restarted. + + If you'd like to have the administrator of the buildslave (or other +people) be notified by email when the buildslave has been missing for +too long, just add the `notify_on_missing=' argument to the +`BuildSlave' definition: + + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing="bob@example.com"), + ] + + By default, this will send email when the buildslave has been +disconnected for more than one hour. Only one email per +connection-loss event will be sent. To change the timeout, use +`missing_timeout=' and give it a number of seconds (the default is +3600). + + You can have the buildmaster send email to multiple recipients: +just provide a list of addresses instead of a single one: + + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing=["bob@example.com", + "alice@example.org"], + missing_timeout=300, # notify after 5 minutes + ), + ] + + The email sent this way will use a MailNotifier (*note +MailNotifier::) status target, if one is configured. This provides a +way for you to control the "from" address of the email, as well as +the relayhost (aka "smarthost") to use as an SMTP server. If no +MailNotifier is configured on this buildmaster, the +buildslave-missing emails will be sent using a default configuration. + + Note that if you want to have a MailNotifier for buildslave-missing +emails but not for regular build emails, just create one with +builders=[], as follows: + + from buildbot.status import mail + m = mail.MailNotifier(fromaddr="buildbot@localhost", builders=[], + relayhost="smtp.example.org") + c['status'].append(m) + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing="bob@example.com"), + ] + + +File: buildbot.info, Node: On-Demand ("Latent") Buildslaves, Next: Defining Global Properties, Prev: Buildslave Specifiers, Up: Configuration + +4.9 On-Demand ("Latent") Buildslaves +==================================== + +The standard buildbot model has slaves started manually. The +previous section described how to configure the master for this +approach. + + Another approach is to let the buildbot master start slaves when +builds are ready, on-demand. Thanks to services such as Amazon Web +Services' Elastic Compute Cloud ("AWS EC2"), this is relatively easy +to set up, and can be very useful for some situations. + + The buildslaves that are started on-demand are called "latent" +buildslaves. As of this writing, buildbot ships with an abstract +base class for building latent buildslaves, and a concrete +implementation for AWS EC2. + +* Menu: + +* Amazon Web Services Elastic Compute Cloud ("AWS EC2"):: +* Dangers with Latent Buildslaves:: +* Writing New Latent Buildslaves:: + + +File: buildbot.info, Node: Amazon Web Services Elastic Compute Cloud ("AWS EC2"), Next: Dangers with Latent Buildslaves, Up: On-Demand ("Latent") Buildslaves + +4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2") +----------------------------------------------------------- + +AWS EC2 is a web service that allows you to start virtual machines in +an Amazon data center. Please see their website for details, incuding +costs. Using the AWS EC2 latent buildslaves involves getting an EC2 +account with AWS and setting up payment; customizing one or more EC2 +machine images ("AMIs") on your desired operating system(s) and +publishing them (privately if needed); and configuring the buildbot +master to know how to start your customized images for +"substantiating" your latent slaves. + +* Menu: + +* Get an AWS EC2 Account:: +* Create an AMI:: +* Configure the Master with an EC2LatentBuildSlave:: + + +File: buildbot.info, Node: Get an AWS EC2 Account, Next: Create an AMI, Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2") + +4.9.1.1 Get an AWS EC2 Account +.............................. + +To start off, to use the AWS EC2 latent buildslave, you need to get +an AWS developer account and sign up for EC2. These instructions may +help you get started: + + * Go to http://aws.amazon.com/ and click to "Sign Up Now" for an + AWS account. + + * Once you are logged into your account, you need to sign up for + EC2. Instructions for how to do this have changed over time + because Amazon changes their website, so the best advice is to + hunt for it. After signing up for EC2, it may say it wants you + to upload an x.509 cert. You will need this to create images + (see below) but it is not technically necessary for the buildbot + master configuration. + + * You must enter a valid credit card before you will be able to + use EC2. Do that under 'Payment Method'. + + * Make sure you're signed up for EC2 by going to 'Your + Account'->'Account Activity' and verifying EC2 is listed. + + +File: buildbot.info, Node: Create an AMI, Next: Configure the Master with an EC2LatentBuildSlave, Prev: Get an AWS EC2 Account, Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2") + +4.9.1.2 Create an AMI +..................... + +Now you need to create an AMI and configure the master. You may need +to run through this cycle a few times to get it working, but these +instructions should get you started. + + Creating an AMI is out of the scope of this document. The EC2 +Getting Started Guide is a good resource for this task. Here are a +few additional hints. + + * When an instance of the image starts, it needs to automatically + start a buildbot slave that connects to your master (to create a + buildbot slave, *note Creating a buildslave::; to make a daemon, + *note Launching the daemons::). + + * You may want to make an instance of the buildbot slave, + configure it as a standard buildslave in the master (i.e., not + as a latent slave), and test and debug it that way before you + turn it into an AMI and convert to a latent slave in the master. + + +File: buildbot.info, Node: Configure the Master with an EC2LatentBuildSlave, Prev: Create an AMI, Up: Amazon Web Services Elastic Compute Cloud ("AWS EC2") + +4.9.1.3 Configure the Master with an EC2LatentBuildSlave +........................................................ + +Now let's assume you have an AMI that should work with the +EC2LatentBuildSlave. It's now time to set up your buildbot master +configuration. + + You will need some information from your AWS account: the "Access +Key Id" and the "Secret Access Key". If you've built the AMI +yourself, you probably already are familiar with these values. If +you have not, and someone has given you access to an AMI, these hints +may help you find the necessary values: + + * While logged into your AWS account, find the "Access + Identifiers" link (either on the left, or via "Your Account" -> + "Access Identifiers". + + * On the page, you'll see alphanumeric values for "Your Access Key + Id:" and "Your Secret Access Key:". Make a note of these. Later + on, we'll call the first one your "identifier" and the second + one your "secret_identifier." + + When creating an EC2LatentBuildSlave in the buildbot master +configuration, the first three arguments are required. The name and +password are the first two arguments, and work the same as with +normal buildslaves. The next argument specifies the type of the EC2 +virtual machine (available options as of this writing include +"m1.small", "m1.large", 'm1.xlarge", "c1.medium", and "c1.xlarge"; +see the EC2 documentation for descriptions of these machines). + + Here is the simplest example of configuring an EC2 latent +buildslave. It specifies all necessary remaining values explicitly in +the instantiation. + + from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345', + identifier='publickey', + secret_identifier='privatekey' + )] + + The "ami" argument specifies the AMI that the master should start. +The "identifier" argument specifies the AWS "Access Key Id," and the +"secret_identifier" specifies the AWS "Secret Access Key." Both the +AMI and the account information can be specified in alternate ways. + + Note that whoever has your identifier and secret_identifier values +can request AWS work charged to your account, so these values need to +be carefully protected. Another way to specify these access keys is +to put them in a separate file. You can then make the access +privileges stricter for this separate file, and potentially let more +people read your main configuration file. + + By default, you can make an .ec2 directory in the home folder of +the user running the buildbot master. In that directory, create a +file called aws_id. The first line of that file should be your +access key id; the second line should be your secret access key id. +Then you can instantiate the build slave as follows. + + from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345')] + + If you want to put the key information in another file, use the +"aws_id_file_path" initialization argument. + + Previous examples used a particular AMI. If the Buildbot master +will be deployed in a process-controlled environment, it may be +convenient to specify the AMI more flexibly. Rather than specifying +an individual AMI, specify one or two AMI filters. + + In all cases, the AMI that sorts last by its location (the S3 +bucket and manifest name) will be preferred. + + One available filter is to specify the acceptable AMI owners, by +AWS account number (the 12 digit number, usually rendered in AWS with +hyphens like "1234-5678-9012", should be entered as in integer). + + from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + valid_ami_owners=[11111111111, + 22222222222], + identifier='publickey', + secret_identifier='privatekey' + ) + + The other available filter is to provide a regular expression +string that will be matched against each AMI's location (the S3 +bucket and manifest name). + + from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') + + The regular expression can specify a group, which will be +preferred for the sorting. Only the first group is used; subsequent +groups are ignored. + + from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') + + If the group can be cast to an integer, it will be. This allows +10 to sort after 1, for instance. + + from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') + + In addition to using the password as a handshake between the +master and the slave, you may want to use a firewall to assert that +only machines from a specific IP can connect as slaves. This is +possible with AWS EC2 by using the Elastic IP feature. To configure, +generate a Elastic IP in AWS, and then specify it in your +configuration using the "elastic_ip" argument. + + from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + 'ami-12345', + identifier='publickey', + secret_identifier='privatekey', + elastic_ip='208.77.188.166' + )] + + The EC2LatentBuildSlave supports all other configuration from the +standard BuildSlave. The "missing_timeout" and "notify_on_missing" +specify how long to wait for an EC2 instance to attach before +considering the attempt to have failed, and email addresses to alert, +respectively. "missing_timeout" defaults to 20 minutes. + + The "build_wait_timeout" allows you to specify how long an +EC2LatentBuildSlave should wait after a build for another build +before it shuts down the EC2 instance. It defaults to 10 minutes. + + "keypair_name" and "security_name" allow you to specify different +names for these AWS EC2 values. They both default to +"latent_buildbot_slave". + + +File: buildbot.info, Node: Dangers with Latent Buildslaves, Next: Writing New Latent Buildslaves, Prev: Amazon Web Services Elastic Compute Cloud ("AWS EC2"), Up: On-Demand ("Latent") Buildslaves + +4.9.2 Dangers with Latent Buildslaves +------------------------------------- + +Any latent build slave that interacts with a for-fee service, such as +the EC2LatentBuildSlave, brings significant risks. As already +identified, the configuraton will need access to account information +that, if obtained by a criminal, can be used to charge services to +your account. Also, bugs in the buildbot software may lead to +unnecessary charges. In particular, if the master neglects to shut +down an instance for some reason, a virtual machine may be running +unnecessarily, charging against your account. Manual and/or automatic +(e.g. nagios with a plugin using a library like boto) double-checking +may be appropriate. + + A comparitively trivial note is that currently if two instances +try to attach to the same latent buildslave, it is likely that the +system will become confused. This should not occur, unless, for +instance, you configure a normal build slave to connect with the +authentication of a latent buildbot. If the situation occurs, stop +all attached instances and restart the master. + + +File: buildbot.info, Node: Writing New Latent Buildslaves, Prev: Dangers with Latent Buildslaves, Up: On-Demand ("Latent") Buildslaves + +4.9.3 Writing New Latent Buildslaves +------------------------------------ + +Writing a new latent buildslave should only require subclassing +`buildbot.buildslave.AbstractLatentBuildSlave' and implementing +start_instance and stop_instance. + + def start_instance(self): + # responsible for starting instance that will try to connect with this + # master. Should return deferred. Problems should use an errback. The + # callback value can be None, or can be an iterable of short strings to + # include in the "substantiate success" status message, such as + # identifying the instance that started. + raise NotImplementedError + + def stop_instance(self, fast=False): + # responsible for shutting down instance. Return a deferred. If `fast`, + # we're trying to shut the master down, so callback as soon as is safe. + # Callback value is ignored. + raise NotImplementedError + + See `buildbot.ec2buildslave.EC2LatentBuildSlave' for an example, +or see the test example `buildbot.test_slaves.FakeLatentBuildSlave'. + + +File: buildbot.info, Node: Defining Global Properties, Next: Defining Builders, Prev: On-Demand ("Latent") Buildslaves, Up: Configuration + +4.10 Defining Global Properties +=============================== + +The `'properties'' configuration key defines a dictionary of +properties that will be available to all builds started by the +buildmaster: + + c['properties'] = { + 'Widget-version' : '1.2', + 'release-stage' : 'alpha' + } + + +File: buildbot.info, Node: Defining Builders, Next: Defining Status Targets, Prev: Defining Global Properties, Up: Configuration + +4.11 Defining Builders +====================== + +The `c['builders']' key is a list of dictionaries which specify the +Builders. The Buildmaster runs a collection of Builders, each of +which handles a single type of build (e.g. full versus quick), on a +single build slave. A Buildbot which makes sure that the latest code +("HEAD") compiles correctly across four separate architecture will +have four Builders, each performing the same build but on different +slaves (one per platform). + + Each Builder gets a separate column in the waterfall display. In +general, each Builder runs independently (although various kinds of +interlocks can cause one Builder to have an effect on another). + + Each Builder specification dictionary has several required keys: + +`name' + This specifies the Builder's name, which is used in status + reports. + +`slavename' + This specifies which buildslave will be used by this Builder. + `slavename' must appear in the `c['slaves']' list. Each + buildslave can accomodate multiple Builders. + +`slavenames' + If you provide `slavenames' instead of `slavename', you can give + a list of buildslaves which are capable of running this Builder. + If multiple buildslaves are available for any given Builder, you + will have some measure of redundancy: in case one slave goes + offline, the others can still keep the Builder working. In + addition, multiple buildslaves will allow multiple simultaneous + builds for the same Builder, which might be useful if you have a + lot of forced or "try" builds taking place. + + If you use this feature, it is important to make sure that the + buildslaves are all, in fact, capable of running the given + build. The slave hosts should be configured similarly, otherwise + you will spend a lot of time trying (unsuccessfully) to + reproduce a failure that only occurs on some of the buildslaves + and not the others. Different platforms, operating systems, + versions of major programs or libraries, all these things mean + you should use separate Builders. + +`builddir' + This specifies the name of a subdirectory (under the base + directory) in which everything related to this builder will be + placed. On the buildmaster, this holds build status information. + On the buildslave, this is where checkouts, compiles, and tests + are run. + +`factory' + This is a `buildbot.process.factory.BuildFactory' instance which + controls how the build is performed. Full details appear in + their own chapter, *Note Build Process::. Parameters like the + location of the CVS repository and the compile-time options used + for the build are generally provided as arguments to the + factory's constructor. + + + Other optional keys may be set on each Builder: + +`category' + If provided, this is a string that identifies a category for the + builder to be a part of. Status clients can limit themselves to a + subset of the available categories. A common use for this is to + add new builders to your setup (for a new module, or for a new + buildslave) that do not work correctly yet and allow you to + integrate them with the active builders. You can put these new + builders in a test category, make your main status clients + ignore them, and have only private status clients pick them up. + As soon as they work, you can move them over to the active + category. + + + +File: buildbot.info, Node: Defining Status Targets, Next: Debug options, Prev: Defining Builders, Up: Configuration + +4.12 Defining Status Targets +============================ + +The Buildmaster has a variety of ways to present build status to +various users. Each such delivery method is a "Status Target" object +in the configuration's `status' list. To add status targets, you just +append more objects to this list: + + c['status'] = [] + + from buildbot.status import html + c['status'].append(html.Waterfall(http_port=8010)) + + from buildbot.status import mail + m = mail.MailNotifier(fromaddr="buildbot@localhost", + extraRecipients=["builds@lists.example.com"], + sendToInterestedUsers=False) + c['status'].append(m) + + from buildbot.status import words + c['status'].append(words.IRC(host="irc.example.com", nick="bb", + channels=["#example"])) + + Status delivery has its own chapter, *Note Status Delivery::, in +which all the built-in status targets are documented. + + +File: buildbot.info, Node: Debug options, Prev: Defining Status Targets, Up: Configuration + +4.13 Debug options +================== + +If you set `c['debugPassword']', then you can connect to the +buildmaster with the diagnostic tool launched by `buildbot +debugclient MASTER:PORT'. From this tool, you can reload the config +file, manually force builds, and inject changes, which may be useful +for testing your buildmaster without actually commiting changes to +your repository (or before you have the Change Sources set up). The +debug tool uses the same port number as the slaves do: +`c['slavePortnum']', and is authenticated with this password. + + c['debugPassword'] = "debugpassword" + + If you set `c['manhole']' to an instance of one of the classes in +`buildbot.manhole', you can telnet or ssh into the buildmaster and +get an interactive Python shell, which may be useful for debugging +buildbot internals. It is probably only useful for buildbot +developers. It exposes full access to the buildmaster's account +(including the ability to modify and delete files), so it should not +be enabled with a weak or easily guessable password. + + There are three separate `Manhole' classes. Two of them use SSH, +one uses unencrypted telnet. Two of them use a username+password +combination to grant access, one of them uses an SSH-style +`authorized_keys' file which contains a list of ssh public keys. + +`manhole.AuthorizedKeysManhole' + You construct this with the name of a file that contains one SSH + public key per line, just like `~/.ssh/authorized_keys'. If you + provide a non-absolute filename, it will be interpreted relative + to the buildmaster's base directory. + +`manhole.PasswordManhole' + This one accepts SSH connections but asks for a username and + password when authenticating. It accepts only one such pair. + +`manhole.TelnetManhole' + This accepts regular unencrypted telnet connections, and asks + for a username/password pair before providing access. Because + this username/password is transmitted in the clear, and because + Manhole access to the buildmaster is equivalent to granting full + shell privileges to both the buildmaster and all the buildslaves + (and to all accounts which then run code produced by the + buildslaves), it is highly recommended that you use one of the + SSH manholes instead. + + + # some examples: + from buildbot import manhole + c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys") + c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword") + c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please") + + The `Manhole' instance can be configured to listen on a specific +port. You may wish to have this listening port bind to the loopback +interface (sometimes known as "lo0", "localhost", or 127.0.0.1) to +restrict access to clients which are running on the same host. + + from buildbot.manhole import PasswordManhole + c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd") + + To have the `Manhole' listen on all interfaces, use `"tcp:9999"' +or simply 9999. This port specification uses +`twisted.application.strports', so you can make it listen on SSL or +even UNIX-domain sockets if you want. + + Note that using any Manhole requires that the TwistedConch package +be installed, and that you be using Twisted version 2.0 or later. + + The buildmaster's SSH server will use a different host key than the +normal sshd running on a typical unix host. This will cause the ssh +client to complain about a "host key mismatch", because it does not +realize there are two separate servers running on the same host. To +avoid this, use a clause like the following in your `.ssh/config' +file: + + Host remotehost-buildbot + HostName remotehost + HostKeyAlias remotehost-buildbot + Port 9999 + # use 'user' if you use PasswordManhole and your name is not 'admin'. + # if you use AuthorizedKeysManhole, this probably doesn't matter. + User admin + + +File: buildbot.info, Node: Getting Source Code Changes, Next: Build Process, Prev: Configuration, Up: Top + +5 Getting Source Code Changes +***************************** + +The most common way to use the Buildbot is centered around the idea of +`Source Trees': a directory tree filled with source code of some form +which can be compiled and/or tested. Some projects use languages that +don't involve any compilation step: nevertheless there may be a +`build' phase where files are copied or rearranged into a form that +is suitable for installation. Some projects do not have unit tests, +and the Buildbot is merely helping to make sure that the sources can +compile correctly. But in all of these cases, the thing-being-tested +is a single source tree. + + A Version Control System mantains a source tree, and tells the +buildmaster when it changes. The first step of each Build is typically +to acquire a copy of some version of this tree. + + This chapter describes how the Buildbot learns about what Changes +have occurred. For more information on VC systems and Changes, see +*note Version Control Systems::. + +* Menu: + +* Change Sources:: +* Choosing ChangeSources:: +* CVSToys - PBService:: +* Mail-parsing ChangeSources:: +* PBChangeSource:: +* P4Source:: +* BonsaiPoller:: +* SVNPoller:: +* MercurialHook:: +* Bzr Hook:: +* Bzr Poller:: + + +File: buildbot.info, Node: Change Sources, Next: Choosing ChangeSources, Prev: Getting Source Code Changes, Up: Getting Source Code Changes + +5.1 Change Sources +================== + +Each Buildmaster watches a single source tree. Changes can be provided +by a variety of ChangeSource types, however any given project will +typically have only a single ChangeSource active. This section +provides a description of all available ChangeSource types and +explains how to set up each of them. + + There are a variety of ChangeSources available, some of which are +meant to be used in conjunction with other tools to deliver Change +events from the VC repository to the buildmaster. + + * CVSToys This ChangeSource opens a TCP connection from the + buildmaster to a waiting FreshCVS daemon that lives on the + repository machine, and subscribes to hear about Changes. + + * MaildirSource This one watches a local maildir-format inbox for + email sent out by the repository when a change is made. When a + message arrives, it is parsed to create the Change object. A + variety of parsing functions are available to accomodate + different email-sending tools. + + * PBChangeSource This ChangeSource listens on a local TCP socket + for inbound connections from a separate tool. Usually, this tool + would be run on the VC repository machine in a commit hook. It + is expected to connect to the TCP socket and send a Change + message over the network connection. The `buildbot sendchange' + command is one example of a tool that knows how to send these + messages, so you can write a commit script for your VC system + that calls it to deliver the Change. There are other tools in + the contrib/ directory that use the same protocol. + + + As a quick guide, here is a list of VC systems and the +ChangeSources that might be useful with them. All of these +ChangeSources are in the `buildbot.changes' module. + +`CVS' + * freshcvs.FreshCVSSource (connected via TCP to the freshcvs + daemon) + + * mail.FCMaildirSource (watching for email sent by a freshcvs + daemon) + + * mail.BonsaiMaildirSource (watching for email sent by Bonsai) + + * mail.SyncmailMaildirSource (watching for email sent by + syncmail) + + * pb.PBChangeSource (listening for connections from `buildbot + sendchange' run in a loginfo script) + + * pb.PBChangeSource (listening for connections from a + long-running `contrib/viewcvspoll.py' polling process which + examines the ViewCVS database directly + +`SVN' + * pb.PBChangeSource (listening for connections from + `contrib/svn_buildbot.py' run in a postcommit script) + + * pb.PBChangeSource (listening for connections from a + long-running `contrib/svn_watcher.py' or + `contrib/svnpoller.py' polling process + + * mail.SVNCommitEmailMaildirSource (watching for email sent + by commit-email.pl) + + * svnpoller.SVNPoller (polling the SVN repository) + +`Darcs' + * pb.PBChangeSource (listening for connections from + `contrib/darcs_buildbot.py' in a commit script + +`Mercurial' + * pb.PBChangeSource (listening for connections from + `contrib/hg_buildbot.py' run in an 'incoming' hook) + + * pb.PBChangeSource (listening for connections from + `buildbot/changes/hgbuildbot.py' run as an in-process + 'changegroup' hook) + +`Arch/Bazaar' + * pb.PBChangeSource (listening for connections from + `contrib/arch_buildbot.py' run in a commit hook) + +`Bzr (the newer Bazaar)' + * pb.PBChangeSource (listening for connections from + `contrib/bzr_buildbot.py' run in a post-change-branch-tip + or commit hook) + + * `contrib/bzr_buildbot.py''s BzrPoller (polling the Bzr + repository) + +`Git' + * pb.PBChangeSource (listening for connections from + `contrib/git_buildbot.py' run in the post-receive hook) + + + All VC systems can be driven by a PBChangeSource and the `buildbot +sendchange' tool run from some form of commit script. If you write +an email parsing function, they can also all be driven by a suitable +`MaildirSource'. + + +File: buildbot.info, Node: Choosing ChangeSources, Next: CVSToys - PBService, Prev: Change Sources, Up: Getting Source Code Changes + +5.2 Choosing ChangeSources +========================== + +The `master.cfg' configuration file has a dictionary key named +`BuildmasterConfig['change_source']', which holds the active +`IChangeSource' object. The config file will typically create an +object from one of the classes described below and stuff it into this +key. + + Each buildmaster typically has just a single ChangeSource, since +it is only watching a single source tree. But if, for some reason, +you need multiple sources, just set `c['change_source']' to a list of +ChangeSources.. it will accept that too. + + s = FreshCVSSourceNewcred(host="host", port=4519, + user="alice", passwd="secret", + prefix="Twisted") + BuildmasterConfig['change_source'] = [s] + + Each source tree has a nominal `top'. Each Change has a list of +filenames, which are all relative to this top location. The +ChangeSource is responsible for doing whatever is necessary to +accomplish this. Most sources have a `prefix' argument: a partial +pathname which is stripped from the front of all filenames provided to +that `ChangeSource'. Files which are outside this sub-tree are +ignored by the changesource: it does not generate Changes for those +files. + + +File: buildbot.info, Node: CVSToys - PBService, Next: Mail-parsing ChangeSources, Prev: Choosing ChangeSources, Up: Getting Source Code Changes + +5.3 CVSToys - PBService +======================= + +The CVSToys (http://purl.net/net/CVSToys) package provides a server +which runs on the machine that hosts the CVS repository it watches. +It has a variety of ways to distribute commit notifications, and +offers a flexible regexp-based way to filter out uninteresting +changes. One of the notification options is named `PBService' and +works by listening on a TCP port for clients. These clients subscribe +to hear about commit notifications. + + The buildmaster has a CVSToys-compatible `PBService' client built +in. There are two versions of it, one for old versions of CVSToys +(1.0.9 and earlier) which used the `oldcred' authentication +framework, and one for newer versions (1.0.10 and later) which use +`newcred'. Both are classes in the `buildbot.changes.freshcvs' +package. + + `FreshCVSSourceNewcred' objects are created with the following +parameters: + +``host' and `port'' + these specify where the CVSToys server can be reached + +``user' and `passwd'' + these specify the login information for the CVSToys server + (`freshcvs'). These must match the server's values, which are + defined in the `freshCfg' configuration file (which lives in the + CVSROOT directory of the repository). + +``prefix'' + this is the prefix to be found and stripped from filenames + delivered by the CVSToys server. Most projects live in + sub-directories of the main repository, as siblings of the + CVSROOT sub-directory, so typically this prefix is set to that + top sub-directory name. + + +Example +======= + +To set up the freshCVS server, add a statement like the following to +your `freshCfg' file: + + pb = ConfigurationSet([ + (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)), + ]) + + This will announce all changes to a client which connects to port +4519 using a username of 'foo' and a password of 'bar'. + + Then add a clause like this to your buildmaster's `master.cfg': + + BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, + "foo", "bar", + prefix="glib/") + + where "cvs.example.com" is the host that is running the FreshCVS +daemon, and "glib" is the top-level directory (relative to the +repository's root) where all your source code lives. Most projects +keep one or more projects in the same repository (along with CVSROOT/ +to hold admin files like loginfo and freshCfg); the prefix= argument +tells the buildmaster to ignore everything outside that directory, +and to strip that common prefix from all pathnames it handles. + + +File: buildbot.info, Node: Mail-parsing ChangeSources, Next: PBChangeSource, Prev: CVSToys - PBService, Up: Getting Source Code Changes + +5.4 Mail-parsing ChangeSources +============================== + +Many projects publish information about changes to their source tree +by sending an email message out to a mailing list, frequently named +PROJECT-commits or PROJECT-changes. Each message usually contains a +description of the change (who made the change, which files were +affected) and sometimes a copy of the diff. Humans can subscribe to +this list to stay informed about what's happening to the source tree. + + The Buildbot can also be subscribed to a -commits mailing list, and +can trigger builds in response to Changes that it hears about. The +buildmaster admin needs to arrange for these email messages to arrive +in a place where the buildmaster can find them, and configure the +buildmaster to parse the messages correctly. Once that is in place, +the email parser will create Change objects and deliver them to the +Schedulers (see *note Change Sources and Schedulers::) just like any +other ChangeSource. + + There are two components to setting up an email-based ChangeSource. +The first is to route the email messages to the buildmaster, which is +done by dropping them into a "maildir". The second is to actually +parse the messages, which is highly dependent upon the tool that was +used to create them. Each VC system has a collection of favorite +change-emailing tools, and each has a slightly different format, so +each has a different parsing function. There is a separate +ChangeSource variant for each parsing function. + + Once you've chosen a maildir location and a parsing function, +create the change source and put it in `c['change_source']': + + from buildbot.changes.mail import SyncmailMaildirSource + c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", + prefix="/trunk/") + +* Menu: + +* Subscribing the Buildmaster:: +* Using Maildirs:: +* Parsing Email Change Messages:: + + +File: buildbot.info, Node: Subscribing the Buildmaster, Next: Using Maildirs, Prev: Mail-parsing ChangeSources, Up: Mail-parsing ChangeSources + +5.4.1 Subscribing the Buildmaster +--------------------------------- + +The recommended way to install the buildbot is to create a dedicated +account for the buildmaster. If you do this, the account will probably +have a distinct email address (perhaps <buildmaster@example.org>). +Then just arrange for this account's email to be delivered to a +suitable maildir (described in the next section). + + If the buildbot does not have its own account, "extension +addresses" can be used to distinguish between email intended for the +buildmaster and email intended for the rest of the account. In most +modern MTAs, the e.g. `foo@example.org' account has control over +every email address at example.org which begins with "foo", such that +email addressed to <account-foo@example.org> can be delivered to a +different destination than <account-bar@example.org>. qmail does this +by using separate .qmail files for the two destinations (`.qmail-foo' +and `.qmail-bar', with `.qmail' controlling the base address and +`.qmail-default' controlling all other extensions). Other MTAs have +similar mechanisms. + + Thus you can assign an extension address like +<foo-buildmaster@example.org> to the buildmaster, and retain +<foo@example.org> for your own use. + + +File: buildbot.info, Node: Using Maildirs, Next: Parsing Email Change Messages, Prev: Subscribing the Buildmaster, Up: Mail-parsing ChangeSources + +5.4.2 Using Maildirs +-------------------- + +A "maildir" is a simple directory structure originally developed for +qmail that allows safe atomic update without locking. Create a base +directory with three subdirectories: "new", "tmp", and "cur". When +messages arrive, they are put into a uniquely-named file (using pids, +timestamps, and random numbers) in "tmp". When the file is complete, +it is atomically renamed into "new". Eventually the buildmaster +notices the file in "new", reads and parses the contents, then moves +it into "cur". A cronjob can be used to delete files in "cur" at +leisure. + + Maildirs are frequently created with the `maildirmake' tool, but a +simple `mkdir -p ~/MAILDIR/{cur,new,tmp}' is pretty much equivalent. + + Many modern MTAs can deliver directly to maildirs. The usual +.forward or .procmailrc syntax is to name the base directory with a +trailing slash, so something like `~/MAILDIR/' . qmail and postfix are +maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail +Delivery Agent). + + For MTAs which cannot put files into maildirs directly, the +"safecat" tool can be executed from a .forward file to accomplish the +same thing. + + The Buildmaster uses the linux DNotify facility to receive +immediate notification when the maildir's "new" directory has +changed. When this facility is not available, it polls the directory +for new messages, every 10 seconds by default. + + +File: buildbot.info, Node: Parsing Email Change Messages, Prev: Using Maildirs, Up: Mail-parsing ChangeSources + +5.4.3 Parsing Email Change Messages +----------------------------------- + +The second component to setting up an email-based ChangeSource is to +parse the actual notices. This is highly dependent upon the VC system +and commit script in use. + + A couple of common tools used to create these change emails are: + +`CVS' + + `CVSToys MailNotifier' + *note FCMaildirSource:: + + `Bonsai notification' + *note BonsaiMaildirSource:: + + `syncmail' + *note SyncmailMaildirSource:: + +`SVN' + + `svnmailer' + http://opensource.perlig.de/en/svnmailer/ + + `commit-email.pl' + *note SVNCommitEmailMaildirSource:: + +`Mercurial' + + `NotifyExtension' + http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension + +`Git' + + `post-receive-email' + http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD + + + The following sections describe the parsers available for each of +these tools. + + Most of these parsers accept a `prefix=' argument, which is used +to limit the set of files that the buildmaster pays attention to. This +is most useful for systems like CVS and SVN which put multiple +projects in a single repository (or use repository names to indicate +branches). Each filename that appears in the email is tested against +the prefix: if the filename does not start with the prefix, the file +is ignored. If the filename _does_ start with the prefix, that prefix +is stripped from the filename before any further processing is done. +Thus the prefix usually ends with a slash. + +* Menu: + +* FCMaildirSource:: +* SyncmailMaildirSource:: +* BonsaiMaildirSource:: +* SVNCommitEmailMaildirSource:: + + +File: buildbot.info, Node: FCMaildirSource, Next: SyncmailMaildirSource, Prev: Parsing Email Change Messages, Up: Parsing Email Change Messages + +5.4.3.1 FCMaildirSource +....................... + +http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/ + + This parser works with the CVSToys `MailNotification' action, +which will send email to a list of recipients for each commit. This +tends to work better than using `/bin/mail' from within the +CVSROOT/loginfo file directly, as CVSToys will batch together all +files changed during the same CVS invocation, and can provide more +information (like creating a ViewCVS URL for each file changed). + + The Buildbot's `FCMaildirSource' knows for to parse these CVSToys +messages and turn them into Change objects. It can be given two +parameters: the directory name of the maildir root, and the prefix to +strip. + + from buildbot.changes.mail import FCMaildirSource + c['change_source'] = FCMaildirSource("~/maildir-buildbot") + + +File: buildbot.info, Node: SyncmailMaildirSource, Next: BonsaiMaildirSource, Prev: FCMaildirSource, Up: Parsing Email Change Messages + +5.4.3.2 SyncmailMaildirSource +............................. + +http://sourceforge.net/projects/cvs-syncmail + + `SyncmailMaildirSource' knows how to parse the message format used +by the CVS "syncmail" script. + + from buildbot.changes.mail import SyncmailMaildirSource + c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot") + + +File: buildbot.info, Node: BonsaiMaildirSource, Next: SVNCommitEmailMaildirSource, Prev: SyncmailMaildirSource, Up: Parsing Email Change Messages + +5.4.3.3 BonsaiMaildirSource +........................... + +http://www.mozilla.org/bonsai.html + + `BonsaiMaildirSource' parses messages sent out by Bonsai, the CVS +tree-management system built by Mozilla. + + from buildbot.changes.mail import BonsaiMaildirSource + c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot") + + +File: buildbot.info, Node: SVNCommitEmailMaildirSource, Prev: BonsaiMaildirSource, Up: Parsing Email Change Messages + +5.4.3.4 SVNCommitEmailMaildirSource +................................... + +`SVNCommitEmailMaildirSource' parses message sent out by the +`commit-email.pl' script, which is included in the Subversion +distribution. + + It does not currently handle branches: all of the Change objects +that it creates will be associated with the default (i.e. trunk) +branch. + + from buildbot.changes.mail import SVNCommitEmailMaildirSource + c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") + + +File: buildbot.info, Node: PBChangeSource, Next: P4Source, Prev: Mail-parsing ChangeSources, Up: Getting Source Code Changes + +5.5 PBChangeSource +================== + +The last kind of ChangeSource actually listens on a TCP port for +clients to connect and push change notices _into_ the Buildmaster. +This is used by the built-in `buildbot sendchange' notification tool, +as well as the VC-specific `contrib/svn_buildbot.py', +`contrib/arch_buildbot.py', `contrib/hg_buildbot.py' tools, and the +`buildbot.changes.hgbuildbot' hook. These tools are run by the +repository (in a commit hook script), and connect to the buildmaster +directly each time a file is comitted. This is also useful for +creating new kinds of change sources that work on a `push' model +instead of some kind of subscription scheme, for example a script +which is run out of an email .forward file. + + This ChangeSource can be configured to listen on its own TCP port, +or it can share the port that the buildmaster is already using for the +buildslaves to connect. (This is possible because the +`PBChangeSource' uses the same protocol as the buildslaves, and they +can be distinguished by the `username' attribute used when the +initial connection is established). It might be useful to have it +listen on a different port if, for example, you wanted to establish +different firewall rules for that port. You could allow only the SVN +repository machine access to the `PBChangeSource' port, while +allowing only the buildslave machines access to the slave port. Or you +could just expose one port and run everything over it. _Note: this +feature is not yet implemented, the PBChangeSource will always share +the slave port and will always have a `user' name of `change', and a +passwd of `changepw'. These limitations will be removed in the +future._. + + The `PBChangeSource' is created with the following arguments. All +are optional. + +``port'' + which port to listen on. If `None' (which is the default), it + shares the port used for buildslave connections. _Not + Implemented, always set to `None'_. + +``user' and `passwd'' + The user/passwd account information that the client program must + use to connect. Defaults to `change' and `changepw'. _Not + Implemented, `user' is currently always set to `change', + `passwd' is always set to `changepw'_. + +``prefix'' + The prefix to be found and stripped from filenames delivered + over the connection. Any filenames which do not start with this + prefix will be removed. If all the filenames in a given Change + are removed, the that whole Change will be dropped. This string + should probably end with a directory separator. + + This is useful for changes coming from version control systems + that represent branches as parent directories within the + repository (like SVN and Perforce). Use a prefix of 'trunk/' or + 'project/branches/foobranch/' to only follow one branch and to + get correct tree-relative filenames. Without a prefix, the + PBChangeSource will probably deliver Changes with filenames like + `trunk/foo.c' instead of just `foo.c'. Of course this also + depends upon the tool sending the Changes in (like `buildbot + sendchange') and what filenames it is delivering: that tool may + be filtering and stripping prefixes at the sending end. + + + +File: buildbot.info, Node: P4Source, Next: BonsaiPoller, Prev: PBChangeSource, Up: Getting Source Code Changes + +5.6 P4Source +============ + +The `P4Source' periodically polls a Perforce +(http://www.perforce.com/) depot for changes. It accepts the +following arguments: + +``p4base'' + The base depot path to watch, without the trailing '/...'. + +``p4port'' + The Perforce server to connect to (as host:port). + +``p4user'' + The Perforce user. + +``p4passwd'' + The Perforce password. + +``p4bin'' + An optional string parameter. Specify the location of the + perforce command line binary (p4). You only need to do this if + the perforce binary is not in the path of the buildbot user. + Defaults to "p4". + +``split_file'' + A function that maps a pathname, without the leading `p4base', + to a (branch, filename) tuple. The default just returns (None, + branchfile), which effectively disables branch support. You + should supply a function which understands your repository + structure. + +``pollinterval'' + How often to poll, in seconds. Defaults to 600 (10 minutes). + +``histmax'' + The maximum number of changes to inspect at a time. If more than + this number occur since the last poll, older changes will be + silently ignored. + +Example +======= + +This configuration uses the `P4PORT', `P4USER', and `P4PASSWD' +specified in the buildmaster's environment. It watches a project in +which the branch name is simply the next path component, and the file +is all path components after. + + import buildbot.changes.p4poller + s = p4poller.P4Source(p4base='//depot/project/', + split_file=lambda branchfile: branchfile.split('/',1), + ) + c['change_source'] = s + + +File: buildbot.info, Node: BonsaiPoller, Next: SVNPoller, Prev: P4Source, Up: Getting Source Code Changes + +5.7 BonsaiPoller +================ + +The `BonsaiPoller' periodically polls a Bonsai server. This is a CGI +script accessed through a web server that provides information about +a CVS tree, for example the Mozilla bonsai server at +`http://bonsai.mozilla.org'. Bonsai servers are usable by both humans +and machines. In this case, the buildbot's change source forms a +query which asks about any files in the specified branch which have +changed since the last query. + + Please take a look at the BonsaiPoller docstring for details about +the arguments it accepts. + + +File: buildbot.info, Node: SVNPoller, Next: MercurialHook, Prev: BonsaiPoller, Up: Getting Source Code Changes + +5.8 SVNPoller +============= + +The `buildbot.changes.svnpoller.SVNPoller' is a ChangeSource which +periodically polls a Subversion (http://subversion.tigris.org/) +repository for new revisions, by running the `svn log' command in a +subshell. It can watch a single branch or multiple branches. + + `SVNPoller' accepts the following arguments: + +`svnurl' + The base URL path to watch, like + `svn://svn.twistedmatrix.com/svn/Twisted/trunk', or + `http://divmod.org/svn/Divmod/', or even + `file:///home/svn/Repository/ProjectA/branches/1.5/'. This must + include the access scheme, the location of the repository (both + the hostname for remote ones, and any additional directory names + necessary to get to the repository), and the sub-path within the + repository's virtual filesystem for the project and branch of + interest. + + The `SVNPoller' will only pay attention to files inside the + subdirectory specified by the complete svnurl. + +`split_file' + A function to convert pathnames into (branch, relative_pathname) + tuples. Use this to explain your repository's branch-naming + policy to `SVNPoller'. This function must accept a single string + and return a two-entry tuple. There are a few utility functions + in `buildbot.changes.svnpoller' that can be used as a + `split_file' function, see below for details. + + The default value always returns (None, path), which indicates + that all files are on the trunk. + + Subclasses of `SVNPoller' can override the `split_file' method + instead of using the `split_file=' argument. + +`svnuser' + An optional string parameter. If set, the `--user' argument will + be added to all `svn' commands. Use this if you have to + authenticate to the svn server before you can do `svn info' or + `svn log' commands. + +`svnpasswd' + Like `svnuser', this will cause a `--password' argument to be + passed to all svn commands. + +`pollinterval' + How often to poll, in seconds. Defaults to 600 (checking once + every 10 minutes). Lower this if you want the buildbot to notice + changes faster, raise it if you want to reduce the network and + CPU load on your svn server. Please be considerate of public SVN + repositories by using a large interval when polling them. + +`histmax' + The maximum number of changes to inspect at a time. Every + POLLINTERVAL seconds, the `SVNPoller' asks for the last HISTMAX + changes and looks through them for any ones it does not already + know about. If more than HISTMAX revisions have been committed + since the last poll, older changes will be silently ignored. + Larger values of histmax will cause more time and memory to be + consumed on each poll attempt. `histmax' defaults to 100. + +`svnbin' + This controls the `svn' executable to use. If subversion is + installed in a weird place on your system (outside of the + buildmaster's `$PATH'), use this to tell `SVNPoller' where to + find it. The default value of "svn" will almost always be + sufficient. + + +Branches +======== + +Each source file that is tracked by a Subversion repository has a +fully-qualified SVN URL in the following form: +(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the +`SVNPoller', you give it a `svnurl' value that includes all of the +REPOURL and possibly some portion of the PROJECT-plus-BRANCH string. +The `SVNPoller' is responsible for producing Changes that contain a +branch name and a FILEPATH (which is relative to the top of a +checked-out tree). The details of how these strings are split up +depend upon how your repository names its branches. + +PROJECT/BRANCHNAME/FILEPATH repositories +---------------------------------------- + +One common layout is to have all the various projects that share a +repository get a single top-level directory each. Then under a given +project's directory, you get two subdirectories, one named "trunk" +and another named "branches". Under "branches" you have a bunch of +other directories, one per branch, with names like "1.5.x" and +"testing". It is also common to see directories like "tags" and +"releases" next to "branches" and "trunk". + + For example, the Twisted project has a subversion server on +"svn.twistedmatrix.com" that hosts several sub-projects. The +repository is available through a SCHEME of "svn:". The primary +sub-project is Twisted, of course, with a repository root of +"svn://svn.twistedmatrix.com/svn/Twisted". Another sub-project is +Informant, with a root of +"svn://svn.twistedmatrix.com/svn/Informant", etc. Inside any +checked-out Twisted tree, there is a file named bin/trial (which is +used to run unit test suites). + + The trunk for Twisted is in +"svn://svn.twistedmatrix.com/svn/Twisted/trunk", and the +fully-qualified SVN URL for the trunk version of `trial' would be +"svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial". The same +SVNURL for that file on a branch named "1.5.x" would be +"svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial". + + To set up a `SVNPoller' that watches the Twisted trunk (and +nothing else), we would use the following: + + from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk") + + In this case, every Change that our `SVNPoller' produces will have +`.branch=None', to indicate that the Change is on the trunk. No +other sub-projects or branches will be tracked. + + If we want our ChangeSource to follow multiple branches, we have +to do two things. First we have to change our `svnurl=' argument to +watch more than just ".../Twisted/trunk". We will set it to +".../Twisted" so that we'll see both the trunk and all the branches. +Second, we have to tell `SVNPoller' how to split the +(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository +out into (BRANCH) and (FILEPATH) pairs. + + We do the latter by providing a "split_file" function. This +function is responsible for splitting something like +"branches/1.5.x/bin/trial" into `branch'="branches/1.5.x" and +`filepath'="bin/trial". This function is always given a string that +names a file relative to the subdirectory pointed to by the +`SVNPoller''s `svnurl=' argument. It is expected to return a +(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the +branch indicated), or None to indicate that the file is outside any +project of interest. + + (note that we want to see "branches/1.5.x" rather than just +"1.5.x" because when we perform the SVN checkout, we will probably +append the branch name to the baseURL, which requires that we keep the +"branches" component in there. Other VC schemes use a different +approach towards branches and may not require this artifact.) + + If your repository uses this same PROJECT/BRANCH/FILEPATH naming +scheme, the following function will work: + + def split_file_branches(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + return (None, '/'.join(pieces[1:])) + elif pieces[0] == 'branches': + return ('/'.join(pieces[0:2]), + '/'.join(pieces[2:])) + else: + return None + + This function is provided as +`buildbot.changes.svnpoller.split_file_branches' for your +convenience. So to have our Twisted-watching `SVNPoller' follow +multiple branches, we would use this: + + from buildbot.changes.svnpoller import SVNPoller, split_file_branches + c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted", + split_file=split_file_branches) + + Changes for all sorts of branches (with names like +"branches/1.5.x", and None to indicate the trunk) will be delivered +to the Schedulers. Each Scheduler is then free to use or ignore each +branch as it sees fit. + +BRANCHNAME/PROJECT/FILEPATH repositories +---------------------------------------- + +Another common way to organize a Subversion repository is to put the +branch name at the top, and the projects underneath. This is +especially frequent when there are a number of related sub-projects +that all get released in a group. + + For example, Divmod.org hosts a project named "Nevow" as well as +one named "Quotient". In a checked-out Nevow tree there is a directory +named "formless" that contains a python source file named +"webform.py". This repository is accessible via webdav (and thus uses +an "http:" scheme) through the divmod.org hostname. There are many +branches in this repository, and they use a (BRANCHNAME)/(PROJECT) +naming policy. + + The fully-qualified SVN URL for the trunk version of webform.py is +`http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py'. You +can do an `svn co' with that URL and get a copy of the latest +version. The 1.5.x branch version of this file would have a URL of +`http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py'. +The whole Nevow trunk would be checked out with +`http://divmod.org/svn/Divmod/trunk/Nevow', while the Quotient trunk +would be checked out using +`http://divmod.org/svn/Divmod/trunk/Quotient'. + + Now suppose we want to have an `SVNPoller' that only cares about +the Nevow trunk. This case looks just like the PROJECT/BRANCH layout +described earlier: + + from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow") + + But what happens when we want to track multiple Nevow branches? We +have to point our `svnurl=' high enough to see all those branches, +but we also don't want to include Quotient changes (since we're only +building Nevow). To accomplish this, we must rely upon the +`split_file' function to help us tell the difference between files +that belong to Nevow and those that belong to Quotient, as well as +figuring out which branch each one is on. + + from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod", + split_file=my_file_splitter) + + The `my_file_splitter' function will be called with +repository-relative pathnames like: + +`trunk/Nevow/formless/webform.py' + This is a Nevow file, on the trunk. We want the Change that + includes this to see a filename of `formless/webform.py"', and a + branch of None + +`branches/1.5.x/Nevow/formless/webform.py' + This is a Nevow file, on a branch. We want to get + branch="branches/1.5.x" and filename="formless/webform.py". + +`trunk/Quotient/setup.py' + This is a Quotient file, so we want to ignore it by having + `my_file_splitter' return None. + +`branches/1.5.x/Quotient/setup.py' + This is also a Quotient file, which should be ignored. + + The following definition for `my_file_splitter' will do the job: + + def my_file_splitter(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + branch = None + pieces.pop(0) # remove 'trunk' + elif pieces[0] == 'branches': + pieces.pop(0) # remove 'branches' + # grab branch name + branch = 'branches/' + pieces.pop(0) + else: + return None # something weird + projectname = pieces.pop(0) + if projectname != 'Nevow': + return None # wrong project + return (branch, '/'.join(pieces)) + + +File: buildbot.info, Node: MercurialHook, Next: Bzr Hook, Prev: SVNPoller, Up: Getting Source Code Changes + +5.9 MercurialHook +================= + +Since Mercurial is written in python, the hook script can invoke +Buildbot's `sendchange' function directly, rather than having to +spawn an external process. This function delivers the same sort of +changes as `buildbot sendchange' and the various hook scripts in +contrib/, so you'll need to add a `pb.PBChangeSource' to your +buildmaster to receive these changes. + + To set this up, first choose a Mercurial repository that represents +your central "official" source tree. This will be the same repository +that your buildslaves will eventually pull from. Install Buildbot on +the machine that hosts this repository, using the same version of +python as Mercurial is using (so that the Mercurial hook can import +code from buildbot). Then add the following to the `.hg/hgrc' file in +that repository, replacing the buildmaster hostname/portnumber as +appropriate for your buildbot: + + [hooks] + changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook + + [hgbuildbot] + master = buildmaster.example.org:9987 + + (Note that Mercurial lets you define multiple `changegroup' hooks +by giving them distinct names, like `changegroup.foo' and +`changegroup.bar', which is why we use `changegroup.buildbot' in this +example. There is nothing magical about the "buildbot" suffix in the +hook name. The `[hgbuildbot]' section _is_ special, however, as it is +the only section that the buildbot hook pays attention to.) + + Also note that this runs as a `changegroup' hook, rather than as +an `incoming' hook. The `changegroup' hook is run with multiple +revisions at a time (say, if multiple revisions are being pushed to +this repository in a single `hg push' command), whereas the +`incoming' hook is run with just one revision at a time. The +`hgbuildbot.hook' function will only work with the `changegroup' hook. + + The `[hgbuildbot]' section has two other parameters that you might +specify, both of which control the name of the branch that is +attached to the changes coming from this hook. + + One common branch naming policy for Mercurial repositories is to +use it just like Darcs: each branch goes into a separate repository, +and all the branches for a single project share a common parent +directory. For example, you might have `/var/repos/PROJECT/trunk/' +and `/var/repos/PROJECT/release'. To use this style, use the +`branchtype = dirname' setting, which simply uses the last component +of the repository's enclosing directory as the branch name: + + [hgbuildbot] + master = buildmaster.example.org:9987 + branchtype = dirname + + Another approach is to use Mercurial's built-in branches (the kind +created with `hg branch' and listed with `hg branches'). This feature +associates persistent names with particular lines of descent within a +single repository. (note that the buildbot `source.Mercurial' +checkout step does not yet support this kind of branch). To have the +commit hook deliver this sort of branch name with the Change object, +use `branchtype = inrepo': + + [hgbuildbot] + master = buildmaster.example.org:9987 + branchtype = inrepo + + Finally, if you want to simply specify the branchname directly, for +all changes, use `branch = BRANCHNAME'. This overrides `branchtype': + + [hgbuildbot] + master = buildmaster.example.org:9987 + branch = trunk + + If you use `branch=' like this, you'll need to put a separate +.hgrc in each repository. If you use `branchtype=', you may be able +to use the same .hgrc for all your repositories, stored in `~/.hgrc' +or `/etc/mercurial/hgrc'. + + +File: buildbot.info, Node: Bzr Hook, Next: Bzr Poller, Prev: MercurialHook, Up: Getting Source Code Changes + +5.10 Bzr Hook +============= + +Bzr is also written in Python, and the Bzr hook depends on Twisted to +send the changes. + + To install, put `contrib/bzr_buildbot.py' in one of your plugins +locations a bzr plugins directory (e.g., `~/.bazaar/plugins'). Then, +in one of your bazaar conf files (e.g., `~/.bazaar/locations.conf'), +set the location you want to connect with buildbot with these keys: + +`buildbot_on' + one of 'commit', 'push, or 'change'. Turns the plugin on to + report changes via commit, changes via push, or any changes to + the trunk. 'change' is recommended. + +`buildbot_server' + (required to send to a buildbot master) the URL of the buildbot + master to which you will connect (as of this writing, the same + server and port to which slaves connect). + +`buildbot_port' + (optional, defaults to 9989) the port of the buildbot master to + which you will connect (as of this writing, the same server and + port to which slaves connect) + +`buildbot_pqm' + (optional, defaults to not pqm) Normally, the user that commits + the revision is the user that is responsible for the change. + When run in a pqm (Patch Queue Manager, see + https://launchpad.net/pqm) environment, the user that commits is + the Patch Queue Manager, and the user that committed the + *parent* revision is responsible for the change. To turn on the + pqm mode, set this value to any of (case-insensitive) "Yes", + "Y", "True", or "T". + +`buildbot_dry_run' + (optional, defaults to not a dry run) Normally, the post-commit + hook will attempt to communicate with the configured buildbot + server and port. If this parameter is included and any of + (case-insensitive) "Yes", "Y", "True", or "T", then the hook + will simply print what it would have sent, but not attempt to + contact the buildbot master. + +`buildbot_send_branch_name' + (optional, defaults to not sending the branch name) If your + buildbot's bzr source build step uses a repourl, do *not* turn + this on. If your buildbot's bzr build step uses a baseURL, then + you may set this value to any of (case-insensitive) "Yes", "Y", + "True", or "T" to have the buildbot master append the branch + name to the baseURL. + + + When buildbot no longer has a hardcoded password, it will be a +configuration option here as well. + + Here's a simple example that you might have in your +`~/.bazaar/locations.conf'. + + [chroot-*:///var/local/myrepo/mybranch] + buildbot_on = change + buildbot_server = localhost + + +File: buildbot.info, Node: Bzr Poller, Prev: Bzr Hook, Up: Getting Source Code Changes + +5.11 Bzr Poller +=============== + +If you cannot insert a Bzr hook in the server, you can use the Bzr +Poller. To use, put `contrib/bzr_buildbot.py' somewhere that your +buildbot configuration can import it. Even putting it in the same +directory as the master.cfg should work. Install the poller in the +buildbot configuration as with any other change source. Minimally, +provide a URL that you want to poll (bzr://, bzr+ssh://, or lp:), +though make sure the buildbot user has necessary privileges. You may +also want to specify these optional values. + +`poll_interval' + The number of seconds to wait between polls. Defaults to 10 + minutes. + +`branch_name' + Any value to be used as the branch name. Defaults to None, or + specify a string, or specify the constants from + `bzr_buildbot.py' SHORT or FULL to get the short branch name or + full branch address. + +`blame_merge_author' + normally, the user that commits the revision is the user that is + responsible for the change. When run in a pqm (Patch Queue + Manager, see https://launchpad.net/pqm) environment, the user + that commits is the Patch Queue Manager, and the user that + committed the merged, *parent* revision is responsible for the + change. set this value to True if this is pointed against a + PQM-managed branch. + + +File: buildbot.info, Node: Build Process, Next: Status Delivery, Prev: Getting Source Code Changes, Up: Top + +6 Build Process +*************** + +A `Build' object is responsible for actually performing a build. It +gets access to a remote `SlaveBuilder' where it may run commands, and +a `BuildStatus' object where it must emit status events. The `Build' +is created by the Builder's `BuildFactory'. + + The default `Build' class is made up of a fixed sequence of +`BuildSteps', executed one after another until all are complete (or +one of them indicates that the build should be halted early). The +default `BuildFactory' creates instances of this `Build' class with a +list of `BuildSteps', so the basic way to configure the build is to +provide a list of `BuildSteps' to your `BuildFactory'. + + More complicated `Build' subclasses can make other decisions: +execute some steps only if certain files were changed, or if certain +previous steps passed or failed. The base class has been written to +allow users to express basic control flow without writing code, but +you can always subclass and customize to achieve more specialized +behavior. + +* Menu: + +* Build Steps:: +* Interlocks:: +* Build Factories:: + + +File: buildbot.info, Node: Build Steps, Next: Interlocks, Prev: Build Process, Up: Build Process + +6.1 Build Steps +=============== + +`BuildStep's are usually specified in the buildmaster's configuration +file, in a list that goes into the `BuildFactory'. The `BuildStep' +instances in this list are used as templates to construct new +independent copies for each build (so that state can be kept on the +`BuildStep' in one build without affecting a later build). Each +`BuildFactory' can be created with a list of steps, or the factory +can be created empty and then steps added to it using the `addStep' +method: + + from buildbot.steps import source, shell + from buildbot.process import factory + + f = factory.BuildFactory() + f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/")) + f.addStep(shell.ShellCommand(command=["make", "all"])) + f.addStep(shell.ShellCommand(command=["make", "test"])) + + In earlier versions (0.7.5 and older), these steps were specified +with a tuple of (step_class, keyword_arguments). Steps can still be +specified this way, but the preferred form is to pass actual +`BuildStep' instances to `addStep', because that gives the +`BuildStep' class a chance to do some validation on the arguments. + + If you have a common set of steps which are used in several +factories, the `addSteps' method may be handy. It takes an iterable +of `BuildStep' instances. + + setup_steps = [ + source.SVN(svnurl="http://svn.example.org/Trunk/") + shell.ShellCommand(command="./setup") + ] + quick = factory.BuildFactory() + quick.addSteps(setup_steps) + quick.addStep(shell.shellCommand(command="make quick")) + + The rest of this section lists all the standard BuildStep objects +available for use in a Build, and the parameters which can be used to +control each. + +* Menu: + +* Common Parameters:: +* Using Build Properties:: +* Source Checkout:: +* ShellCommand:: +* Simple ShellCommand Subclasses:: +* Python BuildSteps:: +* Transferring Files:: +* Steps That Run on the Master:: +* Triggering Schedulers:: +* Writing New BuildSteps:: + + +File: buildbot.info, Node: Common Parameters, Next: Using Build Properties, Prev: Build Steps, Up: Build Steps + +6.1.1 Common Parameters +----------------------- + +The standard `Build' runs a series of `BuildStep's in order, only +stopping when it runs out of steps or if one of them requests that +the build be halted. It collects status information from each one to +create an overall build status (of SUCCESS, WARNINGS, or FAILURE). + + All BuildSteps accept some common parameters. Some of these control +how their individual status affects the overall build. Others are used +to specify which `Locks' (see *note Interlocks::) should be acquired +before allowing the step to run. + + Arguments common to all `BuildStep' subclasses: + +`name' + the name used to describe the step on the status display. It is + also used to give a name to any LogFiles created by this step. + +`haltOnFailure' + if True, a FAILURE of this build step will cause the build to + halt immediately. Steps with `alwaysRun=True' are still run. + Generally speaking, haltOnFailure implies flunkOnFailure (the + default for most BuildSteps). In some cases, particularly series + of tests, it makes sense to haltOnFailure if something fails + early on but not flunkOnFailure. This can be achieved with + haltOnFailure=True, flunkOnFailure=False. + +`flunkOnWarnings' + when True, a WARNINGS or FAILURE of this build step will mark the + overall build as FAILURE. The remaining steps will still be + executed. + +`flunkOnFailure' + when True, a FAILURE of this build step will mark the overall + build as a FAILURE. The remaining steps will still be executed. + +`warnOnWarnings' + when True, a WARNINGS or FAILURE of this build step will mark the + overall build as having WARNINGS. The remaining steps will still + be executed. + +`warnOnFailure' + when True, a FAILURE of this build step will mark the overall + build as having WARNINGS. The remaining steps will still be + executed. + +`alwaysRun' + if True, this build step will always be run, even if a previous + buildstep with `haltOnFailure=True' has failed. + +`locks' + a list of Locks (instances of `buildbot.locks.SlaveLock' or + `buildbot.locks.MasterLock') that should be acquired before + starting this Step. The Locks will be released when the step is + complete. Note that this is a list of actual Lock instances, not + names. Also note that all Locks must have unique names. + + + +File: buildbot.info, Node: Using Build Properties, Next: Source Checkout, Prev: Common Parameters, Up: Build Steps + +6.1.2 Using Build Properties +---------------------------- + +Build properties are a generalized way to provide configuration +information to build steps; see *note Build Properties::. + + Some build properties are inherited from external sources - global +properties, schedulers, or buildslaves. Some build properties are +set when the build starts, such as the SourceStamp information. Other +properties can be set by BuildSteps as they run, for example the +various Source steps will set the `got_revision' property to the +source revision that was actually checked out (which can be useful +when the SourceStamp in use merely requested the "latest revision": +`got_revision' will tell you what was actually built). + + In custom BuildSteps, you can get and set the build properties with +the `getProperty'/`setProperty' methods. Each takes a string for the +name of the property, and returns or accepts an arbitrary(1) object. +For example: + + class MakeTarball(ShellCommand): + def start(self): + if self.getProperty("os") == "win": + self.setCommand([ ... ]) # windows-only command + else: + self.setCommand([ ... ]) # equivalent for other systems + ShellCommand.start(self) + +WithProperties +============== + +You can use build properties in ShellCommands by using the +`WithProperties' wrapper when setting the arguments of the +ShellCommand. This interpolates the named build properties into the +generated shell command. Most step parameters accept +`WithProperties'. Please file bugs for any parameters which do not. + + from buildbot.steps.shell import ShellCommand + from buildbot.process.properties import WithProperties + + f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%s.tar.gz", "revision"), + "source"])) + + If this BuildStep were used in a tree obtained from Subversion, it +would create a tarball with a name like `build-1234.tar.gz'. + + The `WithProperties' function does `printf'-style string +interpolation, using strings obtained by calling +`build.getProperty(propname)'. Note that for every `%s' (or `%d', +etc), you must have exactly one additional argument to indicate which +build property you want to insert. + + You can also use python dictionary-style string interpolation by +using the `%(propname)s' syntax. In this form, the property name goes +in the parentheses, and WithProperties takes _no_ additional +arguments: + + f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%(revision)s.tar.gz"), + "source"])) + + Don't forget the extra "s" after the closing parenthesis! This is +the cause of many confusing errors. + + The dictionary-style interpolation supports a number of more +advanced syntaxes, too. + +`propname:-replacement' + If `propname' exists, substitute its value; otherwise, + substitute `replacement'. `replacement' may be empty + (`%(propname:-)s') + +`propname:+replacement' + If `propname' exists, substitute `replacement'; otherwise, + substitute an empty string. + + + Although these are similar to shell substitutions, no other +substitutions are currently supported, and `replacement' in the above +cannot contain more substitutions. + + Note: like python, you can either do positional-argument +interpolation _or_ keyword-argument interpolation, not both. Thus you +cannot use a string like `WithProperties("foo-%(revision)s-%s", +"branch")'. + +Common Build Properties +======================= + +The following build properties are set when the build is started, and +are available to all steps. + +`branch' + This comes from the build's SourceStamp, and describes which + branch is being checked out. This will be `None' (which + interpolates into `WithProperties' as an empty string) if the + build is on the default branch, which is generally the trunk. + Otherwise it will be a string like "branches/beta1.4". The exact + syntax depends upon the VC system being used. + +`revision' + This also comes from the SourceStamp, and is the revision of the + source code tree that was requested from the VC system. When a + build is requested of a specific revision (as is generally the + case when the build is triggered by Changes), this will contain + the revision specification. This is always a string, although + the syntax depends upon the VC system in use: for SVN it is an + integer, for Mercurial it is a short string, for Darcs it is a + rather large string, etc. + + If the "force build" button was pressed, the revision will be + `None', which means to use the most recent revision available. + This is a "trunk build". This will be interpolated as an empty + string. + +`got_revision' + This is set when a Source step checks out the source tree, and + provides the revision that was actually obtained from the VC + system. In general this should be the same as `revision', + except for trunk builds, where `got_revision' indicates what + revision was current when the checkout was performed. This can + be used to rebuild the same source code later. + + Note that for some VC systems (Darcs in particular), the + revision is a large string containing newlines, and is not + suitable for interpolation into a filename. + +`buildername' + This is a string that indicates which Builder the build was a + part of. The combination of buildername and buildnumber + uniquely identify a build. + +`buildnumber' + Each build gets a number, scoped to the Builder (so the first + build performed on any given Builder will have a build number of + 0). This integer property contains the build's number. + +`slavename' + This is a string which identifies which buildslave the build is + running on. + +`scheduler' + If the build was started from a scheduler, then this property + will contain the name of that scheduler. + + + ---------- Footnotes ---------- + + (1) Build properties are serialized along with the build results, +so they must be serializable. For this reason, the value of any build +property should be simple inert data: strings, numbers, lists, +tuples, and dictionaries. They should not contain class instances. + + +File: buildbot.info, Node: Source Checkout, Next: ShellCommand, Prev: Using Build Properties, Up: Build Steps + +6.1.3 Source Checkout +--------------------- + +The first step of any build is typically to acquire the source code +from which the build will be performed. There are several classes to +handle this, one for each of the different source control system that +Buildbot knows about. For a description of how Buildbot treats source +control in general, see *note Version Control Systems::. + + All source checkout steps accept some common parameters to control +how they get the sources and where they should be placed. The +remaining per-VC-system parameters are mostly to specify where +exactly the sources are coming from. + +`mode' + a string describing the kind of VC operation that is desired. + Defaults to `update'. + + `update' + specifies that the CVS checkout/update should be performed + directly into the workdir. Each build is performed in the + same directory, allowing for incremental builds. This + minimizes disk space, bandwidth, and CPU time. However, it + may encounter problems if the build process does not handle + dependencies properly (sometimes you must do a "clean + build" to make sure everything gets compiled), or if source + files are deleted but generated files can influence test + behavior (e.g. python's .pyc files), or when source + directories are deleted but generated files prevent CVS + from removing them. Builds ought to be correct regardless + of whether they are done "from scratch" or incrementally, + but it is useful to test both kinds: this mode exercises the + incremental-build style. + + `copy' + specifies that the CVS workspace should be maintained in a + separate directory (called the 'copydir'), using checkout + or update as necessary. For each build, a new workdir is + created with a copy of the source tree (rm -rf workdir; cp + -r copydir workdir). This doubles the disk space required, + but keeps the bandwidth low (update instead of a full + checkout). A full 'clean' build is performed each time. This + avoids any generated-file build problems, but is still + occasionally vulnerable to CVS problems such as a + repository being manually rearranged, causing CVS errors on + update which are not an issue with a full checkout. + + `clobber' + specifes that the working directory should be deleted each + time, necessitating a full checkout for each build. This + insures a clean build off a complete checkout, avoiding any + of the problems described above. This mode exercises the + "from-scratch" build style. + + `export' + this is like `clobber', except that the 'cvs export' + command is used to create the working directory. This + command removes all CVS metadata files (the CVS/ + directories) from the tree, which is sometimes useful for + creating source tarballs (to avoid including the metadata + in the tar file). + +`workdir' + like all Steps, this indicates the directory where the build + will take place. Source Steps are special in that they perform + some operations outside of the workdir (like creating the + workdir itself). + +`alwaysUseLatest' + if True, bypass the usual "update to the last Change" behavior, + and always update to the latest changes instead. + +`retry' + If set, this specifies a tuple of `(delay, repeats)' which means + that when a full VC checkout fails, it should be retried up to + REPEATS times, waiting DELAY seconds between attempts. If you + don't provide this, it defaults to `None', which means VC + operations should not be retried. This is provided to make life + easier for buildslaves which are stuck behind poor network + connections. + + + My habit as a developer is to do a `cvs update' and `make' each +morning. Problems can occur, either because of bad code being checked +in, or by incomplete dependencies causing a partial rebuild to fail +where a complete from-scratch build might succeed. A quick Builder +which emulates this incremental-build behavior would use the +`mode='update'' setting. + + On the other hand, other kinds of dependency problems can cause a +clean build to fail where a partial build might succeed. This +frequently results from a link step that depends upon an object file +that was removed from a later version of the tree: in the partial +tree, the object file is still around (even though the Makefiles no +longer know how to create it). + + "official" builds (traceable builds performed from a known set of +source revisions) are always done as clean builds, to make sure it is +not influenced by any uncontrolled factors (like leftover files from a +previous build). A "full" Builder which behaves this way would want +to use the `mode='clobber'' setting. + + Each VC system has a corresponding source checkout class: their +arguments are described on the following pages. + +* Menu: + +* CVS:: +* SVN:: +* Darcs:: +* Mercurial:: +* Arch:: +* Bazaar:: +* Bzr:: +* P4:: +* Git:: + + +File: buildbot.info, Node: CVS, Next: SVN, Prev: Source Checkout, Up: Source Checkout + +6.1.3.1 CVS +........... + +The `CVS' build step performs a CVS (http://www.nongnu.org/cvs/) +checkout or update. It takes the following arguments: + +`cvsroot' + (required): specify the CVSROOT value, which points to a CVS + repository, probably on a remote machine. For example, the + cvsroot value you would use to get a copy of the Buildbot source + code is + `:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot' + +`cvsmodule' + (required): specify the cvs `module', which is generally a + subdirectory of the CVSROOT. The cvsmodule for the Buildbot + source code is `buildbot'. + +`branch' + a string which will be used in a `-r' argument. This is most + useful for specifying a branch to work on. Defaults to `HEAD'. + +`global_options' + a list of flags to be put before the verb in the CVS command. + +`checkoutDelay' + if set, the number of seconds to put between the timestamp of + the last known Change and the value used for the `-D' option. + Defaults to half of the parent Build's treeStableTimer. + + + +File: buildbot.info, Node: SVN, Next: Darcs, Prev: CVS, Up: Source Checkout + +6.1.3.2 SVN +........... + +The `SVN' build step performs a Subversion +(http://subversion.tigris.org) checkout or update. There are two +basic ways of setting up the checkout step, depending upon whether +you are using multiple branches or not. + + If all of your builds use the same branch, then you should create +the `SVN' step with the `svnurl' argument: + +`svnurl' + (required): this specifies the `URL' argument that will be given + to the `svn checkout' command. It dictates both where the + repository is located and which sub-tree should be extracted. In + this respect, it is like a combination of the CVS `cvsroot' and + `cvsmodule' arguments. For example, if you are using a remote + Subversion repository which is accessible through HTTP at a URL + of `http://svn.example.com/repos', and you wanted to check out + the `trunk/calc' sub-tree, you would use + `svnurl="http://svn.example.com/repos/trunk/calc"' as an argument + to your `SVN' step. + + If, on the other hand, you are building from multiple branches, +then you should create the `SVN' step with the `baseURL' and +`defaultBranch' arguments instead: + +`baseURL' + (required): this specifies the base repository URL, to which a + branch name will be appended. It should probably end in a slash. + +`defaultBranch' + this specifies the name of the branch to use when a Build does + not provide one of its own. This will be appended to `baseURL' to + create the string that will be passed to the `svn checkout' + command. + +`username' + if specified, this will be passed to the `svn' binary with a + `--username' option. + +`password' + if specified, this will be passed to the `svn' binary with a + `--password' option. The password itself will be suitably + obfuscated in the logs. + + + If you are using branches, you must also make sure your +`ChangeSource' will report the correct branch names. + +branch example +============== + +Let's suppose that the "MyProject" repository uses branches for the +trunk, for various users' individual development efforts, and for +several new features that will require some amount of work (involving +multiple developers) before they are ready to merge onto the trunk. +Such a repository might be organized as follows: + + svn://svn.example.org/MyProject/trunk + svn://svn.example.org/MyProject/branches/User1/foo + svn://svn.example.org/MyProject/branches/User1/bar + svn://svn.example.org/MyProject/branches/User2/baz + svn://svn.example.org/MyProject/features/newthing + svn://svn.example.org/MyProject/features/otherthing + + Further assume that we want the Buildbot to run tests against the +trunk and against all the feature branches (i.e., do a +checkout/compile/build of branch X when a file has been changed on +branch X, when X is in the set [trunk, features/newthing, +features/otherthing]). We do not want the Buildbot to automatically +build any of the user branches, but it should be willing to build a +user branch when explicitly requested (most likely by the user who +owns that branch). + + There are three things that need to be set up to accomodate this +system. The first is a ChangeSource that is capable of identifying the +branch which owns any given file. This depends upon a user-supplied +function, in an external program that runs in the SVN commit hook and +connects to the buildmaster's `PBChangeSource' over a TCP connection. +(you can use the "`buildbot sendchange'" utility for this purpose, +but you will still need an external program to decide what value +should be passed to the `--branch=' argument). For example, a change +to a file with the SVN url of +"svn://svn.example.org/MyProject/features/newthing/src/foo.c" should +be broken down into a Change instance with +`branch='features/newthing'' and `file='src/foo.c''. + + The second piece is an `AnyBranchScheduler' which will pay +attention to the desired branches. It will not pay attention to the +user branches, so it will not automatically start builds in response +to changes there. The AnyBranchScheduler class requires you to +explicitly list all the branches you want it to use, but it would not +be difficult to write a subclass which used +`branch.startswith('features/'' to remove the need for this explicit +list. Or, if you want to build user branches too, you can use +AnyBranchScheduler with `branches=None' to indicate that you want it +to pay attention to all branches. + + The third piece is an `SVN' checkout step that is configured to +handle the branches correctly, with a `baseURL' value that matches +the way the ChangeSource splits each file's URL into base, branch, +and file. + + from buildbot.changes.pb import PBChangeSource + from buildbot.scheduler import AnyBranchScheduler + from buildbot.process import source, factory + from buildbot.steps import source, shell + + c['change_source'] = PBChangeSource() + s1 = AnyBranchScheduler('main', + ['trunk', 'features/newthing', 'features/otherthing'], + 10*60, ['test-i386', 'test-ppc']) + c['schedulers'] = [s1] + + f = factory.BuildFactory() + f.addStep(source.SVN(mode='update', + baseURL='svn://svn.example.org/MyProject/', + defaultBranch='trunk')) + f.addStep(shell.Compile(command="make all")) + f.addStep(shell.Test(command="make test")) + + c['builders'] = [ + {'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386', + 'factory':f }, + {'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc', + 'factory':f }, + ] + + In this example, when a change arrives with a `branch' attribute +of "trunk", the resulting build will have an SVN step that +concatenates "svn://svn.example.org/MyProject/" (the baseURL) with +"trunk" (the branch name) to get the correct svn command. If the +"newthing" branch has a change to "src/foo.c", then the SVN step will +concatenate "svn://svn.example.org/MyProject/" with +"features/newthing" to get the svnurl for checkout. + + +File: buildbot.info, Node: Darcs, Next: Mercurial, Prev: SVN, Up: Source Checkout + +6.1.3.3 Darcs +............. + +The `Darcs' build step performs a Darcs (http://darcs.net/) checkout +or update. + + Like *Note SVN::, this step can either be configured to always +check out a specific tree, or set up to pull from a particular branch +that gets specified separately for each build. Also like SVN, the +repository URL given to Darcs is created by concatenating a `baseURL' +with the branch name, and if no particular branch is requested, it +uses a `defaultBranch'. The only difference in usage is that each +potential Darcs repository URL must point to a fully-fledged +repository, whereas SVN URLs usually point to sub-trees of the main +Subversion repository. In other words, doing an SVN checkout of +`baseURL' is legal, but silly, since you'd probably wind up with a +copy of every single branch in the whole repository. Doing a Darcs +checkout of `baseURL' is just plain wrong, since the parent directory +of a collection of Darcs repositories is not itself a valid +repository. + + The Darcs step takes the following arguments: + +`repourl' + (required unless `baseURL' is provided): the URL at which the + Darcs source repository is available. + +`baseURL' + (required unless `repourl' is provided): the base repository URL, + to which a branch name will be appended. It should probably end + in a slash. + +`defaultBranch' + (allowed if and only if `baseURL' is provided): this specifies + the name of the branch to use when a Build does not provide one + of its own. This will be appended to `baseURL' to create the + string that will be passed to the `darcs get' command. + + +File: buildbot.info, Node: Mercurial, Next: Arch, Prev: Darcs, Up: Source Checkout + +6.1.3.4 Mercurial +................. + +The `Mercurial' build step performs a Mercurial +(http://selenic.com/mercurial) (aka "hg") checkout or update. + + Branches are handled just like *Note Darcs::. + + The Mercurial step takes the following arguments: + +`repourl' + (required unless `baseURL' is provided): the URL at which the + Mercurial source repository is available. + +`baseURL' + (required unless `repourl' is provided): the base repository URL, + to which a branch name will be appended. It should probably end + in a slash. + +`defaultBranch' + (allowed if and only if `baseURL' is provided): this specifies + the name of the branch to use when a Build does not provide one + of its own. This will be appended to `baseURL' to create the + string that will be passed to the `hg clone' command. + + +File: buildbot.info, Node: Arch, Next: Bazaar, Prev: Mercurial, Up: Source Checkout + +6.1.3.5 Arch +............ + +The `Arch' build step performs an Arch (http://gnuarch.org/) checkout +or update using the `tla' client. It takes the following arguments: + +`url' + (required): this specifies the URL at which the Arch source + archive is available. + +`version' + (required): this specifies which "development line" (like a + branch) should be used. This provides the default branch name, + but individual builds may specify a different one. + +`archive' + (optional): Each repository knows its own archive name. If this + parameter is provided, it must match the repository's archive + name. The parameter is accepted for compatibility with the + `Bazaar' step, below. + + + +File: buildbot.info, Node: Bazaar, Next: Bzr, Prev: Arch, Up: Source Checkout + +6.1.3.6 Bazaar +.............. + +`Bazaar' is an alternate implementation of the Arch VC system, which +uses a client named `baz'. The checkout semantics are just different +enough from `tla' that there is a separate BuildStep for it. + + It takes exactly the same arguments as `Arch', except that the +`archive=' parameter is required. (baz does not emit the archive name +when you do `baz register-archive', so we must provide it ourselves). + + +File: buildbot.info, Node: Bzr, Next: P4, Prev: Bazaar, Up: Source Checkout + +6.1.3.7 Bzr +........... + +`bzr' is a descendant of Arch/Baz, and is frequently referred to as +simply "Bazaar". The repository-vs-workspace model is similar to +Darcs, but it uses a strictly linear sequence of revisions (one +history per branch) like Arch. Branches are put in subdirectories. +This makes it look very much like Mercurial, so it takes the same +arguments: + +`repourl' + (required unless `baseURL' is provided): the URL at which the + Bzr source repository is available. + +`baseURL' + (required unless `repourl' is provided): the base repository URL, + to which a branch name will be appended. It should probably end + in a slash. + +`defaultBranch' + (allowed if and only if `baseURL' is provided): this specifies + the name of the branch to use when a Build does not provide one + of its own. This will be appended to `baseURL' to create the + string that will be passed to the `bzr checkout' command. + + +File: buildbot.info, Node: P4, Next: Git, Prev: Bzr, Up: Source Checkout + +6.1.3.8 P4 +.......... + +The `P4' build step creates a Perforce (http://www.perforce.com/) +client specification and performs an update. + +`p4base' + A view into the Perforce depot without branch name or trailing + "...". Typically "//depot/proj/". + +`defaultBranch' + A branch name to append on build requests if none is specified. + Typically "trunk". + +`p4port' + (optional): the host:port string describing how to get to the P4 + Depot (repository), used as the -p argument for all p4 commands. + +`p4user' + (optional): the Perforce user, used as the -u argument to all p4 + commands. + +`p4passwd' + (optional): the Perforce password, used as the -p argument to + all p4 commands. + +`p4extra_views' + (optional): a list of (depotpath, clientpath) tuples containing + extra views to be mapped into the client specification. Both + will have "/..." appended automatically. The client name and + source directory will be prepended to the client path. + +`p4client' + (optional): The name of the client to use. In mode='copy' and + mode='update', it's particularly important that a unique name is + used for each checkout directory to avoid incorrect + synchronization. For this reason, Python percent substitution + will be performed on this value to replace %(slave)s with the + slave name and %(builder)s with the builder name. The default is + "buildbot_%(slave)s_%(build)s". + + +File: buildbot.info, Node: Git, Prev: P4, Up: Source Checkout + +6.1.3.9 Git +........... + +The `Git' build step clones or updates a Git (http://git.or.cz/) +repository and checks out the specified branch or revision. Note that +the buildbot supports Git version 1.2.0 and later: earlier versions +(such as the one shipped in Ubuntu 'Dapper') do not support the `git +init' command that the buildbot uses. + + The Git step takes the following arguments: + +`repourl' + (required): the URL of the upstream Git repository. + +`branch' + (optional): this specifies the name of the branch to use when a + Build does not provide one of its own. If this this parameter is + not specified, and the Build does not provide a branch, the + "master" branch will be used. + + +File: buildbot.info, Node: ShellCommand, Next: Simple ShellCommand Subclasses, Prev: Source Checkout, Up: Build Steps + +6.1.4 ShellCommand +------------------ + +This is a useful base class for just about everything you might want +to do during a build (except for the initial source checkout). It runs +a single command in a child shell on the buildslave. All stdout/stderr +is recorded into a LogFile. The step finishes with a status of FAILURE +if the command's exit code is non-zero, otherwise it has a status of +SUCCESS. + + The preferred way to specify the command is with a list of argv +strings, since this allows for spaces in filenames and avoids doing +any fragile shell-escaping. You can also specify the command with a +single string, in which case the string is given to '/bin/sh -c +COMMAND' for parsing. + + On Windows, commands are run via `cmd.exe /c' which works well. +However, if you're running a batch file, the error level does not get +propagated correctly unless you add 'call' before your batch file's +name: `cmd=['call', 'myfile.bat', ...]'. + + All ShellCommands are run by default in the "workdir", which +defaults to the "`build'" subdirectory of the slave builder's base +directory. The absolute path of the workdir will thus be the slave's +basedir (set as an option to `buildbot create-slave', *note Creating +a buildslave::) plus the builder's basedir (set in the builder's +`c['builddir']' key in master.cfg) plus the workdir itself (a +class-level attribute of the BuildFactory, defaults to "`build'"). + + `ShellCommand' arguments: + +`command' + a list of strings (preferred) or single string (discouraged) + which specifies the command to be run. A list of strings is + preferred because it can be used directly as an argv array. + Using a single string (with embedded spaces) requires the + buildslave to pass the string to /bin/sh for interpretation, + which raises all sorts of difficult questions about how to + escape or interpret shell metacharacters. + +`env' + a dictionary of environment strings which will be added to the + child command's environment. For example, to run tests with a + different i18n language setting, you might use + + f.addStep(ShellCommand(command=["make", "test"], + env={'LANG': 'fr_FR'})) + + These variable settings will override any existing ones in the + buildslave's environment or the environment specified in the + Builder. The exception is PYTHONPATH, which is merged with + (actually prepended to) any existing $PYTHONPATH setting. The + value is treated as a list of directories to prepend, and a + single string is treated like a one-item list. For example, to + prepend both `/usr/local/lib/python2.3' and + `/home/buildbot/lib/python' to any existing $PYTHONPATH setting, + you would do something like the following: + + f.addStep(ShellCommand( + command=["make", "test"], + env={'PYTHONPATH': ["/usr/local/lib/python2.3", + "/home/buildbot/lib/python"] })) + +`want_stdout' + if False, stdout from the child process is discarded rather than + being sent to the buildmaster for inclusion in the step's + LogFile. + +`want_stderr' + like `want_stdout' but for stderr. Note that commands run through + a PTY do not have separate stdout/stderr streams: both are + merged into stdout. + +`usePTY' + Should this command be run in a `pty'? The default is to + observe the configuration of the client (*note Buildslave + Options::), but specifying `True' or `False' here will override + the default. + + The advantage of using a PTY is that "grandchild" processes are + more likely to be cleaned up if the build is interrupted or + times out (since it enables the use of a "process group" in + which all child processes will be placed). The disadvantages: + some forms of Unix have problems with PTYs, some of your unit + tests may behave differently when run under a PTY (generally + those which check to see if they are being run interactively), + and PTYs will merge the stdout and stderr streams into a single + output stream (which means the red-vs-black coloring in the + logfiles will be lost). + +`logfiles' + Sometimes commands will log interesting data to a local file, + rather than emitting everything to stdout or stderr. For + example, Twisted's "trial" command (which runs unit tests) only + presents summary information to stdout, and puts the rest into a + file named `_trial_temp/test.log'. It is often useful to watch + these files as the command runs, rather than using `/bin/cat' to + dump their contents afterwards. + + The `logfiles=' argument allows you to collect data from these + secondary logfiles in near-real-time, as the step is running. It + accepts a dictionary which maps from a local Log name (which is + how the log data is presented in the build results) to a remote + filename (interpreted relative to the build's working + directory). Each named file will be polled on a regular basis + (every couple of seconds) as the build runs, and any new text + will be sent over to the buildmaster. + + f.addStep(ShellCommand( + command=["make", "test"], + logfiles={"triallog": "_trial_temp/test.log"})) + +`timeout' + if the command fails to produce any output for this many + seconds, it is assumed to be locked up and will be killed. + +`description' + This will be used to describe the command (on the Waterfall + display) while the command is still running. It should be a + single imperfect-tense verb, like "compiling" or "testing". The + preferred form is a list of short strings, which allows the HTML + Waterfall display to create narrower columns by emitting a <br> + tag between each word. You may also provide a single string. + +`descriptionDone' + This will be used to describe the command once it has finished. A + simple noun like "compile" or "tests" should be used. Like + `description', this may either be a list of short strings or a + single string. + + If neither `description' nor `descriptionDone' are set, the + actual command arguments will be used to construct the + description. This may be a bit too wide to fit comfortably on + the Waterfall display. + + f.addStep(ShellCommand(command=["make", "test"], + description=["testing"], + descriptionDone=["tests"])) + +`logEnviron' + If this option is true (the default), then the step's logfile + will describe the environment variables on the slave. In + situations where the environment is not relevant and is long, it + may be easier to set `logEnviron=False'. + + + +File: buildbot.info, Node: Simple ShellCommand Subclasses, Next: Python BuildSteps, Prev: ShellCommand, Up: Build Steps + +6.1.5 Simple ShellCommand Subclasses +------------------------------------ + +Several subclasses of ShellCommand are provided as starting points for +common build steps. These are all very simple: they just override a +few parameters so you don't have to specify them yourself, making the +master.cfg file less verbose. + +* Menu: + +* Configure:: +* Compile:: +* Test:: +* TreeSize:: +* PerlModuleTest:: +* SetProperty:: + + +File: buildbot.info, Node: Configure, Next: Compile, Prev: Simple ShellCommand Subclasses, Up: Simple ShellCommand Subclasses + +6.1.5.1 Configure +................. + +This is intended to handle the `./configure' step from autoconf-style +projects, or the `perl Makefile.PL' step from perl MakeMaker.pm-style +modules. The default command is `./configure' but you can change this +by providing a `command=' parameter. + + +File: buildbot.info, Node: Compile, Next: Test, Prev: Configure, Up: Simple ShellCommand Subclasses + +6.1.5.2 Compile +............... + +This is meant to handle compiling or building a project written in C. +The default command is `make all'. When the compile is finished, the +log file is scanned for GCC warning messages, a summary log is +created with any problems that were seen, and the step is marked as +WARNINGS if any were discovered. The number of warnings is stored in a +Build Property named "warnings-count", which is accumulated over all +Compile steps (so if two warnings are found in one step, and three are +found in another step, the overall build will have a "warnings-count" +property of 5. + + The default regular expression used to detect a warning is +`'.*warning[: ].*'' , which is fairly liberal and may cause +false-positives. To use a different regexp, provide a +`warningPattern=' argument, or use a subclass which sets the +`warningPattern' attribute: + + f.addStep(Compile(command=["make", "test"], + warningPattern="^Warning: ")) + + The `warningPattern=' can also be a pre-compiled python regexp +object: this makes it possible to add flags like `re.I' (to use +case-insensitive matching). + + (TODO: this step needs to be extended to look for GCC error +messages as well, and collect them into a separate logfile, along +with the source code filenames involved). + + +File: buildbot.info, Node: Test, Next: TreeSize, Prev: Compile, Up: Simple ShellCommand Subclasses + +6.1.5.3 Test +............ + +This is meant to handle unit tests. The default command is `make +test', and the `warnOnFailure' flag is set. + + +File: buildbot.info, Node: TreeSize, Next: PerlModuleTest, Prev: Test, Up: Simple ShellCommand Subclasses + +6.1.5.4 TreeSize +................ + +This is a simple command that uses the 'du' tool to measure the size +of the code tree. It puts the size (as a count of 1024-byte blocks, +aka 'KiB' or 'kibibytes') on the step's status text, and sets a build +property named 'tree-size-KiB' with the same value. + + +File: buildbot.info, Node: PerlModuleTest, Next: SetProperty, Prev: TreeSize, Up: Simple ShellCommand Subclasses + +6.1.5.5 PerlModuleTest +...................... + +This is a simple command that knows how to run tests of perl modules. +It parses the output to determine the number of tests passed and +failed and total number executed, saving the results for later query. + + +File: buildbot.info, Node: SetProperty, Prev: PerlModuleTest, Up: Simple ShellCommand Subclasses + +6.1.5.6 SetProperty +................... + +This buildstep is similar to ShellCommand, except that it captures the +output of the command into a property. It is usually used like this: + + f.addStep(SetProperty(command="uname -a", property="uname")) + + This runs `uname -a' and captures its stdout, stripped of leading +and trailing whitespace, in the property "uname". To avoid stripping, +add `strip=False'. The `property' argument can be specified as a +`WithProperties' object. + + The more advanced usage allows you to specify a function to extract +properties from the command output. Here you can use regular +expressions, string interpolation, or whatever you would like. The +function is called with three arguments: the exit status of the +command, its standard output as a string, and its standard error as a +string. It should return a dictionary containing all new properties. + + def glob2list(rc, stdout, stderr): + jpgs = [ l.strip() for l in stdout.split('\n') ] + return { 'jpgs' : jpgs } + f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list)) + + Note that any ordering relationship of the contents of stdout and +stderr is lost. For example, given + + f.addStep(SetProperty( + command="echo output1; echo error >&2; echo output2", + extract_fn=my_extract)) + + Then `my_extract' will see `stdout="output1\noutput2\n"' and +`stderr="error\n"'. + + +File: buildbot.info, Node: Python BuildSteps, Next: Transferring Files, Prev: Simple ShellCommand Subclasses, Up: Build Steps + +6.1.6 Python BuildSteps +----------------------- + +Here are some BuildSteps that are specifcally useful for projects +implemented in Python. + +* Menu: + +* BuildEPYDoc:: +* PyFlakes:: +* PyLint:: + + +File: buildbot.info, Node: BuildEPYDoc, Next: PyFlakes, Up: Python BuildSteps + +6.1.6.1 BuildEPYDoc +................... + +epydoc (http://epydoc.sourceforge.net/) is a tool for generating API +documentation for Python modules from their docstrings. It reads all +the .py files from your source tree, processes the docstrings +therein, and creates a large tree of .html files (or a single .pdf +file). + + The `buildbot.steps.python.BuildEPYDoc' step will run `epydoc' to +produce this API documentation, and will count the errors and +warnings from its output. + + You must supply the command line to be used. The default is `make +epydocs', which assumes that your project has a Makefile with an +"epydocs" target. You might wish to use something like `epydoc -o +apiref source/PKGNAME' instead. You might also want to add `--pdf' to +generate a PDF file instead of a large tree of HTML files. + + The API docs are generated in-place in the build tree (under the +workdir, in the subdirectory controlled by the "-o" argument). To +make them useful, you will probably have to copy them to somewhere +they can be read. A command like `rsync -ad apiref/ +dev.example.com:~public_html/current-apiref/' might be useful. You +might instead want to bundle them into a tarball and publish it in the +same place where the generated install tarball is placed. + + from buildbot.steps.python import BuildEPYDoc + + ... + f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) + + +File: buildbot.info, Node: PyFlakes, Next: PyLint, Prev: BuildEPYDoc, Up: Python BuildSteps + +6.1.6.2 PyFlakes +................ + +PyFlakes (http://divmod.org/trac/wiki/DivmodPyflakes) is a tool to +perform basic static analysis of Python code to look for simple +errors, like missing imports and references of undefined names. It is +like a fast and simple form of the C "lint" program. Other tools +(like pychecker) provide more detailed results but take longer to run. + + The `buildbot.steps.python.PyFlakes' step will run pyflakes and +count the various kinds of errors and warnings it detects. + + You must supply the command line to be used. The default is `make +pyflakes', which assumes you have a top-level Makefile with a +"pyflakes" target. You might want to use something like `pyflakes .' +or `pyflakes src'. + + from buildbot.steps.python import PyFlakes + + ... + f.addStep(PyFlakes(command=["pyflakes", "src"])) + + +File: buildbot.info, Node: PyLint, Prev: PyFlakes, Up: Python BuildSteps + +6.1.6.3 PyLint +.............. + +Similarly, the `buildbot.steps.python.PyLint' step will run pylint and +analyze the results. + + You must supply the command line to be used. There is no default. + + from buildbot.steps.python import PyLint + + ... + f.addStep(PyLint(command=["pylint", "src"])) + + +File: buildbot.info, Node: Transferring Files, Next: Steps That Run on the Master, Prev: Python BuildSteps, Up: Build Steps + +6.1.7 Transferring Files +------------------------ + +Most of the work involved in a build will take place on the +buildslave. But occasionally it is useful to do some work on the +buildmaster side. The most basic way to involve the buildmaster is +simply to move a file from the slave to the master, or vice versa. +There are a pair of BuildSteps named `FileUpload' and `FileDownload' +to provide this functionality. `FileUpload' moves a file _up to_ the +master, while `FileDownload' moves a file _down from_ the master. + + As an example, let's assume that there is a step which produces an +HTML file within the source tree that contains some sort of generated +project documentation. We want to move this file to the buildmaster, +into a `~/public_html' directory, so it can be visible to developers. +This file will wind up in the slave-side working directory under the +name `docs/reference.html'. We want to put it into the master-side +`~/public_html/ref.html'. + + from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import FileUpload + + f.addStep(ShellCommand(command=["make", "docs"])) + f.addStep(FileUpload(slavesrc="docs/reference.html", + masterdest="~/public_html/ref.html")) + + The `masterdest=' argument will be passed to os.path.expanduser, +so things like "~" will be expanded properly. Non-absolute paths will +be interpreted relative to the buildmaster's base directory. +Likewise, the `slavesrc=' argument will be expanded and interpreted +relative to the builder's working directory. + + To move a file from the master to the slave, use the +`FileDownload' command. For example, let's assume that some step +requires a configuration file that, for whatever reason, could not be +recorded in the source code repository or generated on the buildslave +side: + + from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import FileUpload + + f.addStep(FileDownload(mastersrc="~/todays_build_config.txt", + slavedest="build_config.txt")) + f.addStep(ShellCommand(command=["make", "config"])) + + Like `FileUpload', the `mastersrc=' argument is interpreted +relative to the buildmaster's base directory, and the `slavedest=' +argument is relative to the builder's working directory. If the +buildslave is running in `~buildslave', and the builder's "builddir" +is something like `tests-i386', then the workdir is going to be +`~buildslave/tests-i386/build', and a `slavedest=' of `foo/bar.html' +will get put in `~buildslave/tests-i386/build/foo/bar.html'. Both of +these commands will create any missing intervening directories. + +Other Parameters +---------------- + +The `maxsize=' argument lets you set a maximum size for the file to +be transferred. This may help to avoid surprises: transferring a +100MB coredump when you were expecting to move a 10kB status file +might take an awfully long time. The `blocksize=' argument controls +how the file is sent over the network: larger blocksizes are slightly +more efficient but also consume more memory on each end, and there is +a hard-coded limit of about 640kB. + + The `mode=' argument allows you to control the access permissions +of the target file, traditionally expressed as an octal integer. The +most common value is probably 0755, which sets the "x" executable bit +on the file (useful for shell scripts and the like). The default +value for `mode=' is None, which means the permission bits will +default to whatever the umask of the writing process is. The default +umask tends to be fairly restrictive, but at least on the buildslave +you can make it less restrictive with a -umask command-line option at +creation time (*note Buildslave Options::). + +Transfering Directories +----------------------- + +To transfer complete directories from the buildslave to the master, +there is a BuildStep named `DirectoryUpload'. It works like +`FileUpload', just for directories. However it does not support the +`maxsize', `blocksize' and `mode' arguments. As an example, let's +assume an generated project documentation, which consists of many +files (like the output of doxygen or epydoc). We want to move the +entire documentation to the buildmaster, into a `~/public_html/docs' +directory. On the slave-side the directory can be found under `docs': + + from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import DirectoryUpload + + f.addStep(ShellCommand(command=["make", "docs"])) + f.addStep(DirectoryUpload(slavesrc="docs", + masterdest="~/public_html/docs")) + + The DirectoryUpload step will create all necessary directories and +transfers empty directories, too. + + +File: buildbot.info, Node: Steps That Run on the Master, Next: Triggering Schedulers, Prev: Transferring Files, Up: Build Steps + +6.1.8 Steps That Run on the Master +---------------------------------- + +Occasionally, it is useful to execute some task on the master, for +example to create a directory, deploy a build result, or trigger some +other centralized processing. This is possible, in a limited +fashion, with the `MasterShellCommand' step. + + This step operates similarly to a regular `ShellCommand', but +executes on the master, instead of the slave. To be clear, the +enclosing `Build' object must still have a slave object, just as for +any other step - only, in this step, the slave does not do anything. + + In this example, the step renames a tarball based on the day of +the week. + + from buildbot.steps.transfer import FileUpload + from buildbot.steps.master import MasterShellCommand + + f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz", + masterdest="/var/buildoutputs/widgetsoft-new.tar.gz")) + f.addStep(MasterShellCommand(command=""" + cd /var/buildoutputs; + mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz""")) + + +File: buildbot.info, Node: Triggering Schedulers, Next: Writing New BuildSteps, Prev: Steps That Run on the Master, Up: Build Steps + +6.1.9 Triggering Schedulers +--------------------------- + +The counterpart to the Triggerable described in section *note +Triggerable Scheduler:: is the Trigger BuildStep. + + from buildbot.steps.trigger import Trigger + f.addStep(Trigger(schedulerNames=['build-prep'], + waitForFinish=True, + updateSourceStamp=True)) + + The `schedulerNames=' argument lists the Triggerables that should +be triggered when this step is executed. Note that it is possible, +but not advisable, to create a cycle where a build continually +triggers itself, because the schedulers are specified by name. + + If `waitForFinish' is True, then the step will not finish until +all of the builds from the triggered schedulers have finished. If this +argument is False (the default) or not given, then the buildstep +succeeds immediately after triggering the schedulers. + + If `updateSourceStamp' is True (the default), then step updates +the SourceStamp given to the Triggerables to include `got_revision' +(the revision actually used in this build) as `revision' (the +revision to use in the triggered builds). This is useful to ensure +that all of the builds use exactly the same SourceStamp, even if +other Changes have occurred while the build was running. + + +File: buildbot.info, Node: Writing New BuildSteps, Prev: Triggering Schedulers, Up: Build Steps + +6.1.10 Writing New BuildSteps +----------------------------- + +While it is a good idea to keep your build process self-contained in +the source code tree, sometimes it is convenient to put more +intelligence into your Buildbot configuration. One way to do this is +to write a custom BuildStep. Once written, this Step can be used in +the `master.cfg' file. + + The best reason for writing a custom BuildStep is to better parse +the results of the command being run. For example, a BuildStep that +knows about JUnit could look at the logfiles to determine which tests +had been run, how many passed and how many failed, and then report +more detailed information than a simple `rc==0' -based "good/bad" +decision. + +* Menu: + +* Writing BuildStep Constructors:: +* BuildStep LogFiles:: +* Reading Logfiles:: +* Adding LogObservers:: +* BuildStep URLs:: + + +File: buildbot.info, Node: Writing BuildStep Constructors, Next: BuildStep LogFiles, Up: Writing New BuildSteps + +6.1.10.1 Writing BuildStep Constructors +....................................... + +BuildStep classes have some extra equipment, because they are their +own factories. Consider the use of a BuildStep in `master.cfg': + + f.addStep(MyStep(someopt="stuff", anotheropt=1)) + + This creates a single instance of class `MyStep'. However, +Buildbot needs a new object each time the step is executed. this is +accomplished by storing the information required to instantiate a new +object in the `factory' attribute. When the time comes to construct +a new Build, BuildFactory consults this attribute (via +`getStepFactory') and instantiates a new step object. + + When writing a new step class, then, keep in mind are that you +cannot do anything "interesting" in the constructor - limit yourself +to checking and storing arguments. To ensure that these arguments +are provided to any new objects, call `self.addFactoryArguments' with +any keyword arguments your constructor needs. + + Keep a `**kwargs' argument on the end of your options, and pass +that up to the parent class's constructor. + + The whole thing looks like this: + + class Frobinfy(LoggingBuildStep): + def __init__(self, + frob_what="frobee", + frob_how_many=None, + frob_how=None, + **kwargs) + + # check + if frob_how_many is None: + raise TypeError("Frobinfy argument how_many is required") + + # call parent + LoggingBuildStep.__init__(self, **kwargs) + + # and record arguments for later + self.addFactoryArguments( + frob_what=frob_what, + frob_how_many=frob_how_many, + frob_how=frob_how) + + class FastFrobnify(Frobnify): + def __init__(self, + speed=5, + **kwargs) + Frobnify.__init__(self, **kwargs) + self.addFactoryArguments( + speed=speed) + + +File: buildbot.info, Node: BuildStep LogFiles, Next: Reading Logfiles, Prev: Writing BuildStep Constructors, Up: Writing New BuildSteps + +6.1.10.2 BuildStep LogFiles +........................... + +Each BuildStep has a collection of "logfiles". Each one has a short +name, like "stdio" or "warnings". Each LogFile contains an arbitrary +amount of text, usually the contents of some output file generated +during a build or test step, or a record of everything that was +printed to stdout/stderr during the execution of some command. + + These LogFiles are stored to disk, so they can be retrieved later. + + Each can contain multiple "channels", generally limited to three +basic ones: stdout, stderr, and "headers". For example, when a +ShellCommand runs, it writes a few lines to the "headers" channel to +indicate the exact argv strings being run, which directory the command +is being executed in, and the contents of the current environment +variables. Then, as the command runs, it adds a lot of "stdout" and +"stderr" messages. When the command finishes, a final "header" line +is added with the exit code of the process. + + Status display plugins can format these different channels in +different ways. For example, the web page shows LogFiles as text/html, +with header lines in blue text, stdout in black, and stderr in red. A +different URL is available which provides a text/plain format, in +which stdout and stderr are collapsed together, and header lines are +stripped completely. This latter option makes it easy to save the +results to a file and run `grep' or whatever against the output. + + Each BuildStep contains a mapping (implemented in a python +dictionary) from LogFile name to the actual LogFile objects. Status +plugins can get a list of LogFiles to display, for example, a list of +HREF links that, when clicked, provide the full contents of the +LogFile. + +Using LogFiles in custom BuildSteps +=================================== + +The most common way for a custom BuildStep to use a LogFile is to +summarize the results of a ShellCommand (after the command has +finished running). For example, a compile step with thousands of lines +of output might want to create a summary of just the warning messages. +If you were doing this from a shell, you would use something like: + + grep "warning:" output.log >warnings.log + + In a custom BuildStep, you could instead create a "warnings" +LogFile that contained the same text. To do this, you would add code +to your `createSummary' method that pulls lines from the main output +log and creates a new LogFile with the results: + + def createSummary(self, log): + warnings = [] + for line in log.readlines(): + if "warning:" in line: + warnings.append() + self.addCompleteLog('warnings', "".join(warnings)) + + This example uses the `addCompleteLog' method, which creates a new +LogFile, puts some text in it, and then "closes" it, meaning that no +further contents will be added. This LogFile will appear in the HTML +display under an HREF with the name "warnings", since that is the +name of the LogFile. + + You can also use `addHTMLLog' to create a complete (closed) +LogFile that contains HTML instead of plain text. The normal LogFile +will be HTML-escaped if presented through a web page, but the HTML +LogFile will not. At the moment this is only used to present a pretty +HTML representation of an otherwise ugly exception traceback when +something goes badly wrong during the BuildStep. + + In contrast, you might want to create a new LogFile at the +beginning of the step, and add text to it as the command runs. You +can create the LogFile and attach it to the build by calling +`addLog', which returns the LogFile object. You then add text to this +LogFile by calling methods like `addStdout' and `addHeader'. When you +are done, you must call the `finish' method so the LogFile can be +closed. It may be useful to create and populate a LogFile like this +from a LogObserver method *Note Adding LogObservers::. + + The `logfiles=' argument to `ShellCommand' (see *note +ShellCommand::) creates new LogFiles and fills them in realtime by +asking the buildslave to watch a actual file on disk. The buildslave +will look for additions in the target file and report them back to +the BuildStep. These additions will be added to the LogFile by +calling `addStdout'. These secondary LogFiles can be used as the +source of a LogObserver just like the normal "stdio" LogFile. + + +File: buildbot.info, Node: Reading Logfiles, Next: Adding LogObservers, Prev: BuildStep LogFiles, Up: Writing New BuildSteps + +6.1.10.3 Reading Logfiles +......................... + +Once a LogFile has been added to a BuildStep with `addLog()', +`addCompleteLog()', `addHTMLLog()', or `logfiles=', your BuildStep +can retrieve it by using `getLog()': + + class MyBuildStep(ShellCommand): + logfiles = { "nodelog": "_test/node.log" } + + def evaluateCommand(self, cmd): + nodelog = self.getLog("nodelog") + if "STARTED" in nodelog.getText(): + return SUCCESS + else: + return FAILURE + + For a complete list of the methods you can call on a LogFile, +please see the docstrings on the `IStatusLog' class in +`buildbot/interfaces.py'. + + +File: buildbot.info, Node: Adding LogObservers, Next: BuildStep URLs, Prev: Reading Logfiles, Up: Writing New BuildSteps + +6.1.10.4 Adding LogObservers +............................ + +Most shell commands emit messages to stdout or stderr as they operate, +especially if you ask them nicely with a `--verbose' flag of some +sort. They may also write text to a log file while they run. Your +BuildStep can watch this output as it arrives, to keep track of how +much progress the command has made. You can get a better measure of +progress by counting the number of source files compiled or test cases +run than by merely tracking the number of bytes that have been written +to stdout. This improves the accuracy and the smoothness of the ETA +display. + + To accomplish this, you will need to attach a `LogObserver' to one +of the log channels, most commonly to the "stdio" channel but perhaps +to another one which tracks a log file. This observer is given all +text as it is emitted from the command, and has the opportunity to +parse that output incrementally. Once the observer has decided that +some event has occurred (like a source file being compiled), it can +use the `setProgress' method to tell the BuildStep about the progress +that this event represents. + + There are a number of pre-built `LogObserver' classes that you can +choose from (defined in `buildbot.process.buildstep', and of course +you can subclass them to add further customization. The +`LogLineObserver' class handles the grunt work of buffering and +scanning for end-of-line delimiters, allowing your parser to operate +on complete stdout/stderr lines. (Lines longer than a set maximum +length are dropped; the maximum defaults to 16384 bytes, but you can +change it by calling `setMaxLineLength()' on your `LogLineObserver' +instance. Use `sys.maxint' for effective infinity.) + + For example, let's take a look at the `TrialTestCaseCounter', +which is used by the Trial step to count test cases as they are run. +As Trial executes, it emits lines like the following: + + buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK] + buildbot.test.test_config.ConfigTest.testEmpty ... [OK] + buildbot.test.test_config.ConfigTest.testIRC ... [FAIL] + buildbot.test.test_config.ConfigTest.testLocks ... [OK] + + When the tests are finished, trial emits a long line of "======" +and then some lines which summarize the tests that failed. We want to +avoid parsing these trailing lines, because their format is less +well-defined than the "[OK]" lines. + + The parser class looks like this: + + from buildbot.process.buildstep import LogLineObserver + + class TrialTestCaseCounter(LogLineObserver): + _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') + numTests = 0 + finished = False + + def outLineReceived(self, line): + if self.finished: + return + if line.startswith("=" * 40): + self.finished = True + return + + m = self._line_re.search(line.strip()) + if m: + testname, result = m.groups() + self.numTests += 1 + self.step.setProgress('tests', self.numTests) + + This parser only pays attention to stdout, since that's where trial +writes the progress lines. It has a mode flag named `finished' to +ignore everything after the "====" marker, and a scary-looking +regular expression to match each line while hopefully ignoring other +messages that might get displayed as the test runs. + + Each time it identifies a test has been completed, it increments +its counter and delivers the new progress value to the step with +`self.step.setProgress'. This class is specifically measuring +progress along the "tests" metric, in units of test cases (as opposed +to other kinds of progress like the "output" metric, which measures +in units of bytes). The Progress-tracking code uses each progress +metric separately to come up with an overall completion percentage +and an ETA value. + + To connect this parser into the `Trial' BuildStep, +`Trial.__init__' ends with the following clause: + + # this counter will feed Progress along the 'test cases' metric + counter = TrialTestCaseCounter() + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) + + This creates a TrialTestCaseCounter and tells the step that the +counter wants to watch the "stdio" log. The observer is automatically +given a reference to the step in its `.step' attribute. + +A Somewhat Whimsical Example +---------------------------- + +Let's say that we've got some snazzy new unit-test framework called +Framboozle. It's the hottest thing since sliced bread. It slices, it +dices, it runs unit tests like there's no tomorrow. Plus if your unit +tests fail, you can use its name for a Web 2.1 startup company, make +millions of dollars, and hire engineers to fix the bugs for you, while +you spend your afternoons lazily hang-gliding along a scenic pacific +beach, blissfully unconcerned about the state of your tests.(1) + + To run a Framboozle-enabled test suite, you just run the +'framboozler' command from the top of your source code tree. The +'framboozler' command emits a bunch of stuff to stdout, but the most +interesting bit is that it emits the line "FNURRRGH!" every time it +finishes running a test case(2). You'd like to have a test-case +counting LogObserver that watches for these lines and counts them, +because counting them will help the buildbot more accurately +calculate how long the build will take, and this will let you know +exactly how long you can sneak out of the office for your +hang-gliding lessons without anyone noticing that you're gone. + + This will involve writing a new BuildStep (probably named +"Framboozle") which inherits from ShellCommand. The BuildStep class +definition itself will look something like this: + + # START + from buildbot.steps.shell import ShellCommand + from buildbot.process.buildstep import LogLineObserver + + class FNURRRGHCounter(LogLineObserver): + numTests = 0 + def outLineReceived(self, line): + if "FNURRRGH!" in line: + self.numTests += 1 + self.step.setProgress('tests', self.numTests) + + class Framboozle(ShellCommand): + command = ["framboozler"] + + def __init__(self, **kwargs): + ShellCommand.__init__(self, **kwargs) # always upcall! + counter = FNURRRGHCounter()) + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) + # FINISH + + So that's the code that we want to wind up using. How do we +actually deploy it? + + You have a couple of different options. + + Option 1: The simplest technique is to simply put this text +(everything from START to FINISH) in your master.cfg file, somewhere +before the BuildFactory definition where you actually use it in a +clause like: + + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(Framboozle()) + + Remember that master.cfg is secretly just a python program with one +job: populating the BuildmasterConfig dictionary. And python programs +are allowed to define as many classes as they like. So you can define +classes and use them in the same file, just as long as the class is +defined before some other code tries to use it. + + This is easy, and it keeps the point of definition very close to +the point of use, and whoever replaces you after that unfortunate +hang-gliding accident will appreciate being able to easily figure out +what the heck this stupid "Framboozle" step is doing anyways. The +downside is that every time you reload the config file, the Framboozle +class will get redefined, which means that the buildmaster will think +that you've reconfigured all the Builders that use it, even though +nothing changed. Bleh. + + Option 2: Instead, we can put this code in a separate file, and +import it into the master.cfg file just like we would the normal +buildsteps like ShellCommand and SVN. + + Create a directory named ~/lib/python, put everything from START to +FINISH in ~/lib/python/framboozle.py, and run your buildmaster using: + + PYTHONPATH=~/lib/python buildbot start MASTERDIR + + or use the `Makefile.buildbot' to control the way `buildbot start' +works. Or add something like this to something like your ~/.bashrc or +~/.bash_profile or ~/.cshrc: + + export PYTHONPATH=~/lib/python + + Once we've done this, our master.cfg can look like: + + from framboozle import Framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(Framboozle()) + + or: + + import framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(framboozle.Framboozle()) + + (check out the python docs for details about how "import" and +"from A import B" work). + + What we've done here is to tell python that every time it handles +an "import" statement for some named module, it should look in our +~/lib/python/ for that module before it looks anywhere else. After our +directories, it will try in a bunch of standard directories too +(including the one where buildbot is installed). By setting the +PYTHONPATH environment variable, you can add directories to the front +of this search list. + + Python knows that once it "import"s a file, it doesn't need to +re-import it again. This means that reconfiguring the buildmaster +(with "buildbot reconfig", for example) won't make it think the +Framboozle class has changed every time, so the Builders that use it +will not be spuriously restarted. On the other hand, you either have +to start your buildmaster in a slightly weird way, or you have to +modify your environment to set the PYTHONPATH variable. + + Option 3: Install this code into a standard python library +directory + + Find out what your python's standard include path is by asking it: + + 80:warner@luther% python + Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46) + [GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> import pprint + >>> pprint.pprint(sys.path) + ['', + '/usr/lib/python24.zip', + '/usr/lib/python2.4', + '/usr/lib/python2.4/plat-linux2', + '/usr/lib/python2.4/lib-tk', + '/usr/lib/python2.4/lib-dynload', + '/usr/local/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages/Numeric', + '/var/lib/python-support/python2.4', + '/usr/lib/site-python'] + + In this case, putting the code into +/usr/local/lib/python2.4/site-packages/framboozle.py would work just +fine. We can use the same master.cfg "import framboozle" statement as +in Option 2. By putting it in a standard include directory (instead of +the decidedly non-standard ~/lib/python), we don't even have to set +PYTHONPATH to anything special. The downside is that you probably have +to be root to write to one of those standard include directories. + + Option 4: Submit the code for inclusion in the Buildbot +distribution + + Make a fork of buildbot on http://github.com/djmitche/buildbot or +post a patch in a bug at http://buildbot.net. In either case, post a +note about your patch to the mailing list, so others can provide +feedback and, eventually, commit it. + + from buildbot.steps import framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(framboozle.Framboozle()) + + And then you don't even have to install framboozle.py anywhere on +your system, since it will ship with Buildbot. You don't have to be +root, you don't have to set PYTHONPATH. But you do have to make a +good case for Framboozle being worth going into the main +distribution, you'll probably have to provide docs and some unit test +cases, you'll need to figure out what kind of beer the author likes, +and then you'll have to wait until the next release. But in some +environments, all this is easier than getting root on your +buildmaster box, so the tradeoffs may actually be worth it. + + Putting the code in master.cfg (1) makes it available to that +buildmaster instance. Putting it in a file in a personal library +directory (2) makes it available for any buildmasters you might be +running. Putting it in a file in a system-wide shared library +directory (3) makes it available for any buildmasters that anyone on +that system might be running. Getting it into the buildbot's upstream +repository (4) makes it available for any buildmasters that anyone in +the world might be running. It's all a matter of how widely you want +to deploy that new class. + + ---------- Footnotes ---------- + + (1) framboozle.com is still available. Remember, I get 10% :). + + (2) Framboozle gets very excited about running unit tests. + + +File: buildbot.info, Node: BuildStep URLs, Prev: Adding LogObservers, Up: Writing New BuildSteps + +6.1.10.5 BuildStep URLs +....................... + +Each BuildStep has a collection of "links". Like its collection of +LogFiles, each link has a name and a target URL. The web status page +creates HREFs for each link in the same box as it does for LogFiles, +except that the target of the link is the external URL instead of an +internal link to a page that shows the contents of the LogFile. + + These external links can be used to point at build information +hosted on other servers. For example, the test process might produce +an intricate description of which tests passed and failed, or some +sort of code coverage data in HTML form, or a PNG or GIF image with a +graph of memory usage over time. The external link can provide an +easy way for users to navigate from the buildbot's status page to +these external web sites or file servers. Note that the step itself is +responsible for insuring that there will be a document available at +the given URL (perhaps by using `scp' to copy the HTML output to a +`~/public_html/' directory on a remote web server). Calling `addURL' +does not magically populate a web server. + + To set one of these links, the BuildStep should call the `addURL' +method with the name of the link and the target URL. Multiple URLs can +be set. + + In this example, we assume that the `make test' command causes a +collection of HTML files to be created and put somewhere on the +coverage.example.org web server, in a filename that incorporates the +build number. + + class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s" % "buildnumber")] + + def createSummary(self, log): + buildnumber = self.getProperty("buildnumber") + url = "http://coverage.example.org/builds/%s.html" % buildnumber + self.addURL("coverage", url) + + You might also want to extract the URL from some special message +output by the build process itself: + + class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s" % "buildnumber")] + + def createSummary(self, log): + output = StringIO(log.getText()) + for line in output.readlines(): + if line.startswith("coverage-url:"): + url = line[len("coverage-url:"):].strip() + self.addURL("coverage", url) + return + + Note that a build process which emits both stdout and stderr might +cause this line to be split or interleaved between other lines. It +might be necessary to restrict the getText() call to only stdout with +something like this: + + output = StringIO("".join([c[1] + for c in log.getChunks() + if c[0] == LOG_CHANNEL_STDOUT])) + + Of course if the build is run under a PTY, then stdout and stderr +will be merged before the buildbot ever sees them, so such +interleaving will be unavoidable. + + +File: buildbot.info, Node: Interlocks, Next: Build Factories, Prev: Build Steps, Up: Build Process + +6.2 Interlocks +============== + +Until now, we assumed that a master can run builds at any slave +whenever needed or desired. Some times, you want to enforce +additional constraints on builds. For reasons like limited network +bandwidth, old slave machines, or a self-willed data base server, you +may want to limit the number of builds (or build steps) that can +access a resource. + + The mechanism used by Buildbot is known as the read/write lock.(1) +It allows either many readers or a single writer but not a +combination of readers and writers. The general lock has been +modified and extended for use in Buildbot. Firstly, the general lock +allows an infinite number of readers. In Buildbot, we often want to +put an upper limit on the number of readers, for example allowing two +out of five possible builds at the same time. To do this, the lock +counts the number of active readers. Secondly, the terms _read mode_ +and _write mode_ are confusing in Buildbot context. They have been +replaced by _counting mode_ (since the lock counts them) and +_exclusive mode_. As a result of these changes, locks in Buildbot +allow a number of builds (upto some fixed number) in counting mode, +or they allow one build in exclusive mode. + + Often, not all slaves are equal. To allow for this situation, +Buildbot allows to have a separate upper limit on the count for each +slave. In this way, you can have at most 3 concurrent builds at a +fast slave, 2 at a slightly older slave, and 1 at all other slaves. + + The final thing you can specify when you introduce a new lock is +its scope. Some constraints are global - they must be enforced over +all slaves. Other constraints are local to each slave. A _master +lock_ is used for the global constraints. You can ensure for example +that at most one build (of all builds running at all slaves) accesses +the data base server. With a _slave lock_ you can add a limit local +to each slave. With such a lock, you can for example enforce an upper +limit to the number of active builds at a slave, like above. + + Time for a few examples. Below a master lock is defined to protect +a data base, and a slave lock is created to limit the number of +builds at each slave. + + from buildbot import locks + + db_lock = locks.MasterLock("database") + build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = { 'fast': 3, 'new': 2 }) + + After importing locks from buildbot, `db_lock' is defined to be a +master lock. The `"database"' string is used for uniquely identifying +the lock. At the next line, a slave lock called `build_lock' is +created. It is identified by the `"slave_builds"' string. Since the +requirements of the lock are a bit more complicated, two optional +arguments are also specified. The `maxCount' parameter sets the +default limit for builds in counting mode to `1'. For the slave +called `'fast'' however, we want to have at most three builds, and +for the slave called `'new'' the upper limit is two builds running at +the same time. + + The next step is using the locks in builds. Buildbot allows a +lock to be used during an entire build (from beginning to end), or +only during a single build step. In the latter case, the lock is +claimed for use just before the step starts, and released again when +the step ends. To prevent deadlocks,(2) it is not possible to claim +or release locks at other times. + + To use locks, you should add them with a `locks' argument. Each +use of a lock is either in counting mode (that is, possibly shared +with other builds) or in exclusive mode. A build or build step +proceeds only when it has acquired all locks. If a build or step +needs a lot of locks, it may be starved(3) by other builds that need +fewer locks. + + To illustrate use of locks, a few examples. + + from buildbot import locks + from buildbot.steps import source, shell + from buildbot.process import factory + + db_lock = locks.MasterLock("database") + build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = { 'fast': 3, 'new': 2 }) + + f = factory.BuildFactory() + f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk")) + f.addStep(shell.ShellCommand(command="make all")) + f.addStep(shell.ShellCommand(command="make test", + locks=[db_lock.access('exclusive')])) + + b1 = {'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, + 'locks': [build_lock.access('counting')] } + + b2 = {'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f. + 'locks': [build_lock.access('counting')] } + + b3 = {'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f. + 'locks': [build_lock.access('counting')] } + + b4 = {'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f. + 'locks': [build_lock.access('counting')] } + + c['builders'] = [b1, b2, b3, b4] + + Here we have four slaves `b1', `b2', `b3', and `b4'. Each slave +performs the same checkout, make, and test build step sequence. We +want to enforce that at most one test step is executed between all +slaves due to restrictions with the data base server. This is done by +adding the `locks=' parameter with the third step. It takes a list of +locks with their access mode. In this case only the `db_lock' is +needed. The exclusive access mode is used to ensure there is at most +one slave that executes the test step. + + In addition to exclusive accessing the data base, we also want +slaves to stay responsive even under the load of a large number of +builds being triggered. For this purpose, the slave lock called +`build_lock' is defined. Since the restraint holds for entire builds, +the lock is specified in the builder with `'locks': +[build_lock.access('counting')]'. + + ---------- Footnotes ---------- + + (1) See http://en.wikipedia.org/wiki/Read/write_lock_pattern for +more information. + + (2) Deadlock is the situation where two or more slaves each hold a +lock in exclusive mode, and in addition want to claim the lock held by +the other slave exclusively as well. Since locks allow at most one +exclusive user, both slaves will wait forever. + + (3) Starving is the situation that only a few locks are available, +and they are immediately grabbed by another build. As a result, it +may take a long time before all locks needed by the starved build are +free at the same time. + + +File: buildbot.info, Node: Build Factories, Prev: Interlocks, Up: Build Process + +6.3 Build Factories +=================== + +Each Builder is equipped with a "build factory", which is responsible +for producing the actual `Build' objects that perform each build. +This factory is created in the configuration file, and attached to a +Builder through the `factory' element of its dictionary. + + The standard `BuildFactory' object creates `Build' objects by +default. These Builds will each execute a collection of BuildSteps in +a fixed sequence. Each step can affect the results of the build, but +in general there is little intelligence to tie the different steps +together. You can create subclasses of `Build' to implement more +sophisticated build processes, and then use a subclass of +`BuildFactory' (or simply set the `buildClass' attribute) to create +instances of your new Build subclass. + +* Menu: + +* BuildStep Objects:: +* BuildFactory:: +* Process-Specific build factories:: + + +File: buildbot.info, Node: BuildStep Objects, Next: BuildFactory, Prev: Build Factories, Up: Build Factories + +6.3.1 BuildStep Objects +----------------------- + +The steps used by these builds are all subclasses of `BuildStep'. +The standard ones provided with Buildbot are documented later, *Note +Build Steps::. You can also write your own subclasses to use in +builds. + + The basic behavior for a `BuildStep' is to: + + * run for a while, then stop + + * possibly invoke some RemoteCommands on the attached build slave + + * possibly produce a set of log files + + * finish with a status described by one of four values defined in + buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED + + * provide a list of short strings to describe the step + + * define a color (generally green, orange, or red) with which the + step should be displayed + + More sophisticated steps may produce additional information and +provide it to later build steps, or store it in the factory to provide +to later builds. + +* Menu: + +* BuildFactory Attributes:: +* Quick builds:: + + +File: buildbot.info, Node: BuildFactory, Next: Process-Specific build factories, Prev: BuildStep Objects, Up: Build Factories + +6.3.2 BuildFactory +------------------ + +The default `BuildFactory', provided in the +`buildbot.process.factory' module, contains an internal list of +"BuildStep specifications": a list of `(step_class, kwargs)' tuples +for each. These specification tuples are constructed when the config +file is read, by asking the instances passed to `addStep' for their +subclass and arguments. + + When asked to create a Build, the `BuildFactory' puts a copy of +the list of step specifications into the new Build object. When the +Build is actually started, these step specifications are used to +create the actual set of BuildSteps, which are then executed one at a +time. This serves to give each Build an independent copy of each step. +For example, a build which consists of a CVS checkout followed by a +`make build' would be constructed as follows: + + from buildbot.steps import source, shell + from buildbot.process import factory + + f = factory.BuildFactory() + f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update")) + f.addStep(shell.Compile(command=["make", "build"])) + + (To support config files from buildbot-0.7.5 and earlier, +`addStep' also accepts the `f.addStep(shell.Compile, +command=["make","build"])' form, although its use is discouraged +because then the `Compile' step doesn't get to validate or complain +about its arguments until build time. The modern pass-by-instance +approach allows this validation to occur while the config file is +being loaded, where the admin has a better chance of noticing +problems). + + It is also possible to pass a list of steps into the +`BuildFactory' when it is created. Using `addStep' is usually +simpler, but there are cases where is is more convenient to create +the list of steps ahead of time.: + + from buildbot.steps import source, shell + from buildbot.process import factory + + all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), + shell.Compile(command=["make", "build"]), + ] + f = factory.BuildFactory(all_steps) + + Each step can affect the build process in the following ways: + + * If the step's `haltOnFailure' attribute is True, then a failure + in the step (i.e. if it completes with a result of FAILURE) will + cause the whole build to be terminated immediately: no further + steps will be executed, with the exception of steps with + `alwaysRun' set to True. `haltOnFailure' is useful for setup + steps upon which the rest of the build depends: if the CVS + checkout or `./configure' process fails, there is no point in + trying to compile or test the resulting tree. + + * If the step's `alwaysRun' attribute is True, then it will always + be run, regardless of if previous steps have failed. This is + useful for cleanup steps that should always be run to return the + build directory or build slave into a good state. + + * If the `flunkOnFailure' or `flunkOnWarnings' flag is set, then a + result of FAILURE or WARNINGS will mark the build as a whole as + FAILED. However, the remaining steps will still be executed. + This is appropriate for things like multiple testing steps: a + failure in any one of them will indicate that the build has + failed, however it is still useful to run them all to completion. + + * Similarly, if the `warnOnFailure' or `warnOnWarnings' flag is + set, then a result of FAILURE or WARNINGS will mark the build as + having WARNINGS, and the remaining steps will still be executed. + This may be appropriate for certain kinds of optional build or + test steps. For example, a failure experienced while building + documentation files should be made visible with a WARNINGS + result but not be serious enough to warrant marking the whole + build with a FAILURE. + + + In addition, each Step produces its own results, may create +logfiles, etc. However only the flags described above have any effect +on the build as a whole. + + The pre-defined BuildSteps like `CVS' and `Compile' have +reasonably appropriate flags set on them already. For example, without +a source tree there is no point in continuing the build, so the `CVS' +class has the `haltOnFailure' flag set to True. Look in +`buildbot/steps/*.py' to see how the other Steps are marked. + + Each Step is created with an additional `workdir' argument that +indicates where its actions should take place. This is specified as a +subdirectory of the slave builder's base directory, with a default +value of `build'. This is only implemented as a step argument (as +opposed to simply being a part of the base directory) because the +CVS/SVN steps need to perform their checkouts from the parent +directory. + +* Menu: + +* BuildFactory Attributes:: +* Quick builds:: + + +File: buildbot.info, Node: BuildFactory Attributes, Next: Quick builds, Prev: BuildFactory, Up: BuildFactory + +6.3.2.1 BuildFactory Attributes +............................... + +Some attributes from the BuildFactory are copied into each Build. + +`useProgress' + (defaults to True): if True, the buildmaster keeps track of how + long each step takes, so it can provide estimates of how long + future builds will take. If builds are not expected to take a + consistent amount of time (such as incremental builds in which a + random set of files are recompiled or tested each time), this + should be set to False to inhibit progress-tracking. + + + +File: buildbot.info, Node: Quick builds, Prev: BuildFactory Attributes, Up: BuildFactory + +6.3.2.2 Quick builds +.................... + +The difference between a "full build" and a "quick build" is that +quick builds are generally done incrementally, starting with the tree +where the previous build was performed. That simply means that the +source-checkout step should be given a `mode='update'' flag, to do +the source update in-place. + + In addition to that, the `useProgress' flag should be set to +False. Incremental builds will (or at least the ought to) compile as +few files as necessary, so they will take an unpredictable amount of +time to run. Therefore it would be misleading to claim to predict how +long the build will take. + + +File: buildbot.info, Node: Process-Specific build factories, Prev: BuildFactory, Up: Build Factories + +6.3.3 Process-Specific build factories +-------------------------------------- + +Many projects use one of a few popular build frameworks to simplify +the creation and maintenance of Makefiles or other compilation +structures. Buildbot provides several pre-configured BuildFactory +subclasses which let you build these projects with a minimum of fuss. + +* Menu: + +* GNUAutoconf:: +* CPAN:: +* Python distutils:: +* Python/Twisted/trial projects:: + + +File: buildbot.info, Node: GNUAutoconf, Next: CPAN, Prev: Process-Specific build factories, Up: Process-Specific build factories + +6.3.3.1 GNUAutoconf +................... + +GNU Autoconf (http://www.gnu.org/software/autoconf/) is a software +portability tool, intended to make it possible to write programs in C +(and other languages) which will run on a variety of UNIX-like +systems. Most GNU software is built using autoconf. It is frequently +used in combination with GNU automake. These tools both encourage a +build process which usually looks like this: + + % CONFIG_ENV=foo ./configure --with-flags + % make all + % make check + # make install + + (except of course the Buildbot always skips the `make install' +part). + + The Buildbot's `buildbot.process.factory.GNUAutoconf' factory is +designed to build projects which use GNU autoconf and/or automake. The +configuration environment variables, the configure flags, and command +lines used for the compile and test are all configurable, in general +the default values will be suitable. + + Example: + + # use the s() convenience function defined earlier + f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"), + flags=["--disable-nls"]) + + Required Arguments: + +`source' + This argument must be a step specification tuple that provides a + BuildStep to generate the source tree. + + Optional Arguments: + +`configure' + The command used to configure the tree. Defaults to + `./configure'. Accepts either a string or a list of shell argv + elements. + +`configureEnv' + The environment used for the initial configuration step. This + accepts a dictionary which will be merged into the buildslave's + normal environment. This is commonly used to provide things like + `CFLAGS="-O2 -g"' (to turn off debug symbols during the compile). + Defaults to an empty dictionary. + +`configureFlags' + A list of flags to be appended to the argument list of the + configure command. This is commonly used to enable or disable + specific features of the autoconf-controlled package, like + `["--without-x"]' to disable windowing support. Defaults to an + empty list. + +`compile' + this is a shell command or list of argv values which is used to + actually compile the tree. It defaults to `make all'. If set to + None, the compile step is skipped. + +`test' + this is a shell command or list of argv values which is used to + run the tree's self-tests. It defaults to `make check'. If set to + None, the test step is skipped. + + + +File: buildbot.info, Node: CPAN, Next: Python distutils, Prev: GNUAutoconf, Up: Process-Specific build factories + +6.3.3.2 CPAN +............ + +Most Perl modules available from the CPAN (http://www.cpan.org/) +archive use the `MakeMaker' module to provide configuration, build, +and test services. The standard build routine for these modules looks +like: + + % perl Makefile.PL + % make + % make test + # make install + + (except again Buildbot skips the install step) + + Buildbot provides a `CPAN' factory to compile and test these +projects. + + Arguments: +`source' + (required): A step specification tuple, like that used by + GNUAutoconf. + +`perl' + A string which specifies the `perl' executable to use. Defaults + to just `perl'. + + + +File: buildbot.info, Node: Python distutils, Next: Python/Twisted/trial projects, Prev: CPAN, Up: Process-Specific build factories + +6.3.3.3 Python distutils +........................ + +Most Python modules use the `distutils' package to provide +configuration and build services. The standard build process looks +like: + + % python ./setup.py build + % python ./setup.py install + + Unfortunately, although Python provides a standard unit-test +framework named `unittest', to the best of my knowledge `distutils' +does not provide a standardized target to run such unit tests. (Please +let me know if I'm wrong, and I will update this factory.) + + The `Distutils' factory provides support for running the build +part of this process. It accepts the same `source=' parameter as the +other build factories. + + Arguments: +`source' + (required): A step specification tuple, like that used by + GNUAutoconf. + +`python' + A string which specifies the `python' executable to use. Defaults + to just `python'. + +`test' + Provides a shell command which runs unit tests. This accepts + either a string or a list. The default value is None, which + disables the test step (since there is no common default command + to run unit tests in distutils modules). + + + +File: buildbot.info, Node: Python/Twisted/trial projects, Prev: Python distutils, Up: Process-Specific build factories + +6.3.3.4 Python/Twisted/trial projects +..................................... + +Twisted provides a unit test tool named `trial' which provides a few +improvements over Python's built-in `unittest' module. Many python +projects which use Twisted for their networking or application +services also use trial for their unit tests. These modules are +usually built and tested with something like the following: + + % python ./setup.py build + % PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test + % python ./setup.py install + + Unfortunately, the `build/lib' directory into which the +built/copied .py files are placed is actually architecture-dependent, +and I do not yet know of a simple way to calculate its value. For many +projects it is sufficient to import their libraries "in place" from +the tree's base directory (`PYTHONPATH=.'). + + In addition, the PROJECTNAME value where the test files are +located is project-dependent: it is usually just the project's +top-level library directory, as common practice suggests the unit test +files are put in the `test' sub-module. This value cannot be guessed, +the `Trial' class must be told where to find the test files. + + The `Trial' class provides support for building and testing +projects which use distutils and trial. If the test module name is +specified, trial will be invoked. The library path used for testing +can also be set. + + One advantage of trial is that the Buildbot happens to know how to +parse trial output, letting it identify which tests passed and which +ones failed. The Buildbot can then provide fine-grained reports about +how many tests have failed, when individual tests fail when they had +been passing previously, etc. + + Another feature of trial is that you can give it a series of source +.py files, and it will search them for special `test-case-name' tags +that indicate which test cases provide coverage for that file. Trial +can then run just the appropriate tests. This is useful for quick +builds, where you want to only run the test cases that cover the +changed functionality. + + Arguments: +`source' + (required): A step specification tuple, like that used by + GNUAutoconf. + +`buildpython' + A list (argv array) of strings which specifies the `python' + executable to use when building the package. Defaults to just + `['python']'. It may be useful to add flags here, to supress + warnings during compilation of extension modules. This list is + extended with `['./setup.py', 'build']' and then executed in a + ShellCommand. + +`testpath' + Provides a directory to add to `PYTHONPATH' when running the unit + tests, if tests are being run. Defaults to `.' to include the + project files in-place. The generated build library is frequently + architecture-dependent, but may simply be `build/lib' for + pure-python modules. + +`trialpython' + Another list of strings used to build the command that actually + runs trial. This is prepended to the contents of the `trial' + argument below. It may be useful to add `-W' flags here to + supress warnings that occur while tests are being run. Defaults + to an empty list, meaning `trial' will be run without an explicit + interpreter, which is generally what you want if you're using + `/usr/bin/trial' instead of, say, the `./bin/trial' that lives + in the Twisted source tree. + +`trial' + provides the name of the `trial' command. It is occasionally + useful to use an alternate executable, such as `trial2.2' which + might run the tests under an older version of Python. Defaults to + `trial'. + +`tests' + Provides a module name or names which contain the unit tests for + this project. Accepts a string, typically `PROJECTNAME.test', or + a list of strings. Defaults to None, indicating that no tests + should be run. You must either set this or `useTestCaseNames' to + do anyting useful with the Trial factory. + +`useTestCaseNames' + Tells the Step to provide the names of all changed .py files to + trial, so it can look for test-case-name tags and run just the + matching test cases. Suitable for use in quick builds. Defaults + to False. + +`randomly' + If `True', tells Trial (with the `--random=0' argument) to run + the test cases in random order, which sometimes catches subtle + inter-test dependency bugs. Defaults to `False'. + +`recurse' + If `True', tells Trial (with the `--recurse' argument) to look + in all subdirectories for additional test cases. It isn't clear + to me how this works, but it may be useful to deal with the + unknown-PROJECTNAME problem described above, and is currently + used in the Twisted buildbot to accomodate the fact that test + cases are now distributed through multiple + twisted.SUBPROJECT.test directories. + + + Unless one of `trialModule' or `useTestCaseNames' are set, no +tests will be run. + + Some quick examples follow. Most of these examples assume that the +target python code (the "code under test") can be reached directly +from the root of the target tree, rather than being in a `lib/' +subdirectory. + + # Trial(source, tests="toplevel.test") does: + # python ./setup.py build + # PYTHONPATH=. trial -to toplevel.test + + # Trial(source, tests=["toplevel.test", "other.test"]) does: + # python ./setup.py build + # PYTHONPATH=. trial -to toplevel.test other.test + + # Trial(source, useTestCaseNames=True) does: + # python ./setup.py build + # PYTHONPATH=. trial -to --testmodule=foo/bar.py.. (from Changes) + + # Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"): + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. trial -to foo.tests + + # Trial(source, trialpython="python2.3", trial="/usr/bin/trial", + # tests="foo.tests") does: + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests + + # For running trial out of the tree being tested (only useful when the + # tree being built is Twisted itself): + # Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial", + # tests="foo.tests") does: + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests + + If the output directory of `./setup.py build' is known, you can +pull the python code from the built location instead of the source +directories. This should be able to handle variations in where the +source comes from, as well as accomodating binary extension modules: + + # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3') + # does: + # python ./setup.py build + # PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test + + +File: buildbot.info, Node: Status Delivery, Next: Command-line tool, Prev: Build Process, Up: Top + +7 Status Delivery +***************** + +More details are available in the docstrings for each class, use a +command like `pydoc buildbot.status.html.WebStatus' to see them. +Most status delivery objects take a `categories=' argument, which can +contain a list of "category" names: in this case, it will only show +status for Builders that are in one of the named categories. + + (implementor's note: each of these objects should be a +service.MultiService which will be attached to the BuildMaster object +when the configuration is processed. They should use +`self.parent.getStatus()' to get access to the top-level IStatus +object, either inside `startService' or later. They may call +`status.subscribe()' in `startService' to receive notifications of +builder events, in which case they must define `builderAdded' and +related methods. See the docstrings in `buildbot/interfaces.py' for +full details.) + +* Menu: + +* WebStatus:: +* MailNotifier:: +* IRC Bot:: +* PBListener:: +* Writing New Status Plugins:: + + +File: buildbot.info, Node: WebStatus, Next: MailNotifier, Prev: Status Delivery, Up: Status Delivery + +7.1 WebStatus +============= + +The `buildbot.status.html.WebStatus' status target runs a small web +server inside the buildmaster. You can point a browser at this web +server and retrieve information about every build the buildbot knows +about, as well as find out what the buildbot is currently working on. + + The first page you will see is the "Welcome Page", which contains +links to all the other useful pages. This page is simply served from +the `public_html/index.html' file in the buildmaster's base +directory, where it is created by the `buildbot create-master' +command along with the rest of the buildmaster. + + The most complex resource provided by `WebStatus' is the +"Waterfall Display", which shows a time-based chart of events. This +somewhat-busy display provides detailed information about all steps of +all recent builds, and provides hyperlinks to look at individual build +logs and source changes. By simply reloading this page on a regular +basis, you will see a complete description of everything the buildbot +is currently working on. + + There are also pages with more specialized information. For +example, there is a page which shows the last 20 builds performed by +the buildbot, one line each. Each line is a link to detailed +information about that build. By adding query arguments to the URL +used to reach this page, you can narrow the display to builds that +involved certain branches, or which ran on certain Builders. These +pages are described in great detail below. + + When the buildmaster is created, a subdirectory named +`public_html/' is created in its base directory. By default, +`WebStatus' will serve files from this directory: for example, when a +user points their browser at the buildbot's `WebStatus' URL, they +will see the contents of the `public_html/index.html' file. Likewise, +`public_html/robots.txt', `public_html/buildbot.css', and +`public_html/favicon.ico' are all useful things to have in there. +The first time a buildmaster is created, the `public_html' directory +is populated with some sample files, which you will probably want to +customize for your own project. These files are all static: the +buildbot does not modify them in any way as it serves them to HTTP +clients. + + from buildbot.status.html import WebStatus + c['status'].append(WebStatus(8080)) + + Note that the initial robots.txt file has Disallow lines for all of +the dynamically-generated buildbot pages, to discourage web spiders +and search engines from consuming a lot of CPU time as they crawl +through the entire history of your buildbot. If you are running the +buildbot behind a reverse proxy, you'll probably need to put the +robots.txt file somewhere else (at the top level of the parent web +server), and replace the URL prefixes in it with more suitable values. + + If you would like to use an alternative root directory, add the +`public_html=..' option to the `WebStatus' creation: + + c['status'].append(WebStatus(8080, public_html="/var/www/buildbot")) + + In addition, if you are familiar with twisted.web _Resource +Trees_, you can write code to add additional pages at places inside +this web space. Just use `webstatus.putChild' to place these +resources. + + The following section describes the special URLs and the status +views they provide. + +* Menu: + +* WebStatus Configuration Parameters:: +* Buildbot Web Resources:: +* XMLRPC server:: +* HTML Waterfall:: + + +File: buildbot.info, Node: WebStatus Configuration Parameters, Next: Buildbot Web Resources, Prev: WebStatus, Up: WebStatus + +7.1.1 WebStatus Configuration Parameters +---------------------------------------- + +The most common way to run a `WebStatus' is on a regular TCP port. To +do this, just pass in the TCP port number when you create the +`WebStatus' instance; this is called the `http_port' argument: + + from buildbot.status.html import WebStatus + c['status'].append(WebStatus(8080)) + + The `http_port' argument is actually a "strports specification" +for the port that the web server should listen on. This can be a +simple port number, or a string like `tcp:8080:interface=127.0.0.1' +(to limit connections to the loopback interface, and therefore to +clients running on the same host)(1). + + If instead (or in addition) you provide the `distrib_port' +argument, a twisted.web distributed server will be started either on a +TCP port (if `distrib_port' is like `"tcp:12345"') or more likely on +a UNIX socket (if `distrib_port' is like `"unix:/path/to/socket"'). + + The `distrib_port' option means that, on a host with a +suitably-configured twisted-web server, you do not need to consume a +separate TCP port for the buildmaster's status web page. When the web +server is constructed with `mktap web --user', URLs that point to +`http://host/~username/' are dispatched to a sub-server that is +listening on a UNIX socket at `~username/.twisted-web-pb'. On such a +system, it is convenient to create a dedicated `buildbot' user, then +set `distrib_port' to +`"unix:"+os.path.expanduser("~/.twistd-web-pb")'. This configuration +will make the HTML status page available at `http://host/~buildbot/' +. Suitable URL remapping can make it appear at +`http://host/buildbot/', and the right virtual host setup can even +place it at `http://buildbot.host/' . + + The other `WebStatus' argument is `allowForce'. If set to True, +then the web page will provide a "Force Build" button that allows +visitors to manually trigger builds. This is useful for developers to +re-run builds that have failed because of intermittent problems in +the test suite, or because of libraries that were not installed at +the time of the previous build. You may not wish to allow strangers +to cause a build to run: in that case, set this to False to remove +these buttons. The default value is False. + + ---------- Footnotes ---------- + + (1) It may even be possible to provide SSL access by using a +specification like +`"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"', but this is +completely untested + |