diff options
Diffstat (limited to 'buildbot/docs/buildbot.html')
| -rw-r--r-- | buildbot/docs/buildbot.html | 9606 |
1 files changed, 9606 insertions, 0 deletions
diff --git a/buildbot/docs/buildbot.html b/buildbot/docs/buildbot.html new file mode 100644 index 0000000..e4d1409 --- /dev/null +++ b/buildbot/docs/buildbot.html @@ -0,0 +1,9606 @@ +<html lang="en"> +<head> +<title>BuildBot Manual 0.7.10</title> +<meta http-equiv="Content-Type" content="text/html"> +<meta name="description" content="BuildBot Manual 0.7.10"> +<meta name="generator" content="makeinfo 4.11"> +<link title="Top" rel="top" href="#Top"> +<link href="http://www.gnu.org/software/texinfo/" rel="generator-home" title="Texinfo Homepage"> +<!-- +This is the BuildBot manual. + +Copyright (C) 2005,2006 Brian Warner + +Copying and distribution of this file, with or without +modification, are permitted in any medium without royalty +provided the copyright notice and this notice are preserved.--> +<meta http-equiv="Content-Style-Type" content="text/css"> +<style type="text/css"><!-- + pre.display { font-family:inherit } + pre.format { font-family:inherit } + pre.smalldisplay { font-family:inherit; font-size:smaller } + pre.smallformat { font-family:inherit; font-size:smaller } + pre.smallexample { font-size:smaller } + pre.smalllisp { font-size:smaller } + span.sc { font-variant:small-caps } + span.roman { font-family:serif; font-weight:normal; } + span.sansserif { font-family:sans-serif; font-weight:normal; } +--></style> +</head> +<body> +<h1 class="settitle">BuildBot Manual 0.7.10</h1> + <div class="contents"> +<h2>Table of Contents</h2> +<ul> +<li><a name="toc_Top" href="#Top">BuildBot</a> +<li><a name="toc_Introduction" href="#Introduction">1 Introduction</a> +<ul> +<li><a href="#History-and-Philosophy">1.1 History and Philosophy</a> +<li><a href="#System-Architecture">1.2 System Architecture</a> +<ul> +<li><a href="#BuildSlave-Connections">1.2.1 BuildSlave Connections</a> +<li><a href="#Buildmaster-Architecture">1.2.2 Buildmaster Architecture</a> +<li><a href="#Status-Delivery-Architecture">1.2.3 Status Delivery Architecture</a> +</li></ul> +<li><a href="#Control-Flow">1.3 Control Flow</a> +</li></ul> +<li><a name="toc_Installation" href="#Installation">2 Installation</a> +<ul> +<li><a href="#Requirements">2.1 Requirements</a> +<li><a href="#Installing-the-code">2.2 Installing the code</a> +<li><a href="#Creating-a-buildmaster">2.3 Creating a buildmaster</a> +<li><a href="#Upgrading-an-Existing-Buildmaster">2.4 Upgrading an Existing Buildmaster</a> +<li><a href="#Creating-a-buildslave">2.5 Creating a buildslave</a> +<ul> +<li><a href="#Buildslave-Options">2.5.1 Buildslave Options</a> +</li></ul> +<li><a href="#Launching-the-daemons">2.6 Launching the daemons</a> +<li><a href="#Logfiles">2.7 Logfiles</a> +<li><a href="#Shutdown">2.8 Shutdown</a> +<li><a href="#Maintenance">2.9 Maintenance</a> +<li><a href="#Troubleshooting">2.10 Troubleshooting</a> +<ul> +<li><a href="#Starting-the-buildslave">2.10.1 Starting the buildslave</a> +<li><a href="#Connecting-to-the-buildmaster">2.10.2 Connecting to the buildmaster</a> +<li><a href="#Forcing-Builds">2.10.3 Forcing Builds</a> +</li></ul> +</li></ul> +<li><a name="toc_Concepts" href="#Concepts">3 Concepts</a> +<ul> +<li><a href="#Version-Control-Systems">3.1 Version Control Systems</a> +<ul> +<li><a href="#Generalizing-VC-Systems">3.1.1 Generalizing VC Systems</a> +<li><a href="#Source-Tree-Specifications">3.1.2 Source Tree Specifications</a> +<li><a href="#How-Different-VC-Systems-Specify-Sources">3.1.3 How Different VC Systems Specify Sources</a> +<li><a href="#Attributes-of-Changes">3.1.4 Attributes of Changes</a> +</li></ul> +<li><a href="#Schedulers">3.2 Schedulers</a> +<li><a href="#BuildSet">3.3 BuildSet</a> +<li><a href="#BuildRequest">3.4 BuildRequest</a> +<li><a href="#Builder">3.5 Builder</a> +<li><a href="#Users">3.6 Users</a> +<ul> +<li><a href="#Doing-Things-With-Users">3.6.1 Doing Things With Users</a> +<li><a href="#Email-Addresses">3.6.2 Email Addresses</a> +<li><a href="#IRC-Nicknames">3.6.3 IRC Nicknames</a> +<li><a href="#Live-Status-Clients">3.6.4 Live Status Clients</a> +</li></ul> +<li><a href="#Build-Properties">3.7 Build Properties</a> +</li></ul> +<li><a name="toc_Configuration" href="#Configuration">4 Configuration</a> +<ul> +<li><a href="#Config-File-Format">4.1 Config File Format</a> +<li><a href="#Loading-the-Config-File">4.2 Loading the Config File</a> +<li><a href="#Testing-the-Config-File">4.3 Testing the Config File</a> +<li><a href="#Defining-the-Project">4.4 Defining the Project</a> +<li><a href="#Change-Sources-and-Schedulers">4.5 Change Sources and Schedulers</a> +<ul> +<li><a href="#Scheduler-Scheduler">4.5.1 Scheduler Scheduler</a> +<li><a href="#AnyBranchScheduler">4.5.2 AnyBranchScheduler</a> +<li><a href="#Dependent-Scheduler">4.5.3 Dependent Scheduler</a> +<li><a href="#Periodic-Scheduler">4.5.4 Periodic Scheduler</a> +<li><a href="#Nightly-Scheduler">4.5.5 Nightly Scheduler</a> +<li><a href="#Try-Schedulers">4.5.6 Try Schedulers</a> +<li><a href="#Triggerable-Scheduler">4.5.7 Triggerable Scheduler</a> +</li></ul> +<li><a href="#Merging-BuildRequests">4.6 Merging BuildRequests</a> +<li><a href="#Setting-the-slaveport">4.7 Setting the slaveport</a> +<li><a href="#Buildslave-Specifiers">4.8 Buildslave Specifiers</a> +<ul> +<li><a href="#When-Buildslaves-Go-Missing">4.8.1 When Buildslaves Go Missing</a> +</li></ul> +<li><a href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">4.9 On-Demand ("Latent") Buildslaves</a> +<ul> +<li><a href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> +<ul> +<li><a href="#Get-an-AWS-EC2-Account">4.9.1.1 Get an AWS EC2 Account</a> +<li><a href="#Create-an-AMI">4.9.1.2 Create an AMI</a> +<li><a href="#Configure-the-Master-with-an-EC2LatentBuildSlave">4.9.1.3 Configure the Master with an EC2LatentBuildSlave</a> +</li></ul> +<li><a href="#Dangers-with-Latent-Buildslaves">4.9.2 Dangers with Latent Buildslaves</a> +<li><a href="#Writing-New-Latent-Buildslaves">4.9.3 Writing New Latent Buildslaves</a> +</li></ul> +<li><a href="#Defining-Global-Properties">4.10 Defining Global Properties</a> +<li><a href="#Defining-Builders">4.11 Defining Builders</a> +<li><a href="#Defining-Status-Targets">4.12 Defining Status Targets</a> +<li><a href="#Debug-options">4.13 Debug options</a> +</li></ul> +<li><a name="toc_Getting-Source-Code-Changes" href="#Getting-Source-Code-Changes">5 Getting Source Code Changes</a> +<ul> +<li><a href="#Change-Sources">5.1 Change Sources</a> +<li><a href="#Choosing-ChangeSources">5.2 Choosing ChangeSources</a> +<li><a href="#CVSToys-_002d-PBService">5.3 CVSToys - PBService</a> +<li><a href="#Mail_002dparsing-ChangeSources">5.4 Mail-parsing ChangeSources</a> +<ul> +<li><a href="#Subscribing-the-Buildmaster">5.4.1 Subscribing the Buildmaster</a> +<li><a href="#Using-Maildirs">5.4.2 Using Maildirs</a> +<li><a href="#Parsing-Email-Change-Messages">5.4.3 Parsing Email Change Messages</a> +<ul> +<li><a href="#FCMaildirSource">5.4.3.1 FCMaildirSource</a> +<li><a href="#SyncmailMaildirSource">5.4.3.2 SyncmailMaildirSource</a> +<li><a href="#BonsaiMaildirSource">5.4.3.3 BonsaiMaildirSource</a> +<li><a href="#SVNCommitEmailMaildirSource">5.4.3.4 SVNCommitEmailMaildirSource</a> +</li></ul> +</li></ul> +<li><a href="#PBChangeSource">5.5 PBChangeSource</a> +<li><a href="#P4Source">5.6 P4Source</a> +<li><a href="#BonsaiPoller">5.7 BonsaiPoller</a> +<li><a href="#SVNPoller">5.8 SVNPoller</a> +<li><a href="#MercurialHook">5.9 MercurialHook</a> +<li><a href="#Bzr-Hook">5.10 Bzr Hook</a> +<li><a href="#Bzr-Poller">5.11 Bzr Poller</a> +</li></ul> +<li><a name="toc_Build-Process" href="#Build-Process">6 Build Process</a> +<ul> +<li><a href="#Build-Steps">6.1 Build Steps</a> +<ul> +<li><a href="#Common-Parameters">6.1.1 Common Parameters</a> +<li><a href="#Using-Build-Properties">6.1.2 Using Build Properties</a> +<li><a href="#Source-Checkout">6.1.3 Source Checkout</a> +<ul> +<li><a href="#CVS">6.1.3.1 CVS</a> +<li><a href="#SVN">6.1.3.2 SVN</a> +<li><a href="#Darcs">6.1.3.3 Darcs</a> +<li><a href="#Mercurial">6.1.3.4 Mercurial</a> +<li><a href="#Arch">6.1.3.5 Arch</a> +<li><a href="#Bazaar">6.1.3.6 Bazaar</a> +<li><a href="#Bzr">6.1.3.7 Bzr</a> +<li><a href="#P4">6.1.3.8 P4</a> +<li><a href="#Git">6.1.3.9 Git</a> +</li></ul> +<li><a href="#ShellCommand">6.1.4 ShellCommand</a> +<li><a href="#Simple-ShellCommand-Subclasses">6.1.5 Simple ShellCommand Subclasses</a> +<ul> +<li><a href="#Configure">6.1.5.1 Configure</a> +<li><a href="#Compile">6.1.5.2 Compile</a> +<li><a href="#Test">6.1.5.3 Test</a> +<li><a href="#TreeSize">6.1.5.4 TreeSize</a> +<li><a href="#PerlModuleTest">6.1.5.5 PerlModuleTest</a> +<li><a href="#SetProperty">6.1.5.6 SetProperty</a> +</li></ul> +<li><a href="#Python-BuildSteps">6.1.6 Python BuildSteps</a> +<ul> +<li><a href="#BuildEPYDoc">6.1.6.1 BuildEPYDoc</a> +<li><a href="#PyFlakes">6.1.6.2 PyFlakes</a> +<li><a href="#PyLint">6.1.6.3 PyLint</a> +</li></ul> +<li><a href="#Transferring-Files">6.1.7 Transferring Files</a> +<li><a href="#Steps-That-Run-on-the-Master">6.1.8 Steps That Run on the Master</a> +<li><a href="#Triggering-Schedulers">6.1.9 Triggering Schedulers</a> +<li><a href="#Writing-New-BuildSteps">6.1.10 Writing New BuildSteps</a> +<ul> +<li><a href="#Writing-BuildStep-Constructors">6.1.10.1 Writing BuildStep Constructors</a> +<li><a href="#BuildStep-LogFiles">6.1.10.2 BuildStep LogFiles</a> +<li><a href="#Reading-Logfiles">6.1.10.3 Reading Logfiles</a> +<li><a href="#Adding-LogObservers">6.1.10.4 Adding LogObservers</a> +<li><a href="#BuildStep-URLs">6.1.10.5 BuildStep URLs</a> +</li></ul> +</li></ul> +<li><a href="#Interlocks">6.2 Interlocks</a> +<li><a href="#Build-Factories">6.3 Build Factories</a> +<ul> +<li><a href="#BuildStep-Objects">6.3.1 BuildStep Objects</a> +<li><a href="#BuildFactory">6.3.2 BuildFactory</a> +<ul> +<li><a href="#BuildFactory-Attributes">6.3.2.1 BuildFactory Attributes</a> +<li><a href="#Quick-builds">6.3.2.2 Quick builds</a> +</li></ul> +<li><a href="#Process_002dSpecific-build-factories">6.3.3 Process-Specific build factories</a> +<ul> +<li><a href="#GNUAutoconf">6.3.3.1 GNUAutoconf</a> +<li><a href="#CPAN">6.3.3.2 CPAN</a> +<li><a href="#Python-distutils">6.3.3.3 Python distutils</a> +<li><a href="#Python_002fTwisted_002ftrial-projects">6.3.3.4 Python/Twisted/trial projects</a> +</li></ul> +</li></ul> +</li></ul> +<li><a name="toc_Status-Delivery" href="#Status-Delivery">7 Status Delivery</a> +<ul> +<li><a href="#WebStatus">7.1 WebStatus</a> +<ul> +<li><a href="#WebStatus-Configuration-Parameters">7.1.1 WebStatus Configuration Parameters</a> +<li><a href="#Buildbot-Web-Resources">7.1.2 Buildbot Web Resources</a> +<li><a href="#XMLRPC-server">7.1.3 XMLRPC server</a> +<li><a href="#HTML-Waterfall">7.1.4 HTML Waterfall</a> +</li></ul> +<li><a href="#MailNotifier">7.2 MailNotifier</a> +<li><a href="#IRC-Bot">7.3 IRC Bot</a> +<li><a href="#PBListener">7.4 PBListener</a> +<li><a href="#Writing-New-Status-Plugins">7.5 Writing New Status Plugins</a> +</li></ul> +<li><a name="toc_Command_002dline-tool" href="#Command_002dline-tool">8 Command-line tool</a> +<ul> +<li><a href="#Administrator-Tools">8.1 Administrator Tools</a> +<li><a href="#Developer-Tools">8.2 Developer Tools</a> +<ul> +<li><a href="#statuslog">8.2.1 statuslog</a> +<li><a href="#statusgui">8.2.2 statusgui</a> +<li><a href="#try">8.2.3 try</a> +<ul> +<li><a href="#try-_002d_002ddiff">8.2.3.1 try –diff</a> +</li></ul> +</li></ul> +<li><a href="#Other-Tools">8.3 Other Tools</a> +<ul> +<li><a href="#sendchange">8.3.1 sendchange</a> +<li><a href="#debugclient">8.3.2 debugclient</a> +</li></ul> +<li><a href="#_002ebuildbot-config-directory">8.4 .buildbot config directory</a> +</li></ul> +<li><a name="toc_Resources" href="#Resources">9 Resources</a> +<li><a name="toc_Developer_0027s-Appendix" href="#Developer_0027s-Appendix">Developer's Appendix</a> +<li><a name="toc_Index-of-Useful-Classes" href="#Index-of-Useful-Classes">Index of Useful Classes</a> +<li><a name="toc_Index-of-master_002ecfg-keys" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a> +<li><a name="toc_Index" href="#Index">Index</a> +</li></ul> +</div> + + + +<div class="node"> +<p><hr> +<a name="Top"></a> +Next: <a rel="next" accesskey="n" href="#Introduction">Introduction</a>, +Previous: <a rel="previous" accesskey="p" href="#dir">(dir)</a>, +Up: <a rel="up" accesskey="u" href="#dir">(dir)</a> + +</div> + +<h2 class="unnumbered">BuildBot</h2> + +<p>This is the BuildBot manual. + + <p>Copyright (C) 2005,2006 Brian Warner + + <p>Copying and distribution of this file, with or without +modification, are permitted in any medium without royalty +provided the copyright notice and this notice are preserved. + +<ul class="menu"> +<li><a accesskey="1" href="#Introduction">Introduction</a>: What the BuildBot does. +<li><a accesskey="2" href="#Installation">Installation</a>: Creating a buildmaster and buildslaves, + running them. +<li><a accesskey="3" href="#Concepts">Concepts</a>: What goes on in the buildbot's little mind. +<li><a accesskey="4" href="#Configuration">Configuration</a>: Controlling the buildbot. +<li><a accesskey="5" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>: Discovering when to run a build. +<li><a accesskey="6" href="#Build-Process">Build Process</a>: Controlling how each build is run. +<li><a accesskey="7" href="#Status-Delivery">Status Delivery</a>: Telling the world about the build's results. +<li><a accesskey="8" href="#Command_002dline-tool">Command-line tool</a> +<li><a accesskey="9" href="#Resources">Resources</a>: Getting help. +<li><a href="#Developer_0027s-Appendix">Developer's Appendix</a> +<li><a href="#Index-of-Useful-Classes">Index of Useful Classes</a> +<li><a href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a> +<li><a href="#Index">Index</a>: Complete index. + +</li></ul> +<p>--- The Detailed Node Listing --- + +<p>Introduction + +</p> +<ul class="menu"> +<li><a href="#History-and-Philosophy">History and Philosophy</a> +<li><a href="#System-Architecture">System Architecture</a> +<li><a href="#Control-Flow">Control Flow</a> + +</li></ul> +<p>System Architecture + +</p> +<ul class="menu"> +<li><a href="#BuildSlave-Connections">BuildSlave Connections</a> +<li><a href="#Buildmaster-Architecture">Buildmaster Architecture</a> +<li><a href="#Status-Delivery-Architecture">Status Delivery Architecture</a> + +</li></ul> +<p>Installation + +</p> +<ul class="menu"> +<li><a href="#Requirements">Requirements</a> +<li><a href="#Installing-the-code">Installing the code</a> +<li><a href="#Creating-a-buildmaster">Creating a buildmaster</a> +<li><a href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a> +<li><a href="#Creating-a-buildslave">Creating a buildslave</a> +<li><a href="#Launching-the-daemons">Launching the daemons</a> +<li><a href="#Logfiles">Logfiles</a> +<li><a href="#Shutdown">Shutdown</a> +<li><a href="#Maintenance">Maintenance</a> +<li><a href="#Troubleshooting">Troubleshooting</a> + +</li></ul> +<p>Creating a buildslave + +</p> +<ul class="menu"> +<li><a href="#Buildslave-Options">Buildslave Options</a> + +</li></ul> +<p>Troubleshooting + +</p> +<ul class="menu"> +<li><a href="#Starting-the-buildslave">Starting the buildslave</a> +<li><a href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a> +<li><a href="#Forcing-Builds">Forcing Builds</a> + +</li></ul> +<p>Concepts + +</p> +<ul class="menu"> +<li><a href="#Version-Control-Systems">Version Control Systems</a> +<li><a href="#Schedulers">Schedulers</a> +<li><a href="#BuildSet">BuildSet</a> +<li><a href="#BuildRequest">BuildRequest</a> +<li><a href="#Builder">Builder</a> +<li><a href="#Users">Users</a> +<li><a href="#Build-Properties">Build Properties</a> + +</li></ul> +<p>Version Control Systems + +</p> +<ul class="menu"> +<li><a href="#Generalizing-VC-Systems">Generalizing VC Systems</a> +<li><a href="#Source-Tree-Specifications">Source Tree Specifications</a> +<li><a href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a> +<li><a href="#Attributes-of-Changes">Attributes of Changes</a> + +</li></ul> +<p>Users + +</p> +<ul class="menu"> +<li><a href="#Doing-Things-With-Users">Doing Things With Users</a> +<li><a href="#Email-Addresses">Email Addresses</a> +<li><a href="#IRC-Nicknames">IRC Nicknames</a> +<li><a href="#Live-Status-Clients">Live Status Clients</a> + +</li></ul> +<p>Configuration + +</p> +<ul class="menu"> +<li><a href="#Config-File-Format">Config File Format</a> +<li><a href="#Loading-the-Config-File">Loading the Config File</a> +<li><a href="#Testing-the-Config-File">Testing the Config File</a> +<li><a href="#Defining-the-Project">Defining the Project</a> +<li><a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> +<li><a href="#Setting-the-slaveport">Setting the slaveport</a> +<li><a href="#Buildslave-Specifiers">Buildslave Specifiers</a> +<li><a href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> +<li><a href="#Defining-Global-Properties">Defining Global Properties</a> +<li><a href="#Defining-Builders">Defining Builders</a> +<li><a href="#Defining-Status-Targets">Defining Status Targets</a> +<li><a href="#Debug-options">Debug options</a> + +</li></ul> +<p>Change Sources and Schedulers + +</p> +<ul class="menu"> +<li><a href="#Scheduler-Scheduler">Scheduler Scheduler</a> +<li><a href="#AnyBranchScheduler">AnyBranchScheduler</a> +<li><a href="#Dependent-Scheduler">Dependent Scheduler</a> +<li><a href="#Periodic-Scheduler">Periodic Scheduler</a> +<li><a href="#Nightly-Scheduler">Nightly Scheduler</a> +<li><a href="#Try-Schedulers">Try Schedulers</a> +<li><a href="#Triggerable-Scheduler">Triggerable Scheduler</a> + +</li></ul> +<p>Buildslave Specifiers +</p> +<ul class="menu"> +<li><a href="#When-Buildslaves-Go-Missing">When Buildslaves Go Missing</a> + +</li></ul> +<p>On-Demand ("Latent") Buildslaves +</p> +<ul class="menu"> +<li><a href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> +<li><a href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a> +<li><a href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a> + +</li></ul> +<p>Getting Source Code Changes + +</p> +<ul class="menu"> +<li><a href="#Change-Sources">Change Sources</a> +<li><a href="#Choosing-ChangeSources">Choosing ChangeSources</a> +<li><a href="#CVSToys-_002d-PBService">CVSToys - PBService</a> +<li><a href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> +<li><a href="#PBChangeSource">PBChangeSource</a> +<li><a href="#P4Source">P4Source</a> +<li><a href="#BonsaiPoller">BonsaiPoller</a> +<li><a href="#SVNPoller">SVNPoller</a> +<li><a href="#MercurialHook">MercurialHook</a> +<li><a href="#Bzr-Hook">Bzr Hook</a> +<li><a href="#Bzr-Poller">Bzr Poller</a> + +</li></ul> +<p>Mail-parsing ChangeSources + +</p> +<ul class="menu"> +<li><a href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a> +<li><a href="#Using-Maildirs">Using Maildirs</a> +<li><a href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> + +</li></ul> +<p>Parsing Email Change Messages + +</p> +<ul class="menu"> +<li><a href="#FCMaildirSource">FCMaildirSource</a> +<li><a href="#SyncmailMaildirSource">SyncmailMaildirSource</a> +<li><a href="#BonsaiMaildirSource">BonsaiMaildirSource</a> +<li><a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> + +</li></ul> +<p>Build Process + +</p> +<ul class="menu"> +<li><a href="#Build-Steps">Build Steps</a> +<li><a href="#Interlocks">Interlocks</a> +<li><a href="#Build-Factories">Build Factories</a> + +</li></ul> +<p>Build Steps + +</p> +<ul class="menu"> +<li><a href="#Common-Parameters">Common Parameters</a> +<li><a href="#Using-Build-Properties">Using Build Properties</a> +<li><a href="#Source-Checkout">Source Checkout</a> +<li><a href="#ShellCommand">ShellCommand</a> +<li><a href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> +<li><a href="#Python-BuildSteps">Python BuildSteps</a> +<li><a href="#Transferring-Files">Transferring Files</a> +<li><a href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a> +<li><a href="#Triggering-Schedulers">Triggering Schedulers</a> +<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</li></ul> +<p>Source Checkout + +</p> +<ul class="menu"> +<li><a href="#CVS">CVS</a> +<li><a href="#SVN">SVN</a> +<li><a href="#Darcs">Darcs</a> +<li><a href="#Mercurial">Mercurial</a> +<li><a href="#Arch">Arch</a> +<li><a href="#Bazaar">Bazaar</a> +<li><a href="#Bzr">Bzr</a> +<li><a href="#P4">P4</a> +<li><a href="#Git">Git</a> + +</li></ul> +<p>Simple ShellCommand Subclasses + +</p> +<ul class="menu"> +<li><a href="#Configure">Configure</a> +<li><a href="#Compile">Compile</a> +<li><a href="#Test">Test</a> +<li><a href="#TreeSize">TreeSize</a> +<li><a href="#PerlModuleTest">PerlModuleTest</a> +<li><a href="#SetProperty">SetProperty</a> + +</li></ul> +<p>Python BuildSteps + +</p> +<ul class="menu"> +<li><a href="#BuildEPYDoc">BuildEPYDoc</a> +<li><a href="#PyFlakes">PyFlakes</a> +<li><a href="#PyLint">PyLint</a> + +</li></ul> +<p>Writing New BuildSteps + +</p> +<ul class="menu"> +<li><a href="#BuildStep-LogFiles">BuildStep LogFiles</a> +<li><a href="#Reading-Logfiles">Reading Logfiles</a> +<li><a href="#Adding-LogObservers">Adding LogObservers</a> +<li><a href="#BuildStep-URLs">BuildStep URLs</a> + +</li></ul> +<p>Build Factories + +</p> +<ul class="menu"> +<li><a href="#BuildStep-Objects">BuildStep Objects</a> +<li><a href="#BuildFactory">BuildFactory</a> +<li><a href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> + +</li></ul> +<p>BuildStep Objects + +</p> +<ul class="menu"> +<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a> +<li><a href="#Quick-builds">Quick builds</a> + +</li></ul> +<p>BuildFactory + +</p> +<ul class="menu"> +<li><a href="#BuildFactory-Attributes">BuildFactory Attributes</a> +<li><a href="#Quick-builds">Quick builds</a> + +</li></ul> +<p>Process-Specific build factories + +</p> +<ul class="menu"> +<li><a href="#GNUAutoconf">GNUAutoconf</a> +<li><a href="#CPAN">CPAN</a> +<li><a href="#Python-distutils">Python distutils</a> +<li><a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a> + +</li></ul> +<p>Status Delivery + +</p> +<ul class="menu"> +<li><a href="#WebStatus">WebStatus</a> +<li><a href="#MailNotifier">MailNotifier</a> +<li><a href="#IRC-Bot">IRC Bot</a> +<li><a href="#PBListener">PBListener</a> +<li><a href="#Writing-New-Status-Plugins">Writing New Status Plugins</a> + +</li></ul> +<p>WebStatus + +</p> +<ul class="menu"> +<li><a href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a> +<li><a href="#Buildbot-Web-Resources">Buildbot Web Resources</a> +<li><a href="#XMLRPC-server">XMLRPC server</a> +<li><a href="#HTML-Waterfall">HTML Waterfall</a> + +</li></ul> +<p>Command-line tool + +</p> +<ul class="menu"> +<li><a href="#Administrator-Tools">Administrator Tools</a> +<li><a href="#Developer-Tools">Developer Tools</a> +<li><a href="#Other-Tools">Other Tools</a> +<li><a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a> + +</li></ul> +<p>Developer Tools + +</p> +<ul class="menu"> +<li><a href="#statuslog">statuslog</a> +<li><a href="#statusgui">statusgui</a> +<li><a href="#try">try</a> + +</li></ul> +<p>waiting for results + +</p> +<ul class="menu"> +<li><a href="#try-_002d_002ddiff">try --diff</a> + +</li></ul> +<p>Other Tools + +</p> +<ul class="menu"> +<li><a href="#sendchange">sendchange</a> +<li><a href="#debugclient">debugclient</a> + + </ul> + +<div class="node"> +<p><hr> +<a name="Introduction"></a> +Next: <a rel="next" accesskey="n" href="#Installation">Installation</a>, +Previous: <a rel="previous" accesskey="p" href="#Top">Top</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">1 Introduction</h2> + +<p><a name="index-introduction-1"></a> +The BuildBot is a system to automate the compile/test cycle required by most +software projects to validate code changes. By automatically rebuilding and +testing the tree each time something has changed, build problems are +pinpointed quickly, before other developers are inconvenienced by the +failure. The guilty developer can be identified and harassed without human +intervention. By running the builds on a variety of platforms, developers +who do not have the facilities to test their changes everywhere before +checkin will at least know shortly afterwards whether they have broken the +build or not. Warning counts, lint checks, image size, compile time, and +other build parameters can be tracked over time, are more visible, and +are therefore easier to improve. + + <p>The overall goal is to reduce tree breakage and provide a platform to +run tests or code-quality checks that are too annoying or pedantic for +any human to waste their time with. Developers get immediate (and +potentially public) feedback about their changes, encouraging them to +be more careful about testing before checkin. + + <p>Features: + + <ul> +<li>run builds on a variety of slave platforms +<li>arbitrary build process: handles projects using C, Python, whatever +<li>minimal host requirements: python and Twisted +<li>slaves can be behind a firewall if they can still do checkout +<li>status delivery through web page, email, IRC, other protocols +<li>track builds in progress, provide estimated completion time +<li>flexible configuration by subclassing generic build process classes +<li>debug tools to force a new build, submit fake Changes, query slave status +<li>released under the GPL +</ul> + +<ul class="menu"> +<li><a accesskey="1" href="#History-and-Philosophy">History and Philosophy</a> +<li><a accesskey="2" href="#System-Architecture">System Architecture</a> +<li><a accesskey="3" href="#Control-Flow">Control Flow</a> +</ul> + +<div class="node"> +<p><hr> +<a name="History-and-Philosophy"></a> +Next: <a rel="next" accesskey="n" href="#System-Architecture">System Architecture</a>, +Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>, +Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> + +</div> + +<h3 class="section">1.1 History and Philosophy</h3> + +<p><a name="index-Philosophy-of-operation-2"></a> +The Buildbot was inspired by a similar project built for a development +team writing a cross-platform embedded system. The various components +of the project were supposed to compile and run on several flavors of +unix (linux, solaris, BSD), but individual developers had their own +preferences and tended to stick to a single platform. From time to +time, incompatibilities would sneak in (some unix platforms want to +use <code>string.h</code>, some prefer <code>strings.h</code>), and then the tree +would compile for some developers but not others. The buildbot was +written to automate the human process of walking into the office, +updating a tree, compiling (and discovering the breakage), finding the +developer at fault, and complaining to them about the problem they had +introduced. With multiple platforms it was difficult for developers to +do the right thing (compile their potential change on all platforms); +the buildbot offered a way to help. + + <p>Another problem was when programmers would change the behavior of a +library without warning its users, or change internal aspects that +other code was (unfortunately) depending upon. Adding unit tests to +the codebase helps here: if an application's unit tests pass despite +changes in the libraries it uses, you can have more confidence that +the library changes haven't broken anything. Many developers +complained that the unit tests were inconvenient or took too long to +run: having the buildbot run them reduces the developer's workload to +a minimum. + + <p>In general, having more visibility into the project is always good, +and automation makes it easier for developers to do the right thing. +When everyone can see the status of the project, developers are +encouraged to keep the tree in good working order. Unit tests that +aren't run on a regular basis tend to suffer from bitrot just like +code does: exercising them on a regular basis helps to keep them +functioning and useful. + + <p>The current version of the Buildbot is additionally targeted at +distributed free-software projects, where resources and platforms are +only available when provided by interested volunteers. The buildslaves +are designed to require an absolute minimum of configuration, reducing +the effort a potential volunteer needs to expend to be able to +contribute a new test environment to the project. The goal is for +anyone who wishes that a given project would run on their favorite +platform should be able to offer that project a buildslave, running on +that platform, where they can verify that their portability code +works, and keeps working. + +<div class="node"> +<p><hr> +<a name="System-Architecture"></a> +Next: <a rel="next" accesskey="n" href="#Control-Flow">Control Flow</a>, +Previous: <a rel="previous" accesskey="p" href="#History-and-Philosophy">History and Philosophy</a>, +Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> + +</div> + +<!-- node-name, next, previous, up --> +<h3 class="section">1.2 System Architecture</h3> + +<p>The Buildbot consists of a single <code>buildmaster</code> and one or more +<code>buildslaves</code>, connected in a star topology. The buildmaster +makes all decisions about what, when, and how to build. It sends +commands to be run on the build slaves, which simply execute the +commands and return the results. (certain steps involve more local +decision making, where the overhead of sending a lot of commands back +and forth would be inappropriate, but in general the buildmaster is +responsible for everything). + + <p>The buildmaster is usually fed <code>Changes</code> by some sort of version +control system (see <a href="#Change-Sources">Change Sources</a>), which may cause builds to be +run. As the builds are performed, various status messages are +produced, which are then sent to any registered Status Targets +(see <a href="#Status-Delivery">Status Delivery</a>). + +<!-- @image{FILENAME, WIDTH, HEIGHT, ALTTEXT, EXTENSION} --> + <div class="block-image"><img src="images/overview.png" alt="Overview Diagram"></div> + + <p>The buildmaster is configured and maintained by the “buildmaster +admin”, who is generally the project team member responsible for +build process issues. Each buildslave is maintained by a “buildslave +admin”, who do not need to be quite as involved. Generally slaves are +run by anyone who has an interest in seeing the project work well on +their favorite platform. + +<ul class="menu"> +<li><a accesskey="1" href="#BuildSlave-Connections">BuildSlave Connections</a> +<li><a accesskey="2" href="#Buildmaster-Architecture">Buildmaster Architecture</a> +<li><a accesskey="3" href="#Status-Delivery-Architecture">Status Delivery Architecture</a> +</ul> + +<div class="node"> +<p><hr> +<a name="BuildSlave-Connections"></a> +Next: <a rel="next" accesskey="n" href="#Buildmaster-Architecture">Buildmaster Architecture</a>, +Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>, +Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> + +</div> + +<h4 class="subsection">1.2.1 BuildSlave Connections</h4> + +<p>The buildslaves are typically run on a variety of separate machines, +at least one per platform of interest. These machines connect to the +buildmaster over a TCP connection to a publically-visible port. As a +result, the buildslaves can live behind a NAT box or similar +firewalls, as long as they can get to buildmaster. The TCP connections +are initiated by the buildslave and accepted by the buildmaster, but +commands and results travel both ways within this connection. The +buildmaster is always in charge, so all commands travel exclusively +from the buildmaster to the buildslave. + + <p>To perform builds, the buildslaves must typically obtain source code +from a CVS/SVN/etc repository. Therefore they must also be able to +reach the repository. The buildmaster provides instructions for +performing builds, but does not provide the source code itself. + + <div class="block-image"><img src="images/slaves.png" alt="BuildSlave Connections"></div> + +<div class="node"> +<p><hr> +<a name="Buildmaster-Architecture"></a> +Next: <a rel="next" accesskey="n" href="#Status-Delivery-Architecture">Status Delivery Architecture</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildSlave-Connections">BuildSlave Connections</a>, +Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> + +</div> + +<h4 class="subsection">1.2.2 Buildmaster Architecture</h4> + +<p>The Buildmaster consists of several pieces: + + <div class="block-image"><img src="images/master.png" alt="BuildMaster Architecture"></div> + + <ul> +<li>Change Sources, which create a Change object each time something is +modified in the VC repository. Most ChangeSources listen for messages +from a hook script of some sort. Some sources actively poll the +repository on a regular basis. All Changes are fed to the Schedulers. + + <li>Schedulers, which decide when builds should be performed. They collect +Changes into BuildRequests, which are then queued for delivery to +Builders until a buildslave is available. + + <li>Builders, which control exactly <em>how</em> each build is performed +(with a series of BuildSteps, configured in a BuildFactory). Each +Build is run on a single buildslave. + + <li>Status plugins, which deliver information about the build results +through protocols like HTTP, mail, and IRC. + + </ul> + + <div class="block-image"><img src="images/slavebuilder.png" alt="SlaveBuilders"></div> + + <p>Each Builder is configured with a list of BuildSlaves that it will use +for its builds. These buildslaves are expected to behave identically: +the only reason to use multiple BuildSlaves for a single Builder is to +provide a measure of load-balancing. + + <p>Within a single BuildSlave, each Builder creates its own SlaveBuilder +instance. These SlaveBuilders operate independently from each other. +Each gets its own base directory to work in. It is quite common to +have many Builders sharing the same buildslave. For example, there +might be two buildslaves: one for i386, and a second for PowerPC. +There may then be a pair of Builders that do a full compile/test run, +one for each architecture, and a lone Builder that creates snapshot +source tarballs if the full builders complete successfully. The full +builders would each run on a single buildslave, whereas the tarball +creation step might run on either buildslave (since the platform +doesn't matter when creating source tarballs). In this case, the +mapping would look like: + +<pre class="example"> Builder(full-i386) -> BuildSlaves(slave-i386) + Builder(full-ppc) -> BuildSlaves(slave-ppc) + Builder(source-tarball) -> BuildSlaves(slave-i386, slave-ppc) +</pre> + <p>and each BuildSlave would have two SlaveBuilders inside it, one for a +full builder, and a second for the source-tarball builder. + + <p>Once a SlaveBuilder is available, the Builder pulls one or more +BuildRequests off its incoming queue. (It may pull more than one if it +determines that it can merge the requests together; for example, there +may be multiple requests to build the current HEAD revision). These +requests are merged into a single Build instance, which includes the +SourceStamp that describes what exact version of the source code +should be used for the build. The Build is then randomly assigned to a +free SlaveBuilder and the build begins. + + <p>The behaviour when BuildRequests are merged can be customized, see <a href="#Merging-BuildRequests">Merging BuildRequests</a>. + +<div class="node"> +<p><hr> +<a name="Status-Delivery-Architecture"></a> +Previous: <a rel="previous" accesskey="p" href="#Buildmaster-Architecture">Buildmaster Architecture</a>, +Up: <a rel="up" accesskey="u" href="#System-Architecture">System Architecture</a> + +</div> + +<h4 class="subsection">1.2.3 Status Delivery Architecture</h4> + +<p>The buildmaster maintains a central Status object, to which various +status plugins are connected. Through this Status object, a full +hierarchy of build status objects can be obtained. + + <div class="block-image"><img src="images/status.png" alt="Status Delivery"></div> + + <p>The configuration file controls which status plugins are active. Each +status plugin gets a reference to the top-level Status object. From +there they can request information on each Builder, Build, Step, and +LogFile. This query-on-demand interface is used by the html.Waterfall +plugin to create the main status page each time a web browser hits the +main URL. + + <p>The status plugins can also subscribe to hear about new Builds as they +occur: this is used by the MailNotifier to create new email messages +for each recently-completed Build. + + <p>The Status object records the status of old builds on disk in the +buildmaster's base directory. This allows it to return information +about historical builds. + + <p>There are also status objects that correspond to Schedulers and +BuildSlaves. These allow status plugins to report information about +upcoming builds, and the online/offline status of each buildslave. + +<div class="node"> +<p><hr> +<a name="Control-Flow"></a> +Previous: <a rel="previous" accesskey="p" href="#System-Architecture">System Architecture</a>, +Up: <a rel="up" accesskey="u" href="#Introduction">Introduction</a> + +</div> + +<!-- node-name, next, previous, up --> +<h3 class="section">1.3 Control Flow</h3> + +<p>A day in the life of the buildbot: + + <ul> +<li>A developer commits some source code changes to the repository. A hook +script or commit trigger of some sort sends information about this +change to the buildmaster through one of its configured Change +Sources. This notification might arrive via email, or over a network +connection (either initiated by the buildmaster as it “subscribes” +to changes, or by the commit trigger as it pushes Changes towards the +buildmaster). The Change contains information about who made the +change, what files were modified, which revision contains the change, +and any checkin comments. + + <li>The buildmaster distributes this change to all of its configured +Schedulers. Any “important” changes cause the “tree-stable-timer” +to be started, and the Change is added to a list of those that will go +into a new Build. When the timer expires, a Build is started on each +of a set of configured Builders, all compiling/testing the same source +code. Unless configured otherwise, all Builds run in parallel on the +various buildslaves. + + <li>The Build consists of a series of Steps. Each Step causes some number +of commands to be invoked on the remote buildslave associated with +that Builder. The first step is almost always to perform a checkout of +the appropriate revision from the same VC system that produced the +Change. The rest generally perform a compile and run unit tests. As +each Step runs, the buildslave reports back command output and return +status to the buildmaster. + + <li>As the Build runs, status messages like “Build Started”, “Step +Started”, “Build Finished”, etc, are published to a collection of +Status Targets. One of these targets is usually the HTML “Waterfall” +display, which shows a chronological list of events, and summarizes +the results of the most recent build at the top of each column. +Developers can periodically check this page to see how their changes +have fared. If they see red, they know that they've made a mistake and +need to fix it. If they see green, they know that they've done their +duty and don't need to worry about their change breaking anything. + + <li>If a MailNotifier status target is active, the completion of a build +will cause email to be sent to any developers whose Changes were +incorporated into this Build. The MailNotifier can be configured to +only send mail upon failing builds, or for builds which have just +transitioned from passing to failing. Other status targets can provide +similar real-time notification via different communication channels, +like IRC. + + </ul> + +<div class="node"> +<p><hr> +<a name="Installation"></a> +Next: <a rel="next" accesskey="n" href="#Concepts">Concepts</a>, +Previous: <a rel="previous" accesskey="p" href="#Introduction">Introduction</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">2 Installation</h2> + +<ul class="menu"> +<li><a accesskey="1" href="#Requirements">Requirements</a> +<li><a accesskey="2" href="#Installing-the-code">Installing the code</a> +<li><a accesskey="3" href="#Creating-a-buildmaster">Creating a buildmaster</a> +<li><a accesskey="4" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a> +<li><a accesskey="5" href="#Creating-a-buildslave">Creating a buildslave</a> +<li><a accesskey="6" href="#Launching-the-daemons">Launching the daemons</a> +<li><a accesskey="7" href="#Logfiles">Logfiles</a> +<li><a accesskey="8" href="#Shutdown">Shutdown</a> +<li><a accesskey="9" href="#Maintenance">Maintenance</a> +<li><a href="#Troubleshooting">Troubleshooting</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Requirements"></a> +Next: <a rel="next" accesskey="n" href="#Installing-the-code">Installing the code</a>, +Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.1 Requirements</h3> + +<p>At a bare minimum, you'll need the following (for both the buildmaster +and a buildslave): + + <ul> +<li>Python: http://www.python.org + + <p>Buildbot requires python-2.3 or later, and is primarily developed +against python-2.4. It is also tested against python-2.5 . + + <li>Twisted: http://twistedmatrix.com + + <p>Both the buildmaster and the buildslaves require Twisted-2.0.x or +later. It has been tested against all releases of Twisted up to +Twisted-2.5.0 (the most recent as of this writing). As always, the +most recent version is recommended. + + <p>Twisted is delivered as a collection of subpackages. You'll need at +least "Twisted" (the core package), and you'll also want TwistedMail, +TwistedWeb, and TwistedWords (for sending email, serving a web status +page, and delivering build status via IRC, respectively). You might +also want TwistedConch (for the encrypted Manhole debug port). Note +that Twisted requires ZopeInterface to be installed as well. + + </ul> + + <p>Certain other packages may be useful on the system running the +buildmaster: + + <ul> +<li>CVSToys: http://purl.net/net/CVSToys + + <p>If your buildmaster uses FreshCVSSource to receive change notification +from a cvstoys daemon, it will require CVSToys be installed (tested +with CVSToys-1.0.10). If the it doesn't use that source (i.e. if you +only use a mail-parsing change source, or the SVN notification +script), you will not need CVSToys. + + </ul> + + <p>And of course, your project's build process will impose additional +requirements on the buildslaves. These hosts must have all the tools +necessary to compile and test your project's source code. + +<div class="node"> +<p><hr> +<a name="Installing-the-code"></a> +Next: <a rel="next" accesskey="n" href="#Creating-a-buildmaster">Creating a buildmaster</a>, +Previous: <a rel="previous" accesskey="p" href="#Requirements">Requirements</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.2 Installing the code</h3> + +<p><a name="index-installation-3"></a> +The Buildbot is installed using the standard python <code>distutils</code> +module. After unpacking the tarball, the process is: + +<pre class="example"> python setup.py build + python setup.py install +</pre> + <p>where the install step may need to be done as root. This will put the +bulk of the code in somewhere like +/usr/lib/python2.3/site-packages/buildbot . It will also install the +<code>buildbot</code> command-line tool in /usr/bin/buildbot. + + <p>To test this, shift to a different directory (like /tmp), and run: + +<pre class="example"> buildbot --version +</pre> + <p>If it shows you the versions of Buildbot and Twisted, the install went +ok. If it says <code>no such command</code> or it gets an <code>ImportError</code> +when it tries to load the libaries, then something went wrong. +<code>pydoc buildbot</code> is another useful diagnostic tool. + + <p>Windows users will find these files in other places. You will need to +make sure that python can find the libraries, and will probably find +it convenient to have <code>buildbot</code> on your PATH. + + <p>If you wish, you can run the buildbot unit test suite like this: + +<pre class="example"> PYTHONPATH=. trial buildbot.test +</pre> + <p>This should run up to 192 tests, depending upon what VC tools you have +installed. On my desktop machine it takes about five minutes to +complete. Nothing should fail, a few might be skipped. If any of the +tests fail, you should stop and investigate the cause before +continuing the installation process, as it will probably be easier to +track down the bug early. + + <p>If you cannot or do not wish to install the buildbot into a site-wide +location like <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp>, you can also install +it into the account's home directory. Do the install command like +this: + +<pre class="example"> python setup.py install --home=~ +</pre> + <p>That will populate <samp><span class="file">~/lib/python</span></samp> and create +<samp><span class="file">~/bin/buildbot</span></samp>. Make sure this lib directory is on your +<code>PYTHONPATH</code>. + +<div class="node"> +<p><hr> +<a name="Creating-a-buildmaster"></a> +Next: <a rel="next" accesskey="n" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a>, +Previous: <a rel="previous" accesskey="p" href="#Installing-the-code">Installing the code</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.3 Creating a buildmaster</h3> + +<p>As you learned earlier (see <a href="#System-Architecture">System Architecture</a>), the buildmaster +runs on a central host (usually one that is publically visible, so +everybody can check on the status of the project), and controls all +aspects of the buildbot system. Let us call this host +<code>buildbot.example.org</code>. + + <p>You may wish to create a separate user account for the buildmaster, +perhaps named <code>buildmaster</code>. This can help keep your personal +configuration distinct from that of the buildmaster and is useful if +you have to use a mail-based notification system (see <a href="#Change-Sources">Change Sources</a>). However, the Buildbot will work just fine with your regular +user account. + + <p>You need to choose a directory for the buildmaster, called the +<code>basedir</code>. This directory will be owned by the buildmaster, which +will use configuration files therein, and create status files as it +runs. <samp><span class="file">~/Buildbot</span></samp> is a likely value. If you run multiple +buildmasters in the same account, or if you run both masters and +slaves, you may want a more distinctive name like +<samp><span class="file">~/Buildbot/master/gnomovision</span></samp> or +<samp><span class="file">~/Buildmasters/fooproject</span></samp>. If you are using a separate user +account, this might just be <samp><span class="file">~buildmaster/masters/fooproject</span></samp>. + + <p>Once you've picked a directory, use the <samp><span class="command">buildbot +create-master</span></samp> command to create the directory and populate it with +startup files: + +<pre class="example"> buildbot create-master <var>basedir</var> +</pre> + <p>You will need to create a configuration file (see <a href="#Configuration">Configuration</a>) +before starting the buildmaster. Most of the rest of this manual is +dedicated to explaining how to do this. A sample configuration file is +placed in the working directory, named <samp><span class="file">master.cfg.sample</span></samp>, which +can be copied to <samp><span class="file">master.cfg</span></samp> and edited to suit your purposes. + + <p>(Internal details: This command creates a file named +<samp><span class="file">buildbot.tac</span></samp> that contains all the state necessary to create +the buildmaster. Twisted has a tool called <code>twistd</code> which can use +this .tac file to create and launch a buildmaster instance. twistd +takes care of logging and daemonization (running the program in the +background). <samp><span class="file">/usr/bin/buildbot</span></samp> is a front end which runs twistd +for you.) + + <p>In addition to <samp><span class="file">buildbot.tac</span></samp>, a small <samp><span class="file">Makefile.sample</span></samp> is +installed. This can be used as the basis for customized daemon startup, +See <a href="#Launching-the-daemons">Launching the daemons</a>. + +<div class="node"> +<p><hr> +<a name="Upgrading-an-Existing-Buildmaster"></a> +Next: <a rel="next" accesskey="n" href="#Creating-a-buildslave">Creating a buildslave</a>, +Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildmaster">Creating a buildmaster</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.4 Upgrading an Existing Buildmaster</h3> + +<p>If you have just installed a new version of the Buildbot code, and you +have buildmasters that were created using an older version, you'll +need to upgrade these buildmasters before you can use them. The +upgrade process adds and modifies files in the buildmaster's base +directory to make it compatible with the new code. + +<pre class="example"> buildbot upgrade-master <var>basedir</var> +</pre> + <p>This command will also scan your <samp><span class="file">master.cfg</span></samp> file for +incompatbilities (by loading it and printing any errors or deprecation +warnings that occur). Each buildbot release tries to be compatible +with configurations that worked cleanly (i.e. without deprecation +warnings) on the previous release: any functions or classes that are +to be removed will first be deprecated in a release, to give users a +chance to start using their replacement. + + <p>The 0.7.6 release introduced the <samp><span class="file">public_html/</span></samp> directory, which +contains <samp><span class="file">index.html</span></samp> and other files served by the +<code>WebStatus</code> and <code>Waterfall</code> status displays. The +<code>upgrade-master</code> command will create these files if they do not +already exist. It will not modify existing copies, but it will write a +new copy in e.g. <samp><span class="file">index.html.new</span></samp> if the new version differs from +the version that already exists. + + <p>The <code>upgrade-master</code> command is idempotent. It is safe to run it +multiple times. After each upgrade of the buildbot code, you should +use <code>upgrade-master</code> on all your buildmasters. + +<div class="node"> +<p><hr> +<a name="Creating-a-buildslave"></a> +Next: <a rel="next" accesskey="n" href="#Launching-the-daemons">Launching the daemons</a>, +Previous: <a rel="previous" accesskey="p" href="#Upgrading-an-Existing-Buildmaster">Upgrading an Existing Buildmaster</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.5 Creating a buildslave</h3> + +<p>Typically, you will be adding a buildslave to an existing buildmaster, +to provide additional architecture coverage. The buildbot +administrator will give you several pieces of information necessary to +connect to the buildmaster. You should also be somewhat familiar with +the project being tested, so you can troubleshoot build problems +locally. + + <p>The buildbot exists to make sure that the project's stated “how to +build it” process actually works. To this end, the buildslave should +run in an environment just like that of your regular developers. +Typically the project build process is documented somewhere +(<samp><span class="file">README</span></samp>, <samp><span class="file">INSTALL</span></samp>, etc), in a document that should +mention all library dependencies and contain a basic set of build +instructions. This document will be useful as you configure the host +and account in which the buildslave runs. + + <p>Here's a good checklist for setting up a buildslave: + + <ol type=1 start=1> +<li>Set up the account + + <p>It is recommended (although not mandatory) to set up a separate user +account for the buildslave. This account is frequently named +<code>buildbot</code> or <code>buildslave</code>. This serves to isolate your +personal working environment from that of the slave's, and helps to +minimize the security threat posed by letting possibly-unknown +contributors run arbitrary code on your system. The account should +have a minimum of fancy init scripts. + + <li>Install the buildbot code + + <p>Follow the instructions given earlier (see <a href="#Installing-the-code">Installing the code</a>). +If you use a separate buildslave account, and you didn't install the +buildbot code to a shared location, then you will need to install it +with <code>--home=~</code> for each account that needs it. + + <li>Set up the host + + <p>Make sure the host can actually reach the buildmaster. Usually the +buildmaster is running a status webserver on the same machine, so +simply point your web browser at it and see if you can get there. +Install whatever additional packages or libraries the project's +INSTALL document advises. (or not: if your buildslave is supposed to +make sure that building without optional libraries still works, then +don't install those libraries). + + <p>Again, these libraries don't necessarily have to be installed to a +site-wide shared location, but they must be available to your build +process. Accomplishing this is usually very specific to the build +process, so installing them to <samp><span class="file">/usr</span></samp> or <samp><span class="file">/usr/local</span></samp> is +usually the best approach. + + <li>Test the build process + + <p>Follow the instructions in the INSTALL document, in the buildslave's +account. Perform a full CVS (or whatever) checkout, configure, make, +run tests, etc. Confirm that the build works without manual fussing. +If it doesn't work when you do it by hand, it will be unlikely to work +when the buildbot attempts to do it in an automated fashion. + + <li>Choose a base directory + + <p>This should be somewhere in the buildslave's account, typically named +after the project which is being tested. The buildslave will not touch +any file outside of this directory. Something like <samp><span class="file">~/Buildbot</span></samp> +or <samp><span class="file">~/Buildslaves/fooproject</span></samp> is appropriate. + + <li>Get the buildmaster host/port, botname, and password + + <p>When the buildbot admin configures the buildmaster to accept and use +your buildslave, they will provide you with the following pieces of +information: + + <ul> +<li>your buildslave's name +<li>the password assigned to your buildslave +<li>the hostname and port number of the buildmaster, i.e. buildbot.example.org:8007 +</ul> + + <li>Create the buildslave + + <p>Now run the 'buildbot' command as follows: + + <pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var> +</pre> + <p>This will create the base directory and a collection of files inside, +including the <samp><span class="file">buildbot.tac</span></samp> file that contains all the +information you passed to the <code>buildbot</code> command. + + <li>Fill in the hostinfo files + + <p>When it first connects, the buildslave will send a few files up to the +buildmaster which describe the host that it is running on. These files +are presented on the web status display so that developers have more +information to reproduce any test failures that are witnessed by the +buildbot. There are sample files in the <samp><span class="file">info</span></samp> subdirectory of +the buildbot's base directory. You should edit these to correctly +describe you and your host. + + <p><samp><span class="file">BASEDIR/info/admin</span></samp> should contain your name and email address. +This is the “buildslave admin address”, and will be visible from the +build status page (so you may wish to munge it a bit if +address-harvesting spambots are a concern). + + <p><samp><span class="file">BASEDIR/info/host</span></samp> should be filled with a brief description of +the host: OS, version, memory size, CPU speed, versions of relevant +libraries installed, and finally the version of the buildbot code +which is running the buildslave. + + <p>If you run many buildslaves, you may want to create a single +<samp><span class="file">~buildslave/info</span></samp> file and share it among all the buildslaves +with symlinks. + + </ol> + +<ul class="menu"> +<li><a accesskey="1" href="#Buildslave-Options">Buildslave Options</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Buildslave-Options"></a> +Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>, +Up: <a rel="up" accesskey="u" href="#Creating-a-buildslave">Creating a buildslave</a> + +</div> + +<h4 class="subsection">2.5.1 Buildslave Options</h4> + +<p>There are a handful of options you might want to use when creating the +buildslave with the <samp><span class="command">buildbot create-slave <options> DIR <params></span></samp> +command. You can type <samp><span class="command">buildbot create-slave --help</span></samp> for a summary. +To use these, just include them on the <samp><span class="command">buildbot create-slave</span></samp> +command line, like this: + +<pre class="example"> buildbot create-slave --umask=022 ~/buildslave buildmaster.example.org:42012 myslavename mypasswd +</pre> + <dl> +<dt><code>--usepty</code><dd>This is a boolean flag that tells the buildslave whether to launch child +processes in a PTY or with regular pipes (the default) when the master does not +specify. This option is deprecated, as this particular parameter is better +specified on the master. + + <br><dt><code>--umask</code><dd>This is a string (generally an octal representation of an integer) +which will cause the buildslave process' “umask” value to be set +shortly after initialization. The “twistd” daemonization utility +forces the umask to 077 at startup (which means that all files created +by the buildslave or its child processes will be unreadable by any +user other than the buildslave account). If you want build products to +be readable by other accounts, you can add <code>--umask=022</code> to tell +the buildslave to fix the umask after twistd clobbers it. If you want +build products to be <em>writable</em> by other accounts too, use +<code>--umask=000</code>, but this is likely to be a security problem. + + <br><dt><code>--keepalive</code><dd>This is a number that indicates how frequently “keepalive” messages +should be sent from the buildslave to the buildmaster, expressed in +seconds. The default (600) causes a message to be sent to the +buildmaster at least once every 10 minutes. To set this to a lower +value, use e.g. <code>--keepalive=120</code>. + + <p>If the buildslave is behind a NAT box or stateful firewall, these +messages may help to keep the connection alive: some NAT boxes tend to +forget about a connection if it has not been used in a while. When +this happens, the buildmaster will think that the buildslave has +disappeared, and builds will time out. Meanwhile the buildslave will +not realize than anything is wrong. + + <br><dt><code>--maxdelay</code><dd>This is a number that indicates the maximum amount of time the +buildslave will wait between connection attempts, expressed in +seconds. The default (300) causes the buildslave to wait at most 5 +minutes before trying to connect to the buildmaster again. + + <br><dt><code>--log-size</code><dd>This is the size in bytes when to rotate the Twisted log files. + + <br><dt><code>--log-count</code><dd>This is the number of log rotations to keep around. You can either +specify a number or <code>None</code> (the default) to keep all +<samp><span class="file">twistd.log</span></samp> files around. + + </dl> + +<div class="node"> +<p><hr> +<a name="Launching-the-daemons"></a> +Next: <a rel="next" accesskey="n" href="#Logfiles">Logfiles</a>, +Previous: <a rel="previous" accesskey="p" href="#Creating-a-buildslave">Creating a buildslave</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.6 Launching the daemons</h3> + +<p>Both the buildmaster and the buildslave run as daemon programs. To +launch them, pass the working directory to the <code>buildbot</code> +command: + +<pre class="example"> buildbot start <var>BASEDIR</var> +</pre> + <p>This command will start the daemon and then return, so normally it +will not produce any output. To verify that the programs are indeed +running, look for a pair of files named <samp><span class="file">twistd.log</span></samp> and +<samp><span class="file">twistd.pid</span></samp> that should be created in the working directory. +<samp><span class="file">twistd.pid</span></samp> contains the process ID of the newly-spawned daemon. + + <p>When the buildslave connects to the buildmaster, new directories will +start appearing in its base directory. The buildmaster tells the slave +to create a directory for each Builder which will be using that slave. +All build operations are performed within these directories: CVS +checkouts, compiles, and tests. + + <p>Once you get everything running, you will want to arrange for the +buildbot daemons to be started at boot time. One way is to use +<code>cron</code>, by putting them in a @reboot crontab entry<a rel="footnote" href="#fn-1" name="fnd-1"><sup>1</sup></a>: + +<pre class="example"> @reboot buildbot start <var>BASEDIR</var> +</pre> + <p>When you run <samp><span class="command">crontab</span></samp> to set this up, remember to do it as +the buildmaster or buildslave account! If you add this to your crontab +when running as your regular account (or worse yet, root), then the +daemon will run as the wrong user, quite possibly as one with more +authority than you intended to provide. + + <p>It is important to remember that the environment provided to cron jobs +and init scripts can be quite different that your normal runtime. +There may be fewer environment variables specified, and the PATH may +be shorter than usual. It is a good idea to test out this method of +launching the buildslave by using a cron job with a time in the near +future, with the same command, and then check <samp><span class="file">twistd.log</span></samp> to +make sure the slave actually started correctly. Common problems here +are for <samp><span class="file">/usr/local</span></samp> or <samp><span class="file">~/bin</span></samp> to not be on your +<code>PATH</code>, or for <code>PYTHONPATH</code> to not be set correctly. +Sometimes <code>HOME</code> is messed up too. + + <p>To modify the way the daemons are started (perhaps you want to set +some environment variables first, or perform some cleanup each time), +you can create a file named <samp><span class="file">Makefile.buildbot</span></samp> in the base +directory. When the <samp><span class="file">buildbot</span></samp> front-end tool is told to +<samp><span class="command">start</span></samp> the daemon, and it sees this file (and +<samp><span class="file">/usr/bin/make</span></samp> exists), it will do <samp><span class="command">make -f +Makefile.buildbot start</span></samp> instead of its usual action (which involves +running <samp><span class="command">twistd</span></samp>). When the buildmaster or buildslave is +installed, a <samp><span class="file">Makefile.sample</span></samp> is created which implements the +same behavior as the the <samp><span class="file">buildbot</span></samp> tool uses, so if you want to +customize the process, just copy <samp><span class="file">Makefile.sample</span></samp> to +<samp><span class="file">Makefile.buildbot</span></samp> and edit it as necessary. + + <p>Some distributions may include conveniences to make starting buildbot +at boot time easy. For instance, with the default buildbot package in +Debian-based distributions, you may only need to modify +<code>/etc/default/buildbot</code> (see also <code>/etc/init.d/buildbot</code>, which +reads the configuration in <code>/etc/default/buildbot</code>). + +<div class="node"> +<p><hr> +<a name="Logfiles"></a> +Next: <a rel="next" accesskey="n" href="#Shutdown">Shutdown</a>, +Previous: <a rel="previous" accesskey="p" href="#Launching-the-daemons">Launching the daemons</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.7 Logfiles</h3> + +<p><a name="index-logfiles-4"></a> +While a buildbot daemon runs, it emits text to a logfile, named +<samp><span class="file">twistd.log</span></samp>. A command like <code>tail -f twistd.log</code> is useful +to watch the command output as it runs. + + <p>The buildmaster will announce any errors with its configuration file +in the logfile, so it is a good idea to look at the log at startup +time to check for any problems. Most buildmaster activities will cause +lines to be added to the log. + +<div class="node"> +<p><hr> +<a name="Shutdown"></a> +Next: <a rel="next" accesskey="n" href="#Maintenance">Maintenance</a>, +Previous: <a rel="previous" accesskey="p" href="#Logfiles">Logfiles</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.8 Shutdown</h3> + +<p>To stop a buildmaster or buildslave manually, use: + +<pre class="example"> buildbot stop <var>BASEDIR</var> +</pre> + <p>This simply looks for the <samp><span class="file">twistd.pid</span></samp> file and kills whatever +process is identified within. + + <p>At system shutdown, all processes are sent a <code>SIGKILL</code>. The +buildmaster and buildslave will respond to this by shutting down +normally. + + <p>The buildmaster will respond to a <code>SIGHUP</code> by re-reading its +config file. Of course, this only works on unix-like systems with +signal support, and won't work on Windows. The following shortcut is +available: + +<pre class="example"> buildbot reconfig <var>BASEDIR</var> +</pre> + <p>When you update the Buildbot code to a new release, you will need to +restart the buildmaster and/or buildslave before it can take advantage +of the new code. You can do a <code>buildbot stop </code><var>BASEDIR</var> and +<code>buildbot start </code><var>BASEDIR</var> in quick succession, or you can +use the <code>restart</code> shortcut, which does both steps for you: + +<pre class="example"> buildbot restart <var>BASEDIR</var> +</pre> + <p>There are certain configuration changes that are not handled cleanly +by <code>buildbot reconfig</code>. If this occurs, <code>buildbot restart</code> +is a more robust tool to fully switch over to the new configuration. + + <p><code>buildbot restart</code> may also be used to start a stopped Buildbot +instance. This behaviour is useful when writing scripts that stop, start +and restart Buildbot. + + <p>A buildslave may also be gracefully shutdown from the +see <a href="#WebStatus">WebStatus</a> status plugin. This is useful to shutdown a +buildslave without interrupting any current builds. The buildmaster +will wait until the buildslave is finished all its current builds, and +will then tell the buildslave to shutdown. + +<div class="node"> +<p><hr> +<a name="Maintenance"></a> +Next: <a rel="next" accesskey="n" href="#Troubleshooting">Troubleshooting</a>, +Previous: <a rel="previous" accesskey="p" href="#Shutdown">Shutdown</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.9 Maintenance</h3> + +<p>It is a good idea to check the buildmaster's status page every once in +a while, to see if your buildslave is still online. Eventually the +buildbot will probably be enhanced to send you email (via the +<samp><span class="file">info/admin</span></samp> email address) when the slave has been offline for +more than a few hours. + + <p>If you find you can no longer provide a buildslave to the project, please +let the project admins know, so they can put out a call for a +replacement. + + <p>The Buildbot records status and logs output continually, each time a +build is performed. The status tends to be small, but the build logs +can become quite large. Each build and log are recorded in a separate +file, arranged hierarchically under the buildmaster's base directory. +To prevent these files from growing without bound, you should +periodically delete old build logs. A simple cron job to delete +anything older than, say, two weeks should do the job. The only trick +is to leave the <samp><span class="file">buildbot.tac</span></samp> and other support files alone, for +which find's <code>-mindepth</code> argument helps skip everything in the +top directory. You can use something like the following: + +<pre class="example"> @weekly cd BASEDIR && find . -mindepth 2 i-path './public_html/*' -prune -o -type f -mtime +14 -exec rm {} \; + @weekly cd BASEDIR && find twistd.log* -mtime +14 -exec rm {} \; +</pre> + <div class="node"> +<p><hr> +<a name="Troubleshooting"></a> +Previous: <a rel="previous" accesskey="p" href="#Maintenance">Maintenance</a>, +Up: <a rel="up" accesskey="u" href="#Installation">Installation</a> + +</div> + +<h3 class="section">2.10 Troubleshooting</h3> + +<p>Here are a few hints on diagnosing common problems. + +<ul class="menu"> +<li><a accesskey="1" href="#Starting-the-buildslave">Starting the buildslave</a> +<li><a accesskey="2" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a> +<li><a accesskey="3" href="#Forcing-Builds">Forcing Builds</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Starting-the-buildslave"></a> +Next: <a rel="next" accesskey="n" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>, +Previous: <a rel="previous" accesskey="p" href="#Troubleshooting">Troubleshooting</a>, +Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> + +</div> + +<h4 class="subsection">2.10.1 Starting the buildslave</h4> + +<p>Cron jobs are typically run with a minimal shell (<samp><span class="file">/bin/sh</span></samp>, not +<samp><span class="file">/bin/bash</span></samp>), and tilde expansion is not always performed in such +commands. You may want to use explicit paths, because the <code>PATH</code> +is usually quite short and doesn't include anything set by your +shell's startup scripts (<samp><span class="file">.profile</span></samp>, <samp><span class="file">.bashrc</span></samp>, etc). If +you've installed buildbot (or other python libraries) to an unusual +location, you may need to add a <code>PYTHONPATH</code> specification (note +that python will do tilde-expansion on <code>PYTHONPATH</code> elements by +itself). Sometimes it is safer to fully-specify everything: + +<pre class="example"> @reboot PYTHONPATH=~/lib/python /usr/local/bin/buildbot start /usr/home/buildbot/basedir +</pre> + <p>Take the time to get the @reboot job set up. Otherwise, things will work +fine for a while, but the first power outage or system reboot you have will +stop the buildslave with nothing but the cries of sorrowful developers to +remind you that it has gone away. + +<div class="node"> +<p><hr> +<a name="Connecting-to-the-buildmaster"></a> +Next: <a rel="next" accesskey="n" href="#Forcing-Builds">Forcing Builds</a>, +Previous: <a rel="previous" accesskey="p" href="#Starting-the-buildslave">Starting the buildslave</a>, +Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> + +</div> + +<h4 class="subsection">2.10.2 Connecting to the buildmaster</h4> + +<p>If the buildslave cannot connect to the buildmaster, the reason should +be described in the <samp><span class="file">twistd.log</span></samp> logfile. Some common problems +are an incorrect master hostname or port number, or a mistyped bot +name or password. If the buildslave loses the connection to the +master, it is supposed to attempt to reconnect with an +exponentially-increasing backoff. Each attempt (and the time of the +next attempt) will be logged. If you get impatient, just manually stop +and re-start the buildslave. + + <p>When the buildmaster is restarted, all slaves will be disconnected, +and will attempt to reconnect as usual. The reconnect time will depend +upon how long the buildmaster is offline (i.e. how far up the +exponential backoff curve the slaves have travelled). Again, +<code>buildbot stop </code><var>BASEDIR</var><code>; buildbot start </code><var>BASEDIR</var> will +speed up the process. + +<div class="node"> +<p><hr> +<a name="Forcing-Builds"></a> +Previous: <a rel="previous" accesskey="p" href="#Connecting-to-the-buildmaster">Connecting to the buildmaster</a>, +Up: <a rel="up" accesskey="u" href="#Troubleshooting">Troubleshooting</a> + +</div> + +<h4 class="subsection">2.10.3 Forcing Builds</h4> + +<p>From the buildmaster's main status web page, you can force a build to +be run on your build slave. Figure out which column is for a builder +that runs on your slave, click on that builder's name, and the page +that comes up will have a “Force Build” button. Fill in the form, +hit the button, and a moment later you should see your slave's +<samp><span class="file">twistd.log</span></samp> filling with commands being run. Using <code>pstree</code> +or <code>top</code> should also reveal the cvs/make/gcc/etc processes being +run by the buildslave. Note that the same web page should also show +the <samp><span class="file">admin</span></samp> and <samp><span class="file">host</span></samp> information files that you configured +earlier. + +<div class="node"> +<p><hr> +<a name="Concepts"></a> +Next: <a rel="next" accesskey="n" href="#Configuration">Configuration</a>, +Previous: <a rel="previous" accesskey="p" href="#Installation">Installation</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">3 Concepts</h2> + +<p>This chapter defines some of the basic concepts that the Buildbot +uses. You'll need to understand how the Buildbot sees the world to +configure it properly. + +<ul class="menu"> +<li><a accesskey="1" href="#Version-Control-Systems">Version Control Systems</a> +<li><a accesskey="2" href="#Schedulers">Schedulers</a> +<li><a accesskey="3" href="#BuildSet">BuildSet</a> +<li><a accesskey="4" href="#BuildRequest">BuildRequest</a> +<li><a accesskey="5" href="#Builder">Builder</a> +<li><a accesskey="6" href="#Users">Users</a> +<li><a accesskey="7" href="#Build-Properties">Build Properties</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Version-Control-Systems"></a> +Next: <a rel="next" accesskey="n" href="#Schedulers">Schedulers</a>, +Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.1 Version Control Systems</h3> + +<p><a name="index-Version-Control-5"></a> +These source trees come from a Version Control System of some kind. +CVS and Subversion are two popular ones, but the Buildbot supports +others. All VC systems have some notion of an upstream +<code>repository</code> which acts as a server<a rel="footnote" href="#fn-2" name="fnd-2"><sup>2</sup></a>, from which clients +can obtain source trees according to various parameters. The VC +repository provides source trees of various projects, for different +branches, and from various points in time. The first thing we have to +do is to specify which source tree we want to get. + +<ul class="menu"> +<li><a accesskey="1" href="#Generalizing-VC-Systems">Generalizing VC Systems</a> +<li><a accesskey="2" href="#Source-Tree-Specifications">Source Tree Specifications</a> +<li><a accesskey="3" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a> +<li><a accesskey="4" href="#Attributes-of-Changes">Attributes of Changes</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Generalizing-VC-Systems"></a> +Next: <a rel="next" accesskey="n" href="#Source-Tree-Specifications">Source Tree Specifications</a>, +Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>, +Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> + +</div> + +<h4 class="subsection">3.1.1 Generalizing VC Systems</h4> + +<p>For the purposes of the Buildbot, we will try to generalize all VC +systems as having repositories that each provide sources for a variety +of projects. Each project is defined as a directory tree with source +files. The individual files may each have revisions, but we ignore +that and treat the project as a whole as having a set of revisions +(CVS is really the only VC system still in widespread use that has +per-file revisions.. everything modern has moved to atomic tree-wide +changesets). Each time someone commits a change to the project, a new +revision becomes available. These revisions can be described by a +tuple with two items: the first is a branch tag, and the second is +some kind of revision stamp or timestamp. Complex projects may have +multiple branch tags, but there is always a default branch. The +timestamp may be an actual timestamp (such as the -D option to CVS), +or it may be a monotonically-increasing transaction number (such as +the change number used by SVN and P4, or the revision number used by +Arch/Baz/Bazaar, or a labeled tag used in CVS)<a rel="footnote" href="#fn-3" name="fnd-3"><sup>3</sup></a>. The SHA1 revision ID used by Monotone, Mercurial, and Git is +also a kind of revision stamp, in that it specifies a unique copy of +the source tree, as does a Darcs “context” file. + + <p>When we aren't intending to make any changes to the sources we check out +(at least not any that need to be committed back upstream), there are two +basic ways to use a VC system: + + <ul> +<li>Retrieve a specific set of source revisions: some tag or key is used +to index this set, which is fixed and cannot be changed by subsequent +developers committing new changes to the tree. Releases are built from +tagged revisions like this, so that they can be rebuilt again later +(probably with controlled modifications). +<li>Retrieve the latest sources along a specific branch: some tag is used +to indicate which branch is to be used, but within that constraint we want +to get the latest revisions. +</ul> + + <p>Build personnel or CM staff typically use the first approach: the +build that results is (ideally) completely specified by the two +parameters given to the VC system: repository and revision tag. This +gives QA and end-users something concrete to point at when reporting +bugs. Release engineers are also reportedly fond of shipping code that +can be traced back to a concise revision tag of some sort. + + <p>Developers are more likely to use the second approach: each morning +the developer does an update to pull in the changes committed by the +team over the last day. These builds are not easy to fully specify: it +depends upon exactly when you did a checkout, and upon what local +changes the developer has in their tree. Developers do not normally +tag each build they produce, because there is usually significant +overhead involved in creating these tags. Recreating the trees used by +one of these builds can be a challenge. Some VC systems may provide +implicit tags (like a revision number), while others may allow the use +of timestamps to mean “the state of the tree at time X” as opposed +to a tree-state that has been explicitly marked. + + <p>The Buildbot is designed to help developers, so it usually works in +terms of <em>the latest</em> sources as opposed to specific tagged +revisions. However, it would really prefer to build from reproducible +source trees, so implicit revisions are used whenever possible. + +<div class="node"> +<p><hr> +<a name="Source-Tree-Specifications"></a> +Next: <a rel="next" accesskey="n" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>, +Previous: <a rel="previous" accesskey="p" href="#Generalizing-VC-Systems">Generalizing VC Systems</a>, +Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> + +</div> + +<h4 class="subsection">3.1.2 Source Tree Specifications</h4> + +<p>So for the Buildbot's purposes we treat each VC system as a server +which can take a list of specifications as input and produce a source +tree as output. Some of these specifications are static: they are +attributes of the builder and do not change over time. Others are more +variable: each build will have a different value. The repository is +changed over time by a sequence of Changes, each of which represents a +single developer making changes to some set of files. These Changes +are cumulative<a rel="footnote" href="#fn-4" name="fnd-4"><sup>4</sup></a>. + + <p>For normal builds, the Buildbot wants to get well-defined source trees +that contain specific Changes, and exclude other Changes that may have +occurred after the desired ones. We assume that the Changes arrive at +the buildbot (through one of the mechanisms described in see <a href="#Change-Sources">Change Sources</a>) in the same order in which they are committed to the +repository. The Buildbot waits for the tree to become “stable” +before initiating a build, for two reasons. The first is that +developers frequently make multiple related commits in quick +succession, even when the VC system provides ways to make atomic +transactions involving multiple files at the same time. Running a +build in the middle of these sets of changes would use an inconsistent +set of source files, and is likely to fail (and is certain to be less +useful than a build which uses the full set of changes). The +tree-stable-timer is intended to avoid these useless builds that +include some of the developer's changes but not all. The second reason +is that some VC systems (i.e. CVS) do not provide repository-wide +transaction numbers, so that timestamps are the only way to refer to +a specific repository state. These timestamps may be somewhat +ambiguous, due to processing and notification delays. By waiting until +the tree has been stable for, say, 10 minutes, we can choose a +timestamp from the middle of that period to use for our source +checkout, and then be reasonably sure that any clock-skew errors will +not cause the build to be performed on an inconsistent set of source +files. + + <p>The Schedulers always use the tree-stable-timer, with a timeout that +is configured to reflect a reasonable tradeoff between build latency +and change frequency. When the VC system provides coherent +repository-wide revision markers (such as Subversion's revision +numbers, or in fact anything other than CVS's timestamps), the +resulting Build is simply performed against a source tree defined by +that revision marker. When the VC system does not provide this, a +timestamp from the middle of the tree-stable period is used to +generate the source tree<a rel="footnote" href="#fn-5" name="fnd-5"><sup>5</sup></a>. + +<div class="node"> +<p><hr> +<a name="How-Different-VC-Systems-Specify-Sources"></a> +Next: <a rel="next" accesskey="n" href="#Attributes-of-Changes">Attributes of Changes</a>, +Previous: <a rel="previous" accesskey="p" href="#Source-Tree-Specifications">Source Tree Specifications</a>, +Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> + +</div> + +<h4 class="subsection">3.1.3 How Different VC Systems Specify Sources</h4> + +<p>For CVS, the static specifications are <code>repository</code> and +<code>module</code>. In addition to those, each build uses a timestamp (or +omits the timestamp to mean <code>the latest</code>) and <code>branch tag</code> +(which defaults to HEAD). These parameters collectively specify a set +of sources from which a build may be performed. + + <p><a href="http://subversion.tigris.org">Subversion</a> combines the +repository, module, and branch into a single <code>Subversion URL</code> +parameter. Within that scope, source checkouts can be specified by a +numeric <code>revision number</code> (a repository-wide +monotonically-increasing marker, such that each transaction that +changes the repository is indexed by a different revision number), or +a revision timestamp. When branches are used, the repository and +module form a static <code>baseURL</code>, while each build has a +<code>revision number</code> and a <code>branch</code> (which defaults to a +statically-specified <code>defaultBranch</code>). The <code>baseURL</code> and +<code>branch</code> are simply concatenated together to derive the +<code>svnurl</code> to use for the checkout. + + <p><a href="http://www.perforce.com/">Perforce</a> is similar. The server +is specified through a <code>P4PORT</code> parameter. Module and branch +are specified in a single depot path, and revisions are +depot-wide. When branches are used, the <code>p4base</code> and +<code>defaultBranch</code> are concatenated together to produce the depot +path. + + <p><a href="http://wiki.gnuarch.org/">Arch</a> and +<a href="http://bazaar.canonical.com/">Bazaar</a> specify a repository by +URL, as well as a <code>version</code> which is kind of like a branch name. +Arch uses the word <code>archive</code> to represent the repository. Arch +lets you push changes from one archive to another, removing the strict +centralization required by CVS and SVN. It retains the distinction +between repository and working directory that most other VC systems +use. For complex multi-module directory structures, Arch has a +built-in <code>build config</code> layer with which the checkout process has +two steps. First, an initial bootstrap checkout is performed to +retrieve a set of build-config files. Second, one of these files is +used to figure out which archives/modules should be used to populate +subdirectories of the initial checkout. + + <p>Builders which use Arch and Bazaar therefore have a static archive +<code>url</code>, and a default “branch” (which is a string that specifies +a complete category–branch–version triple). Each build can have its +own branch (the category–branch–version string) to override the +default, as well as a revision number (which is turned into a +–patch-NN suffix when performing the checkout). + + <p><a href="http://bazaar-vcs.org">Bzr</a> (which is a descendant of +Arch/Bazaar, and is frequently referred to as “Bazaar”) has the same +sort of repository-vs-workspace model as Arch, but the repository data +can either be stored inside the working directory or kept elsewhere +(either on the same machine or on an entirely different machine). For +the purposes of Buildbot (which never commits changes), the repository +is specified with a URL and a revision number. + + <p>The most common way to obtain read-only access to a bzr tree is via +HTTP, simply by making the repository visible through a web server +like Apache. Bzr can also use FTP and SFTP servers, if the buildslave +process has sufficient privileges to access them. Higher performance +can be obtained by running a special Bazaar-specific server. None of +these matter to the buildbot: the repository URL just has to match the +kind of server being used. The <code>repoURL</code> argument provides the +location of the repository. + + <p>Branches are expressed as subdirectories of the main central +repository, which means that if branches are being used, the BZR step +is given a <code>baseURL</code> and <code>defaultBranch</code> instead of getting +the <code>repoURL</code> argument. + + <p><a href="http://darcs.net/">Darcs</a> doesn't really have the +notion of a single master repository. Nor does it really have +branches. In Darcs, each working directory is also a repository, and +there are operations to push and pull patches from one of these +<code>repositories</code> to another. For the Buildbot's purposes, all you +need to do is specify the URL of a repository that you want to build +from. The build slave will then pull the latest patches from that +repository and build them. Multiple branches are implemented by using +multiple repositories (possibly living on the same server). + + <p>Builders which use Darcs therefore have a static <code>repourl</code> which +specifies the location of the repository. If branches are being used, +the source Step is instead configured with a <code>baseURL</code> and a +<code>defaultBranch</code>, and the two strings are simply concatenated +together to obtain the repository's URL. Each build then has a +specific branch which replaces <code>defaultBranch</code>, or just uses the +default one. Instead of a revision number, each build can have a +“context”, which is a string that records all the patches that are +present in a given tree (this is the output of <samp><span class="command">darcs changes +--context</span></samp>, and is considerably less concise than, e.g. Subversion's +revision number, but the patch-reordering flexibility of Darcs makes +it impossible to provide a shorter useful specification). + + <p><a href="http://selenic.com/mercurial">Mercurial</a> is like Darcs, in that +each branch is stored in a separate repository. The <code>repourl</code>, +<code>baseURL</code>, and <code>defaultBranch</code> arguments are all handled the +same way as with Darcs. The “revision”, however, is the hash +identifier returned by <samp><span class="command">hg identify</span></samp>. + + <p><a href="http://git.or.cz/">Git</a> also follows a decentralized model, and +each repository can have several branches and tags. The source Step is +configured with a static <code>repourl</code> which specifies the location +of the repository. In addition, an optional <code>branch</code> parameter +can be specified to check out code from a specific branch instead of +the default “master” branch. The “revision” is specified as a SHA1 +hash as returned by e.g. <samp><span class="command">git rev-parse</span></samp>. No attempt is made +to ensure that the specified revision is actually a subset of the +specified branch. + +<div class="node"> +<p><hr> +<a name="Attributes-of-Changes"></a> +Previous: <a rel="previous" accesskey="p" href="#How-Different-VC-Systems-Specify-Sources">How Different VC Systems Specify Sources</a>, +Up: <a rel="up" accesskey="u" href="#Version-Control-Systems">Version Control Systems</a> + +</div> + +<h4 class="subsection">3.1.4 Attributes of Changes</h4> + +<h3 class="heading">Who</h3> + +<p>Each Change has a <code>who</code> attribute, which specifies which +developer is responsible for the change. This is a string which comes +from a namespace controlled by the VC repository. Frequently this +means it is a username on the host which runs the repository, but not +all VC systems require this (Arch, for example, uses a fully-qualified +<code>Arch ID</code>, which looks like an email address, as does Darcs). +Each StatusNotifier will map the <code>who</code> attribute into something +appropriate for their particular means of communication: an email +address, an IRC handle, etc. + +<h3 class="heading">Files</h3> + +<p>It also has a list of <code>files</code>, which are just the tree-relative +filenames of any files that were added, deleted, or modified for this +Change. These filenames are used by the <code>fileIsImportant</code> +function (in the Scheduler) to decide whether it is worth triggering a +new build or not, e.g. the function could use the following function +to only run a build if a C file were checked in: + +<pre class="example"> def has_C_files(change): + for name in change.files: + if name.endswith(".c"): + return True + return False +</pre> + <p>Certain BuildSteps can also use the list of changed files +to run a more targeted series of tests, e.g. the +<code>python_twisted.Trial</code> step can run just the unit tests that +provide coverage for the modified .py files instead of running the +full test suite. + +<h3 class="heading">Comments</h3> + +<p>The Change also has a <code>comments</code> attribute, which is a string +containing any checkin comments. + +<h3 class="heading">Revision</h3> + +<p>Each Change can have a <code>revision</code> attribute, which describes how +to get a tree with a specific state: a tree which includes this Change +(and all that came before it) but none that come after it. If this +information is unavailable, the <code>.revision</code> attribute will be +<code>None</code>. These revisions are provided by the ChangeSource, and +consumed by the <code>computeSourceRevision</code> method in the appropriate +<code>step.Source</code> class. + + <dl> +<dt>‘<samp><span class="samp">CVS</span></samp>’<dd><code>revision</code> is an int, seconds since the epoch +<br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd><code>revision</code> is an int, the changeset number (r%d) +<br><dt>‘<samp><span class="samp">Darcs</span></samp>’<dd><code>revision</code> is a large string, the output of <code>darcs changes --context</code> +<br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd><code>revision</code> is a short string (a hash ID), the output of <code>hg identify</code> +<br><dt>‘<samp><span class="samp">Arch/Bazaar</span></samp>’<dd><code>revision</code> is the full revision ID (ending in –patch-%d) +<br><dt>‘<samp><span class="samp">P4</span></samp>’<dd><code>revision</code> is an int, the transaction number +<br><dt>‘<samp><span class="samp">Git</span></samp>’<dd><code>revision</code> is a short string (a SHA1 hash), the output of e.g. +<code>git rev-parse</code> +</dl> + +<h3 class="heading">Branches</h3> + +<p>The Change might also have a <code>branch</code> attribute. This indicates +that all of the Change's files are in the same named branch. The +Schedulers get to decide whether the branch should be built or not. + + <p>For VC systems like CVS, Arch, Monotone, and Git, the <code>branch</code> +name is unrelated to the filename. (that is, the branch name and the +filename inhabit unrelated namespaces). For SVN, branches are +expressed as subdirectories of the repository, so the file's +“svnurl” is a combination of some base URL, the branch name, and the +filename within the branch. (In a sense, the branch name and the +filename inhabit the same namespace). Darcs branches are +subdirectories of a base URL just like SVN. Mercurial branches are the +same as Darcs. + + <dl> +<dt>‘<samp><span class="samp">CVS</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] +<br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd>branch='branches/warner-newfeature', files=['src/foo.c'] +<br><dt>‘<samp><span class="samp">Darcs</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] +<br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] +<br><dt>‘<samp><span class="samp">Arch/Bazaar</span></samp>’<dd>branch='buildbot–usebranches–0', files=['buildbot/master.py'] +<br><dt>‘<samp><span class="samp">Git</span></samp>’<dd>branch='warner-newfeature', files=['src/foo.c'] +</dl> + +<h3 class="heading">Links</h3> + +<!-- TODO: who is using 'links'? how is it being used? --> +<p>Finally, the Change might have a <code>links</code> list, which is intended +to provide a list of URLs to a <em>viewcvs</em>-style web page that +provides more detail for this Change, perhaps including the full file +diffs. + +<div class="node"> +<p><hr> +<a name="Schedulers"></a> +Next: <a rel="next" accesskey="n" href="#BuildSet">BuildSet</a>, +Previous: <a rel="previous" accesskey="p" href="#Version-Control-Systems">Version Control Systems</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.2 Schedulers</h3> + +<p><a name="index-Scheduler-6"></a> +Each Buildmaster has a set of <code>Scheduler</code> objects, each of which +gets a copy of every incoming Change. The Schedulers are responsible +for deciding when Builds should be run. Some Buildbot installations +might have a single Scheduler, while others may have several, each for +a different purpose. + + <p>For example, a “quick” scheduler might exist to give immediate +feedback to developers, hoping to catch obvious problems in the code +that can be detected quickly. These typically do not run the full test +suite, nor do they run on a wide variety of platforms. They also +usually do a VC update rather than performing a brand-new checkout +each time. You could have a “quick” scheduler which used a 30 second +timeout, and feeds a single “quick” Builder that uses a VC +<code>mode='update'</code> setting. + + <p>A separate “full” scheduler would run more comprehensive tests a +little while later, to catch more subtle problems. This scheduler +would have a longer tree-stable-timer, maybe 30 minutes, and would +feed multiple Builders (with a <code>mode=</code> of <code>'copy'</code>, +<code>'clobber'</code>, or <code>'export'</code>). + + <p>The <code>tree-stable-timer</code> and <code>fileIsImportant</code> decisions are +made by the Scheduler. Dependencies are also implemented here. +Periodic builds (those which are run every N seconds rather than after +new Changes arrive) are triggered by a special <code>Periodic</code> +Scheduler subclass. The default Scheduler class can also be told to +watch for specific branches, ignoring Changes on other branches. This +may be useful if you have a trunk and a few release branches which +should be tracked, but when you don't want to have the Buildbot pay +attention to several dozen private user branches. + + <p>When the setup has multiple sources of Changes the <code>category</code> +can be used for <code>Scheduler</code> objects to filter out a subset +of the Changes. Note that not all change sources can attach a category. + + <p>Some Schedulers may trigger builds for other reasons, other than +recent Changes. For example, a Scheduler subclass could connect to a +remote buildmaster and watch for builds of a library to succeed before +triggering a local build that uses that library. + + <p>Each Scheduler creates and submits <code>BuildSet</code> objects to the +<code>BuildMaster</code>, which is then responsible for making sure the +individual <code>BuildRequests</code> are delivered to the target +<code>Builders</code>. + + <p><code>Scheduler</code> instances are activated by placing them in the +<code>c['schedulers']</code> list in the buildmaster config file. Each +Scheduler has a unique name. + +<div class="node"> +<p><hr> +<a name="BuildSet"></a> +Next: <a rel="next" accesskey="n" href="#BuildRequest">BuildRequest</a>, +Previous: <a rel="previous" accesskey="p" href="#Schedulers">Schedulers</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.3 BuildSet</h3> + +<p><a name="index-BuildSet-7"></a> +A <code>BuildSet</code> is the name given to a set of Builds that all +compile/test the same version of the tree on multiple Builders. In +general, all these component Builds will perform the same sequence of +Steps, using the same source code, but on different platforms or +against a different set of libraries. + + <p>The <code>BuildSet</code> is tracked as a single unit, which fails if any of +the component Builds have failed, and therefore can succeed only if +<em>all</em> of the component Builds have succeeded. There are two kinds +of status notification messages that can be emitted for a BuildSet: +the <code>firstFailure</code> type (which fires as soon as we know the +BuildSet will fail), and the <code>Finished</code> type (which fires once +the BuildSet has completely finished, regardless of whether the +overall set passed or failed). + + <p>A <code>BuildSet</code> is created with a <em>source stamp</em> tuple of +(branch, revision, changes, patch), some of which may be None, and a +list of Builders on which it is to be run. They are then given to the +BuildMaster, which is responsible for creating a separate +<code>BuildRequest</code> for each Builder. + + <p>There are a couple of different likely values for the +<code>SourceStamp</code>: + + <dl> +<dt><code>(revision=None, changes=[CHANGES], patch=None)</code><dd>This is a <code>SourceStamp</code> used when a series of Changes have +triggered a build. The VC step will attempt to check out a tree that +contains CHANGES (and any changes that occurred before CHANGES, but +not any that occurred after them). + + <br><dt><code>(revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the default branch. This is the +sort of <code>SourceStamp</code> that would be used on a Build that was +triggered by a user request, or a Periodic scheduler. It is also +possible to configure the VC Source Step to always check out the +latest sources rather than paying attention to the Changes in the +SourceStamp, which will result in same behavior as this. + + <br><dt><code>(branch=BRANCH, revision=None, changes=None, patch=None)</code><dd>This builds the most recent code on the given BRANCH. Again, this is +generally triggered by a user request or Periodic build. + + <br><dt><code>(revision=REV, changes=None, patch=(LEVEL, DIFF))</code><dd>This checks out the tree at the given revision REV, then applies a +patch (using <code>patch -pLEVEL <DIFF</code>). The <a href="#try">try</a> feature uses +this kind of <code>SourceStamp</code>. If <code>patch</code> is None, the patching +step is bypassed. + + </dl> + + <p>The buildmaster is responsible for turning the <code>BuildSet</code> into a +set of <code>BuildRequest</code> objects and queueing them on the +appropriate Builders. + +<div class="node"> +<p><hr> +<a name="BuildRequest"></a> +Next: <a rel="next" accesskey="n" href="#Builder">Builder</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildSet">BuildSet</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.4 BuildRequest</h3> + +<p><a name="index-BuildRequest-8"></a> +A <code>BuildRequest</code> is a request to build a specific set of sources +on a single specific <code>Builder</code>. Each <code>Builder</code> runs the +<code>BuildRequest</code> as soon as it can (i.e. when an associated +buildslave becomes free). <code>BuildRequest</code>s are prioritized from +oldest to newest, so when a buildslave becomes free, the +<code>Builder</code> with the oldest <code>BuildRequest</code> is run. + + <p>The <code>BuildRequest</code> contains the <code>SourceStamp</code> specification. +The actual process of running the build (the series of Steps that will +be executed) is implemented by the <code>Build</code> object. In this future +this might be changed, to have the <code>Build</code> define <em>what</em> +gets built, and a separate <code>BuildProcess</code> (provided by the +Builder) to define <em>how</em> it gets built. + + <p><code>BuildRequest</code> is created with optional <code>Properties</code>. One +of these, <code>owner</code>, is collected by the resultant <code>Build</code> and +added to the set of <em>interested users</em> to which status +notifications will be sent, depending on the configuration for each +status object. + + <p>The <code>BuildRequest</code> may be mergeable with other compatible +<code>BuildRequest</code>s. Builds that are triggered by incoming Changes +will generally be mergeable. Builds that are triggered by user +requests are generally not, unless they are multiple requests to build +the <em>latest sources</em> of the same branch. + +<div class="node"> +<p><hr> +<a name="Builder"></a> +Next: <a rel="next" accesskey="n" href="#Users">Users</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildRequest">BuildRequest</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.5 Builder</h3> + +<p><a name="index-Builder-9"></a> +The <code>Builder</code> is a long-lived object which controls all Builds of +a given type. Each one is created when the config file is first +parsed, and lives forever (or rather until it is removed from the +config file). It mediates the connections to the buildslaves that do +all the work, and is responsible for creating the <code>Build</code> objects +that decide <em>how</em> a build is performed (i.e., which steps are +executed in what order). + + <p>Each <code>Builder</code> gets a unique name, and the path name of a +directory where it gets to do all its work (there is a +buildmaster-side directory for keeping status information, as well as +a buildslave-side directory where the actual checkout/compile/test +commands are executed). It also gets a <code>BuildFactory</code>, which is +responsible for creating new <code>Build</code> instances: because the +<code>Build</code> instance is what actually performs each build, choosing +the <code>BuildFactory</code> is the way to specify what happens each time a +build is done. + + <p>Each <code>Builder</code> is associated with one of more <code>BuildSlaves</code>. +A <code>Builder</code> which is used to perform OS-X builds (as opposed to +Linux or Solaris builds) should naturally be associated with an +OS-X-based buildslave. + + <p>A <code>Builder</code> may be given a set of environment variables to be used +in its see <a href="#ShellCommand">ShellCommand</a>s. These variables will override anything in the +buildslave's environment. Variables passed directly to a ShellCommand will +override variables of the same name passed to the Builder. + + <p>For example, if you a pool of identical slaves it is often easier to manage +variables like PATH from Buildbot rather than manually editing it inside of +the slaves' environment. + +<pre class="example"> f = factory.BuildFactory + f.addStep(ShellCommand( + command=['bash', './configure'])) + f.addStep(Compile()) + + c['builders'] = [ + {'name': 'test', 'slavenames': ['slave1', 'slave2', 'slave3', 'slave4', + 'slave5', 'slave6'], + 'builddir': 'test', 'factory': f', + 'env': {'PATH': '/opt/local/bin:/opt/app/bin:/usr/local/bin:/usr/bin'}} + +</pre> + <div class="node"> +<p><hr> +<a name="Users"></a> +Next: <a rel="next" accesskey="n" href="#Build-Properties">Build Properties</a>, +Previous: <a rel="previous" accesskey="p" href="#Builder">Builder</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.6 Users</h3> + +<p><a name="index-Users-10"></a> +Buildbot has a somewhat limited awareness of <em>users</em>. It assumes +the world consists of a set of developers, each of whom can be +described by a couple of simple attributes. These developers make +changes to the source code, causing builds which may succeed or fail. + + <p>Each developer is primarily known through the source control system. Each +Change object that arrives is tagged with a <code>who</code> field that +typically gives the account name (on the repository machine) of the user +responsible for that change. This string is the primary key by which the +User is known, and is displayed on the HTML status pages and in each Build's +“blamelist”. + + <p>To do more with the User than just refer to them, this username needs to +be mapped into an address of some sort. The responsibility for this mapping +is left up to the status module which needs the address. The core code knows +nothing about email addresses or IRC nicknames, just user names. + +<ul class="menu"> +<li><a accesskey="1" href="#Doing-Things-With-Users">Doing Things With Users</a> +<li><a accesskey="2" href="#Email-Addresses">Email Addresses</a> +<li><a accesskey="3" href="#IRC-Nicknames">IRC Nicknames</a> +<li><a accesskey="4" href="#Live-Status-Clients">Live Status Clients</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Doing-Things-With-Users"></a> +Next: <a rel="next" accesskey="n" href="#Email-Addresses">Email Addresses</a>, +Previous: <a rel="previous" accesskey="p" href="#Users">Users</a>, +Up: <a rel="up" accesskey="u" href="#Users">Users</a> + +</div> + +<h4 class="subsection">3.6.1 Doing Things With Users</h4> + +<p>Each Change has a single User who is responsible for that Change. Most +Builds have a set of Changes: the Build represents the first time these +Changes have been built and tested by the Buildbot. The build has a +“blamelist” that consists of a simple union of the Users responsible +for all the Build's Changes. + + <p>The Build provides (through the IBuildStatus interface) a list of Users +who are “involved” in the build. For now this is equal to the +blamelist, but in the future it will be expanded to include a “build +sheriff” (a person who is “on duty” at that time and responsible for +watching over all builds that occur during their shift), as well as +per-module owners who simply want to keep watch over their domain (chosen by +subdirectory or a regexp matched against the filenames pulled out of the +Changes). The Involved Users are those who probably have an interest in the +results of any given build. + + <p>In the future, Buildbot will acquire the concept of “Problems”, +which last longer than builds and have beginnings and ends. For example, a +test case which passed in one build and then failed in the next is a +Problem. The Problem lasts until the test case starts passing again, at +which point the Problem is said to be “resolved”. + + <p>If there appears to be a code change that went into the tree at the +same time as the test started failing, that Change is marked as being +resposible for the Problem, and the user who made the change is added +to the Problem's “Guilty” list. In addition to this user, there may +be others who share responsibility for the Problem (module owners, +sponsoring developers). In addition to the Responsible Users, there +may be a set of Interested Users, who take an interest in the fate of +the Problem. + + <p>Problems therefore have sets of Users who may want to be kept aware of +the condition of the problem as it changes over time. If configured, the +Buildbot can pester everyone on the Responsible list with increasing +harshness until the problem is resolved, with the most harshness reserved +for the Guilty parties themselves. The Interested Users may merely be told +when the problem starts and stops, as they are not actually responsible for +fixing anything. + +<div class="node"> +<p><hr> +<a name="Email-Addresses"></a> +Next: <a rel="next" accesskey="n" href="#IRC-Nicknames">IRC Nicknames</a>, +Previous: <a rel="previous" accesskey="p" href="#Doing-Things-With-Users">Doing Things With Users</a>, +Up: <a rel="up" accesskey="u" href="#Users">Users</a> + +</div> + +<h4 class="subsection">3.6.2 Email Addresses</h4> + +<p>The <code>buildbot.status.mail.MailNotifier</code> class +(see <a href="#MailNotifier">MailNotifier</a>) provides a status target which can send email +about the results of each build. It accepts a static list of email +addresses to which each message should be delivered, but it can also +be configured to send mail to the Build's Interested Users. To do +this, it needs a way to convert User names into email addresses. + + <p>For many VC systems, the User Name is actually an account name on the +system which hosts the repository. As such, turning the name into an +email address is a simple matter of appending +“@repositoryhost.com”. Some projects use other kinds of mappings +(for example the preferred email address may be at “project.org” +despite the repository host being named “cvs.project.org”), and some +VC systems have full separation between the concept of a user and that +of an account on the repository host (like Perforce). Some systems +(like Arch) put a full contact email address in every change. + + <p>To convert these names to addresses, the MailNotifier uses an EmailLookup +object. This provides a .getAddress method which accepts a name and +(eventually) returns an address. The default <code>MailNotifier</code> +module provides an EmailLookup which simply appends a static string, +configurable when the notifier is created. To create more complex behaviors +(perhaps using an LDAP lookup, or using “finger” on a central host to +determine a preferred address for the developer), provide a different object +as the <code>lookup</code> argument. + + <p>In the future, when the Problem mechanism has been set up, the Buildbot +will need to send mail to arbitrary Users. It will do this by locating a +MailNotifier-like object among all the buildmaster's status targets, and +asking it to send messages to various Users. This means the User-to-address +mapping only has to be set up once, in your MailNotifier, and every email +message the buildbot emits will take advantage of it. + +<div class="node"> +<p><hr> +<a name="IRC-Nicknames"></a> +Next: <a rel="next" accesskey="n" href="#Live-Status-Clients">Live Status Clients</a>, +Previous: <a rel="previous" accesskey="p" href="#Email-Addresses">Email Addresses</a>, +Up: <a rel="up" accesskey="u" href="#Users">Users</a> + +</div> + +<h4 class="subsection">3.6.3 IRC Nicknames</h4> + +<p>Like MailNotifier, the <code>buildbot.status.words.IRC</code> class +provides a status target which can announce the results of each build. It +also provides an interactive interface by responding to online queries +posted in the channel or sent as private messages. + + <p>In the future, the buildbot can be configured map User names to IRC +nicknames, to watch for the recent presence of these nicknames, and to +deliver build status messages to the interested parties. Like +<code>MailNotifier</code> does for email addresses, the <code>IRC</code> object +will have an <code>IRCLookup</code> which is responsible for nicknames. The +mapping can be set up statically, or it can be updated by online users +themselves (by claiming a username with some kind of “buildbot: i am +user warner” commands). + + <p>Once the mapping is established, the rest of the buildbot can ask the +<code>IRC</code> object to send messages to various users. It can report on +the likelihood that the user saw the given message (based upon how long the +user has been inactive on the channel), which might prompt the Problem +Hassler logic to send them an email message instead. + +<div class="node"> +<p><hr> +<a name="Live-Status-Clients"></a> +Previous: <a rel="previous" accesskey="p" href="#IRC-Nicknames">IRC Nicknames</a>, +Up: <a rel="up" accesskey="u" href="#Users">Users</a> + +</div> + +<h4 class="subsection">3.6.4 Live Status Clients</h4> + +<p>The Buildbot also offers a PB-based status client interface which can +display real-time build status in a GUI panel on the developer's desktop. +This interface is normally anonymous, but it could be configured to let the +buildmaster know <em>which</em> developer is using the status client. The +status client could then be used as a message-delivery service, providing an +alternative way to deliver low-latency high-interruption messages to the +developer (like “hey, you broke the build”). + +<div class="node"> +<p><hr> +<a name="Build-Properties"></a> +Previous: <a rel="previous" accesskey="p" href="#Users">Users</a>, +Up: <a rel="up" accesskey="u" href="#Concepts">Concepts</a> + +</div> + +<h3 class="section">3.7 Build Properties</h3> + +<p><a name="index-Properties-11"></a> +Each build has a set of “Build Properties”, which can be used by its +BuildStep to modify their actions. These properties, in the form of +key-value pairs, provide a general framework for dynamically altering +the behavior of a build based on its circumstances. + + <p>Properties come from a number of places: + <ul> +<li>global configuration – +These properties apply to all builds. +<li>schedulers – +A scheduler can specify properties available to all the builds it +starts. +<li>buildslaves – +A buildslave can pass properties on to the builds it performs. +<li>builds – +A build automatically sets a number of properties on itself. +<li>steps – +Steps of a build can set properties that are available to subsequent +steps. In particular, source steps set a number of properties. +</ul> + + <p>Properties are very flexible, and can be used to implement all manner +of functionality. Here are some examples: + + <p>Most Source steps record the revision that they checked out in +the <code>got_revision</code> property. A later step could use this +property to specify the name of a fully-built tarball, dropped in an +easily-acessible directory for later testing. + + <p>Some projects want to perform nightly builds as well as in response +to committed changes. Such a project would run two schedulers, +both pointing to the same set of builders, but could provide an +<code>is_nightly</code> property so that steps can distinguish the nightly +builds, perhaps to run more resource-intensive tests. + + <p>Some projects have different build processes on different systems. +Rather than create a build factory for each slave, the steps can use +buildslave properties to identify the unique aspects of each slave +and adapt the build process dynamically. + +<div class="node"> +<p><hr> +<a name="Configuration"></a> +Next: <a rel="next" accesskey="n" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, +Previous: <a rel="previous" accesskey="p" href="#Concepts">Concepts</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">4 Configuration</h2> + +<p><a name="index-Configuration-12"></a> +The buildbot's behavior is defined by the “config file”, which +normally lives in the <samp><span class="file">master.cfg</span></samp> file in the buildmaster's base +directory (but this can be changed with an option to the +<code>buildbot create-master</code> command). This file completely specifies +which Builders are to be run, which slaves they should use, how +Changes should be tracked, and where the status information is to be +sent. The buildmaster's <samp><span class="file">buildbot.tac</span></samp> file names the base +directory; everything else comes from the config file. + + <p>A sample config file was installed for you when you created the +buildmaster, but you will need to edit it before your buildbot will do +anything useful. + + <p>This chapter gives an overview of the format of this file and the +various sections in it. You will need to read the later chapters to +understand how to fill in each section properly. + +<ul class="menu"> +<li><a accesskey="1" href="#Config-File-Format">Config File Format</a> +<li><a accesskey="2" href="#Loading-the-Config-File">Loading the Config File</a> +<li><a accesskey="3" href="#Testing-the-Config-File">Testing the Config File</a> +<li><a accesskey="4" href="#Defining-the-Project">Defining the Project</a> +<li><a accesskey="5" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> +<li><a accesskey="6" href="#Merging-BuildRequests">Merging BuildRequests</a> +<li><a accesskey="7" href="#Setting-the-slaveport">Setting the slaveport</a> +<li><a accesskey="8" href="#Buildslave-Specifiers">Buildslave Specifiers</a> +<li><a accesskey="9" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> +<li><a href="#Defining-Global-Properties">Defining Global Properties</a> +<li><a href="#Defining-Builders">Defining Builders</a> +<li><a href="#Defining-Status-Targets">Defining Status Targets</a> +<li><a href="#Debug-options">Debug options</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Config-File-Format"></a> +Next: <a rel="next" accesskey="n" href="#Loading-the-Config-File">Loading the Config File</a>, +Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.1 Config File Format</h3> + +<p>The config file is, fundamentally, just a piece of Python code which +defines a dictionary named <code>BuildmasterConfig</code>, with a number of +keys that are treated specially. You don't need to know Python to do +basic configuration, though, you can just copy the syntax of the +sample file. If you <em>are</em> comfortable writing Python code, +however, you can use all the power of a full programming language to +achieve more complicated configurations. + + <p>The <code>BuildmasterConfig</code> name is the only one which matters: all +other names defined during the execution of the file are discarded. +When parsing the config file, the Buildmaster generally compares the +old configuration with the new one and performs the minimum set of +actions necessary to bring the buildbot up to date: Builders which are +not changed are left untouched, and Builders which are modified get to +keep their old event history. + + <p>Basic Python syntax: comments start with a hash character (“#”), +tuples are defined with <code>(parenthesis, pairs)</code>, arrays are +defined with <code>[square, brackets]</code>, tuples and arrays are mostly +interchangeable. Dictionaries (data structures which map “keys” to +“values”) are defined with curly braces: <code>{'key1': 'value1', +'key2': 'value2'} </code>. Function calls (and object instantiation) can use +named parameters, like <code>w = html.Waterfall(http_port=8010)</code>. + + <p>The config file starts with a series of <code>import</code> statements, +which make various kinds of Steps and Status targets available for +later use. The main <code>BuildmasterConfig</code> dictionary is created, +then it is populated with a variety of keys. These keys are broken +roughly into the following sections, each of which is documented in +the rest of this chapter: + + <ul> +<li>Project Definitions +<li>Change Sources / Schedulers +<li>Slaveport +<li>Buildslave Configuration +<li>Builders / Interlocks +<li>Status Targets +<li>Debug options +</ul> + + <p>The config file can use a few names which are placed into its namespace: + + <dl> +<dt><code>basedir</code><dd>the base directory for the buildmaster. This string has not been +expanded, so it may start with a tilde. It needs to be expanded before +use. The config file is located in +<code>os.path.expanduser(os.path.join(basedir, 'master.cfg'))</code> + + </dl> + +<div class="node"> +<p><hr> +<a name="Loading-the-Config-File"></a> +Next: <a rel="next" accesskey="n" href="#Testing-the-Config-File">Testing the Config File</a>, +Previous: <a rel="previous" accesskey="p" href="#Config-File-Format">Config File Format</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.2 Loading the Config File</h3> + +<p>The config file is only read at specific points in time. It is first +read when the buildmaster is launched. Once it is running, there are +various ways to ask it to reload the config file. If you are on the +system hosting the buildmaster, you can send a <code>SIGHUP</code> signal to +it: the <samp><span class="command">buildbot</span></samp> tool has a shortcut for this: + +<pre class="example"> buildbot reconfig <var>BASEDIR</var> +</pre> + <p>This command will show you all of the lines from <samp><span class="file">twistd.log</span></samp> +that relate to the reconfiguration. If there are any problems during +the config-file reload, they will be displayed in these lines. + + <p>The debug tool (<code>buildbot debugclient --master HOST:PORT</code>) has a +“Reload .cfg” button which will also trigger a reload. In the +future, there will be other ways to accomplish this step (probably a +password-protected button on the web page, as well as a privileged IRC +command). + + <p>When reloading the config file, the buildmaster will endeavor to +change as little as possible about the running system. For example, +although old status targets may be shut down and new ones started up, +any status targets that were not changed since the last time the +config file was read will be left running and untouched. Likewise any +Builders which have not been changed will be left running. If a +Builder is modified (say, the build process is changed) while a Build +is currently running, that Build will keep running with the old +process until it completes. Any previously queued Builds (or Builds +which get queued after the reconfig) will use the new process. + +<div class="node"> +<p><hr> +<a name="Testing-the-Config-File"></a> +Next: <a rel="next" accesskey="n" href="#Defining-the-Project">Defining the Project</a>, +Previous: <a rel="previous" accesskey="p" href="#Loading-the-Config-File">Loading the Config File</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.3 Testing the Config File</h3> + +<p>To verify that the config file is well-formed and contains no +deprecated or invalid elements, use the “checkconfig” command: + +<pre class="example"> % buildbot checkconfig master.cfg + Config file is good! +</pre> + <p>If the config file has deprecated features (perhaps because you've +upgraded the buildmaster and need to update the config file to match), +they will be announced by checkconfig. In this case, the config file +will work, but you should really remove the deprecated items and use +the recommended replacements instead: + +<pre class="example"> % buildbot checkconfig master.cfg + /usr/lib/python2.4/site-packages/buildbot/master.py:559: DeprecationWarning: c['sources'] is + deprecated as of 0.7.6 and will be removed by 0.8.0 . Please use c['change_source'] instead. + warnings.warn(m, DeprecationWarning) + Config file is good! +</pre> + <p>If the config file is simply broken, that will be caught too: + +<pre class="example"> % buildbot checkconfig master.cfg + Traceback (most recent call last): + File "/usr/lib/python2.4/site-packages/buildbot/scripts/runner.py", line 834, in doCheckConfig + ConfigLoader(configFile) + File "/usr/lib/python2.4/site-packages/buildbot/scripts/checkconfig.py", line 31, in __init__ + self.loadConfig(configFile) + File "/usr/lib/python2.4/site-packages/buildbot/master.py", line 480, in loadConfig + exec f in localDict + File "/home/warner/BuildBot/master/foolscap/master.cfg", line 90, in ? + c[bogus] = "stuff" + NameError: name 'bogus' is not defined +</pre> + <div class="node"> +<p><hr> +<a name="Defining-the-Project"></a> +Next: <a rel="next" accesskey="n" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, +Previous: <a rel="previous" accesskey="p" href="#Testing-the-Config-File">Testing the Config File</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.4 Defining the Project</h3> + +<p>There are a couple of basic settings that you use to tell the buildbot +what project it is working on. This information is used by status +reporters to let users find out more about the codebase being +exercised by this particular Buildbot installation. + +<pre class="example"> c['projectName'] = "Buildbot" + c['projectURL'] = "http://buildbot.sourceforge.net/" + c['buildbotURL'] = "http://localhost:8010/" +</pre> + <p><a name="index-c_005b_0027projectName_0027_005d-13"></a><code>projectName</code> is a short string will be used to describe the +project that this buildbot is working on. For example, it is used as +the title of the waterfall HTML page. + + <p><a name="index-c_005b_0027projectURL_0027_005d-14"></a><code>projectURL</code> is a string that gives a URL for the project as a +whole. HTML status displays will show <code>projectName</code> as a link to +<code>projectURL</code>, to provide a link from buildbot HTML pages to your +project's home page. + + <p><a name="index-c_005b_0027buildbotURL_0027_005d-15"></a>The <code>buildbotURL</code> string should point to the location where the +buildbot's internal web server (usually the <code>html.Waterfall</code> +page) is visible. This typically uses the port number set when you +create the <code>Waterfall</code> object: the buildbot needs your help to +figure out a suitable externally-visible host name. + + <p>When status notices are sent to users (either by email or over IRC), +<code>buildbotURL</code> will be used to create a URL to the specific build +or problem that they are being notified about. It will also be made +available to queriers (over IRC) who want to find out where to get +more information about this buildbot. + + <p><a name="index-c_005b_0027logCompressionLimit_0027_005d-16"></a>The <code>logCompressionLimit</code> enables bz2-compression of build logs on +disk for logs that are bigger than the given size, or disables that +completely if given <code>False</code>. The default value is 4k, which should +be a reasonable default on most file systems. This setting has no impact +on status plugins, and merely affects the required disk space on the +master for build logs. + +<div class="node"> +<p><hr> +<a name="Change-Sources-and-Schedulers"></a> +Next: <a rel="next" accesskey="n" href="#Merging-BuildRequests">Merging BuildRequests</a>, +Previous: <a rel="previous" accesskey="p" href="#Defining-the-Project">Defining the Project</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.5 Change Sources and Schedulers</h3> + +<p><a name="index-c_005b_0027sources_0027_005d-17"></a><a name="index-c_005b_0027change_005fsource_0027_005d-18"></a> +The <code>c['change_source']</code> key is the ChangeSource +instance<a rel="footnote" href="#fn-6" name="fnd-6"><sup>6</sup></a> that +defines how the buildmaster learns about source code changes. More +information about what goes here is available in See <a href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>. + +<pre class="example"> from buildbot.changes.pb import PBChangeSource + c['change_source'] = PBChangeSource() +</pre> + <p><a name="index-c_005b_0027schedulers_0027_005d-19"></a> +(note: in buildbot-0.7.5 and earlier, this key was named +<code>c['sources']</code>, and required a list. <code>c['sources']</code> is +deprecated as of buildbot-0.7.6 and is scheduled to be removed in a +future release). + + <p><code>c['schedulers']</code> is a list of Scheduler instances, each +of which causes builds to be started on a particular set of +Builders. The two basic Scheduler classes you are likely to start +with are <code>Scheduler</code> and <code>Periodic</code>, but you can write a +customized subclass to implement more complicated build scheduling. + + <p>Scheduler arguments +should always be specified by name (as keyword arguments), to allow +for future expansion: + +<pre class="example"> sched = Scheduler(name="quick", builderNames=['lin', 'win']) +</pre> + <p>All schedulers have several arguments in common: + + <dl> +<dt><code>name</code><dd> +Each Scheduler must have a unique name. This is used in status +displays, and is also available in the build property <code>scheduler</code>. + + <br><dt><code>builderNames</code><dd> +This is the set of builders which this scheduler should trigger, specified +as a list of names (strings). + + <br><dt><code>properties</code><dd><a name="index-Properties-20"></a> +This is a dictionary specifying properties that will be transmitted +to all builds started by this scheduler. + + </dl> + + <p>Here is a brief catalog of the available Scheduler types. All these +Schedulers are classes in <code>buildbot.scheduler</code>, and the +docstrings there are the best source of documentation on the arguments +taken by each one. + +<ul class="menu"> +<li><a accesskey="1" href="#Scheduler-Scheduler">Scheduler Scheduler</a> +<li><a accesskey="2" href="#AnyBranchScheduler">AnyBranchScheduler</a> +<li><a accesskey="3" href="#Dependent-Scheduler">Dependent Scheduler</a> +<li><a accesskey="4" href="#Periodic-Scheduler">Periodic Scheduler</a> +<li><a accesskey="5" href="#Nightly-Scheduler">Nightly Scheduler</a> +<li><a accesskey="6" href="#Try-Schedulers">Try Schedulers</a> +<li><a accesskey="7" href="#Triggerable-Scheduler">Triggerable Scheduler</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Scheduler-Scheduler"></a> +Next: <a rel="next" accesskey="n" href="#AnyBranchScheduler">AnyBranchScheduler</a>, +Previous: <a rel="previous" accesskey="p" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.1 Scheduler Scheduler</h4> + +<p><a name="index-buildbot_002escheduler_002eScheduler-21"></a> +This is the original and still most popular Scheduler class. It follows +exactly one branch, and starts a configurable tree-stable-timer after +each change on that branch. When the timer expires, it starts a build +on some set of Builders. The Scheduler accepts a <code>fileIsImportant</code> +function which can be used to ignore some Changes if they do not +affect any “important” files. + + <p>The arguments to this scheduler are: + + <dl> +<dt><code>name</code> +<br><dt><code>builderNames</code> +<br><dt><code>properties</code> +<br><dt><code>branch</code><dd>This Scheduler will pay attention to a single branch, ignoring Changes +that occur on other branches. Setting <code>branch</code> equal to the +special value of <code>None</code> means it should only pay attention to +the default branch. Note that <code>None</code> is a keyword, not a string, +so you want to use <code>None</code> and not <code>"None"</code>. + + <br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the +build. If new changes are made during this interval, the timer will be +restarted, so really the build will be started after a change and then +after this many seconds of inactivity. + + <br><dt><code>fileIsImportant</code><dd>A callable which takes one argument, a Change instance, and returns +<code>True</code> if the change is worth building, and <code>False</code> if +it is not. Unimportant Changes are accumulated until the build is +triggered by an important change. The default value of None means +that all Changes are important. + + <br><dt><code>categories</code><dd>A list of categories of changes that this scheduler will respond to. If this +is specified, then any non-matching changes are ignored. + + </dl> + + <p>Example: + +<pre class="example"> from buildbot import scheduler + quick = scheduler.Scheduler(name="quick", + branch=None, + treeStableTimer=60, + builderNames=["quick-linux", "quick-netbsd"]) + full = scheduler.Scheduler(name="full", + branch=None, + treeStableTimer=5*60, + builderNames=["full-linux", "full-netbsd", "full-OSX"]) + c['schedulers'] = [quick, full] +</pre> + <p>In this example, the two “quick” builders are triggered 60 seconds +after the tree has been changed. The “full” builds do not run quite +so quickly (they wait 5 minutes), so hopefully if the quick builds +fail due to a missing file or really simple typo, the developer can +discover and fix the problem before the full builds are started. Both +Schedulers only pay attention to the default branch: any changes +on other branches are ignored by these Schedulers. Each Scheduler +triggers a different set of Builders, referenced by name. + +<div class="node"> +<p><hr> +<a name="AnyBranchScheduler"></a> +Next: <a rel="next" accesskey="n" href="#Dependent-Scheduler">Dependent Scheduler</a>, +Previous: <a rel="previous" accesskey="p" href="#Scheduler-Scheduler">Scheduler Scheduler</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.2 AnyBranchScheduler</h4> + +<p><a name="index-buildbot_002escheduler_002eAnyBranchScheduler-22"></a> +This scheduler uses a tree-stable-timer like the default one, but +follows multiple branches at once. Each branch gets a separate timer. + + <p>The arguments to this scheduler are: + + <dl> +<dt><code>name</code> +<br><dt><code>builderNames</code> +<br><dt><code>properties</code> +<br><dt><code>branches</code><dd>This Scheduler will pay attention to any number of branches, ignoring +Changes that occur on other branches. Branches are specified just as +for the <code>Scheduler</code> class. + + <br><dt><code>treeStableTimer</code><dd>The Scheduler will wait for this many seconds before starting the +build. If new changes are made during this interval, the timer will be +restarted, so really the build will be started after a change and then +after this many seconds of inactivity. + + <br><dt><code>fileIsImportant</code><dd>A callable which takes one argument, a Change instance, and returns +<code>True</code> if the change is worth building, and <code>False</code> if +it is not. Unimportant Changes are accumulated until the build is +triggered by an important change. The default value of None means +that all Changes are important. +</dl> + +<div class="node"> +<p><hr> +<a name="Dependent-Scheduler"></a> +Next: <a rel="next" accesskey="n" href="#Periodic-Scheduler">Periodic Scheduler</a>, +Previous: <a rel="previous" accesskey="p" href="#AnyBranchScheduler">AnyBranchScheduler</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.3 Dependent Scheduler</h4> + +<p><a name="index-Dependent-23"></a><a name="index-Dependencies-24"></a><a name="index-buildbot_002escheduler_002eDependent-25"></a> +It is common to wind up with one kind of build which should only be +performed if the same source code was successfully handled by some +other kind of build first. An example might be a packaging step: you +might only want to produce .deb or RPM packages from a tree that was +known to compile successfully and pass all unit tests. You could put +the packaging step in the same Build as the compile and testing steps, +but there might be other reasons to not do this (in particular you +might have several Builders worth of compiles/tests, but only wish to +do the packaging once). Another example is if you want to skip the +“full” builds after a failing “quick” build of the same source +code. Or, if one Build creates a product (like a compiled library) +that is used by some other Builder, you'd want to make sure the +consuming Build is run <em>after</em> the producing one. + + <p>You can use “Dependencies” to express this relationship +to the Buildbot. There is a special kind of Scheduler named +<code>scheduler.Dependent</code> that will watch an “upstream” Scheduler +for builds to complete successfully (on all of its Builders). Each time +that happens, the same source code (i.e. the same <code>SourceStamp</code>) +will be used to start a new set of builds, on a different set of +Builders. This “downstream” scheduler doesn't pay attention to +Changes at all. It only pays attention to the upstream scheduler. + + <p>If the build fails on any of the Builders in the upstream set, +the downstream builds will not fire. Note that, for SourceStamps +generated by a ChangeSource, the <code>revision</code> is None, meaning HEAD. +If any changes are committed between the time the upstream scheduler +begins its build and the time the dependent scheduler begins its +build, then those changes will be included in the downstream build. +See the see <a href="#Triggerable-Scheduler">Triggerable Scheduler</a> for a more flexible dependency +mechanism that can avoid this problem. + + <p>The arguments to this scheduler are: + + <dl> +<dt><code>name</code> +<br><dt><code>builderNames</code> +<br><dt><code>properties</code> +<br><dt><code>upstream</code><dd>The upstream scheduler to watch. Note that this is an “instance”, +not the name of the scheduler. +</dl> + + <p>Example: + +<pre class="example"> from buildbot import scheduler + tests = scheduler.Scheduler("just-tests", None, 5*60, + ["full-linux", "full-netbsd", "full-OSX"]) + package = scheduler.Dependent("build-package", + tests, # upstream scheduler -- no quotes! + ["make-tarball", "make-deb", "make-rpm"]) + c['schedulers'] = [tests, package] +</pre> + <div class="node"> +<p><hr> +<a name="Periodic-Scheduler"></a> +Next: <a rel="next" accesskey="n" href="#Nightly-Scheduler">Nightly Scheduler</a>, +Previous: <a rel="previous" accesskey="p" href="#Dependent-Scheduler">Dependent Scheduler</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.4 Periodic Scheduler</h4> + +<p><a name="index-buildbot_002escheduler_002ePeriodic-26"></a> +This simple scheduler just triggers a build every N seconds. + + <p>The arguments to this scheduler are: + + <dl> +<dt><code>name</code> +<br><dt><code>builderNames</code> +<br><dt><code>properties</code> +<br><dt><code>periodicBuildTimer</code><dd>The time, in seconds, after which to start a build. +</dl> + + <p>Example: + +<pre class="example"> from buildbot import scheduler + nightly = scheduler.Periodic(name="nightly", + builderNames=["full-solaris"], + periodicBuildTimer=24*60*60) + c['schedulers'] = [nightly] +</pre> + <p>The Scheduler in this example just runs the full solaris build once +per day. Note that this Scheduler only lets you control the time +between builds, not the absolute time-of-day of each Build, so this +could easily wind up a “daily” or “every afternoon” scheduler +depending upon when it was first activated. + +<div class="node"> +<p><hr> +<a name="Nightly-Scheduler"></a> +Next: <a rel="next" accesskey="n" href="#Try-Schedulers">Try Schedulers</a>, +Previous: <a rel="previous" accesskey="p" href="#Periodic-Scheduler">Periodic Scheduler</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.5 Nightly Scheduler</h4> + +<p><a name="index-buildbot_002escheduler_002eNightly-27"></a> +This is highly configurable periodic build scheduler, which triggers +a build at particular times of day, week, month, or year. The +configuration syntax is very similar to the well-known <code>crontab</code> +format, in which you provide values for minute, hour, day, and month +(some of which can be wildcards), and a build is triggered whenever +the current time matches the given constraints. This can run a build +every night, every morning, every weekend, alternate Thursdays, +on your boss's birthday, etc. + + <p>Pass some subset of <code>minute</code>, <code>hour</code>, <code>dayOfMonth</code>, +<code>month</code>, and <code>dayOfWeek</code>; each may be a single number or +a list of valid values. The builds will be triggered whenever the +current time matches these values. Wildcards are represented by a +'*' string. All fields default to a wildcard except 'minute', so +with no fields this defaults to a build every hour, on the hour. +The full list of parameters is: + + <dl> +<dt><code>name</code> +<br><dt><code>builderNames</code> +<br><dt><code>properties</code> +<br><dt><code>branch</code><dd>The branch to build, just as for <code>Scheduler</code>. + + <br><dt><code>minute</code><dd>The minute of the hour on which to start the build. This defaults +to 0, meaning an hourly build. + + <br><dt><code>hour</code><dd>The hour of the day on which to start the build, in 24-hour notation. +This defaults to *, meaning every hour. + + <br><dt><code>month</code><dd>The month in which to start the build, with January = 1. This defaults +to *, meaning every month. + + <br><dt><code>dayOfWeek</code><dd>The day of the week to start a build, with Monday = 0. This defauls +to *, meaning every day of the week. + + <br><dt><code>onlyIfChanged</code><dd>If this is true, then builds will not be scheduled at the designated time +unless the source has changed since the previous build. +</dl> + + <p>For example, the following master.cfg clause will cause a build to be +started every night at 3:00am: + +<pre class="example"> s = scheduler.Nightly(name='nightly', + builderNames=['builder1', 'builder2'], + hour=3, + minute=0) +</pre> + <p>This scheduler will perform a build each monday morning at 6:23am and +again at 8:23am, but only if someone has committed code in the interim: + +<pre class="example"> s = scheduler.Nightly(name='BeforeWork', + builderNames=['builder1'], + dayOfWeek=0, + hour=[6,8], + minute=23, + onlyIfChanged=True) +</pre> + <p>The following runs a build every two hours, using Python's <code>range</code> +function: + +<pre class="example"> s = Nightly(name='every2hours', + builderNames=['builder1'], + hour=range(0, 24, 2)) +</pre> + <p>Finally, this example will run only on December 24th: + +<pre class="example"> s = Nightly(name='SleighPreflightCheck', + builderNames=['flying_circuits', 'radar'], + month=12, + dayOfMonth=24, + hour=12, + minute=0) +</pre> + <div class="node"> +<p><hr> +<a name="Try-Schedulers"></a> +Next: <a rel="next" accesskey="n" href="#Triggerable-Scheduler">Triggerable Scheduler</a>, +Previous: <a rel="previous" accesskey="p" href="#Nightly-Scheduler">Nightly Scheduler</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.6 Try Schedulers</h4> + +<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-28"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-29"></a> +This scheduler allows developers to use the <code>buildbot try</code> +command to trigger builds of code they have not yet committed. See +<a href="#try">try</a> for complete details. + + <p>Two implementations are available: <code>Try_Jobdir</code> and +<code>Try_Userpass</code>. The former monitors a job directory, specified +by the <code>jobdir</code> parameter, while the latter listens for PB +connections on a specific <code>port</code>, and authenticates against +<code>userport</code>. + +<div class="node"> +<p><hr> +<a name="Triggerable-Scheduler"></a> +Previous: <a rel="previous" accesskey="p" href="#Try-Schedulers">Try Schedulers</a>, +Up: <a rel="up" accesskey="u" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a> + +</div> + +<h4 class="subsection">4.5.7 Triggerable Scheduler</h4> + +<p><a name="index-Triggers-30"></a><a name="index-buildbot_002escheduler_002eTriggerable-31"></a> +The <code>Triggerable</code> scheduler waits to be triggered by a Trigger +step (see <a href="#Triggering-Schedulers">Triggering Schedulers</a>) in another build. That step +can optionally wait for the scheduler's builds to complete. This +provides two advantages over Dependent schedulers. First, the same +scheduler can be triggered from multiple builds. Second, the ability +to wait for a Triggerable's builds to complete provides a form of +"subroutine call", where one or more builds can "call" a scheduler +to perform some work for them, perhaps on other buildslaves. + + <p>The parameters are just the basics: + + <dl> +<dt><code>name</code><br><dt><code>builderNames</code><br><dt><code>properties</code><dd></dl> + + <p>This class is only useful in conjunction with the <code>Trigger</code> step. +Here is a fully-worked example: + +<pre class="example"> from buildbot import scheduler + from buildbot.process import factory + from buildbot.steps import trigger + + checkin = scheduler.Scheduler(name="checkin", + branch=None, + treeStableTimer=5*60, + builderNames=["checkin"]) + nightly = scheduler.Nightly(name='nightly', + builderNames=['nightly'], + hour=3, + minute=0) + + mktarball = scheduler.Triggerable(name="mktarball", + builderNames=["mktarball"]) + build = scheduler.Triggerable(name="build-all-platforms", + builderNames=["build-all-platforms"]) + test = scheduler.Triggerable(name="distributed-test", + builderNames=["distributed-test"]) + package = scheduler.Triggerable(name="package-all-platforms", + builderNames=["package-all-platforms"]) + + c['schedulers'] = [checkin, nightly, build, test, package] + + # on checkin, make a tarball, build it, and test it + checkin_factory = factory.BuildFactory() + checkin_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) + checkin_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) + checkin_factory.addStep(trigger.Trigger(schedulerNames=['distributed-test'], + waitForFinish=True)) + + # and every night, make a tarball, build it, and package it + nightly_factory = factory.BuildFactory() + nightly_factory.addStep(trigger.Trigger(schedulerNames=['mktarball'], + waitForFinish=True)) + nightly_factory.addStep(trigger.Trigger(schedulerNames=['build-all-platforms'], + waitForFinish=True)) + nightly_factory.addStep(trigger.Trigger(schedulerNames=['package-all-platforms'], + waitForFinish=True)) +</pre> + <div class="node"> +<p><hr> +<a name="Merging-BuildRequests"></a> +Next: <a rel="next" accesskey="n" href="#Setting-the-slaveport">Setting the slaveport</a>, +Previous: <a rel="previous" accesskey="p" href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.6 Merging BuildRequests</h3> + +<p><a name="index-c_005b_0027mergeRequests_0027_005d-32"></a> +By default, buildbot merges BuildRequests that have the compatible +SourceStamps. This behaviour can be customized with the +<code>c['mergeRequests']</code> configuration key. This key specifies a function +which is caleld with three arguments: a <code>Builder</code> and two +<code>BuildRequest</code> objects. It should return true if the requests can be +merged. For example: + +<pre class="example"> def mergeRequests(builder, req1, req2): + """Don't merge buildrequest at all""" + return False + c['mergeRequests'] = mergeRequests +</pre> + <p>In many cases, the details of the SourceStamps and BuildRequests are important. +In this example, only BuildRequests with the same "reason" are merged; thus +developers forcing builds for different reasons will see distinct builds. + +<pre class="example"> def mergeRequests(builder, req1, req2): + if req1.source.canBeMergedWith(req2.source) and req1.reason == req2.reason: + return True + return False + c['mergeRequests'] = mergeRequests +</pre> + <div class="node"> +<p><hr> +<a name="Setting-the-slaveport"></a> +Next: <a rel="next" accesskey="n" href="#Buildslave-Specifiers">Buildslave Specifiers</a>, +Previous: <a rel="previous" accesskey="p" href="#Merging-BuildRequests">Merging BuildRequests</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.7 Setting the slaveport</h3> + +<p><a name="index-c_005b_0027slavePortnum_0027_005d-33"></a> +The buildmaster will listen on a TCP port of your choosing for +connections from buildslaves. It can also use this port for +connections from remote Change Sources, status clients, and debug +tools. This port should be visible to the outside world, and you'll +need to tell your buildslave admins about your choice. + + <p>It does not matter which port you pick, as long it is externally +visible, however you should probably use something larger than 1024, +since most operating systems don't allow non-root processes to bind to +low-numbered ports. If your buildmaster is behind a firewall or a NAT +box of some sort, you may have to configure your firewall to permit +inbound connections to this port. + +<pre class="example"> c['slavePortnum'] = 10000 +</pre> + <p><code>c['slavePortnum']</code> is a <em>strports</em> specification string, +defined in the <code>twisted.application.strports</code> module (try +<samp><span class="command">pydoc twisted.application.strports</span></samp> to get documentation on +the format). This means that you can have the buildmaster listen on a +localhost-only port by doing: + +<pre class="example"> c['slavePortnum'] = "tcp:10000:interface=127.0.0.1" +</pre> + <p>This might be useful if you only run buildslaves on the same machine, +and they are all configured to contact the buildmaster at +<code>localhost:10000</code>. + +<div class="node"> +<p><hr> +<a name="Buildslave-Specifiers"></a> +Next: <a rel="next" accesskey="n" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a>, +Previous: <a rel="previous" accesskey="p" href="#Setting-the-slaveport">Setting the slaveport</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.8 Buildslave Specifiers</h3> + +<p><a name="index-c_005b_0027slaves_0027_005d-34"></a> +The <code>c['slaves']</code> key is a list of known buildslaves. In the common case, +each buildslave is defined by an instance of the BuildSlave class. It +represents a standard, manually started machine that will try to connect to +the buildbot master as a slave. Contrast these with the "on-demand" latent +buildslaves, such as the Amazon Web Service Elastic Compute Cloud latent +buildslave discussed below. + + <p>The BuildSlave class is instantiated with two values: (slavename, +slavepassword). These are the same two values that need to be provided to the +buildslave administrator when they create the buildslave. + + <p>The slavenames must be unique, of course. The password exists to +prevent evildoers from interfering with the buildbot by inserting +their own (broken) buildslaves into the system and thus displacing the +real ones. + + <p>Buildslaves with an unrecognized slavename or a non-matching password +will be rejected when they attempt to connect, and a message +describing the problem will be put in the log file (see <a href="#Logfiles">Logfiles</a>). + +<pre class="example"> from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd') + BuildSlave('bot-bsd', 'bsdpasswd') + ] +</pre> + <p><a name="index-Properties-35"></a><code>BuildSlave</code> objects can also be created with an optional +<code>properties</code> argument, a dictionary specifying properties that +will be available to any builds performed on this slave. For example: + +<pre class="example"> from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + properties={'os':'solaris'}), + ] +</pre> + <p>The <code>BuildSlave</code> constructor can also take an optional +<code>max_builds</code> parameter to limit the number of builds that it +will execute simultaneously: + +<pre class="example"> from buildbot.buildslave import BuildSlave + c['slaves'] = [BuildSlave("bot-linux", "linuxpassword", max_builds=2)] +</pre> + <p>Historical note: in buildbot-0.7.5 and earlier, the <code>c['bots']</code> +key was used instead, and it took a list of (name, password) tuples. +This key is accepted for backwards compatibility, but is deprecated as +of 0.7.6 and will go away in some future release. + +<ul class="menu"> +<li><a accesskey="1" href="#When-Buildslaves-Go-Missing">When Buildslaves Go Missing</a> +</ul> + +<div class="node"> +<p><hr> +<a name="When-Buildslaves-Go-Missing"></a> +Up: <a rel="up" accesskey="u" href="#Buildslave-Specifiers">Buildslave Specifiers</a> + +</div> + +<h4 class="subsection">4.8.1 When Buildslaves Go Missing</h4> + +<p>Sometimes, the buildslaves go away. One very common reason for this is +when the buildslave process is started once (manually) and left +running, but then later the machine reboots and the process is not +automatically restarted. + + <p>If you'd like to have the administrator of the buildslave (or other +people) be notified by email when the buildslave has been missing for +too long, just add the <code>notify_on_missing=</code> argument to the +<code>BuildSlave</code> definition: + +<pre class="example"> c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing="bob@example.com"), + ] +</pre> + <p>By default, this will send email when the buildslave has been +disconnected for more than one hour. Only one email per +connection-loss event will be sent. To change the timeout, use +<code>missing_timeout=</code> and give it a number of seconds (the default +is 3600). + + <p>You can have the buildmaster send email to multiple recipients: just +provide a list of addresses instead of a single one: + +<pre class="example"> c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing=["bob@example.com", + "alice@example.org"], + missing_timeout=300, # notify after 5 minutes + ), + ] +</pre> + <p>The email sent this way will use a MailNotifier (see <a href="#MailNotifier">MailNotifier</a>) +status target, if one is configured. This provides a way for you to +control the “from” address of the email, as well as the relayhost +(aka “smarthost”) to use as an SMTP server. If no MailNotifier is +configured on this buildmaster, the buildslave-missing emails will be +sent using a default configuration. + + <p>Note that if you want to have a MailNotifier for buildslave-missing +emails but not for regular build emails, just create one with +builders=[], as follows: + +<pre class="example"> from buildbot.status import mail + m = mail.MailNotifier(fromaddr="buildbot@localhost", builders=[], + relayhost="smtp.example.org") + c['status'].append(m) + c['slaves'] = [BuildSlave('bot-solaris', 'solarispasswd', + notify_on_missing="bob@example.com"), + ] +</pre> + <div class="node"> +<p><hr> +<a name="On-Demand-(%22Latent%22)-Buildslaves"></a> +<a name="On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves"></a> +Next: <a rel="next" accesskey="n" href="#Defining-Global-Properties">Defining Global Properties</a>, +Previous: <a rel="previous" accesskey="p" href="#Buildslave-Specifiers">Buildslave Specifiers</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.9 On-Demand ("Latent") Buildslaves</h3> + +<p>The standard buildbot model has slaves started manually. The previous section +described how to configure the master for this approach. + + <p>Another approach is to let the buildbot master start slaves when builds are +ready, on-demand. Thanks to services such as Amazon Web Services' Elastic +Compute Cloud ("AWS EC2"), this is relatively easy to set up, and can be +very useful for some situations. + + <p>The buildslaves that are started on-demand are called "latent" buildslaves. +As of this writing, buildbot ships with an abstract base class for building +latent buildslaves, and a concrete implementation for AWS EC2. + +<ul class="menu"> +<li><a accesskey="1" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> +<li><a accesskey="2" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a> +<li><a accesskey="3" href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Amazon-Web-Services-Elastic-Compute-Cloud-(%22AWS-EC2%22)"></a> +<a name="Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029"></a> +Next: <a rel="next" accesskey="n" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a>, +Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> + +</div> + +<h4 class="subsection">4.9.1 Amazon Web Services Elastic Compute Cloud ("AWS EC2")</h4> + +<p><a href="http://aws.amazon.com/ec2/">AWS EC2</a> is a web service that allows you to +start virtual machines in an Amazon data center. Please see their website for +details, incuding costs. Using the AWS EC2 latent buildslaves involves getting +an EC2 account with AWS and setting up payment; customizing one or more EC2 +machine images ("AMIs") on your desired operating system(s) and publishing +them (privately if needed); and configuring the buildbot master to know how to +start your customized images for "substantiating" your latent slaves. + +<ul class="menu"> +<li><a accesskey="1" href="#Get-an-AWS-EC2-Account">Get an AWS EC2 Account</a> +<li><a accesskey="2" href="#Create-an-AMI">Create an AMI</a> +<li><a accesskey="3" href="#Configure-the-Master-with-an-EC2LatentBuildSlave">Configure the Master with an EC2LatentBuildSlave</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Get-an-AWS-EC2-Account"></a> +Next: <a rel="next" accesskey="n" href="#Create-an-AMI">Create an AMI</a>, +Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> + +</div> + +<h5 class="subsubsection">4.9.1.1 Get an AWS EC2 Account</h5> + +<p>To start off, to use the AWS EC2 latent buildslave, you need to get an AWS +developer account and sign up for EC2. These instructions may help you get +started: + + <ul> +<li>Go to http://aws.amazon.com/ and click to "Sign Up Now" for an AWS account. + + <li>Once you are logged into your account, you need to sign up for EC2. +Instructions for how to do this have changed over time because Amazon changes +their website, so the best advice is to hunt for it. After signing up for EC2, +it may say it wants you to upload an x.509 cert. You will need this to create +images (see below) but it is not technically necessary for the buildbot master +configuration. + + <li>You must enter a valid credit card before you will be able to use EC2. Do that +under 'Payment Method'. + + <li>Make sure you're signed up for EC2 by going to 'Your Account'->'Account +Activity' and verifying EC2 is listed. +</ul> + +<div class="node"> +<p><hr> +<a name="Create-an-AMI"></a> +Next: <a rel="next" accesskey="n" href="#Configure-the-Master-with-an-EC2LatentBuildSlave">Configure the Master with an EC2LatentBuildSlave</a>, +Previous: <a rel="previous" accesskey="p" href="#Get-an-AWS-EC2-Account">Get an AWS EC2 Account</a>, +Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> + +</div> + +<h5 class="subsubsection">4.9.1.2 Create an AMI</h5> + +<p>Now you need to create an AMI and configure the master. You may need to +run through this cycle a few times to get it working, but these instructions +should get you started. + + <p>Creating an AMI is out of the scope of this document. The +<a href="http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/">EC2 Getting Started Guide</a> +is a good resource for this task. Here are a few additional hints. + + <ul> +<li>When an instance of the image starts, it needs to automatically start a +buildbot slave that connects to your master (to create a buildbot slave, +see <a href="#Creating-a-buildslave">Creating a buildslave</a>; to make a daemon, +see <a href="#Launching-the-daemons">Launching the daemons</a>). + + <li>You may want to make an instance of the buildbot slave, configure it as a +standard buildslave in the master (i.e., not as a latent slave), and test and +debug it that way before you turn it into an AMI and convert to a latent +slave in the master. +</ul> + +<div class="node"> +<p><hr> +<a name="Configure-the-Master-with-an-EC2LatentBuildSlave"></a> +Previous: <a rel="previous" accesskey="p" href="#Create-an-AMI">Create an AMI</a>, +Up: <a rel="up" accesskey="u" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a> + +</div> + +<h5 class="subsubsection">4.9.1.3 Configure the Master with an EC2LatentBuildSlave</h5> + +<p>Now let's assume you have an AMI that should work with the +EC2LatentBuildSlave. It's now time to set up your buildbot master +configuration. + + <p>You will need some information from your AWS account: the "Access Key Id" and +the "Secret Access Key". If you've built the AMI yourself, you probably +already are familiar with these values. If you have not, and someone has +given you access to an AMI, these hints may help you find the necessary +values: + + <ul> +<li>While logged into your AWS account, find the "Access Identifiers" link (either +on the left, or via "Your Account" -> "Access Identifiers". + + <li>On the page, you'll see alphanumeric values for "Your Access Key Id:" and +"Your Secret Access Key:". Make a note of these. Later on, we'll call the +first one your "identifier" and the second one your "secret_identifier." +</ul> + + <p>When creating an EC2LatentBuildSlave in the buildbot master configuration, +the first three arguments are required. The name and password are the first +two arguments, and work the same as with normal buildslaves. The next +argument specifies the type of the EC2 virtual machine (available options as +of this writing include "m1.small", "m1.large", 'm1.xlarge", "c1.medium", +and "c1.xlarge"; see the EC2 documentation for descriptions of these +machines). + + <p>Here is the simplest example of configuring an EC2 latent buildslave. It +specifies all necessary remaining values explicitly in the instantiation. + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345', + identifier='publickey', + secret_identifier='privatekey' + )] +</pre> + <p>The "ami" argument specifies the AMI that the master should start. The +"identifier" argument specifies the AWS "Access Key Id," and the +"secret_identifier" specifies the AWS "Secret Access Key." Both the AMI and +the account information can be specified in alternate ways. + + <p>Note that whoever has your identifier and secret_identifier values can request +AWS work charged to your account, so these values need to be carefully +protected. Another way to specify these access keys is to put them in a +separate file. You can then make the access privileges stricter for this +separate file, and potentially let more people read your main configuration +file. + + <p>By default, you can make an .ec2 directory in the home folder of the user +running the buildbot master. In that directory, create a file called aws_id. +The first line of that file should be your access key id; the second line +should be your secret access key id. Then you can instantiate the build slave +as follows. + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + ami='ami-12345')] +</pre> + <p>If you want to put the key information in another file, use the +"aws_id_file_path" initialization argument. + + <p>Previous examples used a particular AMI. If the Buildbot master will be +deployed in a process-controlled environment, it may be convenient to +specify the AMI more flexibly. Rather than specifying an individual AMI, +specify one or two AMI filters. + + <p>In all cases, the AMI that sorts last by its location (the S3 bucket and +manifest name) will be preferred. + + <p>One available filter is to specify the acceptable AMI owners, by AWS account +number (the 12 digit number, usually rendered in AWS with hyphens like +"1234-5678-9012", should be entered as in integer). + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + valid_ami_owners=[11111111111, + 22222222222], + identifier='publickey', + secret_identifier='privatekey' + ) +</pre> + <p>The other available filter is to provide a regular expression string that +will be matched against each AMI's location (the S3 bucket and manifest name). + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +</pre> + <p>The regular expression can specify a group, which will be preferred for the +sorting. Only the first group is used; subsequent groups are ignored. + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(.*)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +</pre> + <p>If the group can be cast to an integer, it will be. This allows 10 to sort +after 1, for instance. + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + bot1 = EC2LatentBuildSlave( + 'bot1', 'sekrit', 'm1.large', + valid_ami_location_regex=r'buildbot\-.*\-(\d+)/image.manifest.xml', + identifier='publickey', secret_identifier='privatekey') +</pre> + <p>In addition to using the password as a handshake between the master and the +slave, you may want to use a firewall to assert that only machines from a +specific IP can connect as slaves. This is possible with AWS EC2 by using +the Elastic IP feature. To configure, generate a Elastic IP in AWS, and then +specify it in your configuration using the "elastic_ip" argument. + +<pre class="example"> from buildbot.ec2buildslave import EC2LatentBuildSlave + c['slaves'] = [EC2LatentBuildSlave('bot1', 'sekrit', 'm1.large', + 'ami-12345', + identifier='publickey', + secret_identifier='privatekey', + elastic_ip='208.77.188.166' + )] +</pre> + <p>The EC2LatentBuildSlave supports all other configuration from the standard +BuildSlave. The "missing_timeout" and "notify_on_missing" specify how long +to wait for an EC2 instance to attach before considering the attempt to have +failed, and email addresses to alert, respectively. "missing_timeout" +defaults to 20 minutes. + + <p>The "build_wait_timeout" allows you to specify how long an EC2LatentBuildSlave +should wait after a build for another build before it shuts down the EC2 +instance. It defaults to 10 minutes. + + <p>"keypair_name" and "security_name" allow you to specify different names for +these AWS EC2 values. They both default to "latent_buildbot_slave". + +<div class="node"> +<p><hr> +<a name="Dangers-with-Latent-Buildslaves"></a> +Next: <a rel="next" accesskey="n" href="#Writing-New-Latent-Buildslaves">Writing New Latent Buildslaves</a>, +Previous: <a rel="previous" accesskey="p" href="#Amazon-Web-Services-Elastic-Compute-Cloud-_0028_0022AWS-EC2_0022_0029">Amazon Web Services Elastic Compute Cloud ("AWS EC2")</a>, +Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> + +</div> + +<h4 class="subsection">4.9.2 Dangers with Latent Buildslaves</h4> + +<p>Any latent build slave that interacts with a for-fee service, such as the +EC2LatentBuildSlave, brings significant risks. As already identified, the +configuraton will need access to account information that, if obtained by a +criminal, can be used to charge services to your account. Also, bugs in the +buildbot software may lead to unnecessary charges. In particular, if the +master neglects to shut down an instance for some reason, a virtual machine +may be running unnecessarily, charging against your account. Manual and/or +automatic (e.g. nagios with a plugin using a library like boto) +double-checking may be appropriate. + + <p>A comparitively trivial note is that currently if two instances try to attach +to the same latent buildslave, it is likely that the system will become +confused. This should not occur, unless, for instance, you configure a normal +build slave to connect with the authentication of a latent buildbot. If the +situation occurs, stop all attached instances and restart the master. + +<div class="node"> +<p><hr> +<a name="Writing-New-Latent-Buildslaves"></a> +Previous: <a rel="previous" accesskey="p" href="#Dangers-with-Latent-Buildslaves">Dangers with Latent Buildslaves</a>, +Up: <a rel="up" accesskey="u" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a> + +</div> + +<h4 class="subsection">4.9.3 Writing New Latent Buildslaves</h4> + +<p>Writing a new latent buildslave should only require subclassing +<code>buildbot.buildslave.AbstractLatentBuildSlave</code> and implementing +start_instance and stop_instance. + +<pre class="example"> def start_instance(self): + # responsible for starting instance that will try to connect with this + # master. Should return deferred. Problems should use an errback. The + # callback value can be None, or can be an iterable of short strings to + # include in the "substantiate success" status message, such as + # identifying the instance that started. + raise NotImplementedError + + def stop_instance(self, fast=False): + # responsible for shutting down instance. Return a deferred. If `fast`, + # we're trying to shut the master down, so callback as soon as is safe. + # Callback value is ignored. + raise NotImplementedError +</pre> + <p>See <code>buildbot.ec2buildslave.EC2LatentBuildSlave</code> for an example, or see the +test example <code>buildbot.test_slaves.FakeLatentBuildSlave</code>. + +<div class="node"> +<p><hr> +<a name="Defining-Global-Properties"></a> +Next: <a rel="next" accesskey="n" href="#Defining-Builders">Defining Builders</a>, +Previous: <a rel="previous" accesskey="p" href="#On_002dDemand-_0028_0022Latent_0022_0029-Buildslaves">On-Demand ("Latent") Buildslaves</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.10 Defining Global Properties</h3> + +<p><a name="index-c_005b_0027properties_0027_005d-36"></a><a name="index-Properties-37"></a> +The <code>'properties'</code> configuration key defines a dictionary +of properties that will be available to all builds started by the +buildmaster: + +<pre class="example"> c['properties'] = { + 'Widget-version' : '1.2', + 'release-stage' : 'alpha' + } +</pre> + <div class="node"> +<p><hr> +<a name="Defining-Builders"></a> +Next: <a rel="next" accesskey="n" href="#Defining-Status-Targets">Defining Status Targets</a>, +Previous: <a rel="previous" accesskey="p" href="#Defining-Global-Properties">Defining Global Properties</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.11 Defining Builders</h3> + +<p><a name="index-c_005b_0027builders_0027_005d-38"></a> +The <code>c['builders']</code> key is a list of dictionaries which specify +the Builders. The Buildmaster runs a collection of Builders, each of +which handles a single type of build (e.g. full versus quick), on a +single build slave. A Buildbot which makes sure that the latest code +(“HEAD”) compiles correctly across four separate architecture will +have four Builders, each performing the same build but on different +slaves (one per platform). + + <p>Each Builder gets a separate column in the waterfall display. In +general, each Builder runs independently (although various kinds of +interlocks can cause one Builder to have an effect on another). + + <p>Each Builder specification dictionary has several required keys: + + <dl> +<dt><code>name</code><dd>This specifies the Builder's name, which is used in status +reports. + + <br><dt><code>slavename</code><dd>This specifies which buildslave will be used by this Builder. +<code>slavename</code> must appear in the <code>c['slaves']</code> list. Each +buildslave can accomodate multiple Builders. + + <br><dt><code>slavenames</code><dd>If you provide <code>slavenames</code> instead of <code>slavename</code>, you can +give a list of buildslaves which are capable of running this Builder. +If multiple buildslaves are available for any given Builder, you will +have some measure of redundancy: in case one slave goes offline, the +others can still keep the Builder working. In addition, multiple +buildslaves will allow multiple simultaneous builds for the same +Builder, which might be useful if you have a lot of forced or “try” +builds taking place. + + <p>If you use this feature, it is important to make sure that the +buildslaves are all, in fact, capable of running the given build. The +slave hosts should be configured similarly, otherwise you will spend a +lot of time trying (unsuccessfully) to reproduce a failure that only +occurs on some of the buildslaves and not the others. Different +platforms, operating systems, versions of major programs or libraries, +all these things mean you should use separate Builders. + + <br><dt><code>builddir</code><dd>This specifies the name of a subdirectory (under the base directory) +in which everything related to this builder will be placed. On the +buildmaster, this holds build status information. On the buildslave, +this is where checkouts, compiles, and tests are run. + + <br><dt><code>factory</code><dd>This is a <code>buildbot.process.factory.BuildFactory</code> instance which +controls how the build is performed. Full details appear in their own +chapter, See <a href="#Build-Process">Build Process</a>. Parameters like the location of the CVS +repository and the compile-time options used for the build are +generally provided as arguments to the factory's constructor. + + </dl> + + <p>Other optional keys may be set on each Builder: + + <dl> +<dt><code>category</code><dd>If provided, this is a string that identifies a category for the +builder to be a part of. Status clients can limit themselves to a +subset of the available categories. A common use for this is to add +new builders to your setup (for a new module, or for a new buildslave) +that do not work correctly yet and allow you to integrate them with +the active builders. You can put these new builders in a test +category, make your main status clients ignore them, and have only +private status clients pick them up. As soon as they work, you can +move them over to the active category. + + </dl> + +<div class="node"> +<p><hr> +<a name="Defining-Status-Targets"></a> +Next: <a rel="next" accesskey="n" href="#Debug-options">Debug options</a>, +Previous: <a rel="previous" accesskey="p" href="#Defining-Builders">Defining Builders</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.12 Defining Status Targets</h3> + +<p>The Buildmaster has a variety of ways to present build status to +various users. Each such delivery method is a “Status Target” object +in the configuration's <code>status</code> list. To add status targets, you +just append more objects to this list: + + <p><a name="index-c_005b_0027status_0027_005d-39"></a> +<pre class="example"> c['status'] = [] + + from buildbot.status import html + c['status'].append(html.Waterfall(http_port=8010)) + + from buildbot.status import mail + m = mail.MailNotifier(fromaddr="buildbot@localhost", + extraRecipients=["builds@lists.example.com"], + sendToInterestedUsers=False) + c['status'].append(m) + + from buildbot.status import words + c['status'].append(words.IRC(host="irc.example.com", nick="bb", + channels=["#example"])) +</pre> + <p>Status delivery has its own chapter, See <a href="#Status-Delivery">Status Delivery</a>, in which +all the built-in status targets are documented. + +<div class="node"> +<p><hr> +<a name="Debug-options"></a> +Previous: <a rel="previous" accesskey="p" href="#Defining-Status-Targets">Defining Status Targets</a>, +Up: <a rel="up" accesskey="u" href="#Configuration">Configuration</a> + +</div> + +<h3 class="section">4.13 Debug options</h3> + +<p><a name="index-c_005b_0027debugPassword_0027_005d-40"></a>If you set <code>c['debugPassword']</code>, then you can connect to the +buildmaster with the diagnostic tool launched by <code>buildbot +debugclient MASTER:PORT</code>. From this tool, you can reload the config +file, manually force builds, and inject changes, which may be useful +for testing your buildmaster without actually commiting changes to +your repository (or before you have the Change Sources set up). The +debug tool uses the same port number as the slaves do: +<code>c['slavePortnum']</code>, and is authenticated with this password. + +<pre class="example"> c['debugPassword'] = "debugpassword" +</pre> + <p><a name="index-c_005b_0027manhole_0027_005d-41"></a>If you set <code>c['manhole']</code> to an instance of one of the classes in +<code>buildbot.manhole</code>, you can telnet or ssh into the buildmaster +and get an interactive Python shell, which may be useful for debugging +buildbot internals. It is probably only useful for buildbot +developers. It exposes full access to the buildmaster's account +(including the ability to modify and delete files), so it should not +be enabled with a weak or easily guessable password. + + <p>There are three separate <code>Manhole</code> classes. Two of them use SSH, +one uses unencrypted telnet. Two of them use a username+password +combination to grant access, one of them uses an SSH-style +<samp><span class="file">authorized_keys</span></samp> file which contains a list of ssh public keys. + + <dl> +<dt><code>manhole.AuthorizedKeysManhole</code><dd>You construct this with the name of a file that contains one SSH +public key per line, just like <samp><span class="file">~/.ssh/authorized_keys</span></samp>. If you +provide a non-absolute filename, it will be interpreted relative to +the buildmaster's base directory. + + <br><dt><code>manhole.PasswordManhole</code><dd>This one accepts SSH connections but asks for a username and password +when authenticating. It accepts only one such pair. + + <br><dt><code>manhole.TelnetManhole</code><dd>This accepts regular unencrypted telnet connections, and asks for a +username/password pair before providing access. Because this +username/password is transmitted in the clear, and because Manhole +access to the buildmaster is equivalent to granting full shell +privileges to both the buildmaster and all the buildslaves (and to all +accounts which then run code produced by the buildslaves), it is +highly recommended that you use one of the SSH manholes instead. + + </dl> + +<pre class="example"> # some examples: + from buildbot import manhole + c['manhole'] = manhole.AuthorizedKeysManhole(1234, "authorized_keys") + c['manhole'] = manhole.PasswordManhole(1234, "alice", "mysecretpassword") + c['manhole'] = manhole.TelnetManhole(1234, "bob", "snoop_my_password_please") +</pre> + <p>The <code>Manhole</code> instance can be configured to listen on a specific +port. You may wish to have this listening port bind to the loopback +interface (sometimes known as “lo0”, “localhost”, or 127.0.0.1) to +restrict access to clients which are running on the same host. + +<pre class="example"> from buildbot.manhole import PasswordManhole + c['manhole'] = PasswordManhole("tcp:9999:interface=127.0.0.1","admin","passwd") +</pre> + <p>To have the <code>Manhole</code> listen on all interfaces, use +<code>"tcp:9999"</code> or simply 9999. This port specification uses +<code>twisted.application.strports</code>, so you can make it listen on SSL +or even UNIX-domain sockets if you want. + + <p>Note that using any Manhole requires that the TwistedConch package be +installed, and that you be using Twisted version 2.0 or later. + + <p>The buildmaster's SSH server will use a different host key than the +normal sshd running on a typical unix host. This will cause the ssh +client to complain about a “host key mismatch”, because it does not +realize there are two separate servers running on the same host. To +avoid this, use a clause like the following in your <samp><span class="file">.ssh/config</span></samp> +file: + +<pre class="example"> Host remotehost-buildbot + HostName remotehost + HostKeyAlias remotehost-buildbot + Port 9999 + # use 'user' if you use PasswordManhole and your name is not 'admin'. + # if you use AuthorizedKeysManhole, this probably doesn't matter. + User admin +</pre> + <div class="node"> +<p><hr> +<a name="Getting-Source-Code-Changes"></a> +Next: <a rel="next" accesskey="n" href="#Build-Process">Build Process</a>, +Previous: <a rel="previous" accesskey="p" href="#Configuration">Configuration</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">5 Getting Source Code Changes</h2> + +<p>The most common way to use the Buildbot is centered around the idea of +<code>Source Trees</code>: a directory tree filled with source code of some form +which can be compiled and/or tested. Some projects use languages that don't +involve any compilation step: nevertheless there may be a <code>build</code> phase +where files are copied or rearranged into a form that is suitable for +installation. Some projects do not have unit tests, and the Buildbot is +merely helping to make sure that the sources can compile correctly. But in +all of these cases, the thing-being-tested is a single source tree. + + <p>A Version Control System mantains a source tree, and tells the +buildmaster when it changes. The first step of each Build is typically +to acquire a copy of some version of this tree. + + <p>This chapter describes how the Buildbot learns about what Changes have +occurred. For more information on VC systems and Changes, see +<a href="#Version-Control-Systems">Version Control Systems</a>. + +<ul class="menu"> +<li><a accesskey="1" href="#Change-Sources">Change Sources</a> +<li><a accesskey="2" href="#Choosing-ChangeSources">Choosing ChangeSources</a> +<li><a accesskey="3" href="#CVSToys-_002d-PBService">CVSToys - PBService</a> +<li><a accesskey="4" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> +<li><a accesskey="5" href="#PBChangeSource">PBChangeSource</a> +<li><a accesskey="6" href="#P4Source">P4Source</a> +<li><a accesskey="7" href="#BonsaiPoller">BonsaiPoller</a> +<li><a accesskey="8" href="#SVNPoller">SVNPoller</a> +<li><a accesskey="9" href="#MercurialHook">MercurialHook</a> +<li><a href="#Bzr-Hook">Bzr Hook</a> +<li><a href="#Bzr-Poller">Bzr Poller</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Change-Sources"></a> +Next: <a rel="next" accesskey="n" href="#Choosing-ChangeSources">Choosing ChangeSources</a>, +Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.1 Change Sources</h3> + +<!-- TODO: rework this, the one-buildmaster-one-tree thing isn't quite --> +<!-- so narrow-minded anymore --> +<p>Each Buildmaster watches a single source tree. Changes can be provided +by a variety of ChangeSource types, however any given project will +typically have only a single ChangeSource active. This section +provides a description of all available ChangeSource types and +explains how to set up each of them. + + <p>There are a variety of ChangeSources available, some of which are +meant to be used in conjunction with other tools to deliver Change +events from the VC repository to the buildmaster. + + <ul> +<li>CVSToys +This ChangeSource opens a TCP connection from the buildmaster to a +waiting FreshCVS daemon that lives on the repository machine, and +subscribes to hear about Changes. + + <li>MaildirSource +This one watches a local maildir-format inbox for email sent out by +the repository when a change is made. When a message arrives, it is +parsed to create the Change object. A variety of parsing functions are +available to accomodate different email-sending tools. + + <li>PBChangeSource +This ChangeSource listens on a local TCP socket for inbound +connections from a separate tool. Usually, this tool would be run on +the VC repository machine in a commit hook. It is expected to connect +to the TCP socket and send a Change message over the network +connection. The <samp><span class="command">buildbot sendchange</span></samp> command is one example +of a tool that knows how to send these messages, so you can write a +commit script for your VC system that calls it to deliver the Change. +There are other tools in the contrib/ directory that use the same +protocol. + + </ul> + + <p>As a quick guide, here is a list of VC systems and the ChangeSources +that might be useful with them. All of these ChangeSources are in the +<code>buildbot.changes</code> module. + + <dl> +<dt><code>CVS</code><dd> + <ul> +<li>freshcvs.FreshCVSSource (connected via TCP to the freshcvs daemon) +<li>mail.FCMaildirSource (watching for email sent by a freshcvs daemon) +<li>mail.BonsaiMaildirSource (watching for email sent by Bonsai) +<li>mail.SyncmailMaildirSource (watching for email sent by syncmail) +<li>pb.PBChangeSource (listening for connections from <code>buildbot +sendchange</code> run in a loginfo script) +<li>pb.PBChangeSource (listening for connections from a long-running +<code>contrib/viewcvspoll.py</code> polling process which examines the ViewCVS +database directly +</ul> + + <br><dt><code>SVN</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/svn_buildbot.py</code> run in a postcommit script) +<li>pb.PBChangeSource (listening for connections from a long-running +<code>contrib/svn_watcher.py</code> or <code>contrib/svnpoller.py</code> polling +process +<li>mail.SVNCommitEmailMaildirSource (watching for email sent by commit-email.pl) +<li>svnpoller.SVNPoller (polling the SVN repository) +</ul> + + <br><dt><code>Darcs</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/darcs_buildbot.py</code> in a commit script +</ul> + + <br><dt><code>Mercurial</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/hg_buildbot.py</code> run in an 'incoming' hook) +<li>pb.PBChangeSource (listening for connections from +<code>buildbot/changes/hgbuildbot.py</code> run as an in-process 'changegroup' +hook) +</ul> + + <br><dt><code>Arch/Bazaar</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/arch_buildbot.py</code> run in a commit hook) +</ul> + + <br><dt><code>Bzr (the newer Bazaar)</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/bzr_buildbot.py</code> run in a post-change-branch-tip or commit hook) +<li><code>contrib/bzr_buildbot.py</code>'s BzrPoller (polling the Bzr repository) +</ul> + + <br><dt><code>Git</code><dd> + <ul> +<li>pb.PBChangeSource (listening for connections from +<code>contrib/git_buildbot.py</code> run in the post-receive hook) +</ul> + + </dl> + + <p>All VC systems can be driven by a PBChangeSource and the +<code>buildbot sendchange</code> tool run from some form of commit script. +If you write an email parsing function, they can also all be driven by +a suitable <code>MaildirSource</code>. + +<div class="node"> +<p><hr> +<a name="Choosing-ChangeSources"></a> +Next: <a rel="next" accesskey="n" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>, +Previous: <a rel="previous" accesskey="p" href="#Change-Sources">Change Sources</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.2 Choosing ChangeSources</h3> + +<p>The <code>master.cfg</code> configuration file has a dictionary key named +<code>BuildmasterConfig['change_source']</code>, which holds the active +<code>IChangeSource</code> object. The config file will typically create an +object from one of the classes described below and stuff it into this +key. + + <p>Each buildmaster typically has just a single ChangeSource, since it is +only watching a single source tree. But if, for some reason, you need +multiple sources, just set <code>c['change_source']</code> to a list of +ChangeSources.. it will accept that too. + +<pre class="example"> s = FreshCVSSourceNewcred(host="host", port=4519, + user="alice", passwd="secret", + prefix="Twisted") + BuildmasterConfig['change_source'] = [s] +</pre> + <p>Each source tree has a nominal <code>top</code>. Each Change has a list of +filenames, which are all relative to this top location. The +ChangeSource is responsible for doing whatever is necessary to +accomplish this. Most sources have a <code>prefix</code> argument: a partial +pathname which is stripped from the front of all filenames provided to +that <code>ChangeSource</code>. Files which are outside this sub-tree are +ignored by the changesource: it does not generate Changes for those +files. + +<div class="node"> +<p><hr> +<a name="CVSToys---PBService"></a> +<a name="CVSToys-_002d-PBService"></a> +Next: <a rel="next" accesskey="n" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, +Previous: <a rel="previous" accesskey="p" href="#Choosing-ChangeSources">Choosing ChangeSources</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.3 CVSToys - PBService</h3> + +<p><a name="index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-42"></a> +The <a href="http://purl.net/net/CVSToys">CVSToys</a> package provides a +server which runs on the machine that hosts the CVS repository it +watches. It has a variety of ways to distribute commit notifications, +and offers a flexible regexp-based way to filter out uninteresting +changes. One of the notification options is named <code>PBService</code> and +works by listening on a TCP port for clients. These clients subscribe +to hear about commit notifications. + + <p>The buildmaster has a CVSToys-compatible <code>PBService</code> client built +in. There are two versions of it, one for old versions of CVSToys +(1.0.9 and earlier) which used the <code>oldcred</code> authentication +framework, and one for newer versions (1.0.10 and later) which use +<code>newcred</code>. Both are classes in the +<code>buildbot.changes.freshcvs</code> package. + + <p><code>FreshCVSSourceNewcred</code> objects are created with the following +parameters: + + <dl> +<dt>‘<samp><code>host</code><span class="samp"> and </span><code>port</code></samp>’<dd>these specify where the CVSToys server can be reached + + <br><dt>‘<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>’<dd>these specify the login information for the CVSToys server +(<code>freshcvs</code>). These must match the server's values, which are +defined in the <code>freshCfg</code> configuration file (which lives in the +CVSROOT directory of the repository). + + <br><dt>‘<samp><code>prefix</code></samp>’<dd>this is the prefix to be found and stripped from filenames delivered +by the CVSToys server. Most projects live in sub-directories of the +main repository, as siblings of the CVSROOT sub-directory, so +typically this prefix is set to that top sub-directory name. + + </dl> + +<h3 class="heading">Example</h3> + +<p>To set up the freshCVS server, add a statement like the following to +your <samp><span class="file">freshCfg</span></samp> file: + +<pre class="example"> pb = ConfigurationSet([ + (None, None, None, PBService(userpass=('foo', 'bar'), port=4519)), + ]) +</pre> + <p>This will announce all changes to a client which connects to port 4519 +using a username of 'foo' and a password of 'bar'. + + <p>Then add a clause like this to your buildmaster's <samp><span class="file">master.cfg</span></samp>: + +<pre class="example"> BuildmasterConfig['change_source'] = FreshCVSSource("cvs.example.com", 4519, + "foo", "bar", + prefix="glib/") +</pre> + <p>where "cvs.example.com" is the host that is running the FreshCVS daemon, and +"glib" is the top-level directory (relative to the repository's root) where +all your source code lives. Most projects keep one or more projects in the +same repository (along with CVSROOT/ to hold admin files like loginfo and +freshCfg); the prefix= argument tells the buildmaster to ignore everything +outside that directory, and to strip that common prefix from all pathnames +it handles. + +<div class="node"> +<p><hr> +<a name="Mail-parsing-ChangeSources"></a> +<a name="Mail_002dparsing-ChangeSources"></a> +Next: <a rel="next" accesskey="n" href="#PBChangeSource">PBChangeSource</a>, +Previous: <a rel="previous" accesskey="p" href="#CVSToys-_002d-PBService">CVSToys - PBService</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.4 Mail-parsing ChangeSources</h3> + +<p>Many projects publish information about changes to their source tree +by sending an email message out to a mailing list, frequently named +PROJECT-commits or PROJECT-changes. Each message usually contains a +description of the change (who made the change, which files were +affected) and sometimes a copy of the diff. Humans can subscribe to +this list to stay informed about what's happening to the source tree. + + <p>The Buildbot can also be subscribed to a -commits mailing list, and +can trigger builds in response to Changes that it hears about. The +buildmaster admin needs to arrange for these email messages to arrive +in a place where the buildmaster can find them, and configure the +buildmaster to parse the messages correctly. Once that is in place, +the email parser will create Change objects and deliver them to the +Schedulers (see see <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a>) just +like any other ChangeSource. + + <p>There are two components to setting up an email-based ChangeSource. +The first is to route the email messages to the buildmaster, which is +done by dropping them into a “maildir”. The second is to actually +parse the messages, which is highly dependent upon the tool that was +used to create them. Each VC system has a collection of favorite +change-emailing tools, and each has a slightly different format, so +each has a different parsing function. There is a separate +ChangeSource variant for each parsing function. + + <p>Once you've chosen a maildir location and a parsing function, create +the change source and put it in <code>c['change_source']</code>: + +<pre class="example"> from buildbot.changes.mail import SyncmailMaildirSource + c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot", + prefix="/trunk/") +</pre> + <ul class="menu"> +<li><a accesskey="1" href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a> +<li><a accesskey="2" href="#Using-Maildirs">Using Maildirs</a> +<li><a accesskey="3" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Subscribing-the-Buildmaster"></a> +Next: <a rel="next" accesskey="n" href="#Using-Maildirs">Using Maildirs</a>, +Previous: <a rel="previous" accesskey="p" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, +Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> + +</div> + +<h4 class="subsection">5.4.1 Subscribing the Buildmaster</h4> + +<p>The recommended way to install the buildbot is to create a dedicated +account for the buildmaster. If you do this, the account will probably +have a distinct email address (perhaps +<a href="mailto:buildmaster@example.org">buildmaster@example.org</a>). Then just arrange for this +account's email to be delivered to a suitable maildir (described in +the next section). + + <p>If the buildbot does not have its own account, “extension addresses” +can be used to distinguish between email intended for the buildmaster +and email intended for the rest of the account. In most modern MTAs, +the e.g. <code>foo@example.org</code> account has control over every email +address at example.org which begins with "foo", such that email +addressed to <a href="mailto:account-foo@example.org">account-foo@example.org</a> can be delivered to a +different destination than <a href="mailto:account-bar@example.org">account-bar@example.org</a>. qmail +does this by using separate .qmail files for the two destinations +(<samp><span class="file">.qmail-foo</span></samp> and <samp><span class="file">.qmail-bar</span></samp>, with <samp><span class="file">.qmail</span></samp> +controlling the base address and <samp><span class="file">.qmail-default</span></samp> controlling all +other extensions). Other MTAs have similar mechanisms. + + <p>Thus you can assign an extension address like +<a href="mailto:foo-buildmaster@example.org">foo-buildmaster@example.org</a> to the buildmaster, and retain +<a href="mailto:foo@example.org">foo@example.org</a> for your own use. + +<div class="node"> +<p><hr> +<a name="Using-Maildirs"></a> +Next: <a rel="next" accesskey="n" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a>, +Previous: <a rel="previous" accesskey="p" href="#Subscribing-the-Buildmaster">Subscribing the Buildmaster</a>, +Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> + +</div> + +<h4 class="subsection">5.4.2 Using Maildirs</h4> + +<p>A “maildir” is a simple directory structure originally developed for +qmail that allows safe atomic update without locking. Create a base +directory with three subdirectories: “new”, “tmp”, and “cur”. +When messages arrive, they are put into a uniquely-named file (using +pids, timestamps, and random numbers) in “tmp”. When the file is +complete, it is atomically renamed into “new”. Eventually the +buildmaster notices the file in “new”, reads and parses the +contents, then moves it into “cur”. A cronjob can be used to delete +files in “cur” at leisure. + + <p>Maildirs are frequently created with the <samp><span class="command">maildirmake</span></samp> tool, +but a simple <samp><span class="command">mkdir -p ~/MAILDIR/{cur,new,tmp}</span></samp> is pretty much +equivalent. + + <p>Many modern MTAs can deliver directly to maildirs. The usual .forward +or .procmailrc syntax is to name the base directory with a trailing +slash, so something like <code>~/MAILDIR/</code> . qmail and postfix are +maildir-capable MTAs, and procmail is a maildir-capable MDA (Mail +Delivery Agent). + + <p>For MTAs which cannot put files into maildirs directly, the +“safecat” tool can be executed from a .forward file to accomplish +the same thing. + + <p>The Buildmaster uses the linux DNotify facility to receive immediate +notification when the maildir's “new” directory has changed. When +this facility is not available, it polls the directory for new +messages, every 10 seconds by default. + +<div class="node"> +<p><hr> +<a name="Parsing-Email-Change-Messages"></a> +Previous: <a rel="previous" accesskey="p" href="#Using-Maildirs">Using Maildirs</a>, +Up: <a rel="up" accesskey="u" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a> + +</div> + +<h4 class="subsection">5.4.3 Parsing Email Change Messages</h4> + +<p>The second component to setting up an email-based ChangeSource is to +parse the actual notices. This is highly dependent upon the VC system +and commit script in use. + + <p>A couple of common tools used to create these change emails are: + + <dl> +<dt>‘<samp><span class="samp">CVS</span></samp>’<dd> + <dl> +<dt>‘<samp><span class="samp">CVSToys MailNotifier</span></samp>’<dd><a href="#FCMaildirSource">FCMaildirSource</a> +<br><dt>‘<samp><span class="samp">Bonsai notification</span></samp>’<dd><a href="#BonsaiMaildirSource">BonsaiMaildirSource</a> +<br><dt>‘<samp><span class="samp">syncmail</span></samp>’<dd><a href="#SyncmailMaildirSource">SyncmailMaildirSource</a> +</dl> + + <br><dt>‘<samp><span class="samp">SVN</span></samp>’<dd> + <dl> +<dt>‘<samp><span class="samp">svnmailer</span></samp>’<dd>http://opensource.perlig.de/en/svnmailer/ +<br><dt>‘<samp><span class="samp">commit-email.pl</span></samp>’<dd><a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> +</dl> + + <br><dt>‘<samp><span class="samp">Mercurial</span></samp>’<dd> + <dl> +<dt>‘<samp><span class="samp">NotifyExtension</span></samp>’<dd>http://www.selenic.com/mercurial/wiki/index.cgi/NotifyExtension +</dl> + + <br><dt>‘<samp><span class="samp">Git</span></samp>’<dd> + <dl> +<dt>‘<samp><span class="samp">post-receive-email</span></samp>’<dd>http://git.kernel.org/?p=git/git.git;a=blob;f=contrib/hooks/post-receive-email;hb=HEAD +</dl> + + </dl> + + <p>The following sections describe the parsers available for each of +these tools. + + <p>Most of these parsers accept a <code>prefix=</code> argument, which is used +to limit the set of files that the buildmaster pays attention to. This +is most useful for systems like CVS and SVN which put multiple +projects in a single repository (or use repository names to indicate +branches). Each filename that appears in the email is tested against +the prefix: if the filename does not start with the prefix, the file +is ignored. If the filename <em>does</em> start with the prefix, that +prefix is stripped from the filename before any further processing is +done. Thus the prefix usually ends with a slash. + +<ul class="menu"> +<li><a accesskey="1" href="#FCMaildirSource">FCMaildirSource</a> +<li><a accesskey="2" href="#SyncmailMaildirSource">SyncmailMaildirSource</a> +<li><a accesskey="3" href="#BonsaiMaildirSource">BonsaiMaildirSource</a> +<li><a accesskey="4" href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a> +</ul> + +<div class="node"> +<p><hr> +<a name="FCMaildirSource"></a> +Next: <a rel="next" accesskey="n" href="#SyncmailMaildirSource">SyncmailMaildirSource</a>, +Previous: <a rel="previous" accesskey="p" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a>, +Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> + +</div> + +<h5 class="subsubsection">5.4.3.1 FCMaildirSource</h5> + +<p><a name="index-buildbot_002echanges_002email_002eFCMaildirSource-43"></a> +http://twistedmatrix.com/users/acapnotic/wares/code/CVSToys/ + + <p>This parser works with the CVSToys <code>MailNotification</code> action, +which will send email to a list of recipients for each commit. This +tends to work better than using <code>/bin/mail</code> from within the +CVSROOT/loginfo file directly, as CVSToys will batch together all +files changed during the same CVS invocation, and can provide more +information (like creating a ViewCVS URL for each file changed). + + <p>The Buildbot's <code>FCMaildirSource</code> knows for to parse these CVSToys +messages and turn them into Change objects. It can be given two +parameters: the directory name of the maildir root, and the prefix to +strip. + +<pre class="example"> from buildbot.changes.mail import FCMaildirSource + c['change_source'] = FCMaildirSource("~/maildir-buildbot") +</pre> + <div class="node"> +<p><hr> +<a name="SyncmailMaildirSource"></a> +Next: <a rel="next" accesskey="n" href="#BonsaiMaildirSource">BonsaiMaildirSource</a>, +Previous: <a rel="previous" accesskey="p" href="#FCMaildirSource">FCMaildirSource</a>, +Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> + +</div> + +<h5 class="subsubsection">5.4.3.2 SyncmailMaildirSource</h5> + +<p><a name="index-buildbot_002echanges_002email_002eSyncmailMaildirSource-44"></a> +http://sourceforge.net/projects/cvs-syncmail + + <p><code>SyncmailMaildirSource</code> knows how to parse the message format used by +the CVS “syncmail” script. + +<pre class="example"> from buildbot.changes.mail import SyncmailMaildirSource + c['change_source'] = SyncmailMaildirSource("~/maildir-buildbot") +</pre> + <div class="node"> +<p><hr> +<a name="BonsaiMaildirSource"></a> +Next: <a rel="next" accesskey="n" href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a>, +Previous: <a rel="previous" accesskey="p" href="#SyncmailMaildirSource">SyncmailMaildirSource</a>, +Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> + +</div> + +<h5 class="subsubsection">5.4.3.3 BonsaiMaildirSource</h5> + +<p><a name="index-buildbot_002echanges_002email_002eBonsaiMaildirSource-45"></a> +http://www.mozilla.org/bonsai.html + + <p><code>BonsaiMaildirSource</code> parses messages sent out by Bonsai, the CVS +tree-management system built by Mozilla. + +<pre class="example"> from buildbot.changes.mail import BonsaiMaildirSource + c['change_source'] = BonsaiMaildirSource("~/maildir-buildbot") +</pre> + <div class="node"> +<p><hr> +<a name="SVNCommitEmailMaildirSource"></a> +Previous: <a rel="previous" accesskey="p" href="#BonsaiMaildirSource">BonsaiMaildirSource</a>, +Up: <a rel="up" accesskey="u" href="#Parsing-Email-Change-Messages">Parsing Email Change Messages</a> + +</div> + +<h5 class="subsubsection">5.4.3.4 SVNCommitEmailMaildirSource</h5> + +<p><a name="index-buildbot_002echanges_002email_002eSVNCommitEmailMaildirSource-46"></a> +<code>SVNCommitEmailMaildirSource</code> parses message sent out by the +<code>commit-email.pl</code> script, which is included in the Subversion +distribution. + + <p>It does not currently handle branches: all of the Change objects that +it creates will be associated with the default (i.e. trunk) branch. + +<pre class="example"> from buildbot.changes.mail import SVNCommitEmailMaildirSource + c['change_source'] = SVNCommitEmailMaildirSource("~/maildir-buildbot") +</pre> + <div class="node"> +<p><hr> +<a name="PBChangeSource"></a> +Next: <a rel="next" accesskey="n" href="#P4Source">P4Source</a>, +Previous: <a rel="previous" accesskey="p" href="#Mail_002dparsing-ChangeSources">Mail-parsing ChangeSources</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.5 PBChangeSource</h3> + +<p><a name="index-buildbot_002echanges_002epb_002ePBChangeSource-47"></a> +The last kind of ChangeSource actually listens on a TCP port for +clients to connect and push change notices <em>into</em> the +Buildmaster. This is used by the built-in <code>buildbot sendchange</code> +notification tool, as well as the VC-specific +<samp><span class="file">contrib/svn_buildbot.py</span></samp>, <samp><span class="file">contrib/arch_buildbot.py</span></samp>, +<samp><span class="file">contrib/hg_buildbot.py</span></samp> tools, and the +<code>buildbot.changes.hgbuildbot</code> hook. These tools are run by the +repository (in a commit hook script), and connect to the buildmaster +directly each time a file is comitted. This is also useful for +creating new kinds of change sources that work on a <code>push</code> model +instead of some kind of subscription scheme, for example a script +which is run out of an email .forward file. + + <p>This ChangeSource can be configured to listen on its own TCP port, or +it can share the port that the buildmaster is already using for the +buildslaves to connect. (This is possible because the +<code>PBChangeSource</code> uses the same protocol as the buildslaves, and +they can be distinguished by the <code>username</code> attribute used when +the initial connection is established). It might be useful to have it +listen on a different port if, for example, you wanted to establish +different firewall rules for that port. You could allow only the SVN +repository machine access to the <code>PBChangeSource</code> port, while +allowing only the buildslave machines access to the slave port. Or you +could just expose one port and run everything over it. <em>Note: +this feature is not yet implemented, the PBChangeSource will always +share the slave port and will always have a </em><code>user</code><em> name of +</em><code>change</code><em>, and a passwd of </em><code>changepw</code><em>. These limitations will +be removed in the future.</em>. + + <p>The <code>PBChangeSource</code> is created with the following arguments. All +are optional. + + <dl> +<dt>‘<samp><code>port</code></samp>’<dd>which port to listen on. If <code>None</code> (which is the default), it +shares the port used for buildslave connections. <em>Not +Implemented, always set to </em><code>None</code>. + + <br><dt>‘<samp><code>user</code><span class="samp"> and </span><code>passwd</code></samp>’<dd>The user/passwd account information that the client program must use +to connect. Defaults to <code>change</code> and <code>changepw</code>. <em>Not +Implemented, </em><code>user</code><em> is currently always set to </em><code>change</code><em>, +</em><code>passwd</code><em> is always set to </em><code>changepw</code>. + + <br><dt>‘<samp><code>prefix</code></samp>’<dd>The prefix to be found and stripped from filenames delivered over the +connection. Any filenames which do not start with this prefix will be +removed. If all the filenames in a given Change are removed, the that +whole Change will be dropped. This string should probably end with a +directory separator. + + <p>This is useful for changes coming from version control systems that +represent branches as parent directories within the repository (like +SVN and Perforce). Use a prefix of 'trunk/' or +'project/branches/foobranch/' to only follow one branch and to get +correct tree-relative filenames. Without a prefix, the PBChangeSource +will probably deliver Changes with filenames like <samp><span class="file">trunk/foo.c</span></samp> +instead of just <samp><span class="file">foo.c</span></samp>. Of course this also depends upon the +tool sending the Changes in (like <samp><span class="command">buildbot sendchange</span></samp>) and +what filenames it is delivering: that tool may be filtering and +stripping prefixes at the sending end. + + </dl> + +<div class="node"> +<p><hr> +<a name="P4Source"></a> +Next: <a rel="next" accesskey="n" href="#BonsaiPoller">BonsaiPoller</a>, +Previous: <a rel="previous" accesskey="p" href="#PBChangeSource">PBChangeSource</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.6 P4Source</h3> + +<p><a name="index-buildbot_002echanges_002ep4poller_002eP4Source-48"></a> +The <code>P4Source</code> periodically polls a <a href="http://www.perforce.com/">Perforce</a> depot for changes. It accepts the following arguments: + + <dl> +<dt>‘<samp><code>p4base</code></samp>’<dd>The base depot path to watch, without the trailing '/...'. + + <br><dt>‘<samp><code>p4port</code></samp>’<dd>The Perforce server to connect to (as host:port). + + <br><dt>‘<samp><code>p4user</code></samp>’<dd>The Perforce user. + + <br><dt>‘<samp><code>p4passwd</code></samp>’<dd>The Perforce password. + + <br><dt>‘<samp><code>p4bin</code></samp>’<dd>An optional string parameter. Specify the location of the perforce command +line binary (p4). You only need to do this if the perforce binary is not +in the path of the buildbot user. Defaults to “p4”. + + <br><dt>‘<samp><code>split_file</code></samp>’<dd>A function that maps a pathname, without the leading <code>p4base</code>, to a +(branch, filename) tuple. The default just returns (None, branchfile), +which effectively disables branch support. You should supply a function +which understands your repository structure. + + <br><dt>‘<samp><code>pollinterval</code></samp>’<dd>How often to poll, in seconds. Defaults to 600 (10 minutes). + + <br><dt>‘<samp><code>histmax</code></samp>’<dd>The maximum number of changes to inspect at a time. If more than this +number occur since the last poll, older changes will be silently +ignored. +</dl> + +<h3 class="heading">Example</h3> + +<p>This configuration uses the <code>P4PORT</code>, <code>P4USER</code>, and <code>P4PASSWD</code> +specified in the buildmaster's environment. It watches a project in which the +branch name is simply the next path component, and the file is all path +components after. + +<pre class="example"> import buildbot.changes.p4poller + s = p4poller.P4Source(p4base='//depot/project/', + split_file=lambda branchfile: branchfile.split('/',1), + ) + c['change_source'] = s +</pre> + <div class="node"> +<p><hr> +<a name="BonsaiPoller"></a> +Next: <a rel="next" accesskey="n" href="#SVNPoller">SVNPoller</a>, +Previous: <a rel="previous" accesskey="p" href="#P4Source">P4Source</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.7 BonsaiPoller</h3> + +<p><a name="index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-49"></a> +The <code>BonsaiPoller</code> periodically polls a Bonsai server. This is a +CGI script accessed through a web server that provides information +about a CVS tree, for example the Mozilla bonsai server at +<a href="http://bonsai.mozilla.org">http://bonsai.mozilla.org</a>. Bonsai servers are usable by both +humans and machines. In this case, the buildbot's change source forms +a query which asks about any files in the specified branch which have +changed since the last query. + + <p>Please take a look at the BonsaiPoller docstring for details about the +arguments it accepts. + +<div class="node"> +<p><hr> +<a name="SVNPoller"></a> +Next: <a rel="next" accesskey="n" href="#MercurialHook">MercurialHook</a>, +Previous: <a rel="previous" accesskey="p" href="#BonsaiPoller">BonsaiPoller</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.8 SVNPoller</h3> + +<p><a name="index-buildbot_002echanges_002esvnpoller_002eSVNPoller-50"></a> +The <code>buildbot.changes.svnpoller.SVNPoller</code> is a ChangeSource +which periodically polls a <a href="http://subversion.tigris.org/">Subversion</a> repository for new revisions, by running the <code>svn +log</code> command in a subshell. It can watch a single branch or multiple +branches. + + <p><code>SVNPoller</code> accepts the following arguments: + + <dl> +<dt><code>svnurl</code><dd>The base URL path to watch, like +<code>svn://svn.twistedmatrix.com/svn/Twisted/trunk</code>, or +<code>http://divmod.org/svn/Divmod/</code>, or even +<code>file:///home/svn/Repository/ProjectA/branches/1.5/</code>. This must +include the access scheme, the location of the repository (both the +hostname for remote ones, and any additional directory names necessary +to get to the repository), and the sub-path within the repository's +virtual filesystem for the project and branch of interest. + + <p>The <code>SVNPoller</code> will only pay attention to files inside the +subdirectory specified by the complete svnurl. + + <br><dt><code>split_file</code><dd>A function to convert pathnames into (branch, relative_pathname) +tuples. Use this to explain your repository's branch-naming policy to +<code>SVNPoller</code>. This function must accept a single string and return +a two-entry tuple. There are a few utility functions in +<code>buildbot.changes.svnpoller</code> that can be used as a +<code>split_file</code> function, see below for details. + + <p>The default value always returns (None, path), which indicates that +all files are on the trunk. + + <p>Subclasses of <code>SVNPoller</code> can override the <code>split_file</code> +method instead of using the <code>split_file=</code> argument. + + <br><dt><code>svnuser</code><dd>An optional string parameter. If set, the <code>--user</code> argument will +be added to all <code>svn</code> commands. Use this if you have to +authenticate to the svn server before you can do <code>svn info</code> or +<code>svn log</code> commands. + + <br><dt><code>svnpasswd</code><dd>Like <code>svnuser</code>, this will cause a <code>--password</code> argument to +be passed to all svn commands. + + <br><dt><code>pollinterval</code><dd>How often to poll, in seconds. Defaults to 600 (checking once every 10 +minutes). Lower this if you want the buildbot to notice changes +faster, raise it if you want to reduce the network and CPU load on +your svn server. Please be considerate of public SVN repositories by +using a large interval when polling them. + + <br><dt><code>histmax</code><dd>The maximum number of changes to inspect at a time. Every POLLINTERVAL +seconds, the <code>SVNPoller</code> asks for the last HISTMAX changes and +looks through them for any ones it does not already know about. If +more than HISTMAX revisions have been committed since the last poll, +older changes will be silently ignored. Larger values of histmax will +cause more time and memory to be consumed on each poll attempt. +<code>histmax</code> defaults to 100. + + <br><dt><code>svnbin</code><dd>This controls the <code>svn</code> executable to use. If subversion is +installed in a weird place on your system (outside of the +buildmaster's <code>$PATH</code>), use this to tell <code>SVNPoller</code> where +to find it. The default value of “svn” will almost always be +sufficient. + + </dl> + +<h3 class="heading">Branches</h3> + +<p>Each source file that is tracked by a Subversion repository has a +fully-qualified SVN URL in the following form: +(REPOURL)(PROJECT-plus-BRANCH)(FILEPATH). When you create the +<code>SVNPoller</code>, you give it a <code>svnurl</code> value that includes all +of the REPOURL and possibly some portion of the PROJECT-plus-BRANCH +string. The <code>SVNPoller</code> is responsible for producing Changes that +contain a branch name and a FILEPATH (which is relative to the top of +a checked-out tree). The details of how these strings are split up +depend upon how your repository names its branches. + +<h4 class="subheading">PROJECT/BRANCHNAME/FILEPATH repositories</h4> + +<p>One common layout is to have all the various projects that share a +repository get a single top-level directory each. Then under a given +project's directory, you get two subdirectories, one named “trunk” +and another named “branches”. Under “branches” you have a bunch of +other directories, one per branch, with names like “1.5.x” and +“testing”. It is also common to see directories like “tags” and +“releases” next to “branches” and “trunk”. + + <p>For example, the Twisted project has a subversion server on +“svn.twistedmatrix.com” that hosts several sub-projects. The +repository is available through a SCHEME of “svn:”. The primary +sub-project is Twisted, of course, with a repository root of +“svn://svn.twistedmatrix.com/svn/Twisted”. Another sub-project is +Informant, with a root of +“svn://svn.twistedmatrix.com/svn/Informant”, etc. Inside any +checked-out Twisted tree, there is a file named bin/trial (which is +used to run unit test suites). + + <p>The trunk for Twisted is in +“svn://svn.twistedmatrix.com/svn/Twisted/trunk”, and the +fully-qualified SVN URL for the trunk version of <code>trial</code> would be +“svn://svn.twistedmatrix.com/svn/Twisted/trunk/bin/trial”. The same +SVNURL for that file on a branch named “1.5.x” would be +“svn://svn.twistedmatrix.com/svn/Twisted/branches/1.5.x/bin/trial”. + + <p>To set up a <code>SVNPoller</code> that watches the Twisted trunk (and +nothing else), we would use the following: + +<pre class="example"> from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted/trunk") +</pre> + <p>In this case, every Change that our <code>SVNPoller</code> produces will +have <code>.branch=None</code>, to indicate that the Change is on the trunk. +No other sub-projects or branches will be tracked. + + <p>If we want our ChangeSource to follow multiple branches, we have to do +two things. First we have to change our <code>svnurl=</code> argument to +watch more than just “.../Twisted/trunk”. We will set it to +“.../Twisted” so that we'll see both the trunk and all the branches. +Second, we have to tell <code>SVNPoller</code> how to split the +(PROJECT-plus-BRANCH)(FILEPATH) strings it gets from the repository +out into (BRANCH) and (FILEPATH) pairs. + + <p>We do the latter by providing a “split_file” function. This function +is responsible for splitting something like +“branches/1.5.x/bin/trial” into <code>branch</code>=”branches/1.5.x” and +<code>filepath</code>=”bin/trial”. This function is always given a string +that names a file relative to the subdirectory pointed to by the +<code>SVNPoller</code>'s <code>svnurl=</code> argument. It is expected to return a +(BRANCHNAME, FILEPATH) tuple (in which FILEPATH is relative to the +branch indicated), or None to indicate that the file is outside any +project of interest. + + <p>(note that we want to see “branches/1.5.x” rather than just +“1.5.x” because when we perform the SVN checkout, we will probably +append the branch name to the baseURL, which requires that we keep the +“branches” component in there. Other VC schemes use a different +approach towards branches and may not require this artifact.) + + <p>If your repository uses this same PROJECT/BRANCH/FILEPATH naming +scheme, the following function will work: + +<pre class="example"> def split_file_branches(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + return (None, '/'.join(pieces[1:])) + elif pieces[0] == 'branches': + return ('/'.join(pieces[0:2]), + '/'.join(pieces[2:])) + else: + return None +</pre> + <p>This function is provided as +<code>buildbot.changes.svnpoller.split_file_branches</code> for your +convenience. So to have our Twisted-watching <code>SVNPoller</code> follow +multiple branches, we would use this: + +<pre class="example"> from buildbot.changes.svnpoller import SVNPoller, split_file_branches + c['change_source'] = SVNPoller("svn://svn.twistedmatrix.com/svn/Twisted", + split_file=split_file_branches) +</pre> + <p>Changes for all sorts of branches (with names like “branches/1.5.x”, +and None to indicate the trunk) will be delivered to the Schedulers. +Each Scheduler is then free to use or ignore each branch as it sees +fit. + +<h4 class="subheading">BRANCHNAME/PROJECT/FILEPATH repositories</h4> + +<p>Another common way to organize a Subversion repository is to put the +branch name at the top, and the projects underneath. This is +especially frequent when there are a number of related sub-projects +that all get released in a group. + + <p>For example, Divmod.org hosts a project named “Nevow” as well as one +named “Quotient”. In a checked-out Nevow tree there is a directory +named “formless” that contains a python source file named +“webform.py”. This repository is accessible via webdav (and thus +uses an “http:” scheme) through the divmod.org hostname. There are +many branches in this repository, and they use a +(BRANCHNAME)/(PROJECT) naming policy. + + <p>The fully-qualified SVN URL for the trunk version of webform.py is +<code>http://divmod.org/svn/Divmod/trunk/Nevow/formless/webform.py</code>. +You can do an <code>svn co</code> with that URL and get a copy of the latest +version. The 1.5.x branch version of this file would have a URL of +<code>http://divmod.org/svn/Divmod/branches/1.5.x/Nevow/formless/webform.py</code>. +The whole Nevow trunk would be checked out with +<code>http://divmod.org/svn/Divmod/trunk/Nevow</code>, while the Quotient +trunk would be checked out using +<code>http://divmod.org/svn/Divmod/trunk/Quotient</code>. + + <p>Now suppose we want to have an <code>SVNPoller</code> that only cares about +the Nevow trunk. This case looks just like the PROJECT/BRANCH layout +described earlier: + +<pre class="example"> from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod/trunk/Nevow") +</pre> + <p>But what happens when we want to track multiple Nevow branches? We +have to point our <code>svnurl=</code> high enough to see all those +branches, but we also don't want to include Quotient changes (since +we're only building Nevow). To accomplish this, we must rely upon the +<code>split_file</code> function to help us tell the difference between +files that belong to Nevow and those that belong to Quotient, as well +as figuring out which branch each one is on. + +<pre class="example"> from buildbot.changes.svnpoller import SVNPoller + c['change_source'] = SVNPoller("http://divmod.org/svn/Divmod", + split_file=my_file_splitter) +</pre> + <p>The <code>my_file_splitter</code> function will be called with +repository-relative pathnames like: + + <dl> +<dt><code>trunk/Nevow/formless/webform.py</code><dd>This is a Nevow file, on the trunk. We want the Change that includes this +to see a filename of <code>formless/webform.py"</code>, and a branch of None + + <br><dt><code>branches/1.5.x/Nevow/formless/webform.py</code><dd>This is a Nevow file, on a branch. We want to get +branch=”branches/1.5.x” and filename=”formless/webform.py”. + + <br><dt><code>trunk/Quotient/setup.py</code><dd>This is a Quotient file, so we want to ignore it by having +<code>my_file_splitter</code> return None. + + <br><dt><code>branches/1.5.x/Quotient/setup.py</code><dd>This is also a Quotient file, which should be ignored. +</dl> + + <p>The following definition for <code>my_file_splitter</code> will do the job: + +<pre class="example"> def my_file_splitter(path): + pieces = path.split('/') + if pieces[0] == 'trunk': + branch = None + pieces.pop(0) # remove 'trunk' + elif pieces[0] == 'branches': + pieces.pop(0) # remove 'branches' + # grab branch name + branch = 'branches/' + pieces.pop(0) + else: + return None # something weird + projectname = pieces.pop(0) + if projectname != 'Nevow': + return None # wrong project + return (branch, '/'.join(pieces)) +</pre> + <div class="node"> +<p><hr> +<a name="MercurialHook"></a> +Next: <a rel="next" accesskey="n" href="#Bzr-Hook">Bzr Hook</a>, +Previous: <a rel="previous" accesskey="p" href="#SVNPoller">SVNPoller</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.9 MercurialHook</h3> + +<p>Since Mercurial is written in python, the hook script can invoke +Buildbot's <code>sendchange</code> function directly, rather than having to +spawn an external process. This function delivers the same sort of +changes as <code>buildbot sendchange</code> and the various hook scripts in +contrib/, so you'll need to add a <code>pb.PBChangeSource</code> to your +buildmaster to receive these changes. + + <p>To set this up, first choose a Mercurial repository that represents +your central “official” source tree. This will be the same +repository that your buildslaves will eventually pull from. Install +Buildbot on the machine that hosts this repository, using the same +version of python as Mercurial is using (so that the Mercurial hook +can import code from buildbot). Then add the following to the +<code>.hg/hgrc</code> file in that repository, replacing the buildmaster +hostname/portnumber as appropriate for your buildbot: + +<pre class="example"> [hooks] + changegroup.buildbot = python:buildbot.changes.hgbuildbot.hook + + [hgbuildbot] + master = buildmaster.example.org:9987 +</pre> + <p>(Note that Mercurial lets you define multiple <code>changegroup</code> hooks +by giving them distinct names, like <code>changegroup.foo</code> and +<code>changegroup.bar</code>, which is why we use +<code>changegroup.buildbot</code> in this example. There is nothing magical +about the “buildbot” suffix in the hook name. The +<code>[hgbuildbot]</code> section <em>is</em> special, however, as it is the +only section that the buildbot hook pays attention to.) + + <p>Also note that this runs as a <code>changegroup</code> hook, rather than as +an <code>incoming</code> hook. The <code>changegroup</code> hook is run with +multiple revisions at a time (say, if multiple revisions are being +pushed to this repository in a single <samp><span class="command">hg push</span></samp> command), +whereas the <code>incoming</code> hook is run with just one revision at a +time. The <code>hgbuildbot.hook</code> function will only work with the +<code>changegroup</code> hook. + + <p>The <code>[hgbuildbot]</code> section has two other parameters that you +might specify, both of which control the name of the branch that is +attached to the changes coming from this hook. + + <p>One common branch naming policy for Mercurial repositories is to use +it just like Darcs: each branch goes into a separate repository, and +all the branches for a single project share a common parent directory. +For example, you might have <samp><span class="file">/var/repos/PROJECT/trunk/</span></samp> and +<samp><span class="file">/var/repos/PROJECT/release</span></samp>. To use this style, use the +<code>branchtype = dirname</code> setting, which simply uses the last +component of the repository's enclosing directory as the branch name: + +<pre class="example"> [hgbuildbot] + master = buildmaster.example.org:9987 + branchtype = dirname +</pre> + <p>Another approach is to use Mercurial's built-in branches (the kind +created with <samp><span class="command">hg branch</span></samp> and listed with <samp><span class="command">hg +branches</span></samp>). This feature associates persistent names with particular +lines of descent within a single repository. (note that the buildbot +<code>source.Mercurial</code> checkout step does not yet support this kind +of branch). To have the commit hook deliver this sort of branch name +with the Change object, use <code>branchtype = inrepo</code>: + +<pre class="example"> [hgbuildbot] + master = buildmaster.example.org:9987 + branchtype = inrepo +</pre> + <p>Finally, if you want to simply specify the branchname directly, for +all changes, use <code>branch = BRANCHNAME</code>. This overrides +<code>branchtype</code>: + +<pre class="example"> [hgbuildbot] + master = buildmaster.example.org:9987 + branch = trunk +</pre> + <p>If you use <code>branch=</code> like this, you'll need to put a separate +.hgrc in each repository. If you use <code>branchtype=</code>, you may be +able to use the same .hgrc for all your repositories, stored in +<samp><span class="file">~/.hgrc</span></samp> or <samp><span class="file">/etc/mercurial/hgrc</span></samp>. + +<div class="node"> +<p><hr> +<a name="Bzr-Hook"></a> +Next: <a rel="next" accesskey="n" href="#Bzr-Poller">Bzr Poller</a>, +Previous: <a rel="previous" accesskey="p" href="#MercurialHook">MercurialHook</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.10 Bzr Hook</h3> + +<p>Bzr is also written in Python, and the Bzr hook depends on Twisted to send the +changes. + + <p>To install, put <code>contrib/bzr_buildbot.py</code> in one of your plugins +locations a bzr plugins directory (e.g., +<code>~/.bazaar/plugins</code>). Then, in one of your bazaar conf files (e.g., +<code>~/.bazaar/locations.conf</code>), set the location you want to connect with buildbot +with these keys: + + <dl> +<dt><code>buildbot_on</code><dd>one of 'commit', 'push, or 'change'. Turns the plugin on to report changes via +commit, changes via push, or any changes to the trunk. 'change' is +recommended. + + <br><dt><code>buildbot_server</code><dd>(required to send to a buildbot master) the URL of the buildbot master to +which you will connect (as of this writing, the same server and port to which +slaves connect). + + <br><dt><code>buildbot_port</code><dd>(optional, defaults to 9989) the port of the buildbot master to which you will +connect (as of this writing, the same server and port to which slaves connect) + + <br><dt><code>buildbot_pqm</code><dd>(optional, defaults to not pqm) Normally, the user that commits the revision +is the user that is responsible for the change. When run in a pqm (Patch Queue +Manager, see https://launchpad.net/pqm) environment, the user that commits is +the Patch Queue Manager, and the user that committed the *parent* revision is +responsible for the change. To turn on the pqm mode, set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T". + + <br><dt><code>buildbot_dry_run</code><dd>(optional, defaults to not a dry run) Normally, the post-commit hook will +attempt to communicate with the configured buildbot server and port. If this +parameter is included and any of (case-insensitive) "Yes", "Y", "True", or +"T", then the hook will simply print what it would have sent, but not attempt +to contact the buildbot master. + + <br><dt><code>buildbot_send_branch_name</code><dd>(optional, defaults to not sending the branch name) If your buildbot's bzr +source build step uses a repourl, do *not* turn this on. If your buildbot's +bzr build step uses a baseURL, then you may set this value to any of +(case-insensitive) "Yes", "Y", "True", or "T" to have the buildbot master +append the branch name to the baseURL. + + </dl> + + <p>When buildbot no longer has a hardcoded password, it will be a configuration +option here as well. + + <p>Here's a simple example that you might have in your +<code>~/.bazaar/locations.conf</code>. + +<pre class="example"> [chroot-*:///var/local/myrepo/mybranch] + buildbot_on = change + buildbot_server = localhost +</pre> + <div class="node"> +<p><hr> +<a name="Bzr-Poller"></a> +Previous: <a rel="previous" accesskey="p" href="#Bzr-Hook">Bzr Hook</a>, +Up: <a rel="up" accesskey="u" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a> + +</div> + +<h3 class="section">5.11 Bzr Poller</h3> + +<p>If you cannot insert a Bzr hook in the server, you can use the Bzr Poller. To +use, put <code>contrib/bzr_buildbot.py</code> somewhere that your buildbot +configuration can import it. Even putting it in the same directory as the master.cfg +should work. Install the poller in the buildbot configuration as with any +other change source. Minimally, provide a URL that you want to poll (bzr://, +bzr+ssh://, or lp:), though make sure the buildbot user has necessary +privileges. You may also want to specify these optional values. + + <dl> +<dt><code>poll_interval</code><dd>The number of seconds to wait between polls. Defaults to 10 minutes. + + <br><dt><code>branch_name</code><dd>Any value to be used as the branch name. Defaults to None, or specify a +string, or specify the constants from <code>bzr_buildbot.py</code> SHORT or FULL to +get the short branch name or full branch address. + + <br><dt><code>blame_merge_author</code><dd>normally, the user that commits the revision is the user that is responsible +for the change. When run in a pqm (Patch Queue Manager, see +https://launchpad.net/pqm) environment, the user that commits is the Patch +Queue Manager, and the user that committed the merged, *parent* revision is +responsible for the change. set this value to True if this is pointed against +a PQM-managed branch. +</dl> + +<div class="node"> +<p><hr> +<a name="Build-Process"></a> +Next: <a rel="next" accesskey="n" href="#Status-Delivery">Status Delivery</a>, +Previous: <a rel="previous" accesskey="p" href="#Getting-Source-Code-Changes">Getting Source Code Changes</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">6 Build Process</h2> + +<p>A <code>Build</code> object is responsible for actually performing a build. +It gets access to a remote <code>SlaveBuilder</code> where it may run +commands, and a <code>BuildStatus</code> object where it must emit status +events. The <code>Build</code> is created by the Builder's +<code>BuildFactory</code>. + + <p>The default <code>Build</code> class is made up of a fixed sequence of +<code>BuildSteps</code>, executed one after another until all are complete +(or one of them indicates that the build should be halted early). The +default <code>BuildFactory</code> creates instances of this <code>Build</code> +class with a list of <code>BuildSteps</code>, so the basic way to configure +the build is to provide a list of <code>BuildSteps</code> to your +<code>BuildFactory</code>. + + <p>More complicated <code>Build</code> subclasses can make other decisions: +execute some steps only if certain files were changed, or if certain +previous steps passed or failed. The base class has been written to +allow users to express basic control flow without writing code, but +you can always subclass and customize to achieve more specialized +behavior. + +<ul class="menu"> +<li><a accesskey="1" href="#Build-Steps">Build Steps</a> +<li><a accesskey="2" href="#Interlocks">Interlocks</a> +<li><a accesskey="3" href="#Build-Factories">Build Factories</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Build-Steps"></a> +Next: <a rel="next" accesskey="n" href="#Interlocks">Interlocks</a>, +Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>, +Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> + +</div> + +<h3 class="section">6.1 Build Steps</h3> + +<p><code>BuildStep</code>s are usually specified in the buildmaster's +configuration file, in a list that goes into the <code>BuildFactory</code>. +The <code>BuildStep</code> instances in this list are used as templates to +construct new independent copies for each build (so that state can be +kept on the <code>BuildStep</code> in one build without affecting a later +build). Each <code>BuildFactory</code> can be created with a list of steps, +or the factory can be created empty and then steps added to it using +the <code>addStep</code> method: + +<pre class="example"> from buildbot.steps import source, shell + from buildbot.process import factory + + f = factory.BuildFactory() + f.addStep(source.SVN(svnurl="http://svn.example.org/Trunk/")) + f.addStep(shell.ShellCommand(command=["make", "all"])) + f.addStep(shell.ShellCommand(command=["make", "test"])) +</pre> + <p>In earlier versions (0.7.5 and older), these steps were specified with +a tuple of (step_class, keyword_arguments). Steps can still be +specified this way, but the preferred form is to pass actual +<code>BuildStep</code> instances to <code>addStep</code>, because that gives the +<code>BuildStep</code> class a chance to do some validation on the +arguments. + + <p>If you have a common set of steps which are used in several factories, the +<code>addSteps</code> method may be handy. It takes an iterable of <code>BuildStep</code> +instances. + +<pre class="example"> setup_steps = [ + source.SVN(svnurl="http://svn.example.org/Trunk/") + shell.ShellCommand(command="./setup") + ] + quick = factory.BuildFactory() + quick.addSteps(setup_steps) + quick.addStep(shell.shellCommand(command="make quick")) +</pre> + <p>The rest of this section lists all the standard BuildStep objects +available for use in a Build, and the parameters which can be used to +control each. + +<ul class="menu"> +<li><a accesskey="1" href="#Common-Parameters">Common Parameters</a> +<li><a accesskey="2" href="#Using-Build-Properties">Using Build Properties</a> +<li><a accesskey="3" href="#Source-Checkout">Source Checkout</a> +<li><a accesskey="4" href="#ShellCommand">ShellCommand</a> +<li><a accesskey="5" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> +<li><a accesskey="6" href="#Python-BuildSteps">Python BuildSteps</a> +<li><a accesskey="7" href="#Transferring-Files">Transferring Files</a> +<li><a accesskey="8" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a> +<li><a accesskey="9" href="#Triggering-Schedulers">Triggering Schedulers</a> +<li><a href="#Writing-New-BuildSteps">Writing New BuildSteps</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Common-Parameters"></a> +Next: <a rel="next" accesskey="n" href="#Using-Build-Properties">Using Build Properties</a>, +Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.1 Common Parameters</h4> + +<p>The standard <code>Build</code> runs a series of <code>BuildStep</code>s in order, +only stopping when it runs out of steps or if one of them requests +that the build be halted. It collects status information from each one +to create an overall build status (of SUCCESS, WARNINGS, or FAILURE). + + <p>All BuildSteps accept some common parameters. Some of these control +how their individual status affects the overall build. Others are used +to specify which <code>Locks</code> (see see <a href="#Interlocks">Interlocks</a>) should be +acquired before allowing the step to run. + + <p>Arguments common to all <code>BuildStep</code> subclasses: + + <dl> +<dt><code>name</code><dd>the name used to describe the step on the status display. It is also +used to give a name to any LogFiles created by this step. + + <br><dt><code>haltOnFailure</code><dd>if True, a FAILURE of this build step will cause the build to halt +immediately. Steps with <code>alwaysRun=True</code> are still run. Generally +speaking, haltOnFailure implies flunkOnFailure (the default for most +BuildSteps). In some cases, particularly series of tests, it makes sense +to haltOnFailure if something fails early on but not flunkOnFailure. +This can be achieved with haltOnFailure=True, flunkOnFailure=False. + + <br><dt><code>flunkOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the +overall build as FAILURE. The remaining steps will still be executed. + + <br><dt><code>flunkOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as +a FAILURE. The remaining steps will still be executed. + + <br><dt><code>warnOnWarnings</code><dd>when True, a WARNINGS or FAILURE of this build step will mark the +overall build as having WARNINGS. The remaining steps will still be +executed. + + <br><dt><code>warnOnFailure</code><dd>when True, a FAILURE of this build step will mark the overall build as +having WARNINGS. The remaining steps will still be executed. + + <br><dt><code>alwaysRun</code><dd>if True, this build step will always be run, even if a previous buildstep +with <code>haltOnFailure=True</code> has failed. + + <br><dt><code>locks</code><dd>a list of Locks (instances of <code>buildbot.locks.SlaveLock</code> or +<code>buildbot.locks.MasterLock</code>) that should be acquired before +starting this Step. The Locks will be released when the step is +complete. Note that this is a list of actual Lock instances, not +names. Also note that all Locks must have unique names. + + </dl> + +<div class="node"> +<p><hr> +<a name="Using-Build-Properties"></a> +Next: <a rel="next" accesskey="n" href="#Source-Checkout">Source Checkout</a>, +Previous: <a rel="previous" accesskey="p" href="#Common-Parameters">Common Parameters</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.2 Using Build Properties</h4> + +<p><a name="index-Properties-51"></a> +Build properties are a generalized way to provide configuration +information to build steps; see <a href="#Build-Properties">Build Properties</a>. + + <p>Some build properties are inherited from external sources – global +properties, schedulers, or buildslaves. Some build properties are +set when the build starts, such as the SourceStamp information. Other +properties can be set by BuildSteps as they run, for example the +various Source steps will set the <code>got_revision</code> property to the +source revision that was actually checked out (which can be useful +when the SourceStamp in use merely requested the “latest revision”: +<code>got_revision</code> will tell you what was actually built). + + <p>In custom BuildSteps, you can get and set the build properties with +the <code>getProperty</code>/<code>setProperty</code> methods. Each takes a string +for the name of the property, and returns or accepts an +arbitrary<a rel="footnote" href="#fn-7" name="fnd-7"><sup>7</sup></a> object. For example: + +<pre class="example"> class MakeTarball(ShellCommand): + def start(self): + if self.getProperty("os") == "win": + self.setCommand([ ... ]) # windows-only command + else: + self.setCommand([ ... ]) # equivalent for other systems + ShellCommand.start(self) +</pre> + <h3 class="heading">WithProperties</h3> + +<p><a name="index-WithProperties-52"></a> +You can use build properties in ShellCommands by using the +<code>WithProperties</code> wrapper when setting the arguments of +the ShellCommand. This interpolates the named build properties +into the generated shell command. Most step parameters accept +<code>WithProperties</code>. Please file bugs for any parameters which +do not. + +<pre class="example"> from buildbot.steps.shell import ShellCommand + from buildbot.process.properties import WithProperties + + f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%s.tar.gz", "revision"), + "source"])) +</pre> + <p>If this BuildStep were used in a tree obtained from Subversion, it +would create a tarball with a name like <samp><span class="file">build-1234.tar.gz</span></samp>. + + <p>The <code>WithProperties</code> function does <code>printf</code>-style string +interpolation, using strings obtained by calling +<code>build.getProperty(propname)</code>. Note that for every <code>%s</code> (or +<code>%d</code>, etc), you must have exactly one additional argument to +indicate which build property you want to insert. + + <p>You can also use python dictionary-style string interpolation by using +the <code>%(propname)s</code> syntax. In this form, the property name goes +in the parentheses, and WithProperties takes <em>no</em> additional +arguments: + +<pre class="example"> f.addStep(ShellCommand( + command=["tar", "czf", + WithProperties("build-%(revision)s.tar.gz"), + "source"])) +</pre> + <p>Don't forget the extra “s” after the closing parenthesis! This is +the cause of many confusing errors. + + <p>The dictionary-style interpolation supports a number of more advanced +syntaxes, too. + + <dl> +<dt><code>propname:-replacement</code><dd>If <code>propname</code> exists, substitute its value; otherwise, +substitute <code>replacement</code>. <code>replacement</code> may be empty +(<code>%(propname:-)s</code>) + + <br><dt><code>propname:+replacement</code><dd>If <code>propname</code> exists, substitute <code>replacement</code>; otherwise, +substitute an empty string. + + </dl> + + <p>Although these are similar to shell substitutions, no other +substitutions are currently supported, and <code>replacement</code> in the +above cannot contain more substitutions. + + <p>Note: like python, you can either do positional-argument interpolation +<em>or</em> keyword-argument interpolation, not both. Thus you cannot use +a string like <code>WithProperties("foo-%(revision)s-%s", "branch")</code>. + +<h3 class="heading">Common Build Properties</h3> + +<p>The following build properties are set when the build is started, and +are available to all steps. + + <dl> +<dt><code>branch</code><dd> +This comes from the build's SourceStamp, and describes which branch is +being checked out. This will be <code>None</code> (which interpolates into +<code>WithProperties</code> as an empty string) if the build is on the +default branch, which is generally the trunk. Otherwise it will be a +string like “branches/beta1.4”. The exact syntax depends upon the VC +system being used. + + <br><dt><code>revision</code><dd> +This also comes from the SourceStamp, and is the revision of the source code +tree that was requested from the VC system. When a build is requested of a +specific revision (as is generally the case when the build is triggered by +Changes), this will contain the revision specification. This is always a +string, although the syntax depends upon the VC system in use: for SVN it is an +integer, for Mercurial it is a short string, for Darcs it is a rather large +string, etc. + + <p>If the “force build” button was pressed, the revision will be <code>None</code>, +which means to use the most recent revision available. This is a “trunk +build”. This will be interpolated as an empty string. + + <br><dt><code>got_revision</code><dd> +This is set when a Source step checks out the source tree, and +provides the revision that was actually obtained from the VC system. +In general this should be the same as <code>revision</code>, except for +trunk builds, where <code>got_revision</code> indicates what revision was +current when the checkout was performed. This can be used to rebuild +the same source code later. + + <p>Note that for some VC systems (Darcs in particular), the revision is a +large string containing newlines, and is not suitable for interpolation +into a filename. + + <br><dt><code>buildername</code><dd> +This is a string that indicates which Builder the build was a part of. +The combination of buildername and buildnumber uniquely identify a +build. + + <br><dt><code>buildnumber</code><dd> +Each build gets a number, scoped to the Builder (so the first build +performed on any given Builder will have a build number of 0). This +integer property contains the build's number. + + <br><dt><code>slavename</code><dd> +This is a string which identifies which buildslave the build is +running on. + + <br><dt><code>scheduler</code><dd> +If the build was started from a scheduler, then this property will +contain the name of that scheduler. + + </dl> + +<div class="node"> +<p><hr> +<a name="Source-Checkout"></a> +Next: <a rel="next" accesskey="n" href="#ShellCommand">ShellCommand</a>, +Previous: <a rel="previous" accesskey="p" href="#Using-Build-Properties">Using Build Properties</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.3 Source Checkout</h4> + +<p>The first step of any build is typically to acquire the source code +from which the build will be performed. There are several classes to +handle this, one for each of the different source control system that +Buildbot knows about. For a description of how Buildbot treats source +control in general, see <a href="#Version-Control-Systems">Version Control Systems</a>. + + <p>All source checkout steps accept some common parameters to control how +they get the sources and where they should be placed. The remaining +per-VC-system parameters are mostly to specify where exactly the +sources are coming from. + + <dl> +<dt><code>mode</code><dd> +a string describing the kind of VC operation that is desired. Defaults +to <code>update</code>. + + <dl> +<dt><code>update</code><dd>specifies that the CVS checkout/update should be performed directly +into the workdir. Each build is performed in the same directory, +allowing for incremental builds. This minimizes disk space, bandwidth, +and CPU time. However, it may encounter problems if the build process +does not handle dependencies properly (sometimes you must do a “clean +build” to make sure everything gets compiled), or if source files are +deleted but generated files can influence test behavior (e.g. python's +.pyc files), or when source directories are deleted but generated +files prevent CVS from removing them. Builds ought to be correct +regardless of whether they are done “from scratch” or incrementally, +but it is useful to test both kinds: this mode exercises the +incremental-build style. + + <br><dt><code>copy</code><dd>specifies that the CVS workspace should be maintained in a separate +directory (called the 'copydir'), using checkout or update as +necessary. For each build, a new workdir is created with a copy of the +source tree (rm -rf workdir; cp -r copydir workdir). This doubles the +disk space required, but keeps the bandwidth low (update instead of a +full checkout). A full 'clean' build is performed each time. This +avoids any generated-file build problems, but is still occasionally +vulnerable to CVS problems such as a repository being manually +rearranged, causing CVS errors on update which are not an issue with a +full checkout. + + <!-- TODO: something is screwy about this, revisit. Is it the source --> + <!-- directory or the working directory that is deleted each time? --> + <br><dt><code>clobber</code><dd>specifes that the working directory should be deleted each time, +necessitating a full checkout for each build. This insures a clean +build off a complete checkout, avoiding any of the problems described +above. This mode exercises the “from-scratch” build style. + + <br><dt><code>export</code><dd>this is like <code>clobber</code>, except that the 'cvs export' command is +used to create the working directory. This command removes all CVS +metadata files (the CVS/ directories) from the tree, which is +sometimes useful for creating source tarballs (to avoid including the +metadata in the tar file). +</dl> + + <br><dt><code>workdir</code><dd>like all Steps, this indicates the directory where the build will take +place. Source Steps are special in that they perform some operations +outside of the workdir (like creating the workdir itself). + + <br><dt><code>alwaysUseLatest</code><dd>if True, bypass the usual “update to the last Change” behavior, and +always update to the latest changes instead. + + <br><dt><code>retry</code><dd>If set, this specifies a tuple of <code>(delay, repeats)</code> which means +that when a full VC checkout fails, it should be retried up to +<var>repeats</var> times, waiting <var>delay</var> seconds between attempts. If +you don't provide this, it defaults to <code>None</code>, which means VC +operations should not be retried. This is provided to make life easier +for buildslaves which are stuck behind poor network connections. + + </dl> + + <p>My habit as a developer is to do a <code>cvs update</code> and <code>make</code> each +morning. Problems can occur, either because of bad code being checked in, or +by incomplete dependencies causing a partial rebuild to fail where a +complete from-scratch build might succeed. A quick Builder which emulates +this incremental-build behavior would use the <code>mode='update'</code> +setting. + + <p>On the other hand, other kinds of dependency problems can cause a clean +build to fail where a partial build might succeed. This frequently results +from a link step that depends upon an object file that was removed from a +later version of the tree: in the partial tree, the object file is still +around (even though the Makefiles no longer know how to create it). + + <p>“official” builds (traceable builds performed from a known set of +source revisions) are always done as clean builds, to make sure it is +not influenced by any uncontrolled factors (like leftover files from a +previous build). A “full” Builder which behaves this way would want +to use the <code>mode='clobber'</code> setting. + + <p>Each VC system has a corresponding source checkout class: their +arguments are described on the following pages. + +<ul class="menu"> +<li><a accesskey="1" href="#CVS">CVS</a> +<li><a accesskey="2" href="#SVN">SVN</a> +<li><a accesskey="3" href="#Darcs">Darcs</a> +<li><a accesskey="4" href="#Mercurial">Mercurial</a> +<li><a accesskey="5" href="#Arch">Arch</a> +<li><a accesskey="6" href="#Bazaar">Bazaar</a> +<li><a accesskey="7" href="#Bzr">Bzr</a> +<li><a accesskey="8" href="#P4">P4</a> +<li><a accesskey="9" href="#Git">Git</a> +</ul> + +<div class="node"> +<p><hr> +<a name="CVS"></a> +Next: <a rel="next" accesskey="n" href="#SVN">SVN</a>, +Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.1 CVS</h5> + +<p><a name="index-CVS-Checkout-53"></a><a name="index-buildbot_002esteps_002esource_002eCVS-54"></a> + + <p>The <code>CVS</code> build step performs a <a href="http://www.nongnu.org/cvs/">CVS</a> checkout or update. It takes the following arguments: + + <dl> +<dt><code>cvsroot</code><dd>(required): specify the CVSROOT value, which points to a CVS +repository, probably on a remote machine. For example, the cvsroot +value you would use to get a copy of the Buildbot source code is +<code>:pserver:anonymous@cvs.sourceforge.net:/cvsroot/buildbot</code> + + <br><dt><code>cvsmodule</code><dd>(required): specify the cvs <code>module</code>, which is generally a +subdirectory of the CVSROOT. The cvsmodule for the Buildbot source +code is <code>buildbot</code>. + + <br><dt><code>branch</code><dd>a string which will be used in a <code>-r</code> argument. This is most +useful for specifying a branch to work on. Defaults to <code>HEAD</code>. + + <br><dt><code>global_options</code><dd>a list of flags to be put before the verb in the CVS command. + + <br><dt><code>checkoutDelay</code><dd>if set, the number of seconds to put between the timestamp of the last +known Change and the value used for the <code>-D</code> option. Defaults to +half of the parent Build's treeStableTimer. + + </dl> + +<div class="node"> +<p><hr> +<a name="SVN"></a> +Next: <a rel="next" accesskey="n" href="#Darcs">Darcs</a>, +Previous: <a rel="previous" accesskey="p" href="#CVS">CVS</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.2 SVN</h5> + +<p><a name="index-SVN-Checkout-55"></a><a name="index-buildbot_002esteps_002esource_002eSVN-56"></a> + + <p>The <code>SVN</code> build step performs a +<a href="http://subversion.tigris.org">Subversion</a> checkout or update. +There are two basic ways of setting up the checkout step, depending +upon whether you are using multiple branches or not. + + <p>If all of your builds use the same branch, then you should create the +<code>SVN</code> step with the <code>svnurl</code> argument: + + <dl> +<dt><code>svnurl</code><dd>(required): this specifies the <code>URL</code> argument that will be given +to the <code>svn checkout</code> command. It dictates both where the +repository is located and which sub-tree should be extracted. In this +respect, it is like a combination of the CVS <code>cvsroot</code> and +<code>cvsmodule</code> arguments. For example, if you are using a remote +Subversion repository which is accessible through HTTP at a URL of +<code>http://svn.example.com/repos</code>, and you wanted to check out the +<code>trunk/calc</code> sub-tree, you would use +<code>svnurl="http://svn.example.com/repos/trunk/calc"</code> as an argument +to your <code>SVN</code> step. +</dl> + + <p>If, on the other hand, you are building from multiple branches, then +you should create the <code>SVN</code> step with the <code>baseURL</code> and +<code>defaultBranch</code> arguments instead: + + <dl> +<dt><code>baseURL</code><dd>(required): this specifies the base repository URL, to which a branch +name will be appended. It should probably end in a slash. + + <br><dt><code>defaultBranch</code><dd>this specifies the name of the branch to use when a Build does not +provide one of its own. This will be appended to <code>baseURL</code> to +create the string that will be passed to the <code>svn checkout</code> +command. + + <br><dt><code>username</code><dd>if specified, this will be passed to the <code>svn</code> binary with a +<code>--username</code> option. + + <br><dt><code>password</code><dd>if specified, this will be passed to the <code>svn</code> binary with a +<code>--password</code> option. The password itself will be suitably obfuscated in +the logs. + + </dl> + + <p>If you are using branches, you must also make sure your +<code>ChangeSource</code> will report the correct branch names. + +<h3 class="heading">branch example</h3> + +<p>Let's suppose that the “MyProject” repository uses branches for the +trunk, for various users' individual development efforts, and for +several new features that will require some amount of work (involving +multiple developers) before they are ready to merge onto the trunk. +Such a repository might be organized as follows: + +<pre class="example"> svn://svn.example.org/MyProject/trunk + svn://svn.example.org/MyProject/branches/User1/foo + svn://svn.example.org/MyProject/branches/User1/bar + svn://svn.example.org/MyProject/branches/User2/baz + svn://svn.example.org/MyProject/features/newthing + svn://svn.example.org/MyProject/features/otherthing +</pre> + <p>Further assume that we want the Buildbot to run tests against the +trunk and against all the feature branches (i.e., do a +checkout/compile/build of branch X when a file has been changed on +branch X, when X is in the set [trunk, features/newthing, +features/otherthing]). We do not want the Buildbot to automatically +build any of the user branches, but it should be willing to build a +user branch when explicitly requested (most likely by the user who +owns that branch). + + <p>There are three things that need to be set up to accomodate this +system. The first is a ChangeSource that is capable of identifying the +branch which owns any given file. This depends upon a user-supplied +function, in an external program that runs in the SVN commit hook and +connects to the buildmaster's <code>PBChangeSource</code> over a TCP +connection. (you can use the “<code>buildbot sendchange</code>” utility +for this purpose, but you will still need an external program to +decide what value should be passed to the <code>--branch=</code> argument). +For example, a change to a file with the SVN url of +“svn://svn.example.org/MyProject/features/newthing/src/foo.c” should +be broken down into a Change instance with +<code>branch='features/newthing'</code> and <code>file='src/foo.c'</code>. + + <p>The second piece is an <code>AnyBranchScheduler</code> which will pay +attention to the desired branches. It will not pay attention to the +user branches, so it will not automatically start builds in response +to changes there. The AnyBranchScheduler class requires you to +explicitly list all the branches you want it to use, but it would not +be difficult to write a subclass which used +<code>branch.startswith('features/'</code> to remove the need for this +explicit list. Or, if you want to build user branches too, you can use +AnyBranchScheduler with <code>branches=None</code> to indicate that you want +it to pay attention to all branches. + + <p>The third piece is an <code>SVN</code> checkout step that is configured to +handle the branches correctly, with a <code>baseURL</code> value that +matches the way the ChangeSource splits each file's URL into base, +branch, and file. + +<pre class="example"> from buildbot.changes.pb import PBChangeSource + from buildbot.scheduler import AnyBranchScheduler + from buildbot.process import source, factory + from buildbot.steps import source, shell + + c['change_source'] = PBChangeSource() + s1 = AnyBranchScheduler('main', + ['trunk', 'features/newthing', 'features/otherthing'], + 10*60, ['test-i386', 'test-ppc']) + c['schedulers'] = [s1] + + f = factory.BuildFactory() + f.addStep(source.SVN(mode='update', + baseURL='svn://svn.example.org/MyProject/', + defaultBranch='trunk')) + f.addStep(shell.Compile(command="make all")) + f.addStep(shell.Test(command="make test")) + + c['builders'] = [ + {'name':'test-i386', 'slavename':'bot-i386', 'builddir':'test-i386', + 'factory':f }, + {'name':'test-ppc', 'slavename':'bot-ppc', 'builddir':'test-ppc', + 'factory':f }, + ] +</pre> + <p>In this example, when a change arrives with a <code>branch</code> attribute +of “trunk”, the resulting build will have an SVN step that +concatenates “svn://svn.example.org/MyProject/” (the baseURL) with +“trunk” (the branch name) to get the correct svn command. If the +“newthing” branch has a change to “src/foo.c”, then the SVN step +will concatenate “svn://svn.example.org/MyProject/” with +“features/newthing” to get the svnurl for checkout. + +<div class="node"> +<p><hr> +<a name="Darcs"></a> +Next: <a rel="next" accesskey="n" href="#Mercurial">Mercurial</a>, +Previous: <a rel="previous" accesskey="p" href="#SVN">SVN</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.3 Darcs</h5> + +<p><a name="index-Darcs-Checkout-57"></a><a name="index-buildbot_002esteps_002esource_002eDarcs-58"></a> + + <p>The <code>Darcs</code> build step performs a +<a href="http://darcs.net/">Darcs</a> checkout or update. + + <p>Like See <a href="#SVN">SVN</a>, this step can either be configured to always check +out a specific tree, or set up to pull from a particular branch that +gets specified separately for each build. Also like SVN, the +repository URL given to Darcs is created by concatenating a +<code>baseURL</code> with the branch name, and if no particular branch is +requested, it uses a <code>defaultBranch</code>. The only difference in +usage is that each potential Darcs repository URL must point to a +fully-fledged repository, whereas SVN URLs usually point to sub-trees +of the main Subversion repository. In other words, doing an SVN +checkout of <code>baseURL</code> is legal, but silly, since you'd probably +wind up with a copy of every single branch in the whole repository. +Doing a Darcs checkout of <code>baseURL</code> is just plain wrong, since +the parent directory of a collection of Darcs repositories is not +itself a valid repository. + + <p>The Darcs step takes the following arguments: + + <dl> +<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the +Darcs source repository is available. + + <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + + <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to <code>baseURL</code> to create the string that +will be passed to the <code>darcs get</code> command. +</dl> + +<div class="node"> +<p><hr> +<a name="Mercurial"></a> +Next: <a rel="next" accesskey="n" href="#Arch">Arch</a>, +Previous: <a rel="previous" accesskey="p" href="#Darcs">Darcs</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.4 Mercurial</h5> + +<p><a name="index-Mercurial-Checkout-59"></a><a name="index-buildbot_002esteps_002esource_002eMercurial-60"></a> + + <p>The <code>Mercurial</code> build step performs a +<a href="http://selenic.com/mercurial">Mercurial</a> (aka “hg”) checkout +or update. + + <p>Branches are handled just like See <a href="#Darcs">Darcs</a>. + + <p>The Mercurial step takes the following arguments: + + <dl> +<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the +Mercurial source repository is available. + + <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + + <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to <code>baseURL</code> to create the string that +will be passed to the <code>hg clone</code> command. +</dl> + +<div class="node"> +<p><hr> +<a name="Arch"></a> +Next: <a rel="next" accesskey="n" href="#Bazaar">Bazaar</a>, +Previous: <a rel="previous" accesskey="p" href="#Mercurial">Mercurial</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.5 Arch</h5> + +<p><a name="index-Arch-Checkout-61"></a><a name="index-buildbot_002esteps_002esource_002eArch-62"></a> + + <p>The <code>Arch</code> build step performs an <a href="http://gnuarch.org/">Arch</a> checkout or update using the <code>tla</code> client. It takes the +following arguments: + + <dl> +<dt><code>url</code><dd>(required): this specifies the URL at which the Arch source archive is +available. + + <br><dt><code>version</code><dd>(required): this specifies which “development line” (like a branch) +should be used. This provides the default branch name, but individual +builds may specify a different one. + + <br><dt><code>archive</code><dd>(optional): Each repository knows its own archive name. If this +parameter is provided, it must match the repository's archive name. +The parameter is accepted for compatibility with the <code>Bazaar</code> +step, below. + + </dl> + +<div class="node"> +<p><hr> +<a name="Bazaar"></a> +Next: <a rel="next" accesskey="n" href="#Bzr">Bzr</a>, +Previous: <a rel="previous" accesskey="p" href="#Arch">Arch</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.6 Bazaar</h5> + +<p><a name="index-Bazaar-Checkout-63"></a><a name="index-buildbot_002esteps_002esource_002eBazaar-64"></a> + + <p><code>Bazaar</code> is an alternate implementation of the Arch VC system, +which uses a client named <code>baz</code>. The checkout semantics are just +different enough from <code>tla</code> that there is a separate BuildStep for +it. + + <p>It takes exactly the same arguments as <code>Arch</code>, except that the +<code>archive=</code> parameter is required. (baz does not emit the archive +name when you do <code>baz register-archive</code>, so we must provide it +ourselves). + +<div class="node"> +<p><hr> +<a name="Bzr"></a> +Next: <a rel="next" accesskey="n" href="#P4">P4</a>, +Previous: <a rel="previous" accesskey="p" href="#Bazaar">Bazaar</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.7 Bzr</h5> + +<p><a name="index-Bzr-Checkout-65"></a><a name="index-buildbot_002esteps_002esource_002eBzr-66"></a> +<code>bzr</code> is a descendant of Arch/Baz, and is frequently referred to +as simply “Bazaar”. The repository-vs-workspace model is similar to +Darcs, but it uses a strictly linear sequence of revisions (one +history per branch) like Arch. Branches are put in subdirectories. +This makes it look very much like Mercurial, so it takes the same +arguments: + + <dl> +<dt><code>repourl</code><dd>(required unless <code>baseURL</code> is provided): the URL at which the +Bzr source repository is available. + + <br><dt><code>baseURL</code><dd>(required unless <code>repourl</code> is provided): the base repository URL, +to which a branch name will be appended. It should probably end in a +slash. + + <br><dt><code>defaultBranch</code><dd>(allowed if and only if <code>baseURL</code> is provided): this specifies +the name of the branch to use when a Build does not provide one of its +own. This will be appended to <code>baseURL</code> to create the string that +will be passed to the <code>bzr checkout</code> command. +</dl> + +<div class="node"> +<p><hr> +<a name="P4"></a> +Next: <a rel="next" accesskey="n" href="#Git">Git</a>, +Previous: <a rel="previous" accesskey="p" href="#Bzr">Bzr</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.8 P4</h5> + +<p><a name="index-Perforce-Update-67"></a><a name="index-buildbot_002esteps_002esource_002eP4-68"></a><!-- TODO @bsindex buildbot.steps.source.P4Sync --> + + <p>The <code>P4</code> build step creates a <a href="http://www.perforce.com/">Perforce</a> client specification and performs an update. + + <dl> +<dt><code>p4base</code><dd>A view into the Perforce depot without branch name or trailing "...". +Typically "//depot/proj/". +<br><dt><code>defaultBranch</code><dd>A branch name to append on build requests if none is specified. +Typically "trunk". +<br><dt><code>p4port</code><dd>(optional): the host:port string describing how to get to the P4 Depot +(repository), used as the -p argument for all p4 commands. +<br><dt><code>p4user</code><dd>(optional): the Perforce user, used as the -u argument to all p4 +commands. +<br><dt><code>p4passwd</code><dd>(optional): the Perforce password, used as the -p argument to all p4 +commands. +<br><dt><code>p4extra_views</code><dd>(optional): a list of (depotpath, clientpath) tuples containing extra +views to be mapped into the client specification. Both will have +"/..." appended automatically. The client name and source directory +will be prepended to the client path. +<br><dt><code>p4client</code><dd>(optional): The name of the client to use. In mode='copy' and +mode='update', it's particularly important that a unique name is used +for each checkout directory to avoid incorrect synchronization. For +this reason, Python percent substitution will be performed on this value +to replace %(slave)s with the slave name and %(builder)s with the +builder name. The default is "buildbot_%(slave)s_%(build)s". +</dl> + +<div class="node"> +<p><hr> +<a name="Git"></a> +Previous: <a rel="previous" accesskey="p" href="#P4">P4</a>, +Up: <a rel="up" accesskey="u" href="#Source-Checkout">Source Checkout</a> + +</div> + +<h5 class="subsubsection">6.1.3.9 Git</h5> + +<p><a name="index-Git-Checkout-69"></a><a name="index-buildbot_002esteps_002esource_002eGit-70"></a> +The <code>Git</code> build step clones or updates a <a href="http://git.or.cz/">Git</a> repository and checks out the specified branch or revision. Note +that the buildbot supports Git version 1.2.0 and later: earlier +versions (such as the one shipped in Ubuntu 'Dapper') do not support +the <samp><span class="command">git init</span></samp> command that the buildbot uses. + + <p>The Git step takes the following arguments: + + <dl> +<dt><code>repourl</code><dd>(required): the URL of the upstream Git repository. + + <br><dt><code>branch</code><dd>(optional): this specifies the name of the branch to use when a Build +does not provide one of its own. If this this parameter is not +specified, and the Build does not provide a branch, the “master” +branch will be used. +</dl> + +<div class="node"> +<p><hr> +<a name="ShellCommand"></a> +Next: <a rel="next" accesskey="n" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, +Previous: <a rel="previous" accesskey="p" href="#Source-Checkout">Source Checkout</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.4 ShellCommand</h4> + +<p><a name="index-buildbot_002esteps_002eshell_002eShellCommand-71"></a><!-- TODO @bsindex buildbot.steps.shell.TreeSize --> + + <p>This is a useful base class for just about everything you might want +to do during a build (except for the initial source checkout). It runs +a single command in a child shell on the buildslave. All stdout/stderr +is recorded into a LogFile. The step finishes with a status of FAILURE +if the command's exit code is non-zero, otherwise it has a status of +SUCCESS. + + <p>The preferred way to specify the command is with a list of argv strings, +since this allows for spaces in filenames and avoids doing any fragile +shell-escaping. You can also specify the command with a single string, in +which case the string is given to '/bin/sh -c COMMAND' for parsing. + + <p>On Windows, commands are run via <code>cmd.exe /c</code> which works well. However, +if you're running a batch file, the error level does not get propagated +correctly unless you add 'call' before your batch file's name: +<code>cmd=['call', 'myfile.bat', ...]</code>. + + <p>All ShellCommands are run by default in the “workdir”, which +defaults to the “<samp><span class="file">build</span></samp>” subdirectory of the slave builder's +base directory. The absolute path of the workdir will thus be the +slave's basedir (set as an option to <code>buildbot create-slave</code>, +see <a href="#Creating-a-buildslave">Creating a buildslave</a>) plus the builder's basedir (set in the +builder's <code>c['builddir']</code> key in master.cfg) plus the workdir +itself (a class-level attribute of the BuildFactory, defaults to +“<samp><span class="file">build</span></samp>”). + + <p><code>ShellCommand</code> arguments: + + <dl> +<dt><code>command</code><dd>a list of strings (preferred) or single string (discouraged) which +specifies the command to be run. A list of strings is preferred +because it can be used directly as an argv array. Using a single +string (with embedded spaces) requires the buildslave to pass the +string to /bin/sh for interpretation, which raises all sorts of +difficult questions about how to escape or interpret shell +metacharacters. + + <br><dt><code>env</code><dd>a dictionary of environment strings which will be added to the child +command's environment. For example, to run tests with a different i18n +language setting, you might use + + <pre class="example"> f.addStep(ShellCommand(command=["make", "test"], + env={'LANG': 'fr_FR'})) +</pre> + <p>These variable settings will override any existing ones in the +buildslave's environment or the environment specified in the +Builder. The exception is PYTHONPATH, which is merged +with (actually prepended to) any existing $PYTHONPATH setting. The +value is treated as a list of directories to prepend, and a single +string is treated like a one-item list. For example, to prepend both +<samp><span class="file">/usr/local/lib/python2.3</span></samp> and <samp><span class="file">/home/buildbot/lib/python</span></samp> +to any existing $PYTHONPATH setting, you would do something like the +following: + + <pre class="example"> f.addStep(ShellCommand( + command=["make", "test"], + env={'PYTHONPATH': ["/usr/local/lib/python2.3", + "/home/buildbot/lib/python"] })) +</pre> + <br><dt><code>want_stdout</code><dd>if False, stdout from the child process is discarded rather than being +sent to the buildmaster for inclusion in the step's LogFile. + + <br><dt><code>want_stderr</code><dd>like <code>want_stdout</code> but for stderr. Note that commands run through +a PTY do not have separate stdout/stderr streams: both are merged into +stdout. + + <br><dt><code>usePTY</code><dd>Should this command be run in a <code>pty</code>? The default is to observe the +configuration of the client (see <a href="#Buildslave-Options">Buildslave Options</a>), but specifying +<code>True</code> or <code>False</code> here will override the default. + + <p>The advantage of using a PTY is that “grandchild” processes are more likely +to be cleaned up if the build is interrupted or times out (since it enables the +use of a “process group” in which all child processes will be placed). The +disadvantages: some forms of Unix have problems with PTYs, some of your unit +tests may behave differently when run under a PTY (generally those which check +to see if they are being run interactively), and PTYs will merge the stdout and +stderr streams into a single output stream (which means the red-vs-black +coloring in the logfiles will be lost). + + <br><dt><code>logfiles</code><dd>Sometimes commands will log interesting data to a local file, rather +than emitting everything to stdout or stderr. For example, Twisted's +“trial” command (which runs unit tests) only presents summary +information to stdout, and puts the rest into a file named +<samp><span class="file">_trial_temp/test.log</span></samp>. It is often useful to watch these files +as the command runs, rather than using <samp><span class="command">/bin/cat</span></samp> to dump +their contents afterwards. + + <p>The <code>logfiles=</code> argument allows you to collect data from these +secondary logfiles in near-real-time, as the step is running. It +accepts a dictionary which maps from a local Log name (which is how +the log data is presented in the build results) to a remote filename +(interpreted relative to the build's working directory). Each named +file will be polled on a regular basis (every couple of seconds) as +the build runs, and any new text will be sent over to the buildmaster. + + <pre class="example"> f.addStep(ShellCommand( + command=["make", "test"], + logfiles={"triallog": "_trial_temp/test.log"})) +</pre> + <br><dt><code>timeout</code><dd>if the command fails to produce any output for this many seconds, it +is assumed to be locked up and will be killed. + + <br><dt><code>description</code><dd>This will be used to describe the command (on the Waterfall display) +while the command is still running. It should be a single +imperfect-tense verb, like “compiling” or “testing”. The preferred +form is a list of short strings, which allows the HTML Waterfall +display to create narrower columns by emitting a <br> tag between each +word. You may also provide a single string. + + <br><dt><code>descriptionDone</code><dd>This will be used to describe the command once it has finished. A +simple noun like “compile” or “tests” should be used. Like +<code>description</code>, this may either be a list of short strings or a +single string. + + <p>If neither <code>description</code> nor <code>descriptionDone</code> are set, the +actual command arguments will be used to construct the description. +This may be a bit too wide to fit comfortably on the Waterfall +display. + + <pre class="example"> f.addStep(ShellCommand(command=["make", "test"], + description=["testing"], + descriptionDone=["tests"])) +</pre> + <br><dt><code>logEnviron</code><dd>If this option is true (the default), then the step's logfile will describe the +environment variables on the slave. In situations where the environment is not +relevant and is long, it may be easier to set <code>logEnviron=False</code>. + + </dl> + +<div class="node"> +<p><hr> +<a name="Simple-ShellCommand-Subclasses"></a> +Next: <a rel="next" accesskey="n" href="#Python-BuildSteps">Python BuildSteps</a>, +Previous: <a rel="previous" accesskey="p" href="#ShellCommand">ShellCommand</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.5 Simple ShellCommand Subclasses</h4> + +<p>Several subclasses of ShellCommand are provided as starting points for +common build steps. These are all very simple: they just override a few +parameters so you don't have to specify them yourself, making the master.cfg +file less verbose. + +<ul class="menu"> +<li><a accesskey="1" href="#Configure">Configure</a> +<li><a accesskey="2" href="#Compile">Compile</a> +<li><a accesskey="3" href="#Test">Test</a> +<li><a accesskey="4" href="#TreeSize">TreeSize</a> +<li><a accesskey="5" href="#PerlModuleTest">PerlModuleTest</a> +<li><a accesskey="6" href="#SetProperty">SetProperty</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Configure"></a> +Next: <a rel="next" accesskey="n" href="#Compile">Compile</a>, +Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.1 Configure</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002eConfigure-72"></a> +This is intended to handle the <code>./configure</code> step from +autoconf-style projects, or the <code>perl Makefile.PL</code> step from perl +MakeMaker.pm-style modules. The default command is <code>./configure</code> +but you can change this by providing a <code>command=</code> parameter. + +<div class="node"> +<p><hr> +<a name="Compile"></a> +Next: <a rel="next" accesskey="n" href="#Test">Test</a>, +Previous: <a rel="previous" accesskey="p" href="#Configure">Configure</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.2 Compile</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002eCompile-73"></a> +This is meant to handle compiling or building a project written in C. +The default command is <code>make all</code>. When the compile is finished, +the log file is scanned for GCC warning messages, a summary log is +created with any problems that were seen, and the step is marked as +WARNINGS if any were discovered. The number of warnings is stored in a +Build Property named “warnings-count”, which is accumulated over all +Compile steps (so if two warnings are found in one step, and three are +found in another step, the overall build will have a +“warnings-count” property of 5. + + <p>The default regular expression used to detect a warning is +<code>'.*warning[: ].*'</code> , which is fairly liberal and may cause +false-positives. To use a different regexp, provide a +<code>warningPattern=</code> argument, or use a subclass which sets the +<code>warningPattern</code> attribute: + +<pre class="example"> f.addStep(Compile(command=["make", "test"], + warningPattern="^Warning: ")) +</pre> + <p>The <code>warningPattern=</code> can also be a pre-compiled python regexp +object: this makes it possible to add flags like <code>re.I</code> (to use +case-insensitive matching). + + <p>(TODO: this step needs to be extended to look for GCC error messages +as well, and collect them into a separate logfile, along with the +source code filenames involved). + +<div class="node"> +<p><hr> +<a name="Test"></a> +Next: <a rel="next" accesskey="n" href="#TreeSize">TreeSize</a>, +Previous: <a rel="previous" accesskey="p" href="#Compile">Compile</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.3 Test</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002eTest-74"></a> +This is meant to handle unit tests. The default command is <code>make +test</code>, and the <code>warnOnFailure</code> flag is set. + +<div class="node"> +<p><hr> +<a name="TreeSize"></a> +Next: <a rel="next" accesskey="n" href="#PerlModuleTest">PerlModuleTest</a>, +Previous: <a rel="previous" accesskey="p" href="#Test">Test</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.4 TreeSize</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002eTreeSize-75"></a> +This is a simple command that uses the 'du' tool to measure the size +of the code tree. It puts the size (as a count of 1024-byte blocks, +aka 'KiB' or 'kibibytes') on the step's status text, and sets a build +property named 'tree-size-KiB' with the same value. + +<div class="node"> +<p><hr> +<a name="PerlModuleTest"></a> +Next: <a rel="next" accesskey="n" href="#SetProperty">SetProperty</a>, +Previous: <a rel="previous" accesskey="p" href="#TreeSize">TreeSize</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.5 PerlModuleTest</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002ePerlModuleTest-76"></a> +This is a simple command that knows how to run tests of perl modules. +It parses the output to determine the number of tests passed and +failed and total number executed, saving the results for later query. + +<div class="node"> +<p><hr> +<a name="SetProperty"></a> +Previous: <a rel="previous" accesskey="p" href="#PerlModuleTest">PerlModuleTest</a>, +Up: <a rel="up" accesskey="u" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a> + +</div> + +<h5 class="subsubsection">6.1.5.6 SetProperty</h5> + +<p><a name="index-buildbot_002esteps_002eshell_002eSetProperty-77"></a> +This buildstep is similar to ShellCommand, except that it captures the +output of the command into a property. It is usually used like this: + +<pre class="example"> f.addStep(SetProperty(command="uname -a", property="uname")) +</pre> + <p>This runs <code>uname -a</code> and captures its stdout, stripped of leading +and trailing whitespace, in the property "uname". To avoid stripping, +add <code>strip=False</code>. The <code>property</code> argument can be specified +as a <code>WithProperties</code> object. + + <p>The more advanced usage allows you to specify a function to extract +properties from the command output. Here you can use regular +expressions, string interpolation, or whatever you would like. +The function is called with three arguments: the exit status of the +command, its standard output as a string, and its standard error as +a string. It should return a dictionary containing all new properties. + +<pre class="example"> def glob2list(rc, stdout, stderr): + jpgs = [ l.strip() for l in stdout.split('\n') ] + return { 'jpgs' : jpgs } + f.addStep(SetProperty(command="ls -1 *.jpg", extract_fn=glob2list)) +</pre> + <p>Note that any ordering relationship of the contents of stdout and +stderr is lost. For example, given + +<pre class="example"> f.addStep(SetProperty( + command="echo output1; echo error >&2; echo output2", + extract_fn=my_extract)) +</pre> + <p>Then <code>my_extract</code> will see <code>stdout="output1\noutput2\n"</code> +and <code>stderr="error\n"</code>. + +<div class="node"> +<p><hr> +<a name="Python-BuildSteps"></a> +Next: <a rel="next" accesskey="n" href="#Transferring-Files">Transferring Files</a>, +Previous: <a rel="previous" accesskey="p" href="#Simple-ShellCommand-Subclasses">Simple ShellCommand Subclasses</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.6 Python BuildSteps</h4> + +<p>Here are some BuildSteps that are specifcally useful for projects +implemented in Python. + +<ul class="menu"> +<li><a accesskey="1" href="#BuildEPYDoc">BuildEPYDoc</a> +<li><a accesskey="2" href="#PyFlakes">PyFlakes</a> +<li><a accesskey="3" href="#PyLint">PyLint</a> +</ul> + +<div class="node"> +<p><hr> +<a name="BuildEPYDoc"></a> +Next: <a rel="next" accesskey="n" href="#PyFlakes">PyFlakes</a>, +Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.6.1 BuildEPYDoc</h5> + +<p><a name="index-buildbot_002esteps_002epython_002eBuildEPYDoc-78"></a> +<a href="http://epydoc.sourceforge.net/">epydoc</a> is a tool for generating +API documentation for Python modules from their docstrings. It reads +all the .py files from your source tree, processes the docstrings +therein, and creates a large tree of .html files (or a single .pdf +file). + + <p>The <code>buildbot.steps.python.BuildEPYDoc</code> step will run +<samp><span class="command">epydoc</span></samp> to produce this API documentation, and will count the +errors and warnings from its output. + + <p>You must supply the command line to be used. The default is +<samp><span class="command">make epydocs</span></samp>, which assumes that your project has a Makefile +with an “epydocs” target. You might wish to use something like +<samp><span class="command">epydoc -o apiref source/PKGNAME</span></samp> instead. You might also want +to add <samp><span class="command">--pdf</span></samp> to generate a PDF file instead of a large tree +of HTML files. + + <p>The API docs are generated in-place in the build tree (under the +workdir, in the subdirectory controlled by the “-o” argument). To +make them useful, you will probably have to copy them to somewhere +they can be read. A command like <samp><span class="command">rsync -ad apiref/ +dev.example.com:~public_html/current-apiref/</span></samp> might be useful. You +might instead want to bundle them into a tarball and publish it in the +same place where the generated install tarball is placed. + +<pre class="example"> from buildbot.steps.python import BuildEPYDoc + + ... + f.addStep(BuildEPYDoc(command=["epydoc", "-o", "apiref", "source/mypkg"])) +</pre> + <div class="node"> +<p><hr> +<a name="PyFlakes"></a> +Next: <a rel="next" accesskey="n" href="#PyLint">PyLint</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildEPYDoc">BuildEPYDoc</a>, +Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.6.2 PyFlakes</h5> + +<p><a name="index-buildbot_002esteps_002epython_002ePyFlakes-79"></a> +<a href="http://divmod.org/trac/wiki/DivmodPyflakes">PyFlakes</a> is a tool +to perform basic static analysis of Python code to look for simple +errors, like missing imports and references of undefined names. It is +like a fast and simple form of the C “lint” program. Other tools +(like pychecker) provide more detailed results but take longer to run. + + <p>The <code>buildbot.steps.python.PyFlakes</code> step will run pyflakes and +count the various kinds of errors and warnings it detects. + + <p>You must supply the command line to be used. The default is +<samp><span class="command">make pyflakes</span></samp>, which assumes you have a top-level Makefile +with a “pyflakes” target. You might want to use something like +<samp><span class="command">pyflakes .</span></samp> or <samp><span class="command">pyflakes src</span></samp>. + +<pre class="example"> from buildbot.steps.python import PyFlakes + + ... + f.addStep(PyFlakes(command=["pyflakes", "src"])) +</pre> + <div class="node"> +<p><hr> +<a name="PyLint"></a> +Previous: <a rel="previous" accesskey="p" href="#PyFlakes">PyFlakes</a>, +Up: <a rel="up" accesskey="u" href="#Python-BuildSteps">Python BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.6.3 PyLint</h5> + +<p><a name="index-buildbot_002esteps_002epython_002ePyLint-80"></a> +Similarly, the <code>buildbot.steps.python.PyLint</code> step will run pylint and +analyze the results. + + <p>You must supply the command line to be used. There is no default. + +<pre class="example"> from buildbot.steps.python import PyLint + + ... + f.addStep(PyLint(command=["pylint", "src"])) +</pre> + <div class="node"> +<p><hr> +<a name="Transferring-Files"></a> +Next: <a rel="next" accesskey="n" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a>, +Previous: <a rel="previous" accesskey="p" href="#Python-BuildSteps">Python BuildSteps</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.7 Transferring Files</h4> + +<p><a name="index-File-Transfer-81"></a><a name="index-buildbot_002esteps_002etransfer_002eFileUpload-82"></a><a name="index-buildbot_002esteps_002etransfer_002eFileDownload-83"></a><a name="index-buildbot_002esteps_002etransfer_002eDirectoryUpload-84"></a> +Most of the work involved in a build will take place on the +buildslave. But occasionally it is useful to do some work on the +buildmaster side. The most basic way to involve the buildmaster is +simply to move a file from the slave to the master, or vice versa. +There are a pair of BuildSteps named <code>FileUpload</code> and +<code>FileDownload</code> to provide this functionality. <code>FileUpload</code> +moves a file <em>up to</em> the master, while <code>FileDownload</code> moves +a file <em>down from</em> the master. + + <p>As an example, let's assume that there is a step which produces an +HTML file within the source tree that contains some sort of generated +project documentation. We want to move this file to the buildmaster, +into a <samp><span class="file">~/public_html</span></samp> directory, so it can be visible to +developers. This file will wind up in the slave-side working directory +under the name <samp><span class="file">docs/reference.html</span></samp>. We want to put it into the +master-side <samp><span class="file">~/public_html/ref.html</span></samp>. + +<pre class="example"> from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import FileUpload + + f.addStep(ShellCommand(command=["make", "docs"])) + f.addStep(FileUpload(slavesrc="docs/reference.html", + masterdest="~/public_html/ref.html")) +</pre> + <p>The <code>masterdest=</code> argument will be passed to os.path.expanduser, +so things like “~” will be expanded properly. Non-absolute paths +will be interpreted relative to the buildmaster's base directory. +Likewise, the <code>slavesrc=</code> argument will be expanded and +interpreted relative to the builder's working directory. + + <p>To move a file from the master to the slave, use the +<code>FileDownload</code> command. For example, let's assume that some step +requires a configuration file that, for whatever reason, could not be +recorded in the source code repository or generated on the buildslave +side: + +<pre class="example"> from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import FileUpload + + f.addStep(FileDownload(mastersrc="~/todays_build_config.txt", + slavedest="build_config.txt")) + f.addStep(ShellCommand(command=["make", "config"])) +</pre> + <p>Like <code>FileUpload</code>, the <code>mastersrc=</code> argument is interpreted +relative to the buildmaster's base directory, and the +<code>slavedest=</code> argument is relative to the builder's working +directory. If the buildslave is running in <samp><span class="file">~buildslave</span></samp>, and the +builder's “builddir” is something like <samp><span class="file">tests-i386</span></samp>, then the +workdir is going to be <samp><span class="file">~buildslave/tests-i386/build</span></samp>, and a +<code>slavedest=</code> of <samp><span class="file">foo/bar.html</span></samp> will get put in +<samp><span class="file">~buildslave/tests-i386/build/foo/bar.html</span></samp>. Both of these commands +will create any missing intervening directories. + +<h4 class="subheading">Other Parameters</h4> + +<p>The <code>maxsize=</code> argument lets you set a maximum size for the file +to be transferred. This may help to avoid surprises: transferring a +100MB coredump when you were expecting to move a 10kB status file +might take an awfully long time. The <code>blocksize=</code> argument +controls how the file is sent over the network: larger blocksizes are +slightly more efficient but also consume more memory on each end, and +there is a hard-coded limit of about 640kB. + + <p>The <code>mode=</code> argument allows you to control the access permissions +of the target file, traditionally expressed as an octal integer. The +most common value is probably 0755, which sets the “x” executable +bit on the file (useful for shell scripts and the like). The default +value for <code>mode=</code> is None, which means the permission bits will +default to whatever the umask of the writing process is. The default +umask tends to be fairly restrictive, but at least on the buildslave +you can make it less restrictive with a –umask command-line option at +creation time (see <a href="#Buildslave-Options">Buildslave Options</a>). + +<h4 class="subheading">Transfering Directories</h4> + +<p>To transfer complete directories from the buildslave to the master, there +is a BuildStep named <code>DirectoryUpload</code>. It works like <code>FileUpload</code>, +just for directories. However it does not support the <code>maxsize</code>, +<code>blocksize</code> and <code>mode</code> arguments. As an example, let's assume an +generated project documentation, which consists of many files (like the output +of doxygen or epydoc). We want to move the entire documentation to the +buildmaster, into a <code>~/public_html/docs</code> directory. On the slave-side +the directory can be found under <code>docs</code>: + +<pre class="example"> from buildbot.steps.shell import ShellCommand + from buildbot.steps.transfer import DirectoryUpload + + f.addStep(ShellCommand(command=["make", "docs"])) + f.addStep(DirectoryUpload(slavesrc="docs", + masterdest="~/public_html/docs")) +</pre> + <p>The DirectoryUpload step will create all necessary directories and +transfers empty directories, too. + +<div class="node"> +<p><hr> +<a name="Steps-That-Run-on-the-Master"></a> +Next: <a rel="next" accesskey="n" href="#Triggering-Schedulers">Triggering Schedulers</a>, +Previous: <a rel="previous" accesskey="p" href="#Transferring-Files">Transferring Files</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.8 Steps That Run on the Master</h4> + +<p>Occasionally, it is useful to execute some task on the master, for example to +create a directory, deploy a build result, or trigger some other centralized +processing. This is possible, in a limited fashion, with the +<code>MasterShellCommand</code> step. + + <p>This step operates similarly to a regular <code>ShellCommand</code>, but executes on +the master, instead of the slave. To be clear, the enclosing <code>Build</code> +object must still have a slave object, just as for any other step – only, in +this step, the slave does not do anything. + + <p>In this example, the step renames a tarball based on the day of the week. + +<pre class="example"> from buildbot.steps.transfer import FileUpload + from buildbot.steps.master import MasterShellCommand + + f.addStep(FileUpload(slavesrc="widgetsoft.tar.gz", + masterdest="/var/buildoutputs/widgetsoft-new.tar.gz")) + f.addStep(MasterShellCommand(command=""" + cd /var/buildoutputs; + mv widgetsoft-new.tar.gz widgetsoft-`date +%a`.tar.gz""")) +</pre> + <div class="node"> +<p><hr> +<a name="Triggering-Schedulers"></a> +Next: <a rel="next" accesskey="n" href="#Writing-New-BuildSteps">Writing New BuildSteps</a>, +Previous: <a rel="previous" accesskey="p" href="#Steps-That-Run-on-the-Master">Steps That Run on the Master</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.9 Triggering Schedulers</h4> + +<p>The counterpart to the Triggerable described in section +see <a href="#Triggerable-Scheduler">Triggerable Scheduler</a> is the Trigger BuildStep. + +<pre class="example"> from buildbot.steps.trigger import Trigger + f.addStep(Trigger(schedulerNames=['build-prep'], + waitForFinish=True, + updateSourceStamp=True)) +</pre> + <p>The <code>schedulerNames=</code> argument lists the Triggerables +that should be triggered when this step is executed. Note that +it is possible, but not advisable, to create a cycle where a build +continually triggers itself, because the schedulers are specified +by name. + + <p>If <code>waitForFinish</code> is True, then the step will not finish until +all of the builds from the triggered schedulers have finished. If this +argument is False (the default) or not given, then the buildstep +succeeds immediately after triggering the schedulers. + + <p>If <code>updateSourceStamp</code> is True (the default), then step updates +the SourceStamp given to the Triggerables to include +<code>got_revision</code> (the revision actually used in this build) as +<code>revision</code> (the revision to use in the triggered builds). This is +useful to ensure that all of the builds use exactly the same +SourceStamp, even if other Changes have occurred while the build was +running. + +<div class="node"> +<p><hr> +<a name="Writing-New-BuildSteps"></a> +Previous: <a rel="previous" accesskey="p" href="#Triggering-Schedulers">Triggering Schedulers</a>, +Up: <a rel="up" accesskey="u" href="#Build-Steps">Build Steps</a> + +</div> + +<h4 class="subsection">6.1.10 Writing New BuildSteps</h4> + +<p>While it is a good idea to keep your build process self-contained in +the source code tree, sometimes it is convenient to put more +intelligence into your Buildbot configuration. One way to do this is +to write a custom BuildStep. Once written, this Step can be used in +the <samp><span class="file">master.cfg</span></samp> file. + + <p>The best reason for writing a custom BuildStep is to better parse the +results of the command being run. For example, a BuildStep that knows +about JUnit could look at the logfiles to determine which tests had +been run, how many passed and how many failed, and then report more +detailed information than a simple <code>rc==0</code> -based “good/bad” +decision. + +<ul class="menu"> +<li><a accesskey="1" href="#Writing-BuildStep-Constructors">Writing BuildStep Constructors</a> +<li><a accesskey="2" href="#BuildStep-LogFiles">BuildStep LogFiles</a> +<li><a accesskey="3" href="#Reading-Logfiles">Reading Logfiles</a> +<li><a accesskey="4" href="#Adding-LogObservers">Adding LogObservers</a> +<li><a accesskey="5" href="#BuildStep-URLs">BuildStep URLs</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Writing-BuildStep-Constructors"></a> +Next: <a rel="next" accesskey="n" href="#BuildStep-LogFiles">BuildStep LogFiles</a>, +Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.10.1 Writing BuildStep Constructors</h5> + +<p>BuildStep classes have some extra equipment, because they are their own +factories. Consider the use of a BuildStep in <samp><span class="file">master.cfg</span></samp>: + +<pre class="example"> f.addStep(MyStep(someopt="stuff", anotheropt=1)) +</pre> + <p>This creates a single instance of class <code>MyStep</code>. However, Buildbot needs +a new object each time the step is executed. this is accomplished by storing +the information required to instantiate a new object in the <code>factory</code> +attribute. When the time comes to construct a new Build, BuildFactory consults +this attribute (via <code>getStepFactory</code>) and instantiates a new step object. + + <p>When writing a new step class, then, keep in mind are that you cannot do +anything "interesting" in the constructor – limit yourself to checking and +storing arguments. To ensure that these arguments are provided to any new +objects, call <code>self.addFactoryArguments</code> with any keyword arguments your +constructor needs. + + <p>Keep a <code>**kwargs</code> argument on the end of your options, and pass that up to +the parent class's constructor. + + <p>The whole thing looks like this: + +<pre class="example"> class Frobinfy(LoggingBuildStep): + def __init__(self, + frob_what="frobee", + frob_how_many=None, + frob_how=None, + **kwargs) + + # check + if frob_how_many is None: + raise TypeError("Frobinfy argument how_many is required") + + # call parent + LoggingBuildStep.__init__(self, **kwargs) + + # and record arguments for later + self.addFactoryArguments( + frob_what=frob_what, + frob_how_many=frob_how_many, + frob_how=frob_how) + + class FastFrobnify(Frobnify): + def __init__(self, + speed=5, + **kwargs) + Frobnify.__init__(self, **kwargs) + self.addFactoryArguments( + speed=speed) +</pre> + <div class="node"> +<p><hr> +<a name="BuildStep-LogFiles"></a> +Next: <a rel="next" accesskey="n" href="#Reading-Logfiles">Reading Logfiles</a>, +Previous: <a rel="previous" accesskey="p" href="#Writing-BuildStep-Constructors">Writing BuildStep Constructors</a>, +Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.10.2 BuildStep LogFiles</h5> + +<p>Each BuildStep has a collection of “logfiles”. Each one has a short +name, like “stdio” or “warnings”. Each LogFile contains an +arbitrary amount of text, usually the contents of some output file +generated during a build or test step, or a record of everything that +was printed to stdout/stderr during the execution of some command. + + <p>These LogFiles are stored to disk, so they can be retrieved later. + + <p>Each can contain multiple “channels”, generally limited to three +basic ones: stdout, stderr, and “headers”. For example, when a +ShellCommand runs, it writes a few lines to the “headers” channel to +indicate the exact argv strings being run, which directory the command +is being executed in, and the contents of the current environment +variables. Then, as the command runs, it adds a lot of “stdout” and +“stderr” messages. When the command finishes, a final “header” +line is added with the exit code of the process. + + <p>Status display plugins can format these different channels in +different ways. For example, the web page shows LogFiles as text/html, +with header lines in blue text, stdout in black, and stderr in red. A +different URL is available which provides a text/plain format, in +which stdout and stderr are collapsed together, and header lines are +stripped completely. This latter option makes it easy to save the +results to a file and run <samp><span class="command">grep</span></samp> or whatever against the +output. + + <p>Each BuildStep contains a mapping (implemented in a python dictionary) +from LogFile name to the actual LogFile objects. Status plugins can +get a list of LogFiles to display, for example, a list of HREF links +that, when clicked, provide the full contents of the LogFile. + +<h3 class="heading">Using LogFiles in custom BuildSteps</h3> + +<p>The most common way for a custom BuildStep to use a LogFile is to +summarize the results of a ShellCommand (after the command has +finished running). For example, a compile step with thousands of lines +of output might want to create a summary of just the warning messages. +If you were doing this from a shell, you would use something like: + +<pre class="example"> grep "warning:" output.log >warnings.log +</pre> + <p>In a custom BuildStep, you could instead create a “warnings” LogFile +that contained the same text. To do this, you would add code to your +<code>createSummary</code> method that pulls lines from the main output log +and creates a new LogFile with the results: + +<pre class="example"> def createSummary(self, log): + warnings = [] + for line in log.readlines(): + if "warning:" in line: + warnings.append() + self.addCompleteLog('warnings', "".join(warnings)) +</pre> + <p>This example uses the <code>addCompleteLog</code> method, which creates a +new LogFile, puts some text in it, and then “closes” it, meaning +that no further contents will be added. This LogFile will appear in +the HTML display under an HREF with the name “warnings”, since that +is the name of the LogFile. + + <p>You can also use <code>addHTMLLog</code> to create a complete (closed) +LogFile that contains HTML instead of plain text. The normal LogFile +will be HTML-escaped if presented through a web page, but the HTML +LogFile will not. At the moment this is only used to present a pretty +HTML representation of an otherwise ugly exception traceback when +something goes badly wrong during the BuildStep. + + <p>In contrast, you might want to create a new LogFile at the beginning +of the step, and add text to it as the command runs. You can create +the LogFile and attach it to the build by calling <code>addLog</code>, which +returns the LogFile object. You then add text to this LogFile by +calling methods like <code>addStdout</code> and <code>addHeader</code>. When you +are done, you must call the <code>finish</code> method so the LogFile can be +closed. It may be useful to create and populate a LogFile like this +from a LogObserver method See <a href="#Adding-LogObservers">Adding LogObservers</a>. + + <p>The <code>logfiles=</code> argument to <code>ShellCommand</code> (see +see <a href="#ShellCommand">ShellCommand</a>) creates new LogFiles and fills them in realtime +by asking the buildslave to watch a actual file on disk. The +buildslave will look for additions in the target file and report them +back to the BuildStep. These additions will be added to the LogFile by +calling <code>addStdout</code>. These secondary LogFiles can be used as the +source of a LogObserver just like the normal “stdio” LogFile. + +<div class="node"> +<p><hr> +<a name="Reading-Logfiles"></a> +Next: <a rel="next" accesskey="n" href="#Adding-LogObservers">Adding LogObservers</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildStep-LogFiles">BuildStep LogFiles</a>, +Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.10.3 Reading Logfiles</h5> + +<p>Once a LogFile has been added to a BuildStep with <code>addLog()</code>, +<code>addCompleteLog()</code>, <code>addHTMLLog()</code>, or <code>logfiles=</code>, +your BuildStep can retrieve it by using <code>getLog()</code>: + +<pre class="example"> class MyBuildStep(ShellCommand): + logfiles = { "nodelog": "_test/node.log" } + + def evaluateCommand(self, cmd): + nodelog = self.getLog("nodelog") + if "STARTED" in nodelog.getText(): + return SUCCESS + else: + return FAILURE +</pre> + <p>For a complete list of the methods you can call on a LogFile, please +see the docstrings on the <code>IStatusLog</code> class in +<samp><span class="file">buildbot/interfaces.py</span></samp>. + +<div class="node"> +<p><hr> +<a name="Adding-LogObservers"></a> +Next: <a rel="next" accesskey="n" href="#BuildStep-URLs">BuildStep URLs</a>, +Previous: <a rel="previous" accesskey="p" href="#Reading-Logfiles">Reading Logfiles</a>, +Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.10.4 Adding LogObservers</h5> + +<p><a name="index-LogObserver-85"></a><a name="index-LogLineObserver-86"></a> +Most shell commands emit messages to stdout or stderr as they operate, +especially if you ask them nicely with a <code>--verbose</code> flag of some +sort. They may also write text to a log file while they run. Your +BuildStep can watch this output as it arrives, to keep track of how +much progress the command has made. You can get a better measure of +progress by counting the number of source files compiled or test cases +run than by merely tracking the number of bytes that have been written +to stdout. This improves the accuracy and the smoothness of the ETA +display. + + <p>To accomplish this, you will need to attach a <code>LogObserver</code> to +one of the log channels, most commonly to the “stdio” channel but +perhaps to another one which tracks a log file. This observer is given +all text as it is emitted from the command, and has the opportunity to +parse that output incrementally. Once the observer has decided that +some event has occurred (like a source file being compiled), it can +use the <code>setProgress</code> method to tell the BuildStep about the +progress that this event represents. + + <p>There are a number of pre-built <code>LogObserver</code> classes that you +can choose from (defined in <code>buildbot.process.buildstep</code>, and of +course you can subclass them to add further customization. The +<code>LogLineObserver</code> class handles the grunt work of buffering and +scanning for end-of-line delimiters, allowing your parser to operate +on complete stdout/stderr lines. (Lines longer than a set maximum +length are dropped; the maximum defaults to 16384 bytes, but you can +change it by calling <code>setMaxLineLength()</code> on your +<code>LogLineObserver</code> instance. Use <code>sys.maxint</code> for effective +infinity.) + + <p>For example, let's take a look at the <code>TrialTestCaseCounter</code>, +which is used by the Trial step to count test cases as they are run. +As Trial executes, it emits lines like the following: + +<pre class="example"> buildbot.test.test_config.ConfigTest.testDebugPassword ... [OK] + buildbot.test.test_config.ConfigTest.testEmpty ... [OK] + buildbot.test.test_config.ConfigTest.testIRC ... [FAIL] + buildbot.test.test_config.ConfigTest.testLocks ... [OK] +</pre> + <p>When the tests are finished, trial emits a long line of “======” and +then some lines which summarize the tests that failed. We want to +avoid parsing these trailing lines, because their format is less +well-defined than the “[OK]” lines. + + <p>The parser class looks like this: + +<pre class="example"> from buildbot.process.buildstep import LogLineObserver + + class TrialTestCaseCounter(LogLineObserver): + _line_re = re.compile(r'^([\w\.]+) \.\.\. \[([^\]]+)\]$') + numTests = 0 + finished = False + + def outLineReceived(self, line): + if self.finished: + return + if line.startswith("=" * 40): + self.finished = True + return + + m = self._line_re.search(line.strip()) + if m: + testname, result = m.groups() + self.numTests += 1 + self.step.setProgress('tests', self.numTests) +</pre> + <p>This parser only pays attention to stdout, since that's where trial +writes the progress lines. It has a mode flag named <code>finished</code> to +ignore everything after the “====” marker, and a scary-looking +regular expression to match each line while hopefully ignoring other +messages that might get displayed as the test runs. + + <p>Each time it identifies a test has been completed, it increments its +counter and delivers the new progress value to the step with +<code>self.step.setProgress</code>. This class is specifically measuring +progress along the “tests” metric, in units of test cases (as +opposed to other kinds of progress like the “output” metric, which +measures in units of bytes). The Progress-tracking code uses each +progress metric separately to come up with an overall completion +percentage and an ETA value. + + <p>To connect this parser into the <code>Trial</code> BuildStep, +<code>Trial.__init__</code> ends with the following clause: + +<pre class="example"> # this counter will feed Progress along the 'test cases' metric + counter = TrialTestCaseCounter() + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) +</pre> + <p>This creates a TrialTestCaseCounter and tells the step that the +counter wants to watch the “stdio” log. The observer is +automatically given a reference to the step in its <code>.step</code> +attribute. + +<h4 class="subheading">A Somewhat Whimsical Example</h4> + +<p>Let's say that we've got some snazzy new unit-test framework called +Framboozle. It's the hottest thing since sliced bread. It slices, it +dices, it runs unit tests like there's no tomorrow. Plus if your unit +tests fail, you can use its name for a Web 2.1 startup company, make +millions of dollars, and hire engineers to fix the bugs for you, while +you spend your afternoons lazily hang-gliding along a scenic pacific +beach, blissfully unconcerned about the state of your +tests.<a rel="footnote" href="#fn-8" name="fnd-8"><sup>8</sup></a> + + <p>To run a Framboozle-enabled test suite, you just run the 'framboozler' +command from the top of your source code tree. The 'framboozler' +command emits a bunch of stuff to stdout, but the most interesting bit +is that it emits the line "FNURRRGH!" every time it finishes running a +test case<a rel="footnote" href="#fn-9" name="fnd-9"><sup>9</sup></a>. You'd like to have a test-case counting LogObserver that +watches for these lines and counts them, because counting them will +help the buildbot more accurately calculate how long the build will +take, and this will let you know exactly how long you can sneak out of +the office for your hang-gliding lessons without anyone noticing that +you're gone. + + <p>This will involve writing a new BuildStep (probably named +"Framboozle") which inherits from ShellCommand. The BuildStep class +definition itself will look something like this: + +<pre class="example"> # START + from buildbot.steps.shell import ShellCommand + from buildbot.process.buildstep import LogLineObserver + + class FNURRRGHCounter(LogLineObserver): + numTests = 0 + def outLineReceived(self, line): + if "FNURRRGH!" in line: + self.numTests += 1 + self.step.setProgress('tests', self.numTests) + + class Framboozle(ShellCommand): + command = ["framboozler"] + + def __init__(self, **kwargs): + ShellCommand.__init__(self, **kwargs) # always upcall! + counter = FNURRRGHCounter()) + self.addLogObserver('stdio', counter) + self.progressMetrics += ('tests',) + # FINISH +</pre> + <p>So that's the code that we want to wind up using. How do we actually +deploy it? + + <p>You have a couple of different options. + + <p>Option 1: The simplest technique is to simply put this text +(everything from START to FINISH) in your master.cfg file, somewhere +before the BuildFactory definition where you actually use it in a +clause like: + +<pre class="example"> f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(Framboozle()) +</pre> + <p>Remember that master.cfg is secretly just a python program with one +job: populating the BuildmasterConfig dictionary. And python programs +are allowed to define as many classes as they like. So you can define +classes and use them in the same file, just as long as the class is +defined before some other code tries to use it. + + <p>This is easy, and it keeps the point of definition very close to the +point of use, and whoever replaces you after that unfortunate +hang-gliding accident will appreciate being able to easily figure out +what the heck this stupid "Framboozle" step is doing anyways. The +downside is that every time you reload the config file, the Framboozle +class will get redefined, which means that the buildmaster will think +that you've reconfigured all the Builders that use it, even though +nothing changed. Bleh. + + <p>Option 2: Instead, we can put this code in a separate file, and import +it into the master.cfg file just like we would the normal buildsteps +like ShellCommand and SVN. + + <p>Create a directory named ~/lib/python, put everything from START to +FINISH in ~/lib/python/framboozle.py, and run your buildmaster using: + +<pre class="example"> PYTHONPATH=~/lib/python buildbot start MASTERDIR +</pre> + <p>or use the <samp><span class="file">Makefile.buildbot</span></samp> to control the way +<samp><span class="command">buildbot start</span></samp> works. Or add something like this to +something like your ~/.bashrc or ~/.bash_profile or ~/.cshrc: + +<pre class="example"> export PYTHONPATH=~/lib/python +</pre> + <p>Once we've done this, our master.cfg can look like: + +<pre class="example"> from framboozle import Framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(Framboozle()) +</pre> + <p>or: + +<pre class="example"> import framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(framboozle.Framboozle()) +</pre> + <p>(check out the python docs for details about how "import" and "from A +import B" work). + + <p>What we've done here is to tell python that every time it handles an +"import" statement for some named module, it should look in our +~/lib/python/ for that module before it looks anywhere else. After our +directories, it will try in a bunch of standard directories too +(including the one where buildbot is installed). By setting the +PYTHONPATH environment variable, you can add directories to the front +of this search list. + + <p>Python knows that once it "import"s a file, it doesn't need to +re-import it again. This means that reconfiguring the buildmaster +(with "buildbot reconfig", for example) won't make it think the +Framboozle class has changed every time, so the Builders that use it +will not be spuriously restarted. On the other hand, you either have +to start your buildmaster in a slightly weird way, or you have to +modify your environment to set the PYTHONPATH variable. + + <p>Option 3: Install this code into a standard python library directory + + <p>Find out what your python's standard include path is by asking it: + +<pre class="example"> 80:warner@luther% python + Python 2.4.4c0 (#2, Oct 2 2006, 00:57:46) + [GCC 4.1.2 20060928 (prerelease) (Debian 4.1.1-15)] on linux2 + Type "help", "copyright", "credits" or "license" for more information. + >>> import sys + >>> import pprint + >>> pprint.pprint(sys.path) + ['', + '/usr/lib/python24.zip', + '/usr/lib/python2.4', + '/usr/lib/python2.4/plat-linux2', + '/usr/lib/python2.4/lib-tk', + '/usr/lib/python2.4/lib-dynload', + '/usr/local/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages', + '/usr/lib/python2.4/site-packages/Numeric', + '/var/lib/python-support/python2.4', + '/usr/lib/site-python'] +</pre> + <p>In this case, putting the code into +/usr/local/lib/python2.4/site-packages/framboozle.py would work just +fine. We can use the same master.cfg "import framboozle" statement as +in Option 2. By putting it in a standard include directory (instead of +the decidedly non-standard ~/lib/python), we don't even have to set +PYTHONPATH to anything special. The downside is that you probably have +to be root to write to one of those standard include directories. + + <p>Option 4: Submit the code for inclusion in the Buildbot distribution + + <p>Make a fork of buildbot on http://github.com/djmitche/buildbot or post a patch +in a bug at http://buildbot.net. In either case, post a note about your patch +to the mailing list, so others can provide feedback and, eventually, commit it. + +<pre class="example"> from buildbot.steps import framboozle + f = BuildFactory() + f.addStep(SVN(svnurl="stuff")) + f.addStep(framboozle.Framboozle()) +</pre> + <p>And then you don't even have to install framboozle.py anywhere on your +system, since it will ship with Buildbot. You don't have to be root, +you don't have to set PYTHONPATH. But you do have to make a good case +for Framboozle being worth going into the main distribution, you'll +probably have to provide docs and some unit test cases, you'll need to +figure out what kind of beer the author likes, and then you'll have to +wait until the next release. But in some environments, all this is +easier than getting root on your buildmaster box, so the tradeoffs may +actually be worth it. + + <p>Putting the code in master.cfg (1) makes it available to that +buildmaster instance. Putting it in a file in a personal library +directory (2) makes it available for any buildmasters you might be +running. Putting it in a file in a system-wide shared library +directory (3) makes it available for any buildmasters that anyone on +that system might be running. Getting it into the buildbot's upstream +repository (4) makes it available for any buildmasters that anyone in +the world might be running. It's all a matter of how widely you want +to deploy that new class. + +<div class="node"> +<p><hr> +<a name="BuildStep-URLs"></a> +Previous: <a rel="previous" accesskey="p" href="#Adding-LogObservers">Adding LogObservers</a>, +Up: <a rel="up" accesskey="u" href="#Writing-New-BuildSteps">Writing New BuildSteps</a> + +</div> + +<h5 class="subsubsection">6.1.10.5 BuildStep URLs</h5> + +<p><a name="index-links-87"></a><a name="index-BuildStep-URLs-88"></a><a name="index-addURL-89"></a> +Each BuildStep has a collection of “links”. Like its collection of +LogFiles, each link has a name and a target URL. The web status page +creates HREFs for each link in the same box as it does for LogFiles, +except that the target of the link is the external URL instead of an +internal link to a page that shows the contents of the LogFile. + + <p>These external links can be used to point at build information hosted +on other servers. For example, the test process might produce an +intricate description of which tests passed and failed, or some sort +of code coverage data in HTML form, or a PNG or GIF image with a graph +of memory usage over time. The external link can provide an easy way +for users to navigate from the buildbot's status page to these +external web sites or file servers. Note that the step itself is +responsible for insuring that there will be a document available at +the given URL (perhaps by using <samp><span class="command">scp</span></samp> to copy the HTML output +to a <samp><span class="file">~/public_html/</span></samp> directory on a remote web server). Calling +<code>addURL</code> does not magically populate a web server. + + <p>To set one of these links, the BuildStep should call the <code>addURL</code> +method with the name of the link and the target URL. Multiple URLs can +be set. + + <p>In this example, we assume that the <samp><span class="command">make test</span></samp> command causes +a collection of HTML files to be created and put somewhere on the +coverage.example.org web server, in a filename that incorporates the +build number. + +<pre class="example"> class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s" % "buildnumber")] + + def createSummary(self, log): + buildnumber = self.getProperty("buildnumber") + url = "http://coverage.example.org/builds/%s.html" % buildnumber + self.addURL("coverage", url) +</pre> + <p>You might also want to extract the URL from some special message +output by the build process itself: + +<pre class="example"> class TestWithCodeCoverage(BuildStep): + command = ["make", "test", + WithProperties("buildnum=%s" % "buildnumber")] + + def createSummary(self, log): + output = StringIO(log.getText()) + for line in output.readlines(): + if line.startswith("coverage-url:"): + url = line[len("coverage-url:"):].strip() + self.addURL("coverage", url) + return +</pre> + <p>Note that a build process which emits both stdout and stderr might +cause this line to be split or interleaved between other lines. It +might be necessary to restrict the getText() call to only stdout with +something like this: + +<pre class="example"> output = StringIO("".join([c[1] + for c in log.getChunks() + if c[0] == LOG_CHANNEL_STDOUT])) +</pre> + <p>Of course if the build is run under a PTY, then stdout and stderr will +be merged before the buildbot ever sees them, so such interleaving +will be unavoidable. + +<div class="node"> +<p><hr> +<a name="Interlocks"></a> +Next: <a rel="next" accesskey="n" href="#Build-Factories">Build Factories</a>, +Previous: <a rel="previous" accesskey="p" href="#Build-Steps">Build Steps</a>, +Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> + +</div> + +<h3 class="section">6.2 Interlocks</h3> + +<p><a name="index-locks-90"></a><a name="index-buildbot_002elocks_002eMasterLock-91"></a><a name="index-buildbot_002elocks_002eSlaveLock-92"></a><a name="index-buildbot_002elocks_002eLockAccess-93"></a> +Until now, we assumed that a master can run builds at any slave whenever +needed or desired. Some times, you want to enforce additional constraints on +builds. For reasons like limited network bandwidth, old slave machines, or a +self-willed data base server, you may want to limit the number of builds (or +build steps) that can access a resource. + + <p>The mechanism used by Buildbot is known as the read/write lock.<a rel="footnote" href="#fn-10" name="fnd-10"><sup>10</sup></a> It +allows either many readers or a single writer but not a combination of readers +and writers. The general lock has been modified and extended for use in +Buildbot. Firstly, the general lock allows an infinite number of readers. In +Buildbot, we often want to put an upper limit on the number of readers, for +example allowing two out of five possible builds at the same time. To do this, +the lock counts the number of active readers. Secondly, the terms <em>read +mode</em> and <em>write mode</em> are confusing in Buildbot context. They have been +replaced by <em>counting mode</em> (since the lock counts them) and <em>exclusive +mode</em>. As a result of these changes, locks in Buildbot allow a number of +builds (upto some fixed number) in counting mode, or they allow one build in +exclusive mode. + + <p>Often, not all slaves are equal. To allow for this situation, Buildbot allows +to have a separate upper limit on the count for each slave. In this way, you +can have at most 3 concurrent builds at a fast slave, 2 at a slightly older +slave, and 1 at all other slaves. + + <p>The final thing you can specify when you introduce a new lock is its scope. +Some constraints are global – they must be enforced over all slaves. Other +constraints are local to each slave. A <em>master lock</em> is used for the +global constraints. You can ensure for example that at most one build (of all +builds running at all slaves) accesses the data base server. With a +<em>slave lock</em> you can add a limit local to each slave. With such a lock, +you can for example enforce an upper limit to the number of active builds at a +slave, like above. + + <p>Time for a few examples. Below a master lock is defined to protect a data base, +and a slave lock is created to limit the number of builds at each slave. + +<pre class="example"> from buildbot import locks + + db_lock = locks.MasterLock("database") + build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = { 'fast': 3, 'new': 2 }) +</pre> + <p>After importing locks from buildbot, <code>db_lock</code> is defined to be a master +lock. The <code>"database"</code> string is used for uniquely identifying the lock. +At the next line, a slave lock called <code>build_lock</code> is created. It is +identified by the <code>"slave_builds"</code> string. Since the requirements of the +lock are a bit more complicated, two optional arguments are also specified. The +<code>maxCount</code> parameter sets the default limit for builds in counting mode to +<code>1</code>. For the slave called <code>'fast'</code> however, we want to have at most +three builds, and for the slave called <code>'new'</code> the upper limit is two +builds running at the same time. + + <p>The next step is using the locks in builds. Buildbot allows a lock to be used +during an entire build (from beginning to end), or only during a single build +step. In the latter case, the lock is claimed for use just before the step +starts, and released again when the step ends. To prevent +deadlocks,<a rel="footnote" href="#fn-11" name="fnd-11"><sup>11</sup></a> it is not possible to claim or release +locks at other times. + + <p>To use locks, you should add them with a <code>locks</code> argument. +Each use of a lock is either in counting mode (that is, possibly shared with +other builds) or in exclusive mode. A build or build step proceeds only when it +has acquired all locks. If a build or step needs a lot of locks, it may be +starved<a rel="footnote" href="#fn-12" name="fnd-12"><sup>12</sup></a> by other builds that need fewer locks. + + <p>To illustrate use of locks, a few examples. + +<pre class="example"> from buildbot import locks + from buildbot.steps import source, shell + from buildbot.process import factory + + db_lock = locks.MasterLock("database") + build_lock = locks.SlaveLock("slave_builds", + maxCount = 1, + maxCountForSlave = { 'fast': 3, 'new': 2 }) + + f = factory.BuildFactory() + f.addStep(source.SVN(svnurl="http://example.org/svn/Trunk")) + f.addStep(shell.ShellCommand(command="make all")) + f.addStep(shell.ShellCommand(command="make test", + locks=[db_lock.access('exclusive')])) + + b1 = {'name': 'full1', 'slavename': 'fast', 'builddir': 'f1', 'factory': f, + 'locks': [build_lock.access('counting')] } + + b2 = {'name': 'full2', 'slavename': 'new', 'builddir': 'f2', 'factory': f. + 'locks': [build_lock.access('counting')] } + + b3 = {'name': 'full3', 'slavename': 'old', 'builddir': 'f3', 'factory': f. + 'locks': [build_lock.access('counting')] } + + b4 = {'name': 'full4', 'slavename': 'other', 'builddir': 'f4', 'factory': f. + 'locks': [build_lock.access('counting')] } + + c['builders'] = [b1, b2, b3, b4] +</pre> + <p>Here we have four slaves <code>b1</code>, <code>b2</code>, <code>b3</code>, and <code>b4</code>. Each +slave performs the same checkout, make, and test build step sequence. +We want to enforce that at most one test step is executed between all slaves due +to restrictions with the data base server. This is done by adding the +<code>locks=</code> parameter with the third step. It takes a list of locks with their +access mode. In this case only the <code>db_lock</code> is needed. The exclusive +access mode is used to ensure there is at most one slave that executes the test +step. + + <p>In addition to exclusive accessing the data base, we also want slaves to stay +responsive even under the load of a large number of builds being triggered. +For this purpose, the slave lock called <code>build_lock</code> is defined. Since +the restraint holds for entire builds, the lock is specified in the builder +with <code>'locks': [build_lock.access('counting')]</code>. +<div class="node"> +<p><hr> +<a name="Build-Factories"></a> +Previous: <a rel="previous" accesskey="p" href="#Interlocks">Interlocks</a>, +Up: <a rel="up" accesskey="u" href="#Build-Process">Build Process</a> + +</div> + +<h3 class="section">6.3 Build Factories</h3> + +<p>Each Builder is equipped with a “build factory”, which is +responsible for producing the actual <code>Build</code> objects that perform +each build. This factory is created in the configuration file, and +attached to a Builder through the <code>factory</code> element of its +dictionary. + + <p>The standard <code>BuildFactory</code> object creates <code>Build</code> objects +by default. These Builds will each execute a collection of BuildSteps +in a fixed sequence. Each step can affect the results of the build, +but in general there is little intelligence to tie the different steps +together. You can create subclasses of <code>Build</code> to implement more +sophisticated build processes, and then use a subclass of +<code>BuildFactory</code> (or simply set the <code>buildClass</code> attribute) to +create instances of your new Build subclass. + +<ul class="menu"> +<li><a accesskey="1" href="#BuildStep-Objects">BuildStep Objects</a> +<li><a accesskey="2" href="#BuildFactory">BuildFactory</a> +<li><a accesskey="3" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> +</ul> + +<div class="node"> +<p><hr> +<a name="BuildStep-Objects"></a> +Next: <a rel="next" accesskey="n" href="#BuildFactory">BuildFactory</a>, +Previous: <a rel="previous" accesskey="p" href="#Build-Factories">Build Factories</a>, +Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> + +</div> + +<h4 class="subsection">6.3.1 BuildStep Objects</h4> + +<p>The steps used by these builds are all subclasses of <code>BuildStep</code>. +The standard ones provided with Buildbot are documented later, +See <a href="#Build-Steps">Build Steps</a>. You can also write your own subclasses to use in +builds. + + <p>The basic behavior for a <code>BuildStep</code> is to: + + <ul> +<li>run for a while, then stop +<li>possibly invoke some RemoteCommands on the attached build slave +<li>possibly produce a set of log files +<li>finish with a status described by one of four values defined in +buildbot.status.builder: SUCCESS, WARNINGS, FAILURE, SKIPPED +<li>provide a list of short strings to describe the step +<li>define a color (generally green, orange, or red) with which the +step should be displayed +</ul> + + <p>More sophisticated steps may produce additional information and +provide it to later build steps, or store it in the factory to provide +to later builds. + +<ul class="menu"> +<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a> +<li><a accesskey="2" href="#Quick-builds">Quick builds</a> +</ul> + +<div class="node"> +<p><hr> +<a name="BuildFactory"></a> +Next: <a rel="next" accesskey="n" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildStep-Objects">BuildStep Objects</a>, +Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> + +</div> + +<h4 class="subsection">6.3.2 BuildFactory</h4> + +<p><a name="index-buildbot_002eprocess_002efactory_002eBuildFactory-94"></a><a name="index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-95"></a><!-- TODO: what is BasicSVN anyway? --> +<a name="index-buildbot_002eprocess_002efactory_002eBasicSVN-96"></a> +The default <code>BuildFactory</code>, provided in the +<code>buildbot.process.factory</code> module, contains an internal list of +“BuildStep specifications”: a list of <code>(step_class, kwargs)</code> +tuples for each. These specification tuples are constructed when the +config file is read, by asking the instances passed to <code>addStep</code> +for their subclass and arguments. + + <p>When asked to create a Build, the <code>BuildFactory</code> puts a copy of +the list of step specifications into the new Build object. When the +Build is actually started, these step specifications are used to +create the actual set of BuildSteps, which are then executed one at a +time. This serves to give each Build an independent copy of each step. +For example, a build which consists of a CVS checkout followed by a +<code>make build</code> would be constructed as follows: + +<pre class="example"> from buildbot.steps import source, shell + from buildbot.process import factory + + f = factory.BuildFactory() + f.addStep(source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update")) + f.addStep(shell.Compile(command=["make", "build"])) +</pre> + <p>(To support config files from buildbot-0.7.5 and earlier, +<code>addStep</code> also accepts the <code>f.addStep(shell.Compile, +command=["make","build"])</code> form, although its use is discouraged +because then the <code>Compile</code> step doesn't get to validate or +complain about its arguments until build time. The modern +pass-by-instance approach allows this validation to occur while the +config file is being loaded, where the admin has a better chance of +noticing problems). + + <p>It is also possible to pass a list of steps into the +<code>BuildFactory</code> when it is created. Using <code>addStep</code> is +usually simpler, but there are cases where is is more convenient to +create the list of steps ahead of time.: + +<pre class="example"> from buildbot.steps import source, shell + from buildbot.process import factory + + all_steps = [source.CVS(cvsroot=CVSROOT, cvsmodule="project", mode="update"), + shell.Compile(command=["make", "build"]), + ] + f = factory.BuildFactory(all_steps) +</pre> + <p>Each step can affect the build process in the following ways: + + <ul> +<li>If the step's <code>haltOnFailure</code> attribute is True, then a failure +in the step (i.e. if it completes with a result of FAILURE) will cause +the whole build to be terminated immediately: no further steps will be +executed, with the exception of steps with <code>alwaysRun</code> set to +True. <code>haltOnFailure</code> is useful for setup steps upon which the +rest of the build depends: if the CVS checkout or <code>./configure</code> +process fails, there is no point in trying to compile or test the +resulting tree. + + <li>If the step's <code>alwaysRun</code> attribute is True, then it will always +be run, regardless of if previous steps have failed. This is useful +for cleanup steps that should always be run to return the build +directory or build slave into a good state. + + <li>If the <code>flunkOnFailure</code> or <code>flunkOnWarnings</code> flag is set, +then a result of FAILURE or WARNINGS will mark the build as a whole as +FAILED. However, the remaining steps will still be executed. This is +appropriate for things like multiple testing steps: a failure in any +one of them will indicate that the build has failed, however it is +still useful to run them all to completion. + + <li>Similarly, if the <code>warnOnFailure</code> or <code>warnOnWarnings</code> flag +is set, then a result of FAILURE or WARNINGS will mark the build as +having WARNINGS, and the remaining steps will still be executed. This +may be appropriate for certain kinds of optional build or test steps. +For example, a failure experienced while building documentation files +should be made visible with a WARNINGS result but not be serious +enough to warrant marking the whole build with a FAILURE. + + </ul> + + <p>In addition, each Step produces its own results, may create logfiles, +etc. However only the flags described above have any effect on the +build as a whole. + + <p>The pre-defined BuildSteps like <code>CVS</code> and <code>Compile</code> have +reasonably appropriate flags set on them already. For example, without +a source tree there is no point in continuing the build, so the +<code>CVS</code> class has the <code>haltOnFailure</code> flag set to True. Look +in <samp><span class="file">buildbot/steps/*.py</span></samp> to see how the other Steps are +marked. + + <p>Each Step is created with an additional <code>workdir</code> argument that +indicates where its actions should take place. This is specified as a +subdirectory of the slave builder's base directory, with a default +value of <code>build</code>. This is only implemented as a step argument (as +opposed to simply being a part of the base directory) because the +CVS/SVN steps need to perform their checkouts from the parent +directory. + +<ul class="menu"> +<li><a accesskey="1" href="#BuildFactory-Attributes">BuildFactory Attributes</a> +<li><a accesskey="2" href="#Quick-builds">Quick builds</a> +</ul> + +<div class="node"> +<p><hr> +<a name="BuildFactory-Attributes"></a> +Next: <a rel="next" accesskey="n" href="#Quick-builds">Quick builds</a>, +Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>, +Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a> + +</div> + +<h5 class="subsubsection">6.3.2.1 BuildFactory Attributes</h5> + +<p>Some attributes from the BuildFactory are copied into each Build. + + <p><a name="index-treeStableTimer-97"></a> + <dl> +<dt><code>useProgress</code><dd>(defaults to True): if True, the buildmaster keeps track of how long +each step takes, so it can provide estimates of how long future builds +will take. If builds are not expected to take a consistent amount of +time (such as incremental builds in which a random set of files are +recompiled or tested each time), this should be set to False to +inhibit progress-tracking. + + </dl> + +<div class="node"> +<p><hr> +<a name="Quick-builds"></a> +Previous: <a rel="previous" accesskey="p" href="#BuildFactory-Attributes">BuildFactory Attributes</a>, +Up: <a rel="up" accesskey="u" href="#BuildFactory">BuildFactory</a> + +</div> + +<h5 class="subsubsection">6.3.2.2 Quick builds</h5> + +<p><a name="index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-98"></a> +The difference between a “full build” and a “quick build” is that +quick builds are generally done incrementally, starting with the tree +where the previous build was performed. That simply means that the +source-checkout step should be given a <code>mode='update'</code> flag, to +do the source update in-place. + + <p>In addition to that, the <code>useProgress</code> flag should be set to +False. Incremental builds will (or at least the ought to) compile as +few files as necessary, so they will take an unpredictable amount of +time to run. Therefore it would be misleading to claim to predict how +long the build will take. + +<div class="node"> +<p><hr> +<a name="Process-Specific-build-factories"></a> +<a name="Process_002dSpecific-build-factories"></a> +Previous: <a rel="previous" accesskey="p" href="#BuildFactory">BuildFactory</a>, +Up: <a rel="up" accesskey="u" href="#Build-Factories">Build Factories</a> + +</div> + +<h4 class="subsection">6.3.3 Process-Specific build factories</h4> + +<p>Many projects use one of a few popular build frameworks to simplify +the creation and maintenance of Makefiles or other compilation +structures. Buildbot provides several pre-configured BuildFactory +subclasses which let you build these projects with a minimum of fuss. + +<ul class="menu"> +<li><a accesskey="1" href="#GNUAutoconf">GNUAutoconf</a> +<li><a accesskey="2" href="#CPAN">CPAN</a> +<li><a accesskey="3" href="#Python-distutils">Python distutils</a> +<li><a accesskey="4" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a> +</ul> + +<div class="node"> +<p><hr> +<a name="GNUAutoconf"></a> +Next: <a rel="next" accesskey="n" href="#CPAN">CPAN</a>, +Previous: <a rel="previous" accesskey="p" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a>, +Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> + +</div> + +<h5 class="subsubsection">6.3.3.1 GNUAutoconf</h5> + +<p><a name="index-buildbot_002eprocess_002efactory_002eGNUAutoconf-99"></a> +<a href="http://www.gnu.org/software/autoconf/">GNU Autoconf</a> is a +software portability tool, intended to make it possible to write +programs in C (and other languages) which will run on a variety of +UNIX-like systems. Most GNU software is built using autoconf. It is +frequently used in combination with GNU automake. These tools both +encourage a build process which usually looks like this: + +<pre class="example"> % CONFIG_ENV=foo ./configure --with-flags + % make all + % make check + # make install +</pre> + <p>(except of course the Buildbot always skips the <code>make install</code> +part). + + <p>The Buildbot's <code>buildbot.process.factory.GNUAutoconf</code> factory is +designed to build projects which use GNU autoconf and/or automake. The +configuration environment variables, the configure flags, and command +lines used for the compile and test are all configurable, in general +the default values will be suitable. + + <p>Example: + +<pre class="example"> # use the s() convenience function defined earlier + f = factory.GNUAutoconf(source=s(step.SVN, svnurl=URL, mode="copy"), + flags=["--disable-nls"]) +</pre> + <p>Required Arguments: + + <dl> +<dt><code>source</code><dd>This argument must be a step specification tuple that provides a +BuildStep to generate the source tree. +</dl> + + <p>Optional Arguments: + + <dl> +<dt><code>configure</code><dd>The command used to configure the tree. Defaults to +<code>./configure</code>. Accepts either a string or a list of shell argv +elements. + + <br><dt><code>configureEnv</code><dd>The environment used for the initial configuration step. This accepts +a dictionary which will be merged into the buildslave's normal +environment. This is commonly used to provide things like +<code>CFLAGS="-O2 -g"</code> (to turn off debug symbols during the compile). +Defaults to an empty dictionary. + + <br><dt><code>configureFlags</code><dd>A list of flags to be appended to the argument list of the configure +command. This is commonly used to enable or disable specific features +of the autoconf-controlled package, like <code>["--without-x"]</code> to +disable windowing support. Defaults to an empty list. + + <br><dt><code>compile</code><dd>this is a shell command or list of argv values which is used to +actually compile the tree. It defaults to <code>make all</code>. If set to +None, the compile step is skipped. + + <br><dt><code>test</code><dd>this is a shell command or list of argv values which is used to run +the tree's self-tests. It defaults to <code>make check</code>. If set to +None, the test step is skipped. + + </dl> + +<div class="node"> +<p><hr> +<a name="CPAN"></a> +Next: <a rel="next" accesskey="n" href="#Python-distutils">Python distutils</a>, +Previous: <a rel="previous" accesskey="p" href="#GNUAutoconf">GNUAutoconf</a>, +Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> + +</div> + +<h5 class="subsubsection">6.3.3.2 CPAN</h5> + +<p><a name="index-buildbot_002eprocess_002efactory_002eCPAN-100"></a> +Most Perl modules available from the <a href="http://www.cpan.org/">CPAN</a> +archive use the <code>MakeMaker</code> module to provide configuration, +build, and test services. The standard build routine for these modules +looks like: + +<pre class="example"> % perl Makefile.PL + % make + % make test + # make install +</pre> + <p>(except again Buildbot skips the install step) + + <p>Buildbot provides a <code>CPAN</code> factory to compile and test these +projects. + + <p>Arguments: + <dl> +<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. + + <br><dt><code>perl</code><dd>A string which specifies the <code>perl</code> executable to use. Defaults +to just <code>perl</code>. + + </dl> + +<div class="node"> +<p><hr> +<a name="Python-distutils"></a> +Next: <a rel="next" accesskey="n" href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a>, +Previous: <a rel="previous" accesskey="p" href="#CPAN">CPAN</a>, +Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> + +</div> + +<h5 class="subsubsection">6.3.3.3 Python distutils</h5> + +<p><a name="index-buildbot_002eprocess_002efactory_002eDistutils-101"></a> +Most Python modules use the <code>distutils</code> package to provide +configuration and build services. The standard build process looks +like: + +<pre class="example"> % python ./setup.py build + % python ./setup.py install +</pre> + <p>Unfortunately, although Python provides a standard unit-test framework +named <code>unittest</code>, to the best of my knowledge <code>distutils</code> +does not provide a standardized target to run such unit tests. (Please +let me know if I'm wrong, and I will update this factory.) + + <p>The <code>Distutils</code> factory provides support for running the build +part of this process. It accepts the same <code>source=</code> parameter as +the other build factories. + + <p>Arguments: + <dl> +<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. + + <br><dt><code>python</code><dd>A string which specifies the <code>python</code> executable to use. Defaults +to just <code>python</code>. + + <br><dt><code>test</code><dd>Provides a shell command which runs unit tests. This accepts either a +string or a list. The default value is None, which disables the test +step (since there is no common default command to run unit tests in +distutils modules). + + </dl> + +<div class="node"> +<p><hr> +<a name="Python%2fTwisted%2ftrial-projects"></a> +<a name="Python_002fTwisted_002ftrial-projects"></a> +Previous: <a rel="previous" accesskey="p" href="#Python-distutils">Python distutils</a>, +Up: <a rel="up" accesskey="u" href="#Process_002dSpecific-build-factories">Process-Specific build factories</a> + +</div> + +<h5 class="subsubsection">6.3.3.4 Python/Twisted/trial projects</h5> + +<p><a name="index-buildbot_002eprocess_002efactory_002eTrial-102"></a><!-- TODO: document these steps better --> +<a name="index-buildbot_002esteps_002epython_005ftwisted_002eHLint-103"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eTrial-104"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-105"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-106"></a><a name="index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-107"></a> +Twisted provides a unit test tool named <code>trial</code> which provides a +few improvements over Python's built-in <code>unittest</code> module. Many +python projects which use Twisted for their networking or application +services also use trial for their unit tests. These modules are +usually built and tested with something like the following: + +<pre class="example"> % python ./setup.py build + % PYTHONPATH=build/lib.linux-i686-2.3 trial -v PROJECTNAME.test + % python ./setup.py install +</pre> + <p>Unfortunately, the <samp><span class="file">build/lib</span></samp> directory into which the +built/copied .py files are placed is actually architecture-dependent, +and I do not yet know of a simple way to calculate its value. For many +projects it is sufficient to import their libraries “in place” from +the tree's base directory (<code>PYTHONPATH=.</code>). + + <p>In addition, the <var>PROJECTNAME</var> value where the test files are +located is project-dependent: it is usually just the project's +top-level library directory, as common practice suggests the unit test +files are put in the <code>test</code> sub-module. This value cannot be +guessed, the <code>Trial</code> class must be told where to find the test +files. + + <p>The <code>Trial</code> class provides support for building and testing +projects which use distutils and trial. If the test module name is +specified, trial will be invoked. The library path used for testing +can also be set. + + <p>One advantage of trial is that the Buildbot happens to know how to +parse trial output, letting it identify which tests passed and which +ones failed. The Buildbot can then provide fine-grained reports about +how many tests have failed, when individual tests fail when they had +been passing previously, etc. + + <p>Another feature of trial is that you can give it a series of source +.py files, and it will search them for special <code>test-case-name</code> +tags that indicate which test cases provide coverage for that file. +Trial can then run just the appropriate tests. This is useful for +quick builds, where you want to only run the test cases that cover the +changed functionality. + + <p>Arguments: + <dl> +<dt><code>source</code><dd>(required): A step specification tuple, like that used by GNUAutoconf. + + <br><dt><code>buildpython</code><dd>A list (argv array) of strings which specifies the <code>python</code> +executable to use when building the package. Defaults to just +<code>['python']</code>. It may be useful to add flags here, to supress +warnings during compilation of extension modules. This list is +extended with <code>['./setup.py', 'build']</code> and then executed in a +ShellCommand. + + <br><dt><code>testpath</code><dd>Provides a directory to add to <code>PYTHONPATH</code> when running the unit +tests, if tests are being run. Defaults to <code>.</code> to include the +project files in-place. The generated build library is frequently +architecture-dependent, but may simply be <samp><span class="file">build/lib</span></samp> for +pure-python modules. + + <br><dt><code>trialpython</code><dd>Another list of strings used to build the command that actually runs +trial. This is prepended to the contents of the <code>trial</code> argument +below. It may be useful to add <code>-W</code> flags here to supress +warnings that occur while tests are being run. Defaults to an empty +list, meaning <code>trial</code> will be run without an explicit +interpreter, which is generally what you want if you're using +<samp><span class="file">/usr/bin/trial</span></samp> instead of, say, the <samp><span class="file">./bin/trial</span></samp> that +lives in the Twisted source tree. + + <br><dt><code>trial</code><dd>provides the name of the <code>trial</code> command. It is occasionally +useful to use an alternate executable, such as <code>trial2.2</code> which +might run the tests under an older version of Python. Defaults to +<code>trial</code>. + + <br><dt><code>tests</code><dd>Provides a module name or names which contain the unit tests for this +project. Accepts a string, typically <code>PROJECTNAME.test</code>, or a +list of strings. Defaults to None, indicating that no tests should be +run. You must either set this or <code>useTestCaseNames</code> to do anyting +useful with the Trial factory. + + <br><dt><code>useTestCaseNames</code><dd>Tells the Step to provide the names of all changed .py files to trial, +so it can look for test-case-name tags and run just the matching test +cases. Suitable for use in quick builds. Defaults to False. + + <br><dt><code>randomly</code><dd>If <code>True</code>, tells Trial (with the <code>--random=0</code> argument) to +run the test cases in random order, which sometimes catches subtle +inter-test dependency bugs. Defaults to <code>False</code>. + + <br><dt><code>recurse</code><dd>If <code>True</code>, tells Trial (with the <code>--recurse</code> argument) to +look in all subdirectories for additional test cases. It isn't clear +to me how this works, but it may be useful to deal with the +unknown-PROJECTNAME problem described above, and is currently used in +the Twisted buildbot to accomodate the fact that test cases are now +distributed through multiple twisted.SUBPROJECT.test directories. + + </dl> + + <p>Unless one of <code>trialModule</code> or <code>useTestCaseNames</code> +are set, no tests will be run. + + <p>Some quick examples follow. Most of these examples assume that the +target python code (the “code under test”) can be reached directly +from the root of the target tree, rather than being in a <samp><span class="file">lib/</span></samp> +subdirectory. + +<pre class="example"> # Trial(source, tests="toplevel.test") does: + # python ./setup.py build + # PYTHONPATH=. trial -to toplevel.test + + # Trial(source, tests=["toplevel.test", "other.test"]) does: + # python ./setup.py build + # PYTHONPATH=. trial -to toplevel.test other.test + + # Trial(source, useTestCaseNames=True) does: + # python ./setup.py build + # PYTHONPATH=. trial -to --testmodule=foo/bar.py.. (from Changes) + + # Trial(source, buildpython=["python2.3", "-Wall"], tests="foo.tests"): + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. trial -to foo.tests + + # Trial(source, trialpython="python2.3", trial="/usr/bin/trial", + # tests="foo.tests") does: + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. python2.3 /usr/bin/trial -to foo.tests + + # For running trial out of the tree being tested (only useful when the + # tree being built is Twisted itself): + # Trial(source, trialpython=["python2.3", "-Wall"], trial="./bin/trial", + # tests="foo.tests") does: + # python2.3 -Wall ./setup.py build + # PYTHONPATH=. python2.3 -Wall ./bin/trial -to foo.tests +</pre> + <p>If the output directory of <code>./setup.py build</code> is known, you can +pull the python code from the built location instead of the source +directories. This should be able to handle variations in where the +source comes from, as well as accomodating binary extension modules: + +<pre class="example"> # Trial(source,tests="toplevel.test",testpath='build/lib.linux-i686-2.3') + # does: + # python ./setup.py build + # PYTHONPATH=build/lib.linux-i686-2.3 trial -to toplevel.test +</pre> + <div class="node"> +<p><hr> +<a name="Status-Delivery"></a> +Next: <a rel="next" accesskey="n" href="#Command_002dline-tool">Command-line tool</a>, +Previous: <a rel="previous" accesskey="p" href="#Build-Process">Build Process</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">7 Status Delivery</h2> + +<p>More details are available in the docstrings for each class, use a +command like <code>pydoc buildbot.status.html.WebStatus</code> to see them. +Most status delivery objects take a <code>categories=</code> argument, which +can contain a list of “category” names: in this case, it will only +show status for Builders that are in one of the named categories. + + <p>(implementor's note: each of these objects should be a +service.MultiService which will be attached to the BuildMaster object +when the configuration is processed. They should use +<code>self.parent.getStatus()</code> to get access to the top-level IStatus +object, either inside <code>startService</code> or later. They may call +<code>status.subscribe()</code> in <code>startService</code> to receive +notifications of builder events, in which case they must define +<code>builderAdded</code> and related methods. See the docstrings in +<samp><span class="file">buildbot/interfaces.py</span></samp> for full details.) + +<ul class="menu"> +<li><a accesskey="1" href="#WebStatus">WebStatus</a> +<li><a accesskey="2" href="#MailNotifier">MailNotifier</a> +<li><a accesskey="3" href="#IRC-Bot">IRC Bot</a> +<li><a accesskey="4" href="#PBListener">PBListener</a> +<li><a accesskey="5" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a> +</ul> + +<!-- @node Email Delivery, , Status Delivery, Status Delivery --> +<!-- @subsection Email Delivery --> +<!-- DOCUMENT THIS --> +<div class="node"> +<p><hr> +<a name="WebStatus"></a> +Next: <a rel="next" accesskey="n" href="#MailNotifier">MailNotifier</a>, +Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>, +Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> + +</div> + +<h3 class="section">7.1 WebStatus</h3> + +<p><a name="index-WebStatus-108"></a><a name="index-buildbot_002estatus_002eweb_002ebaseweb_002eWebStatus-109"></a> +The <code>buildbot.status.html.WebStatus</code> status target runs a small +web server inside the buildmaster. You can point a browser at this web +server and retrieve information about every build the buildbot knows +about, as well as find out what the buildbot is currently working on. + + <p>The first page you will see is the “Welcome Page”, which contains +links to all the other useful pages. This page is simply served from +the <samp><span class="file">public_html/index.html</span></samp> file in the buildmaster's base +directory, where it is created by the <samp><span class="command">buildbot create-master</span></samp> +command along with the rest of the buildmaster. + + <p>The most complex resource provided by <code>WebStatus</code> is the +“Waterfall Display”, which shows a time-based chart of events. This +somewhat-busy display provides detailed information about all steps of +all recent builds, and provides hyperlinks to look at individual build +logs and source changes. By simply reloading this page on a regular +basis, you will see a complete description of everything the buildbot +is currently working on. + + <p>There are also pages with more specialized information. For example, +there is a page which shows the last 20 builds performed by the +buildbot, one line each. Each line is a link to detailed information +about that build. By adding query arguments to the URL used to reach +this page, you can narrow the display to builds that involved certain +branches, or which ran on certain Builders. These pages are described +in great detail below. + + <p>When the buildmaster is created, a subdirectory named +<samp><span class="file">public_html/</span></samp> is created in its base directory. By default, <code>WebStatus</code> +will serve files from this directory: for example, when a user points +their browser at the buildbot's <code>WebStatus</code> URL, they will see +the contents of the <samp><span class="file">public_html/index.html</span></samp> file. Likewise, +<samp><span class="file">public_html/robots.txt</span></samp>, <samp><span class="file">public_html/buildbot.css</span></samp>, and +<samp><span class="file">public_html/favicon.ico</span></samp> are all useful things to have in there. +The first time a buildmaster is created, the <samp><span class="file">public_html</span></samp> +directory is populated with some sample files, which you will probably +want to customize for your own project. These files are all static: +the buildbot does not modify them in any way as it serves them to HTTP +clients. + +<pre class="example"> from buildbot.status.html import WebStatus + c['status'].append(WebStatus(8080)) +</pre> + <p>Note that the initial robots.txt file has Disallow lines for all of +the dynamically-generated buildbot pages, to discourage web spiders +and search engines from consuming a lot of CPU time as they crawl +through the entire history of your buildbot. If you are running the +buildbot behind a reverse proxy, you'll probably need to put the +robots.txt file somewhere else (at the top level of the parent web +server), and replace the URL prefixes in it with more suitable values. + + <p>If you would like to use an alternative root directory, add the +<code>public_html=..</code> option to the <code>WebStatus</code> creation: + +<pre class="example"> c['status'].append(WebStatus(8080, public_html="/var/www/buildbot")) +</pre> + <p>In addition, if you are familiar with twisted.web <em>Resource +Trees</em>, you can write code to add additional pages at places inside +this web space. Just use <code>webstatus.putChild</code> to place these +resources. + + <p>The following section describes the special URLs and the status views +they provide. + +<ul class="menu"> +<li><a accesskey="1" href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a> +<li><a accesskey="2" href="#Buildbot-Web-Resources">Buildbot Web Resources</a> +<li><a accesskey="3" href="#XMLRPC-server">XMLRPC server</a> +<li><a accesskey="4" href="#HTML-Waterfall">HTML Waterfall</a> +</ul> + +<div class="node"> +<p><hr> +<a name="WebStatus-Configuration-Parameters"></a> +Next: <a rel="next" accesskey="n" href="#Buildbot-Web-Resources">Buildbot Web Resources</a>, +Previous: <a rel="previous" accesskey="p" href="#WebStatus">WebStatus</a>, +Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> + +</div> + +<h4 class="subsection">7.1.1 WebStatus Configuration Parameters</h4> + +<p>The most common way to run a <code>WebStatus</code> is on a regular TCP +port. To do this, just pass in the TCP port number when you create the +<code>WebStatus</code> instance; this is called the <code>http_port</code> argument: + +<pre class="example"> from buildbot.status.html import WebStatus + c['status'].append(WebStatus(8080)) +</pre> + <p>The <code>http_port</code> argument is actually a “strports specification” +for the port that the web server should listen on. This can be a +simple port number, or a string like +<code>tcp:8080:interface=127.0.0.1</code> (to limit connections to the +loopback interface, and therefore to clients running on the same +host)<a rel="footnote" href="#fn-13" name="fnd-13"><sup>13</sup></a>. + + <p>If instead (or in addition) you provide the <code>distrib_port</code> +argument, a twisted.web distributed server will be started either on a +TCP port (if <code>distrib_port</code> is like <code>"tcp:12345"</code>) or more +likely on a UNIX socket (if <code>distrib_port</code> is like +<code>"unix:/path/to/socket"</code>). + + <p>The <code>distrib_port</code> option means that, on a host with a +suitably-configured twisted-web server, you do not need to consume a +separate TCP port for the buildmaster's status web page. When the web +server is constructed with <code>mktap web --user</code>, URLs that point to +<code>http://host/~username/</code> are dispatched to a sub-server that is +listening on a UNIX socket at <code>~username/.twisted-web-pb</code>. On +such a system, it is convenient to create a dedicated <code>buildbot</code> +user, then set <code>distrib_port</code> to +<code>"unix:"+os.path.expanduser("~/.twistd-web-pb")</code>. This +configuration will make the HTML status page available at +<code>http://host/~buildbot/</code> . Suitable URL remapping can make it +appear at <code>http://host/buildbot/</code>, and the right virtual host +setup can even place it at <code>http://buildbot.host/</code> . + + <p>The other <code>WebStatus</code> argument is <code>allowForce</code>. If set to +True, then the web page will provide a “Force Build” button that +allows visitors to manually trigger builds. This is useful for +developers to re-run builds that have failed because of intermittent +problems in the test suite, or because of libraries that were not +installed at the time of the previous build. You may not wish to allow +strangers to cause a build to run: in that case, set this to False to +remove these buttons. The default value is False. + +<div class="node"> +<p><hr> +<a name="Buildbot-Web-Resources"></a> +Next: <a rel="next" accesskey="n" href="#XMLRPC-server">XMLRPC server</a>, +Previous: <a rel="previous" accesskey="p" href="#WebStatus-Configuration-Parameters">WebStatus Configuration Parameters</a>, +Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> + +</div> + +<h4 class="subsection">7.1.2 Buildbot Web Resources</h4> + +<p>Certain URLs are “magic”, and the pages they serve are created by +code in various classes in the <samp><span class="file">buildbot.status.web</span></samp> package +instead of being read from disk. The most common way to access these +pages is for the buildmaster admin to write or modify the +<samp><span class="file">index.html</span></samp> page to contain links to them. Of course other +project web pages can contain links to these buildbot pages as well. + + <p>Many pages can be modified by adding query arguments to the URL. For +example, a page which shows the results of the most recent build +normally does this for all builders at once. But by appending +“?builder=i386” to the end of the URL, the page will show only the +results for the “i386” builder. When used in this way, you can add +multiple “builder=” arguments to see multiple builders. Remembering +that URL query arguments are separated <em>from each other</em> with +ampersands, a URL that ends in “?builder=i386&builder=ppc” would +show builds for just those two Builders. + + <p>The <code>branch=</code> query argument can be used on some pages. This +filters the information displayed by that page down to only the builds +or changes which involved the given branch. Use <code>branch=trunk</code> to +reference the trunk: if you aren't intentionally using branches, +you're probably using trunk. Multiple <code>branch=</code> arguments can be +used to examine multiple branches at once (so appending +<code>?branch=foo&branch=bar</code> to the URL will show builds involving +either branch). No <code>branch=</code> arguments means to show builds and +changes for all branches. + + <p>Some pages may include the Builder name or the build number in the +main part of the URL itself. For example, a page that describes Build +#7 of the “i386” builder would live at <samp><span class="file">/builders/i386/builds/7</span></samp>. + + <p>The table below lists all of the internal pages and the URLs that can +be used to access them. + + <p>NOTE: of the pages described here, <code>/slave_status_timeline</code> and +<code>/last_build</code> have not yet been implemented, and <code>/xmlrpc</code> +has only a few methods so far. Future releases will improve this. + + <dl> +<dt><code>/waterfall</code><dd> +This provides a chronologically-oriented display of the activity of +all builders. It is the same display used by the Waterfall display. + + <p>By adding one or more “builder=” query arguments, the Waterfall is +restricted to only showing information about the given Builders. By +adding one or more “branch=” query arguments, the display is +restricted to showing information about the given branches. In +addition, adding one or more “category=” query arguments to the URL +will limit the display to Builders that were defined with one of the +given categories. + + <p>A 'show_events=true' query argument causes the display to include +non-Build events, like slaves attaching and detaching, as well as +reconfiguration events. 'show_events=false' hides these events. The +default is to show them. + + <p>The <code>last_time=</code>, <code>first_time=</code>, and <code>show_time=</code> +arguments will control what interval of time is displayed. The default +is to show the latest events, but these can be used to look at earlier +periods in history. The <code>num_events=</code> argument also provides a +limit on the size of the displayed page. + + <p>The Waterfall has references to resources many of the other portions +of the URL space: <samp><span class="file">/builders</span></samp> for access to individual builds, +<samp><span class="file">/changes</span></samp> for access to information about source code changes, +etc. + + <br><dt><code>/rss</code><dd> +This provides a rss feed summarizing all failed builds. The same +query-arguments used by 'waterfall' can be added to filter the +feed output. + + <br><dt><code>/atom</code><dd> +This provides an atom feed summarizing all failed builds. The same +query-arguments used by 'waterfall' can be added to filter the feed +output. + + <br><dt><code>/builders/$BUILDERNAME</code><dd> +This describes the given Builder, and provides buttons to force a build. + + <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM</code><dd> +This describes a specific Build. + + <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME</code><dd> +This describes a specific BuildStep. + + <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME</code><dd> +This provides an HTML representation of a specific logfile. + + <br><dt><code>/builders/$BUILDERNAME/builds/$BUILDNUM/steps/$STEPNAME/logs/$LOGNAME/text</code><dd> +This returns the logfile as plain text, without any HTML coloring +markup. It also removes the “headers”, which are the lines that +describe what command was run and what the environment variable +settings were like. This maybe be useful for saving to disk and +feeding to tools like 'grep'. + + <br><dt><code>/changes</code><dd> +This provides a brief description of the ChangeSource in use +(see <a href="#Change-Sources">Change Sources</a>). + + <br><dt><code>/changes/NN</code><dd> +This shows detailed information about the numbered Change: who was the +author, what files were changed, what revision number was represented, +etc. + + <br><dt><code>/buildslaves</code><dd> +This summarizes each BuildSlave, including which Builders are +configured to use it, whether the buildslave is currently connected or +not, and host information retrieved from the buildslave itself. + + <br><dt><code>/one_line_per_build</code><dd> +This page shows one line of text for each build, merging information +from all Builders<a rel="footnote" href="#fn-14" name="fnd-14"><sup>14</sup></a>. Each line specifies +the name of the Builder, the number of the Build, what revision it +used, and a summary of the results. Successful builds are in green, +while failing builds are in red. The date and time of the build are +added to the right-hand edge of the line. The lines are ordered by +build finish timestamp. + + <p>One or more <code>builder=</code> or <code>branch=</code> arguments can be used to +restrict the list. In addition, a <code>numbuilds=</code> argument will +control how many lines are displayed (20 by default). + + <br><dt><code>/one_box_per_builder</code><dd> +This page shows a small table, with one box for each Builder, +containing the results of the most recent Build. It does not show the +individual steps, or the current status. This is a simple summary of +buildbot status: if this page is green, then all tests are passing. + + <p>As with <code>/one_line_per_build</code>, this page will also honor +<code>builder=</code> and <code>branch=</code> arguments. + + <br><dt><code>/about</code><dd> +This page gives a brief summary of the Buildbot itself: software +version, versions of some libraries that the Buildbot depends upon, +etc. It also contains a link to the buildbot.net home page. + + <br><dt><code>/slave_status_timeline</code><dd> +(note: this page has not yet been implemented) + + <p>This provides a chronological display of configuration and operational +events: master startup/shutdown, slave connect/disconnect, and +config-file changes. When a config-file reload is abandoned because of +an error in the config file, the error is displayed on this page. + + <p>This page does not show any builds. + + <br><dt><code>/last_build/$BUILDERNAME/status.png</code><dd> +This returns a PNG image that describes the results of the most recent +build, which can be referenced in an IMG tag by other pages, perhaps +from a completely different site. Use it as you would a webcounter. + + </dl> + + <p>There are also a set of web-status resources that are intended for use +by other programs, rather than humans. + + <dl> +<dt><code>/xmlrpc</code><dd> +This runs an XML-RPC server which can be used to query status +information about various builds. See <a href="#XMLRPC-server">XMLRPC server</a> for more +details. + + </dl> + +<div class="node"> +<p><hr> +<a name="XMLRPC-server"></a> +Next: <a rel="next" accesskey="n" href="#HTML-Waterfall">HTML Waterfall</a>, +Previous: <a rel="previous" accesskey="p" href="#Buildbot-Web-Resources">Buildbot Web Resources</a>, +Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> + +</div> + +<h4 class="subsection">7.1.3 XMLRPC server</h4> + +<p>When using WebStatus, the buildbot runs an XML-RPC server at +<samp><span class="file">/xmlrpc</span></samp> that can be used by other programs to query build +status. The following table lists the methods that can be invoked +using this interface. + + <dl> +<dt><code>getAllBuildsInInterval(start, stop)</code><dd> +Return a list of builds that have completed after the 'start' +timestamp and before the 'stop' timestamp. This looks at all Builders. + + <p>The timestamps are integers, interpreted as standard unix timestamps +(seconds since epoch). + + <p>Each Build is returned as a tuple in the form: <code>(buildername, +buildnumber, build_end, branchname, revision, results, text)</code> + + <p>The buildnumber is an integer. 'build_end' is an integer (seconds +since epoch) specifying when the build finished. + + <p>The branchname is a string, which may be an empty string to indicate +None (i.e. the default branch). The revision is a string whose meaning +is specific to the VC system in use, and comes from the 'got_revision' +build property. The results are expressed as a string, one of +('success', 'warnings', 'failure', 'exception'). The text is a list of +short strings that ought to be joined by spaces and include slightly +more data about the results of the build. + + <br><dt><code>getBuild(builder_name, build_number)</code><dd> +Return information about a specific build. + + <p>This returns a dictionary (aka “struct” in XMLRPC terms) with +complete information about the build. It does not include the contents +of the log files, but it has just about everything else. + + </dl> + +<div class="node"> +<p><hr> +<a name="HTML-Waterfall"></a> +Previous: <a rel="previous" accesskey="p" href="#XMLRPC-server">XMLRPC server</a>, +Up: <a rel="up" accesskey="u" href="#WebStatus">WebStatus</a> + +</div> + +<h4 class="subsection">7.1.4 HTML Waterfall</h4> + +<p><a name="index-Waterfall-110"></a><a name="index-buildbot_002estatus_002ehtml_002eWaterfall-111"></a> +The <code>Waterfall</code> status target, deprecated as of 0.7.6, is a +subset of the regular <code>WebStatus</code> resource (see <a href="#WebStatus">WebStatus</a>). +This section (and the <code>Waterfall</code> class itself) will be removed +from a future release. + +<pre class="example"> from buildbot.status import html + w = html.WebStatus(http_port=8080) + c['status'].append(w) +</pre> + <div class="node"> +<p><hr> +<a name="MailNotifier"></a> +Next: <a rel="next" accesskey="n" href="#IRC-Bot">IRC Bot</a>, +Previous: <a rel="previous" accesskey="p" href="#WebStatus">WebStatus</a>, +Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> + +</div> + +<h3 class="section">7.2 MailNotifier</h3> + +<p><a name="index-email-112"></a><a name="index-mail-113"></a><a name="index-buildbot_002estatus_002email_002eMailNotifier-114"></a> +The buildbot can also send email when builds finish. The most common +use of this is to tell developers when their change has caused the +build to fail. It is also quite common to send a message to a mailing +list (usually named “builds” or similar) about every build. + + <p>The <code>MailNotifier</code> status target is used to accomplish this. You +configure it by specifying who mail should be sent to, under what +circumstances mail should be sent, and how to deliver the mail. It can +be configured to only send out mail for certain builders, and only +send messages when the build fails, or when the builder transitions +from success to failure. It can also be configured to include various +build logs in each message. + + <p>By default, the message will be sent to the Interested Users list +(see <a href="#Doing-Things-With-Users">Doing Things With Users</a>), which includes all developers who +made changes in the build. You can add additional recipients with the +extraRecipients argument. + + <p>Each MailNotifier sends mail to a single set of recipients. To send +different kinds of mail to different recipients, use multiple +MailNotifiers. + + <p>The following simple example will send an email upon the completion of +each build, to just those developers whose Changes were included in +the build. The email contains a description of the Build, its results, +and URLs where more information can be obtained. + +<pre class="example"> from buildbot.status.mail import MailNotifier + mn = MailNotifier(fromaddr="buildbot@example.org", lookup="example.org") + c['status'].append(mn) +</pre> + <p>To get a simple one-message-per-build (say, for a mailing list), use +the following form instead. This form does not send mail to individual +developers (and thus does not need the <code>lookup=</code> argument, +explained below), instead it only ever sends mail to the “extra +recipients” named in the arguments: + +<pre class="example"> mn = MailNotifier(fromaddr="buildbot@example.org", + sendToInterestedUsers=False, + extraRecipients=['listaddr@example.org']) +</pre> + <p>In some cases it is desirable to have different information then what +is provided in a standard MailNotifier message. For this purpose +MailNotifier provides the argument customMesg (a function) which allows +for the creation of messages with unique content. + + <p>For example it can be useful to display the last few lines of a log file +and recent changes when a builder fails: + +<pre class="example"> def message(attrs): + logLines = 10 + text = list() + text.append("STATUS: %s" % attrs['result'].title()) + text.append("") + text.extend([c.asText() for c in attrs['changes']]) + text.append("") + name, url, lines = attrs['logs'][-1] + text.append("Last %d lines of '%s':" % (logLines, name)) + text.extend(["\t%s\n" % line for line in lines[len(lines)-logLines:]]) + text.append("") + text.append("-buildbot") + return ("\n".join(text), 'plain') + + mn = MailNotifier(fromaddr="buildbot@example.org", + sendToInterestedUsers=False, + mode='problem', + extraRecipients=['listaddr@example.org'], + customMesg=message) +</pre> + <p>A customMesg function takes a single dict argument (see below) and returns a +tuple of strings. The first string is the complete text of the message and the +second is the message type ('plain' or 'html'). The 'html' type should be used +when generating an HTML message: + +<pre class="example"> def message(attrs): + logLines = 10 + text = list() + text.append('<h4>Build status %s.</h4>' % (attrs['result'].title())) + if attrs['changes']: + text.append('<h4>Recent Changes:</h4>') + text.extend([c.asHTML() for c in attrs['changes']]) + name, url, lines = attrs['logs'][-1] + text.append('<h4>Last %d lines of "%s":</h4>' % (logLines, name)) + text.append('<p>') + text.append('<br>'.join([line for line in lines[len(lines)-logLines:]])) + text.append('</p>') + text.append('<br><br>') + text.append('Full log at: %s' % url) + text.append('<br><br>') + text.append('<b>-buildbot</b>') + return ('\n'.join(text), 'html') +</pre> + <h3 class="heading">MailNotifier arguments</h3> + + <dl> +<dt><code>fromaddr</code><dd>The email address to be used in the 'From' header. + + <br><dt><code>sendToInterestedUsers</code><dd>(boolean). If True (the default), send mail to all of the Interested +Users. If False, only send mail to the extraRecipients list. + + <br><dt><code>extraRecipients</code><dd>(tuple of strings). A list of email addresses to which messages should +be sent (in addition to the InterestedUsers list, which includes any +developers who made Changes that went into this build). It is a good +idea to create a small mailing list and deliver to that, then let +subscribers come and go as they please. + + <br><dt><code>subject</code><dd>(string). A string to be used as the subject line of the message. +<code>%(builder)s</code> will be replaced with the name of the builder which +provoked the message. + + <br><dt><code>mode</code><dd>(string). Default to 'all'. One of: + <dl> +<dt><code>all</code><dd>Send mail about all builds, bothpassing and failing +<br><dt><code>failing</code><dd>Only send mail about builds which fail +<br><dt><code>problem</code><dd>Only send mail about a build which failed when the previous build has passed. +If your builds usually pass, then this will only send mail when a problem +occurs. +</dl> + + <br><dt><code>builders</code><dd>(list of strings). A list of builder names for which mail should be +sent. Defaults to None (send mail for all builds). Use either builders +or categories, but not both. + + <br><dt><code>categories</code><dd>(list of strings). A list of category names to serve status +information for. Defaults to None (all categories). Use either +builders or categories, but not both. + + <br><dt><code>addLogs</code><dd>(boolean). If True, include all build logs as attachments to the +messages. These can be quite large. This can also be set to a list of +log names, to send a subset of the logs. Defaults to False. + + <br><dt><code>relayhost</code><dd>(string). The host to which the outbound SMTP connection should be +made. Defaults to 'localhost' + + <br><dt><code>lookup</code><dd>(implementor of <code>IEmailLookup</code>). Object which provides +IEmailLookup, which is responsible for mapping User names (which come +from the VC system) into valid email addresses. If not provided, the +notifier will only be able to send mail to the addresses in the +extraRecipients list. Most of the time you can use a simple Domain +instance. As a shortcut, you can pass as string: this will be treated +as if you had provided Domain(str). For example, +lookup='twistedmatrix.com' will allow mail to be sent to all +developers whose SVN usernames match their twistedmatrix.com account +names. See buildbot/status/mail.py for more details. + + <br><dt><code>customMesg</code><dd>This is a optional function that can be used to generate a custom mail +message. The customMesg function takes a single dict and must return a +tuple containing the message text and type ('html' or 'plain'). Below is a list +of availale keys in the dict passed to customMesg: + + <dl> +<dt><code>builderName</code><dd>(str) Name of the builder that generated this event. +<br><dt><code>projectName</code><dd>(str) Name of the project. +<br><dt><code>mode</code><dd>(str) Mode set in MailNotifier. (failing, passing, problem). +<br><dt><code>result</code><dd>(str) Builder result as a string. 'success', 'warnings', 'failure', 'skipped', or 'exception' +<br><dt><code>buildURL</code><dd>(str) URL to build page. +<br><dt><code>buildbotURL</code><dd>(str) URL to buildbot main page. +<br><dt><code>buildText</code><dd>(str) Build text from build.getText(). +<br><dt><code>slavename</code><dd>(str) Slavename. +<br><dt><code>reason</code><dd>(str) Build reason from build.getReason(). +<br><dt><code>responsibleUsers</code><dd>(List of str) List of responsible users. +<br><dt><code>branch</code><dd>(str) Name of branch used. If no SourceStamp exists branch +is an empty string. +<br><dt><code>revision</code><dd>(str) Name of revision used. If no SourceStamp exists revision +is an empty string. +<br><dt><code>patch</code><dd>(str) Name of patch used. If no SourceStamp exists patch +is an empty string. +<br><dt><code>changes</code><dd>(list of objs) List of change objects from SourceStamp. A change +object has the following useful information: + <dl> +<dt><code>who</code><dd>(str) who made this change +<br><dt><code>revision</code><dd>(str) what VC revision is this change +<br><dt><code>branch</code><dd>(str) on what branch did this change occur +<br><dt><code>when</code><dd>(str) when did this change occur +<br><dt><code>files</code><dd>(list of str) what files were affected in this change +<br><dt><code>comments</code><dd>(str) comments reguarding the change. +</dl> + The functions asText and asHTML return a list of strings with +the above information formatted. +<br><dt><code>logs</code><dd>(List of Tuples) List of tuples where each tuple contains the log name, log url, +and log contents as a list of strings. +</dl> + </dl> + +<div class="node"> +<p><hr> +<a name="IRC-Bot"></a> +Next: <a rel="next" accesskey="n" href="#PBListener">PBListener</a>, +Previous: <a rel="previous" accesskey="p" href="#MailNotifier">MailNotifier</a>, +Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> + +</div> + +<h3 class="section">7.3 IRC Bot</h3> + +<p><a name="index-IRC-115"></a><a name="index-buildbot_002estatus_002ewords_002eIRC-116"></a> + + <p>The <code>buildbot.status.words.IRC</code> status target creates an IRC bot +which will attach to certain channels and be available for status +queries. It can also be asked to announce builds as they occur, or be +told to shut up. + +<pre class="example"> from buildbot.status import words + irc = words.IRC("irc.example.org", "botnickname", + channels=["channel1", "channel2"], + password="mysecretpassword", + notify_events={ + 'exception': 1, + 'successToFailure': 1, + 'failureToSuccess': 1, + }) + c['status'].append(irc) +</pre> + <p>Take a look at the docstring for <code>words.IRC</code> for more details on +configuring this service. The <code>password</code> argument, if provided, +will be sent to Nickserv to claim the nickname: some IRC servers will +not allow clients to send private messages until they have logged in +with a password. + + <p>To use the service, you address messages at the buildbot, either +normally (<code>botnickname: status</code>) or with private messages +(<code>/msg botnickname status</code>). The buildbot will respond in kind. + + <p>Some of the commands currently available: + + <dl> +<dt><code>list builders</code><dd>Emit a list of all configured builders +<br><dt><code>status BUILDER</code><dd>Announce the status of a specific Builder: what it is doing right now. +<br><dt><code>status all</code><dd>Announce the status of all Builders +<br><dt><code>watch BUILDER</code><dd>If the given Builder is currently running, wait until the Build is +finished and then announce the results. +<br><dt><code>last BUILDER</code><dd>Return the results of the last build to run on the given Builder. +<br><dt><code>join CHANNEL</code><dd>Join the given IRC channel +<br><dt><code>leave CHANNEL</code><dd>Leave the given IRC channel +<br><dt><code>notify on|off|list EVENT</code><dd>Report events relating to builds. If the command is issued as a +private message, then the report will be sent back as a private +message to the user who issued the command. Otherwise, the report +will be sent to the channel. Available events to be notified are: + + <dl> +<dt><code>started</code><dd>A build has started +<br><dt><code>finished</code><dd>A build has finished +<br><dt><code>success</code><dd>A build finished successfully +<br><dt><code>failed</code><dd>A build failed +<br><dt><code>exception</code><dd>A build generated and exception +<br><dt><code>xToY</code><dd>The previous build was x, but this one is Y, where x and Y are each +one of success, warnings, failure, exception (except Y is +capitalized). For example: successToFailure will notify if the +previous build was successful, but this one failed +</dl> + + <br><dt><code>help COMMAND</code><dd>Describe a command. Use <code>help commands</code> to get a list of known +commands. +<br><dt><code>source</code><dd>Announce the URL of the Buildbot's home page. +<br><dt><code>version</code><dd>Announce the version of this Buildbot. +</dl> + + <p>Additionally, the config file may specify default notification options +as shown in the example earlier. + + <p>If the <code>allowForce=True</code> option was used, some addtional commands +will be available: + + <dl> +<dt><code>force build BUILDER REASON</code><dd>Tell the given Builder to start a build of the latest code. The user +requesting the build and REASON are recorded in the Build status. The +buildbot will announce the build's status when it finishes. + + <br><dt><code>stop build BUILDER REASON</code><dd>Terminate any running build in the given Builder. REASON will be added +to the build status to explain why it was stopped. You might use this +if you committed a bug, corrected it right away, and don't want to +wait for the first build (which is destined to fail) to complete +before starting the second (hopefully fixed) build. +</dl> + +<div class="node"> +<p><hr> +<a name="PBListener"></a> +Next: <a rel="next" accesskey="n" href="#Writing-New-Status-Plugins">Writing New Status Plugins</a>, +Previous: <a rel="previous" accesskey="p" href="#IRC-Bot">IRC Bot</a>, +Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> + +</div> + +<h3 class="section">7.4 PBListener</h3> + +<p><a name="index-PBListener-117"></a><a name="index-buildbot_002estatus_002eclient_002ePBListener-118"></a> + +<pre class="example"> import buildbot.status.client + pbl = buildbot.status.client.PBListener(port=int, user=str, + passwd=str) + c['status'].append(pbl) +</pre> + <p>This sets up a PB listener on the given TCP port, to which a PB-based +status client can connect and retrieve status information. +<code>buildbot statusgui</code> (see <a href="#statusgui">statusgui</a>) is an example of such a +status client. The <code>port</code> argument can also be a strports +specification string. + +<div class="node"> +<p><hr> +<a name="Writing-New-Status-Plugins"></a> +Previous: <a rel="previous" accesskey="p" href="#PBListener">PBListener</a>, +Up: <a rel="up" accesskey="u" href="#Status-Delivery">Status Delivery</a> + +</div> + +<h3 class="section">7.5 Writing New Status Plugins</h3> + +<p>TODO: this needs a lot more examples + + <p>Each status plugin is an object which provides the +<code>twisted.application.service.IService</code> interface, which creates a +tree of Services with the buildmaster at the top [not strictly true]. +The status plugins are all children of an object which implements +<code>buildbot.interfaces.IStatus</code>, the main status object. From this +object, the plugin can retrieve anything it wants about current and +past builds. It can also subscribe to hear about new and upcoming +builds. + + <p>Status plugins which only react to human queries (like the Waterfall +display) never need to subscribe to anything: they are idle until +someone asks a question, then wake up and extract the information they +need to answer it, then they go back to sleep. Plugins which need to +act spontaneously when builds complete (like the MailNotifier plugin) +need to subscribe to hear about new builds. + + <p>If the status plugin needs to run network services (like the HTTP +server used by the Waterfall plugin), they can be attached as Service +children of the plugin itself, using the <code>IServiceCollection</code> +interface. + +<div class="node"> +<p><hr> +<a name="Command-line-tool"></a> +<a name="Command_002dline-tool"></a> +Next: <a rel="next" accesskey="n" href="#Resources">Resources</a>, +Previous: <a rel="previous" accesskey="p" href="#Status-Delivery">Status Delivery</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">8 Command-line tool</h2> + +<p>The <samp><span class="command">buildbot</span></samp> command-line tool can be used to start or stop a +buildmaster or buildbot, and to interact with a running buildmaster. +Some of its subcommands are intended for buildmaster admins, while +some are for developers who are editing the code that the buildbot is +monitoring. + +<ul class="menu"> +<li><a accesskey="1" href="#Administrator-Tools">Administrator Tools</a> +<li><a accesskey="2" href="#Developer-Tools">Developer Tools</a> +<li><a accesskey="3" href="#Other-Tools">Other Tools</a> +<li><a accesskey="4" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a> +</ul> + +<div class="node"> +<p><hr> +<a name="Administrator-Tools"></a> +Next: <a rel="next" accesskey="n" href="#Developer-Tools">Developer Tools</a>, +Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>, +Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> + +</div> + +<h3 class="section">8.1 Administrator Tools</h3> + +<p>The following <samp><span class="command">buildbot</span></samp> sub-commands are intended for +buildmaster administrators: + +<h3 class="heading">create-master</h3> + +<p>This creates a new directory and populates it with files that allow it +to be used as a buildmaster's base directory. + +<pre class="example"> buildbot create-master BASEDIR +</pre> + <h3 class="heading">create-slave</h3> + +<p>This creates a new directory and populates it with files that let it +be used as a buildslave's base directory. You must provide several +arguments, which are used to create the initial <samp><span class="file">buildbot.tac</span></samp> +file. + +<pre class="example"> buildbot create-slave <var>BASEDIR</var> <var>MASTERHOST</var>:<var>PORT</var> <var>SLAVENAME</var> <var>PASSWORD</var> +</pre> + <h3 class="heading">start</h3> + +<p>This starts a buildmaster or buildslave which was already created in +the given base directory. The daemon is launched in the background, +with events logged to a file named <samp><span class="file">twistd.log</span></samp>. + +<pre class="example"> buildbot start BASEDIR +</pre> + <h3 class="heading">stop</h3> + +<p>This terminates the daemon (either buildmaster or buildslave) running +in the given directory. + +<pre class="example"> buildbot stop BASEDIR +</pre> + <h3 class="heading">sighup</h3> + +<p>This sends a SIGHUP to the buildmaster running in the given directory, +which causes it to re-read its <samp><span class="file">master.cfg</span></samp> file. + +<pre class="example"> buildbot sighup BASEDIR +</pre> + <div class="node"> +<p><hr> +<a name="Developer-Tools"></a> +Next: <a rel="next" accesskey="n" href="#Other-Tools">Other Tools</a>, +Previous: <a rel="previous" accesskey="p" href="#Administrator-Tools">Administrator Tools</a>, +Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> + +</div> + +<h3 class="section">8.2 Developer Tools</h3> + +<p>These tools are provided for use by the developers who are working on +the code that the buildbot is monitoring. + +<ul class="menu"> +<li><a accesskey="1" href="#statuslog">statuslog</a> +<li><a accesskey="2" href="#statusgui">statusgui</a> +<li><a accesskey="3" href="#try">try</a> +</ul> + +<div class="node"> +<p><hr> +<a name="statuslog"></a> +Next: <a rel="next" accesskey="n" href="#statusgui">statusgui</a>, +Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>, +Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> + +</div> + +<h4 class="subsection">8.2.1 statuslog</h4> + +<pre class="example"> buildbot statuslog --master <var>MASTERHOST</var>:<var>PORT</var> +</pre> + <p>This command starts a simple text-based status client, one which just +prints out a new line each time an event occurs on the buildmaster. + + <p>The <samp><span class="option">--master</span></samp> option provides the location of the +<code>buildbot.status.client.PBListener</code> status port, used to deliver +build information to realtime status clients. The option is always in +the form of a string, with hostname and port number separated by a +colon (<code>HOSTNAME:PORTNUM</code>). Note that this port is <em>not</em> the +same as the slaveport (although a future version may allow the same +port number to be used for both purposes). If you get an error message +to the effect of “Failure: twisted.cred.error.UnauthorizedLogin:”, +this may indicate that you are connecting to the slaveport rather than +a <code>PBListener</code> port. + + <p>The <samp><span class="option">--master</span></samp> option can also be provided by the +<code>masterstatus</code> name in <samp><span class="file">.buildbot/options</span></samp> (see <a href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>). + +<div class="node"> +<p><hr> +<a name="statusgui"></a> +Next: <a rel="next" accesskey="n" href="#try">try</a>, +Previous: <a rel="previous" accesskey="p" href="#statuslog">statuslog</a>, +Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> + +</div> + +<h4 class="subsection">8.2.2 statusgui</h4> + +<p><a name="index-statusgui-119"></a> +If you have set up a PBListener (see <a href="#PBListener">PBListener</a>), you will be able +to monitor your Buildbot using a simple Gtk+ application invoked with +the <code>buildbot statusgui</code> command: + +<pre class="example"> buildbot statusgui --master <var>MASTERHOST</var>:<var>PORT</var> +</pre> + <p>This command starts a simple Gtk+-based status client, which contains +a few boxes for each Builder that change color as events occur. It +uses the same <samp><span class="option">--master</span></samp> argument as the <samp><span class="command">buildbot +statuslog</span></samp> command (see <a href="#statuslog">statuslog</a>). + +<div class="node"> +<p><hr> +<a name="try"></a> +Previous: <a rel="previous" accesskey="p" href="#statusgui">statusgui</a>, +Up: <a rel="up" accesskey="u" href="#Developer-Tools">Developer Tools</a> + +</div> + +<h4 class="subsection">8.2.3 try</h4> + +<p>This lets a developer to ask the question “What would happen if I +committed this patch right now?”. It runs the unit test suite (across +multiple build platforms) on the developer's current code, allowing +them to make sure they will not break the tree when they finally +commit their changes. + + <p>The <samp><span class="command">buildbot try</span></samp> command is meant to be run from within a +developer's local tree, and starts by figuring out the base revision +of that tree (what revision was current the last time the tree was +updated), and a patch that can be applied to that revision of the tree +to make it match the developer's copy. This (revision, patch) pair is +then sent to the buildmaster, which runs a build with that +SourceStamp. If you want, the tool will emit status messages as the +builds run, and will not terminate until the first failure has been +detected (or the last success). + + <p>There is an alternate form which accepts a pre-made patch file +(typically the output of a command like 'svn diff'). This “–diff” +form does not require a local tree to run from. See See <a href="#try-_002d_002ddiff">try –diff</a>. + + <p>For this command to work, several pieces must be in place: + +<h3 class="heading">TryScheduler</h3> + +<p><a name="index-buildbot_002escheduler_002eTry_005fJobdir-120"></a><a name="index-buildbot_002escheduler_002eTry_005fUserpass-121"></a> +The buildmaster must have a <code>scheduler.Try</code> instance in +the config file's <code>c['schedulers']</code> list. This lets the +administrator control who may initiate these “trial” builds, which +branches are eligible for trial builds, and which Builders should be +used for them. + + <p>The <code>TryScheduler</code> has various means to accept build requests: +all of them enforce more security than the usual buildmaster ports do. +Any source code being built can be used to compromise the buildslave +accounts, but in general that code must be checked out from the VC +repository first, so only people with commit privileges can get +control of the buildslaves. The usual force-build control channels can +waste buildslave time but do not allow arbitrary commands to be +executed by people who don't have those commit privileges. However, +the source code patch that is provided with the trial build does not +have to go through the VC system first, so it is important to make +sure these builds cannot be abused by a non-committer to acquire as +much control over the buildslaves as a committer has. Ideally, only +developers who have commit access to the VC repository would be able +to start trial builds, but unfortunately the buildmaster does not, in +general, have access to VC system's user list. + + <p>As a result, the <code>TryScheduler</code> requires a bit more +configuration. There are currently two ways to set this up: + + <dl> +<dt><strong>jobdir (ssh)</strong><dd> +This approach creates a command queue directory, called the +“jobdir”, in the buildmaster's working directory. The buildmaster +admin sets the ownership and permissions of this directory to only +grant write access to the desired set of developers, all of whom must +have accounts on the machine. The <code>buildbot try</code> command creates +a special file containing the source stamp information and drops it in +the jobdir, just like a standard maildir. When the buildmaster notices +the new file, it unpacks the information inside and starts the builds. + + <p>The config file entries used by 'buildbot try' either specify a local +queuedir (for which write and mv are used) or a remote one (using scp +and ssh). + + <p>The advantage of this scheme is that it is quite secure, the +disadvantage is that it requires fiddling outside the buildmaster +config (to set the permissions on the jobdir correctly). If the +buildmaster machine happens to also house the VC repository, then it +can be fairly easy to keep the VC userlist in sync with the +trial-build userlist. If they are on different machines, this will be +much more of a hassle. It may also involve granting developer accounts +on a machine that would not otherwise require them. + + <p>To implement this, the buildslave invokes 'ssh -l username host +buildbot tryserver ARGS', passing the patch contents over stdin. The +arguments must include the inlet directory and the revision +information. + + <br><dt><strong>user+password (PB)</strong><dd> +In this approach, each developer gets a username/password pair, which +are all listed in the buildmaster's configuration file. When the +developer runs <code>buildbot try</code>, their machine connects to the +buildmaster via PB and authenticates themselves using that username +and password, then sends a PB command to start the trial build. + + <p>The advantage of this scheme is that the entire configuration is +performed inside the buildmaster's config file. The disadvantages are +that it is less secure (while the “cred” authentication system does +not expose the password in plaintext over the wire, it does not offer +most of the other security properties that SSH does). In addition, the +buildmaster admin is responsible for maintaining the username/password +list, adding and deleting entries as developers come and go. + + </dl> + + <p>For example, to set up the “jobdir” style of trial build, using a +command queue directory of <samp><span class="file">MASTERDIR/jobdir</span></samp> (and assuming that +all your project developers were members of the <code>developers</code> unix +group), you would first create that directory (with <samp><span class="command">mkdir +MASTERDIR/jobdir MASTERDIR/jobdir/new MASTERDIR/jobdir/cur +MASTERDIR/jobdir/tmp; chgrp developers MASTERDIR/jobdir +MASTERDIR/jobdir/*; chmod g+rwx,o-rwx MASTERDIR/jobdir +MASTERDIR/jobdir/*</span></samp>), and then use the following scheduler in the +buildmaster's config file: + +<pre class="example"> from buildbot.scheduler import Try_Jobdir + s = Try_Jobdir("try1", ["full-linux", "full-netbsd", "full-OSX"], + jobdir="jobdir") + c['schedulers'] = [s] +</pre> + <p>Note that you must create the jobdir before telling the buildmaster to +use this configuration, otherwise you will get an error. Also remember +that the buildmaster must be able to read and write to the jobdir as +well. Be sure to watch the <samp><span class="file">twistd.log</span></samp> file (see <a href="#Logfiles">Logfiles</a>) +as you start using the jobdir, to make sure the buildmaster is happy +with it. + + <p>To use the username/password form of authentication, create a +<code>Try_Userpass</code> instance instead. It takes the same +<code>builderNames</code> argument as the <code>Try_Jobdir</code> form, but +accepts an addtional <code>port</code> argument (to specify the TCP port to +listen on) and a <code>userpass</code> list of username/password pairs to +accept. Remember to use good passwords for this: the security of the +buildslave accounts depends upon it: + +<pre class="example"> from buildbot.scheduler import Try_Userpass + s = Try_Userpass("try2", ["full-linux", "full-netbsd", "full-OSX"], + port=8031, userpass=[("alice","pw1"), ("bob", "pw2")] ) + c['schedulers'] = [s] +</pre> + <p>Like most places in the buildbot, the <code>port</code> argument takes a +strports specification. See <code>twisted.application.strports</code> for +details. + +<h3 class="heading">locating the master</h3> + +<p>The <samp><span class="command">try</span></samp> command needs to be told how to connect to the +<code>TryScheduler</code>, and must know which of the authentication +approaches described above is in use by the buildmaster. You specify +the approach by using <samp><span class="option">--connect=ssh</span></samp> or <samp><span class="option">--connect=pb</span></samp> +(or <code>try_connect = 'ssh'</code> or <code>try_connect = 'pb'</code> in +<samp><span class="file">.buildbot/options</span></samp>). + + <p>For the PB approach, the command must be given a <samp><span class="option">--master</span></samp> +argument (in the form HOST:PORT) that points to TCP port that you +picked in the <code>Try_Userpass</code> scheduler. It also takes a +<samp><span class="option">--username</span></samp> and <samp><span class="option">--passwd</span></samp> pair of arguments that match +one of the entries in the buildmaster's <code>userpass</code> list. These +arguments can also be provided as <code>try_master</code>, +<code>try_username</code>, and <code>try_password</code> entries in the +<samp><span class="file">.buildbot/options</span></samp> file. + + <p>For the SSH approach, the command must be given <samp><span class="option">--tryhost</span></samp>, +<samp><span class="option">--username</span></samp>, and optionally <samp><span class="option">--password</span></samp> (TODO: +really?) to get to the buildmaster host. It must also be given +<samp><span class="option">--trydir</span></samp>, which points to the inlet directory configured +above. The trydir can be relative to the user's home directory, but +most of the time you will use an explicit path like +<samp><span class="file">~buildbot/project/trydir</span></samp>. These arguments can be provided in +<samp><span class="file">.buildbot/options</span></samp> as <code>try_host</code>, <code>try_username</code>, +<code>try_password</code>, and <code>try_dir</code>. + + <p>In addition, the SSH approach needs to connect to a PBListener status +port, so it can retrieve and report the results of the build (the PB +approach uses the existing connection to retrieve status information, +so this step is not necessary). This requires a <samp><span class="option">--master</span></samp> +argument, or a <code>masterstatus</code> entry in <samp><span class="file">.buildbot/options</span></samp>, +in the form of a HOSTNAME:PORT string. + +<h3 class="heading">choosing the Builders</h3> + +<p>A trial build is performed on multiple Builders at the same time, and +the developer gets to choose which Builders are used (limited to a set +selected by the buildmaster admin with the TryScheduler's +<code>builderNames=</code> argument). The set you choose will depend upon +what your goals are: if you are concerned about cross-platform +compatibility, you should use multiple Builders, one from each +platform of interest. You might use just one builder if that platform +has libraries or other facilities that allow better test coverage than +what you can accomplish on your own machine, or faster test runs. + + <p>The set of Builders to use can be specified with multiple +<samp><span class="option">--builder</span></samp> arguments on the command line. It can also be +specified with a single <code>try_builders</code> option in +<samp><span class="file">.buildbot/options</span></samp> that uses a list of strings to specify all +the Builder names: + +<pre class="example"> try_builders = ["full-OSX", "full-win32", "full-linux"] +</pre> + <h3 class="heading">specifying the VC system</h3> + +<p>The <samp><span class="command">try</span></samp> command also needs to know how to take the +developer's current tree and extract the (revision, patch) +source-stamp pair. Each VC system uses a different process, so you +start by telling the <samp><span class="command">try</span></samp> command which VC system you are +using, with an argument like <samp><span class="option">--vc=cvs</span></samp> or <samp><span class="option">--vc=tla</span></samp>. +This can also be provided as <code>try_vc</code> in +<samp><span class="file">.buildbot/options</span></samp>. + + <p>The following names are recognized: <code>cvs</code> <code>svn</code> <code>baz</code> +<code>tla</code> <code>hg</code> <code>darcs</code> + +<h3 class="heading">finding the top of the tree</h3> + +<p>Some VC systems (notably CVS and SVN) track each directory +more-or-less independently, which means the <samp><span class="command">try</span></samp> command +needs to move up to the top of the project tree before it will be able +to construct a proper full-tree patch. To accomplish this, the +<samp><span class="command">try</span></samp> command will crawl up through the parent directories +until it finds a marker file. The default name for this marker file is +<samp><span class="file">.buildbot-top</span></samp>, so when you are using CVS or SVN you should +<code>touch .buildbot-top</code> from the top of your tree before running +<samp><span class="command">buildbot try</span></samp>. Alternatively, you can use a filename like +<samp><span class="file">ChangeLog</span></samp> or <samp><span class="file">README</span></samp>, since many projects put one of +these files in their top-most directory (and nowhere else). To set +this filename, use <samp><span class="option">--try-topfile=ChangeLog</span></samp>, or set it in the +options file with <code>try_topfile = 'ChangeLog'</code>. + + <p>You can also manually set the top of the tree with +<samp><span class="option">--try-topdir=~/trees/mytree</span></samp>, or <code>try_topdir = +'~/trees/mytree'</code>. If you use <code>try_topdir</code>, in a +<samp><span class="file">.buildbot/options</span></samp> file, you will need a separate options file +for each tree you use, so it may be more convenient to use the +<code>try_topfile</code> approach instead. + + <p>Other VC systems which work on full projects instead of individual +directories (tla, baz, darcs, monotone, mercurial, git) do not require +<samp><span class="command">try</span></samp> to know the top directory, so the <samp><span class="option">--try-topfile</span></samp> +and <samp><span class="option">--try-topdir</span></samp> arguments will be ignored. +<!-- is this true? I think I currently require topdirs all the time. --> + + <p>If the <samp><span class="command">try</span></samp> command cannot find the top directory, it will +abort with an error message. + +<h3 class="heading">determining the branch name</h3> + +<p>Some VC systems record the branch information in a way that “try” +can locate it, in particular Arch (both <samp><span class="command">tla</span></samp> and +<samp><span class="command">baz</span></samp>). For the others, if you are using something other than +the default branch, you will have to tell the buildbot which branch +your tree is using. You can do this with either the <samp><span class="option">--branch</span></samp> +argument, or a <samp><span class="option">try_branch</span></samp> entry in the +<samp><span class="file">.buildbot/options</span></samp> file. + +<h3 class="heading">determining the revision and patch</h3> + +<p>Each VC system has a separate approach for determining the tree's base +revision and computing a patch. + + <dl> +<dt><code>CVS</code><dd> +<samp><span class="command">try</span></samp> pretends that the tree is up to date. It converts the +current time into a <code>-D</code> time specification, uses it as the base +revision, and computes the diff between the upstream tree as of that +point in time versus the current contents. This works, more or less, +but requires that the local clock be in reasonably good sync with the +repository. + + <br><dt><code>SVN</code><dd><samp><span class="command">try</span></samp> does a <code>svn status -u</code> to find the latest +repository revision number (emitted on the last line in the “Status +against revision: NN” message). It then performs an <code>svn diff +-rNN</code> to find out how your tree differs from the repository version, +and sends the resulting patch to the buildmaster. If your tree is not +up to date, this will result in the “try” tree being created with +the latest revision, then <em>backwards</em> patches applied to bring it +“back” to the version you actually checked out (plus your actual +code changes), but this will still result in the correct tree being +used for the build. + + <br><dt><code>baz</code><dd><samp><span class="command">try</span></samp> does a <code>baz tree-id</code> to determine the +fully-qualified version and patch identifier for the tree +(ARCHIVE/VERSION–patch-NN), and uses the VERSION–patch-NN component +as the base revision. It then does a <code>baz diff</code> to obtain the +patch. + + <br><dt><code>tla</code><dd><samp><span class="command">try</span></samp> does a <code>tla tree-version</code> to get the +fully-qualified version identifier (ARCHIVE/VERSION), then takes the +first line of <code>tla logs --reverse</code> to figure out the base +revision. Then it does <code>tla changes --diffs</code> to obtain the patch. + + <br><dt><code>Darcs</code><dd><code>darcs changes --context</code> emits a text file that contains a list +of all patches back to and including the last tag was made. This text +file (plus the location of a repository that contains all these +patches) is sufficient to re-create the tree. Therefore the contents +of this “context” file <em>are</em> the revision stamp for a +Darcs-controlled source tree. + + <p>So <samp><span class="command">try</span></samp> does a <code>darcs changes --context</code> to determine +what your tree's base revision is, and then does a <code>darcs diff +-u</code> to compute the patch relative to that revision. + + <br><dt><code>Mercurial</code><dd><code>hg identify</code> emits a short revision ID (basically a truncated +SHA1 hash of the current revision's contents), which is used as the +base revision. <code>hg diff</code> then provides the patch relative to that +revision. For <samp><span class="command">try</span></samp> to work, your working directory must only +have patches that are available from the same remotely-available +repository that the build process' <code>step.Mercurial</code> will use. + + <br><dt><code>Git</code><dd><code>git branch -v</code> lists all the branches available in the local +repository along with the revision ID it points to and a short summary +of the last commit. The line containing the currently checked out +branch begins with '* ' (star and space) while all the others start +with ' ' (two spaces). <samp><span class="command">try</span></samp> scans for this line and extracts +the branch name and revision from it. Then it generates a diff against +the base revision. +<!-- TODO: I'm not sure if this actually works the way it's intended --> +<!-- since the extracted base revision might not actually exist in the --> +<!-- upstream repository. Perhaps we need to add a -remote option to --> +<!-- specify the remote tracking branch to generate a diff against. --> + + <!-- TODO: monotone --> + </dl> + +<h3 class="heading">waiting for results</h3> + +<p>If you provide the <samp><span class="option">--wait</span></samp> option (or <code>try_wait = True</code> +in <samp><span class="file">.buildbot/options</span></samp>), the <samp><span class="command">buildbot try</span></samp> command will +wait until your changes have either been proven good or bad before +exiting. Unless you use the <samp><span class="option">--quiet</span></samp> option (or +<code>try_quiet=True</code>), it will emit a progress message every 60 +seconds until the builds have completed. + +<ul class="menu"> +<li><a accesskey="1" href="#try-_002d_002ddiff">try --diff</a> +</ul> + +<div class="node"> +<p><hr> +<a name="try---diff"></a> +<a name="try-_002d_002ddiff"></a> +Previous: <a rel="previous" accesskey="p" href="#try">try</a>, +Up: <a rel="up" accesskey="u" href="#try">try</a> + +</div> + +<h5 class="subsubsection">8.2.3.1 try –diff</h5> + +<p>Sometimes you might have a patch from someone else that you want to +submit to the buildbot. For example, a user may have created a patch +to fix some specific bug and sent it to you by email. You've inspected +the patch and suspect that it might do the job (and have at least +confirmed that it doesn't do anything evil). Now you want to test it +out. + + <p>One approach would be to check out a new local tree, apply the patch, +run your local tests, then use “buildbot try” to run the tests on +other platforms. An alternate approach is to use the <samp><span class="command">buildbot +try --diff</span></samp> form to have the buildbot test the patch without using a +local tree. + + <p>This form takes a <samp><span class="option">--diff</span></samp> argument which points to a file that +contains the patch you want to apply. By default this patch will be +applied to the TRUNK revision, but if you give the optional +<samp><span class="option">--baserev</span></samp> argument, a tree of the given revision will be used +as a starting point instead of TRUNK. + + <p>You can also use <samp><span class="command">buildbot try --diff=-</span></samp> to read the patch +from stdin. + + <p>Each patch has a “patchlevel” associated with it. This indicates the +number of slashes (and preceding pathnames) that should be stripped +before applying the diff. This exactly corresponds to the <samp><span class="option">-p</span></samp> +or <samp><span class="option">--strip</span></samp> argument to the <samp><span class="command">patch</span></samp> utility. By +default <samp><span class="command">buildbot try --diff</span></samp> uses a patchlevel of 0, but you +can override this with the <samp><span class="option">-p</span></samp> argument. + + <p>When you use <samp><span class="option">--diff</span></samp>, you do not need to use any of the other +options that relate to a local tree, specifically <samp><span class="option">--vc</span></samp>, +<samp><span class="option">--try-topfile</span></samp>, or <samp><span class="option">--try-topdir</span></samp>. These options will +be ignored. Of course you must still specify how to get to the +buildmaster (with <samp><span class="option">--connect</span></samp>, <samp><span class="option">--tryhost</span></samp>, etc). + +<div class="node"> +<p><hr> +<a name="Other-Tools"></a> +Next: <a rel="next" accesskey="n" href="#g_t_002ebuildbot-config-directory">.buildbot config directory</a>, +Previous: <a rel="previous" accesskey="p" href="#Developer-Tools">Developer Tools</a>, +Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> + +</div> + +<h3 class="section">8.3 Other Tools</h3> + +<p>These tools are generally used by buildmaster administrators. + +<ul class="menu"> +<li><a accesskey="1" href="#sendchange">sendchange</a> +<li><a accesskey="2" href="#debugclient">debugclient</a> +</ul> + +<div class="node"> +<p><hr> +<a name="sendchange"></a> +Next: <a rel="next" accesskey="n" href="#debugclient">debugclient</a>, +Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>, +Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a> + +</div> + +<h4 class="subsection">8.3.1 sendchange</h4> + +<p>This command is used to tell the buildmaster about source changes. It +is intended to be used from within a commit script, installed on the +VC server. It requires that you have a PBChangeSource +(see <a href="#PBChangeSource">PBChangeSource</a>) running in the buildmaster (by being set in +<code>c['change_source']</code>). + +<pre class="example"> buildbot sendchange --master <var>MASTERHOST</var>:<var>PORT</var> --username <var>USER</var> <var>FILENAMES..</var> +</pre> + <p>There are other (optional) arguments which can influence the +<code>Change</code> that gets submitted: + + <dl> +<dt><code>--branch</code><dd>This provides the (string) branch specifier. If omitted, it defaults +to None, indicating the “default branch”. All files included in this +Change must be on the same branch. + + <br><dt><code>--category</code><dd>This provides the (string) category specifier. If omitted, it defaults +to None, indicating “no category”. The category property is used +by Schedulers to filter what changes they listen to. + + <br><dt><code>--revision_number</code><dd>This provides a (numeric) revision number for the change, used for VC systems +that use numeric transaction numbers (like Subversion). + + <br><dt><code>--revision</code><dd>This provides a (string) revision specifier, for VC systems that use +strings (Arch would use something like patch-42 etc). + + <br><dt><code>--revision_file</code><dd>This provides a filename which will be opened and the contents used as +the revision specifier. This is specifically for Darcs, which uses the +output of <samp><span class="command">darcs changes --context</span></samp> as a revision specifier. +This context file can be a couple of kilobytes long, spanning a couple +lines per patch, and would be a hassle to pass as a command-line +argument. + + <br><dt><code>--comments</code><dd>This provides the change comments as a single argument. You may want +to use <samp><span class="option">--logfile</span></samp> instead. + + <br><dt><code>--logfile</code><dd>This instructs the tool to read the change comments from the given +file. If you use <code>-</code> as the filename, the tool will read the +change comments from stdin. +</dl> + +<div class="node"> +<p><hr> +<a name="debugclient"></a> +Previous: <a rel="previous" accesskey="p" href="#sendchange">sendchange</a>, +Up: <a rel="up" accesskey="u" href="#Other-Tools">Other Tools</a> + +</div> + +<h4 class="subsection">8.3.2 debugclient</h4> + +<pre class="example"> buildbot debugclient --master <var>MASTERHOST</var>:<var>PORT</var> --passwd <var>DEBUGPW</var> +</pre> + <p>This launches a small Gtk+/Glade-based debug tool, connecting to the +buildmaster's “debug port”. This debug port shares the same port +number as the slaveport (see <a href="#Setting-the-slaveport">Setting the slaveport</a>), but the +<code>debugPort</code> is only enabled if you set a debug password in the +buildmaster's config file (see <a href="#Debug-options">Debug options</a>). The +<samp><span class="option">--passwd</span></samp> option must match the <code>c['debugPassword']</code> +value. + + <p><samp><span class="option">--master</span></samp> can also be provided in <samp><span class="file">.debug/options</span></samp> by the +<code>master</code> key. <samp><span class="option">--passwd</span></samp> can be provided by the +<code>debugPassword</code> key. + + <p>The <code>Connect</code> button must be pressed before any of the other +buttons will be active. This establishes the connection to the +buildmaster. The other sections of the tool are as follows: + + <dl> +<dt><code>Reload .cfg</code><dd>Forces the buildmaster to reload its <samp><span class="file">master.cfg</span></samp> file. This is +equivalent to sending a SIGHUP to the buildmaster, but can be done +remotely through the debug port. Note that it is a good idea to be +watching the buildmaster's <samp><span class="file">twistd.log</span></samp> as you reload the config +file, as any errors which are detected in the config file will be +announced there. + + <br><dt><code>Rebuild .py</code><dd>(not yet implemented). The idea here is to use Twisted's “rebuild” +facilities to replace the buildmaster's running code with a new +version. Even if this worked, it would only be used by buildbot +developers. + + <br><dt><code>poke IRC</code><dd>This locates a <code>words.IRC</code> status target and causes it to emit a +message on all the channels to which it is currently connected. This +was used to debug a problem in which the buildmaster lost the +connection to the IRC server and did not attempt to reconnect. + + <br><dt><code>Commit</code><dd>This allows you to inject a Change, just as if a real one had been +delivered by whatever VC hook you are using. You can set the name of +the committed file and the name of the user who is doing the commit. +Optionally, you can also set a revision for the change. If the +revision you provide looks like a number, it will be sent as an +integer, otherwise it will be sent as a string. + + <br><dt><code>Force Build</code><dd>This lets you force a Builder (selected by name) to start a build of +the current source tree. + + <br><dt><code>Currently</code><dd>(obsolete). This was used to manually set the status of the given +Builder, but the status-assignment code was changed in an incompatible +way and these buttons are no longer meaningful. + + </dl> + +<div class="node"> +<p><hr> +<a name=".buildbot-config-directory"></a> +<a name="g_t_002ebuildbot-config-directory"></a> +Previous: <a rel="previous" accesskey="p" href="#Other-Tools">Other Tools</a>, +Up: <a rel="up" accesskey="u" href="#Command_002dline-tool">Command-line tool</a> + +</div> + +<h3 class="section">8.4 .buildbot config directory</h3> + +<p>Many of the <samp><span class="command">buildbot</span></samp> tools must be told how to contact the +buildmaster that they interact with. This specification can be +provided as a command-line argument, but most of the time it will be +easier to set them in an “options” file. The <samp><span class="command">buildbot</span></samp> +command will look for a special directory named <samp><span class="file">.buildbot</span></samp>, +starting from the current directory (where the command was run) and +crawling upwards, eventually looking in the user's home directory. It +will look for a file named <samp><span class="file">options</span></samp> in this directory, and will +evaluate it as a python script, looking for certain names to be set. +You can just put simple <code>name = 'value'</code> pairs in this file to +set the options. + + <p>For a description of the names used in this file, please see the +documentation for the individual <samp><span class="command">buildbot</span></samp> sub-commands. The +following is a brief sample of what this file's contents could be. + +<pre class="example"> # for status-reading tools + masterstatus = 'buildbot.example.org:12345' + # for 'sendchange' or the debug port + master = 'buildbot.example.org:18990' + debugPassword = 'eiv7Po' +</pre> + <dl> +<dt><code>masterstatus</code><dd>Location of the <code>client.PBListener</code> status port, used by +<samp><span class="command">statuslog</span></samp> and <samp><span class="command">statusgui</span></samp>. + + <br><dt><code>master</code><dd>Location of the <code>debugPort</code> (for <samp><span class="command">debugclient</span></samp>). Also the +location of the <code>pb.PBChangeSource</code> (for <samp><span class="command">sendchange</span></samp>). +Usually shares the slaveport, but a future version may make it +possible to have these listen on a separate port number. + + <br><dt><code>debugPassword</code><dd>Must match the value of <code>c['debugPassword']</code>, used to protect the +debug port, for the <samp><span class="command">debugclient</span></samp> command. + + <br><dt><code>username</code><dd>Provides a default username for the <samp><span class="command">sendchange</span></samp> command. + + </dl> + + <p>The following options are used by the <code>buildbot try</code> command +(see <a href="#try">try</a>): + + <dl> +<dt><code>try_connect</code><dd>This specifies how the “try” command should deliver its request to +the buildmaster. The currently accepted values are “ssh” and “pb”. +<br><dt><code>try_builders</code><dd>Which builders should be used for the “try” build. +<br><dt><code>try_vc</code><dd>This specifies the version control system being used. +<br><dt><code>try_branch</code><dd>This indicates that the current tree is on a non-trunk branch. +<br><dt><code>try_topdir</code><br><dt><code>try_topfile</code><dd>Use <code>try_topdir</code> to explicitly indicate the top of your working +tree, or <code>try_topfile</code> to name a file that will only be found in +that top-most directory. + + <br><dt><code>try_host</code><br><dt><code>try_username</code><br><dt><code>try_dir</code><dd>When try_connect is “ssh”, the command will pay attention to +<code>try_host</code>, <code>try_username</code>, and <code>try_dir</code>. + + <br><dt><code>try_username</code><br><dt><code>try_password</code><br><dt><code>try_master</code><dd>Instead, when <code>try_connect</code> is “pb”, the command will pay +attention to <code>try_username</code>, <code>try_password</code>, and +<code>try_master</code>. + + <br><dt><code>try_wait</code><br><dt><code>masterstatus</code><dd><code>try_wait</code> and <code>masterstatus</code> are used to ask the “try” +command to wait for the requested build to complete. + + </dl> + +<div class="node"> +<p><hr> +<a name="Resources"></a> +Next: <a rel="next" accesskey="n" href="#Developer_0027s-Appendix">Developer's Appendix</a>, +Previous: <a rel="previous" accesskey="p" href="#Command_002dline-tool">Command-line tool</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="chapter">9 Resources</h2> + +<p>The Buildbot's home page is at <a href="http://buildbot.sourceforge.net/">http://buildbot.sourceforge.net/</a> + + <p>For configuration questions and general discussion, please use the +<code>buildbot-devel</code> mailing list. The subscription instructions and +archives are available at +<a href="http://lists.sourceforge.net/lists/listinfo/buildbot-devel">http://lists.sourceforge.net/lists/listinfo/buildbot-devel</a> + +<div class="node"> +<p><hr> +<a name="Developer's-Appendix"></a> +<a name="Developer_0027s-Appendix"></a> +Next: <a rel="next" accesskey="n" href="#Index-of-Useful-Classes">Index of Useful Classes</a>, +Previous: <a rel="previous" accesskey="p" href="#Resources">Resources</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="unnumbered">Developer's Appendix</h2> + +<p>This appendix contains random notes about the implementation of the +Buildbot, and is likely to only be of use to people intending to +extend the Buildbot's internals. + + <p>The buildmaster consists of a tree of Service objects, which is shaped +as follows: + +<pre class="example"> BuildMaster + ChangeMaster (in .change_svc) + [IChangeSource instances] + [IScheduler instances] (in .schedulers) + BotMaster (in .botmaster) + [IBuildSlave instances] + [IStatusTarget instances] (in .statusTargets) +</pre> + <p>The BotMaster has a collection of Builder objects as values of its +<code>.builders</code> dictionary. + +<div class="node"> +<p><hr> +<a name="Index-of-Useful-Classes"></a> +Next: <a rel="next" accesskey="n" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>, +Previous: <a rel="previous" accesskey="p" href="#Developer_0027s-Appendix">Developer's Appendix</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="unnumbered">Index of Useful Classes</h2> + +<p>This is a list of all user-visible classes. There are the ones that +are useful in <samp><span class="file">master.cfg</span></samp>, the buildmaster's configuration file. +Classes that are not listed here are generally internal things that +admins are unlikely to have much use for. + +<h3 class="heading">Change Sources</h3> + +<ul class="index-cs" compact> +<li><a href="#index-buildbot_002echanges_002ebonsaipoller_002eBonsaiPoller-49"><code>buildbot.changes.bonsaipoller.BonsaiPoller</code></a>: <a href="#BonsaiPoller">BonsaiPoller</a></li> +<li><a href="#index-buildbot_002echanges_002efreshcvs_002eFreshCVSSource-42"><code>buildbot.changes.freshcvs.FreshCVSSource</code></a>: <a href="#CVSToys-_002d-PBService">CVSToys - PBService</a></li> +<li><a href="#index-buildbot_002echanges_002email_002eBonsaiMaildirSource-45"><code>buildbot.changes.mail.BonsaiMaildirSource</code></a>: <a href="#BonsaiMaildirSource">BonsaiMaildirSource</a></li> +<li><a href="#index-buildbot_002echanges_002email_002eFCMaildirSource-43"><code>buildbot.changes.mail.FCMaildirSource</code></a>: <a href="#FCMaildirSource">FCMaildirSource</a></li> +<li><a href="#index-buildbot_002echanges_002email_002eSVNCommitEmailMaildirSource-46"><code>buildbot.changes.mail.SVNCommitEmailMaildirSource</code></a>: <a href="#SVNCommitEmailMaildirSource">SVNCommitEmailMaildirSource</a></li> +<li><a href="#index-buildbot_002echanges_002email_002eSyncmailMaildirSource-44"><code>buildbot.changes.mail.SyncmailMaildirSource</code></a>: <a href="#SyncmailMaildirSource">SyncmailMaildirSource</a></li> +<li><a href="#index-buildbot_002echanges_002ep4poller_002eP4Source-48"><code>buildbot.changes.p4poller.P4Source</code></a>: <a href="#P4Source">P4Source</a></li> +<li><a href="#index-buildbot_002echanges_002epb_002ePBChangeSource-47"><code>buildbot.changes.pb.PBChangeSource</code></a>: <a href="#PBChangeSource">PBChangeSource</a></li> +<li><a href="#index-buildbot_002echanges_002esvnpoller_002eSVNPoller-50"><code>buildbot.changes.svnpoller.SVNPoller</code></a>: <a href="#SVNPoller">SVNPoller</a></li> + </ul><h3 class="heading">Schedulers and Locks</h3> + + + +<ul class="index-sl" compact> +<li><a href="#index-buildbot_002elocks_002eLockAccess-93"><code>buildbot.locks.LockAccess</code></a>: <a href="#Interlocks">Interlocks</a></li> +<li><a href="#index-buildbot_002elocks_002eMasterLock-91"><code>buildbot.locks.MasterLock</code></a>: <a href="#Interlocks">Interlocks</a></li> +<li><a href="#index-buildbot_002elocks_002eSlaveLock-92"><code>buildbot.locks.SlaveLock</code></a>: <a href="#Interlocks">Interlocks</a></li> +<li><a href="#index-buildbot_002escheduler_002eAnyBranchScheduler-22"><code>buildbot.scheduler.AnyBranchScheduler</code></a>: <a href="#AnyBranchScheduler">AnyBranchScheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002eDependent-25"><code>buildbot.scheduler.Dependent</code></a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002eNightly-27"><code>buildbot.scheduler.Nightly</code></a>: <a href="#Nightly-Scheduler">Nightly Scheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002ePeriodic-26"><code>buildbot.scheduler.Periodic</code></a>: <a href="#Periodic-Scheduler">Periodic Scheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002eScheduler-21"><code>buildbot.scheduler.Scheduler</code></a>: <a href="#Scheduler-Scheduler">Scheduler Scheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002eTriggerable-31"><code>buildbot.scheduler.Triggerable</code></a>: <a href="#Triggerable-Scheduler">Triggerable Scheduler</a></li> +<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-120"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#try">try</a></li> +<li><a href="#index-buildbot_002escheduler_002eTry_005fJobdir-28"><code>buildbot.scheduler.Try_Jobdir</code></a>: <a href="#Try-Schedulers">Try Schedulers</a></li> +<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-121"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#try">try</a></li> +<li><a href="#index-buildbot_002escheduler_002eTry_005fUserpass-29"><code>buildbot.scheduler.Try_Userpass</code></a>: <a href="#Try-Schedulers">Try Schedulers</a></li> + </ul><h3 class="heading">Build Factories</h3> + + + +<ul class="index-bf" compact> +<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicBuildFactory-95"><code>buildbot.process.factory.BasicBuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eBasicSVN-96"><code>buildbot.process.factory.BasicSVN</code></a>: <a href="#BuildFactory">BuildFactory</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eBuildFactory-94"><code>buildbot.process.factory.BuildFactory</code></a>: <a href="#BuildFactory">BuildFactory</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eCPAN-100"><code>buildbot.process.factory.CPAN</code></a>: <a href="#CPAN">CPAN</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eDistutils-101"><code>buildbot.process.factory.Distutils</code></a>: <a href="#Python-distutils">Python distutils</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eGNUAutoconf-99"><code>buildbot.process.factory.GNUAutoconf</code></a>: <a href="#GNUAutoconf">GNUAutoconf</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eQuickBuildFactory-98"><code>buildbot.process.factory.QuickBuildFactory</code></a>: <a href="#Quick-builds">Quick builds</a></li> +<li><a href="#index-buildbot_002eprocess_002efactory_002eTrial-102"><code>buildbot.process.factory.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> + </ul><h3 class="heading">Build Steps</h3> + + + +<ul class="index-bs" compact> +<li><a href="#index-buildbot_002esteps_002emaxq_002eMaxQ-123"><code>buildbot.steps.maxq.MaxQ</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li> +<li><a href="#index-buildbot_002esteps_002epython_002eBuildEPYDoc-78"><code>buildbot.steps.python.BuildEPYDoc</code></a>: <a href="#BuildEPYDoc">BuildEPYDoc</a></li> +<li><a href="#index-buildbot_002esteps_002epython_002ePyFlakes-79"><code>buildbot.steps.python.PyFlakes</code></a>: <a href="#PyFlakes">PyFlakes</a></li> +<li><a href="#index-buildbot_002esteps_002epython_002ePyLint-80"><code>buildbot.steps.python.PyLint</code></a>: <a href="#PyLint">PyLint</a></li> +<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eBuildDebs-106"><code>buildbot.steps.python_twisted.BuildDebs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> +<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eHLint-103"><code>buildbot.steps.python_twisted.HLint</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> +<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eProcessDocs-105"><code>buildbot.steps.python_twisted.ProcessDocs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> +<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eRemovePYCs-107"><code>buildbot.steps.python_twisted.RemovePYCs</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> +<li><a href="#index-buildbot_002esteps_002epython_005ftwisted_002eTrial-104"><code>buildbot.steps.python_twisted.Trial</code></a>: <a href="#Python_002fTwisted_002ftrial-projects">Python/Twisted/trial projects</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eCompile-73"><code>buildbot.steps.shell.Compile</code></a>: <a href="#Compile">Compile</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eConfigure-72"><code>buildbot.steps.shell.Configure</code></a>: <a href="#Configure">Configure</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002ePerlModuleTest-76"><code>buildbot.steps.shell.PerlModuleTest</code></a>: <a href="#PerlModuleTest">PerlModuleTest</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eSetProperty-77"><code>buildbot.steps.shell.SetProperty</code></a>: <a href="#SetProperty">SetProperty</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eShellCommand-71"><code>buildbot.steps.shell.ShellCommand</code></a>: <a href="#ShellCommand">ShellCommand</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eTest-74"><code>buildbot.steps.shell.Test</code></a>: <a href="#Test">Test</a></li> +<li><a href="#index-buildbot_002esteps_002eshell_002eTreeSize-75"><code>buildbot.steps.shell.TreeSize</code></a>: <a href="#TreeSize">TreeSize</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eArch-62"><code>buildbot.steps.source.Arch</code></a>: <a href="#Arch">Arch</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eBazaar-64"><code>buildbot.steps.source.Bazaar</code></a>: <a href="#Bazaar">Bazaar</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eBzr-66"><code>buildbot.steps.source.Bzr</code></a>: <a href="#Bzr">Bzr</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eCVS-54"><code>buildbot.steps.source.CVS</code></a>: <a href="#CVS">CVS</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eDarcs-58"><code>buildbot.steps.source.Darcs</code></a>: <a href="#Darcs">Darcs</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eGit-122"><code>buildbot.steps.source.Git</code></a>: <a href="#Index-of-Useful-Classes">Index of Useful Classes</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eGit-70"><code>buildbot.steps.source.Git</code></a>: <a href="#Git">Git</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eMercurial-60"><code>buildbot.steps.source.Mercurial</code></a>: <a href="#Mercurial">Mercurial</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eP4-68"><code>buildbot.steps.source.P4</code></a>: <a href="#P4">P4</a></li> +<li><a href="#index-buildbot_002esteps_002esource_002eSVN-56"><code>buildbot.steps.source.SVN</code></a>: <a href="#SVN">SVN</a></li> +<li><a href="#index-buildbot_002esteps_002etransfer_002eDirectoryUpload-84"><code>buildbot.steps.transfer.DirectoryUpload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> +<li><a href="#index-buildbot_002esteps_002etransfer_002eFileDownload-83"><code>buildbot.steps.transfer.FileDownload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> +<li><a href="#index-buildbot_002esteps_002etransfer_002eFileUpload-82"><code>buildbot.steps.transfer.FileUpload</code></a>: <a href="#Transferring-Files">Transferring Files</a></li> + </ul><!-- undocumented steps --> +<p><a name="index-buildbot_002esteps_002esource_002eGit-122"></a><a name="index-buildbot_002esteps_002emaxq_002eMaxQ-123"></a> + +<h3 class="heading">Status Targets</h3> + + + +<ul class="index-st" compact> +<li><a href="#index-buildbot_002estatus_002eclient_002ePBListener-118"><code>buildbot.status.client.PBListener</code></a>: <a href="#PBListener">PBListener</a></li> +<li><a href="#index-buildbot_002estatus_002ehtml_002eWaterfall-111"><code>buildbot.status.html.Waterfall</code></a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li> +<li><a href="#index-buildbot_002estatus_002email_002eMailNotifier-114"><code>buildbot.status.mail.MailNotifier</code></a>: <a href="#MailNotifier">MailNotifier</a></li> +<li><a href="#index-buildbot_002estatus_002eweb_002ebaseweb_002eWebStatus-109"><code>buildbot.status.web.baseweb.WebStatus</code></a>: <a href="#WebStatus">WebStatus</a></li> +<li><a href="#index-buildbot_002estatus_002ewords_002eIRC-116"><code>buildbot.status.words.IRC</code></a>: <a href="#IRC-Bot">IRC Bot</a></li> + </ul><!-- TODO: undocumented targets --> +<div class="node"> +<p><hr> +<a name="Index-of-master.cfg-keys"></a> +<a name="Index-of-master_002ecfg-keys"></a> +Next: <a rel="next" accesskey="n" href="#Index">Index</a>, +Previous: <a rel="previous" accesskey="p" href="#Index-of-Useful-Classes">Index of Useful Classes</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="unnumbered">Index of master.cfg keys</h2> + +<p>This is a list of all of the significant keys in master.cfg . Recall +that master.cfg is effectively a small python program with exactly one +responsibility: create a dictionary named <code>BuildmasterConfig</code>. +The keys of this dictionary are listed here. The beginning of the +master.cfg file typically starts with something like: + +<pre class="example"> BuildmasterConfig = c = {} +</pre> + <p>Therefore a config key of <code>change_source</code> will usually appear in +master.cfg as <code>c['change_source']</code>. + + + +<ul class="index-bc" compact> +<li><a href="#index-c_005b_0027buildbotURL_0027_005d-15"><code>c['buildbotURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> +<li><a href="#index-c_005b_0027builders_0027_005d-38"><code>c['builders']</code></a>: <a href="#Defining-Builders">Defining Builders</a></li> +<li><a href="#index-c_005b_0027change_005fsource_0027_005d-18"><code>c['change_source']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> +<li><a href="#index-c_005b_0027debugPassword_0027_005d-40"><code>c['debugPassword']</code></a>: <a href="#Debug-options">Debug options</a></li> +<li><a href="#index-c_005b_0027logCompressionLimit_0027_005d-16"><code>c['logCompressionLimit']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> +<li><a href="#index-c_005b_0027manhole_0027_005d-41"><code>c['manhole']</code></a>: <a href="#Debug-options">Debug options</a></li> +<li><a href="#index-c_005b_0027mergeRequests_0027_005d-32"><code>c['mergeRequests']</code></a>: <a href="#Merging-BuildRequests">Merging BuildRequests</a></li> +<li><a href="#index-c_005b_0027projectName_0027_005d-13"><code>c['projectName']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> +<li><a href="#index-c_005b_0027projectURL_0027_005d-14"><code>c['projectURL']</code></a>: <a href="#Defining-the-Project">Defining the Project</a></li> +<li><a href="#index-c_005b_0027properties_0027_005d-36"><code>c['properties']</code></a>: <a href="#Defining-Global-Properties">Defining Global Properties</a></li> +<li><a href="#index-c_005b_0027schedulers_0027_005d-19"><code>c['schedulers']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> +<li><a href="#index-c_005b_0027slavePortnum_0027_005d-33"><code>c['slavePortnum']</code></a>: <a href="#Setting-the-slaveport">Setting the slaveport</a></li> +<li><a href="#index-c_005b_0027slaves_0027_005d-34"><code>c['slaves']</code></a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li> +<li><a href="#index-c_005b_0027sources_0027_005d-17"><code>c['sources']</code></a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> +<li><a href="#index-c_005b_0027status_0027_005d-39"><code>c['status']</code></a>: <a href="#Defining-Status-Targets">Defining Status Targets</a></li> + </ul><div class="node"> +<p><hr> +<a name="Index"></a> +Previous: <a rel="previous" accesskey="p" href="#Index-of-master_002ecfg-keys">Index of master.cfg keys</a>, +Up: <a rel="up" accesskey="u" href="#Top">Top</a> + +</div> + +<h2 class="unnumbered">Index</h2> + + + +<ul class="index-cp" compact> +<li><a href="#index-addURL-89">addURL</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> +<li><a href="#index-Arch-Checkout-61">Arch Checkout</a>: <a href="#Arch">Arch</a></li> +<li><a href="#index-Bazaar-Checkout-63">Bazaar Checkout</a>: <a href="#Bazaar">Bazaar</a></li> +<li><a href="#index-Builder-9">Builder</a>: <a href="#Builder">Builder</a></li> +<li><a href="#index-BuildRequest-8">BuildRequest</a>: <a href="#BuildRequest">BuildRequest</a></li> +<li><a href="#index-BuildSet-7">BuildSet</a>: <a href="#BuildSet">BuildSet</a></li> +<li><a href="#index-BuildStep-URLs-88">BuildStep URLs</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> +<li><a href="#index-Bzr-Checkout-65">Bzr Checkout</a>: <a href="#Bzr">Bzr</a></li> +<li><a href="#index-Configuration-12">Configuration</a>: <a href="#Configuration">Configuration</a></li> +<li><a href="#index-CVS-Checkout-53">CVS Checkout</a>: <a href="#CVS">CVS</a></li> +<li><a href="#index-Darcs-Checkout-57">Darcs Checkout</a>: <a href="#Darcs">Darcs</a></li> +<li><a href="#index-Dependencies-24">Dependencies</a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> +<li><a href="#index-Dependent-23">Dependent</a>: <a href="#Dependent-Scheduler">Dependent Scheduler</a></li> +<li><a href="#index-email-112">email</a>: <a href="#MailNotifier">MailNotifier</a></li> +<li><a href="#index-File-Transfer-81">File Transfer</a>: <a href="#Transferring-Files">Transferring Files</a></li> +<li><a href="#index-Git-Checkout-69">Git Checkout</a>: <a href="#Git">Git</a></li> +<li><a href="#index-installation-3">installation</a>: <a href="#Installing-the-code">Installing the code</a></li> +<li><a href="#index-introduction-1">introduction</a>: <a href="#Introduction">Introduction</a></li> +<li><a href="#index-IRC-115">IRC</a>: <a href="#IRC-Bot">IRC Bot</a></li> +<li><a href="#index-links-87">links</a>: <a href="#BuildStep-URLs">BuildStep URLs</a></li> +<li><a href="#index-locks-90">locks</a>: <a href="#Interlocks">Interlocks</a></li> +<li><a href="#index-logfiles-4">logfiles</a>: <a href="#Logfiles">Logfiles</a></li> +<li><a href="#index-LogLineObserver-86">LogLineObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li> +<li><a href="#index-LogObserver-85">LogObserver</a>: <a href="#Adding-LogObservers">Adding LogObservers</a></li> +<li><a href="#index-mail-113">mail</a>: <a href="#MailNotifier">MailNotifier</a></li> +<li><a href="#index-Mercurial-Checkout-59">Mercurial Checkout</a>: <a href="#Mercurial">Mercurial</a></li> +<li><a href="#index-PBListener-117">PBListener</a>: <a href="#PBListener">PBListener</a></li> +<li><a href="#index-Perforce-Update-67">Perforce Update</a>: <a href="#P4">P4</a></li> +<li><a href="#index-Philosophy-of-operation-2">Philosophy of operation</a>: <a href="#History-and-Philosophy">History and Philosophy</a></li> +<li><a href="#index-Properties-51">Properties</a>: <a href="#Using-Build-Properties">Using Build Properties</a></li> +<li><a href="#index-Properties-37">Properties</a>: <a href="#Defining-Global-Properties">Defining Global Properties</a></li> +<li><a href="#index-Properties-35">Properties</a>: <a href="#Buildslave-Specifiers">Buildslave Specifiers</a></li> +<li><a href="#index-Properties-20">Properties</a>: <a href="#Change-Sources-and-Schedulers">Change Sources and Schedulers</a></li> +<li><a href="#index-Properties-11">Properties</a>: <a href="#Build-Properties">Build Properties</a></li> +<li><a href="#index-Scheduler-6">Scheduler</a>: <a href="#Schedulers">Schedulers</a></li> +<li><a href="#index-statusgui-119">statusgui</a>: <a href="#statusgui">statusgui</a></li> +<li><a href="#index-SVN-Checkout-55">SVN Checkout</a>: <a href="#SVN">SVN</a></li> +<li><a href="#index-treeStableTimer-97">treeStableTimer</a>: <a href="#BuildFactory-Attributes">BuildFactory Attributes</a></li> +<li><a href="#index-Triggers-30">Triggers</a>: <a href="#Triggerable-Scheduler">Triggerable Scheduler</a></li> +<li><a href="#index-Users-10">Users</a>: <a href="#Users">Users</a></li> +<li><a href="#index-Version-Control-5">Version Control</a>: <a href="#Version-Control-Systems">Version Control Systems</a></li> +<li><a href="#index-Waterfall-110">Waterfall</a>: <a href="#HTML-Waterfall">HTML Waterfall</a></li> +<li><a href="#index-WebStatus-108">WebStatus</a>: <a href="#WebStatus">WebStatus</a></li> +<li><a href="#index-WithProperties-52">WithProperties</a>: <a href="#Using-Build-Properties">Using Build Properties</a></li> + </ul><div class="footnote"> +<hr> +<a name="texinfo-footnotes-in-document"></a><h4>Footnotes</h4><p class="footnote"><small>[<a name="fn-1" href="#fnd-1">1</a>]</small> this +@reboot syntax is understood by Vixie cron, which is the flavor +usually provided with linux systems. Other unices may have a cron that +doesn't understand @reboot</p> + + <p class="footnote"><small>[<a name="fn-2" href="#fnd-2">2</a>]</small> except Darcs, but +since the Buildbot never modifies its local source tree we can ignore +the fact that Darcs uses a less centralized model</p> + + <p class="footnote"><small>[<a name="fn-3" href="#fnd-3">3</a>]</small> many VC +systems provide more complexity than this: in particular the local +views that P4 and ClearCase can assemble out of various source +directories are more complex than we're prepared to take advantage of +here</p> + + <p class="footnote"><small>[<a name="fn-4" href="#fnd-4">4</a>]</small> Monotone's <em>multiple heads</em> feature +violates this assumption of cumulative Changes, but in most situations +the changes don't occur frequently enough for this to be a significant +problem</p> + + <p class="footnote"><small>[<a name="fn-5" href="#fnd-5">5</a>]</small> this <code>checkoutDelay</code> defaults +to half the tree-stable timer, but it can be overridden with an +argument to the Source Step</p> + + <p class="footnote"><small>[<a name="fn-6" href="#fnd-6">6</a>]</small> To be precise, it is an object or a list of objects +which all implement the <code>buildbot.interfaces.IChangeSource</code> +Interface. It is unusual to have multiple ChangeSources, so this key +accepts either a single ChangeSource or a sequence of them.</p> + + <p class="footnote"><small>[<a name="fn-7" href="#fnd-7">7</a>]</small> Build properties are serialized along with the +build results, so they must be serializable. For this reason, the +value of any build property should be simple inert data: strings, +numbers, lists, tuples, and dictionaries. They should not contain +class instances.</p> + + <p class="footnote"><small>[<a name="fn-8" href="#fnd-8">8</a>]</small> framboozle.com is still available. Remember, I get 10% +:).</p> + + <p class="footnote"><small>[<a name="fn-9" href="#fnd-9">9</a>]</small> Framboozle gets very excited about running unit +tests.</p> + + <p class="footnote"><small>[<a name="fn-10" href="#fnd-10">10</a>]</small> See +http://en.wikipedia.org/wiki/Read/write_lock_pattern for more information.</p> + + <p class="footnote"><small>[<a name="fn-11" href="#fnd-11">11</a>]</small> Deadlock is the situation where two or more slaves each +hold a lock in exclusive mode, and in addition want to claim the lock held by +the other slave exclusively as well. Since locks allow at most one exclusive +user, both slaves will wait forever.</p> + + <p class="footnote"><small>[<a name="fn-12" href="#fnd-12">12</a>]</small> Starving is the situation that only a few locks are available, +and they are immediately grabbed by another build. As a result, it may take a +long time before all locks needed by the starved build are free at the same +time.</p> + + <p class="footnote"><small>[<a name="fn-13" href="#fnd-13">13</a>]</small> It may even be possible to provide SSL access by using +a specification like +<code>"ssl:12345:privateKey=mykey.pen:certKey=cert.pem"</code>, but this is +completely untested</p> + + <p class="footnote"><small>[<a name="fn-14" href="#fnd-14">14</a>]</small> Apparently this is the same way +http://buildd.debian.org displays build status</p> + + <hr></div> + +</body></html> + |